I was doing some performance testing on this site, and while I was impressed, I felt like I could wring some more speed out of it. I decided to switch from Heroku’s Static Buildpack to a simple Nginx webserver using Heroku’s nice Nginx Buildpack. In theory this should be a little quicker and lighter.
The first step was to get Nginx set up on the Heroku App. I had the static buildpack already, so I simply had to remove it and add the Nginx Buildpack.
The next step is to modify the Procfile to run Ngnix in Solo mode:
web: bin/start-nginx-solo
Then I created a config directory and put my nginx.conf.erb file in it. The Nginx buildpack will grab that file, process it, and then use it to run Nginx on the dyno.
I found this config to work well:
daemon off;
# Heroku dynos have at least 4 cores
worker_processes <%= ENV['NGINX_WORKERS'] || 4 %>;
events {
use epoll;
accept_mutex on;
worker_connections <%= ENV['NGINX_WORKER_CONNECTIONS'] || 1024 %>;
}
http {
gzip on;
gzip_comp_level 2;
gzip_min_length 512;
server_tokens off;
log_format main '$time_iso8601 - $status $request - client IP: $http_x_forwarded_for - <%= ENV['DYNO'] %> to $upstream_addr - upstream status: $upstream_status, upstream_response_time $upstream_response_time, request_time $request_time';
access_log /dev/stdout main;
error_log /dev/stdout notice;
log_not_found on;
include mime.types;
default_type application/octet-stream;
sendfile on;
# Must read the body in 5 seconds.
client_body_timeout <%= ENV['NGINX_CLIENT_BODY_TIMEOUT'] || 5 %>;
server {
listen <%= ENV["PORT"] %>;
error_page 404 /404.html;
error_page 403 /403.html;
port_in_redirect off;
location / {
root _site/;
}
}
}
The important parts are the port_in_redirect off; and the root _site/; directives. The port_in_redirect tells Nginx to not embed it’s port into any redirects, and root tells Nginx where the static files are.
Well, is it fast?
I ran ab on the site before and after, and there is a large improvement. Larger then I expected.
The old configuration had a perc99 under load of 1.7 seconds per request, and the new has a perc99 of 0.6 seconds. More then a second per request faster!
Old Config
Concurrency Level: 100
Time taken for tests: 7.284 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 6922000 bytes
HTML transferred: 6722000 bytes
Requests per second: 137.28 [#/sec] (mean)
Time per request: 728.426 [ms] (mean)
Time per request: 7.284 [ms] (mean, across all concurrent requests)
Transfer rate: 928.00 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 307 449 233.8 380 1612
Processing: 103 204 84.1 212 1026
Waiting: 103 201 83.5 211 1026
Total: 424 653 247.2 591 1939
Percentage of the requests served within a certain time (ms)
50% 591
66% 627
75% 658
80% 670
90% 822
95% 1084
98% 1738
99% 1769
100% 1939 (longest request)
New Config
Concurrency Level: 100
Time taken for tests: 26.680 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 34870000 bytes
HTML transferred: 33650000 bytes
Requests per second: 187.41 [#/sec] (mean)
Time per request: 533.592 [ms] (mean)
Time per request: 5.336 [ms] (mean, across all concurrent requests)
Transfer rate: 1276.36 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 297 385 42.1 378 620
Processing: 102 136 33.6 122 641
Waiting: 102 133 30.5 120 641
Total: 424 520 51.2 506 986
Percentage of the requests served within a certain time (ms)
50% 506
66% 524
75% 536
80% 544
90% 577
95% 643
98% 685
99% 696
100% 986 (longest request)
But what about SSL?
One of the things that I really liked about the Static Buildpack is that it is based on Rack, and I could use the rack-ssl-enforcer gem to make sure that all requests were directed to the encrypted version of the site. I did some investigation and found the following stanza in the server block would work with the Heroku router:
if ($http_x_forwarded_proto != "https") {
return 301 https://$host$request_uri;
}
A redirect would be preferable, however since we are behind the Heroku Router, a simple redirect seems to enter into an infinite redirect loop. The if loop slows down processing considerably.
With SSL Redirect
Concurrency Level: 100
Time taken for tests: 26.618 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 34870000 bytes
HTML transferred: 33650000 bytes
Requests per second: 187.85 [#/sec] (mean)
Time per request: 532.352 [ms] (mean)
Time per request: 5.324 [ms] (mean, across all concurrent requests)
Transfer rate: 1279.33 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 305 385 40.6 381 1618
Processing: 101 134 31.4 123 471
Waiting: 101 131 28.0 120 469
Total: 422 519 48.5 510 1740
Percentage of the requests served within a certain time (ms)
50% 510
66% 523
75% 533
80% 544
90% 569
95% 596
98% 630
99% 704
100% 1740 (longest request)
Still respectable, but you can see the impact of using the if statement in the Nginx config. The perc99 is only a few milliseconds slower with the SSL redirect, but milliseconds count!
I will continue to research and see if I can remove that if statement, and find a cleaner way to implement this.
One of the other uses for Nginx on Heroku is a on Dyno proxy and static file server. This experiment shows how efficient Nginx can be at hosting the static assets of a complex web application, like Ruby. Using Nginx as a front end to proxy requests, and serve static files offloads traffic from the language specific app server, and can improve overall performance of a web application.
Update 10/10/19
I quickly found out that I still needed the static buildpack in my development and staging environments. It’s used to build out the static assets that Nginx serves, after all.
If you refer back to How I post an article you will know that I use a Heroku Pipeline to manage posting. The nice thing is that I don’t need the Static buildpack or Ruby on my production site, as the slug is promoted intact between the Staging and Production apps. So I have Ruby and the Static buildpack installed and running in staging, but only the Nginx buildpack associated with the production app.
Upon writing this I am not even sure if I need any buildpacks associated with the prod app, as the slug already contains a complete built environment. More testing is needed.