Acorns

I just ran a Rails 4 app through WebPageTest.org and it walloped me with a big red “F” (fail) for my app’s use of keep-alives. I felt a bit miffed. I had done some basic work to ensure decent performance, like:

• Recompressing images
• Designing critical-path server code for consistently fast first-byte times
• Using asset pipeline to concatentate and minify scripts and CSS
• Using heroku_rails_deflate to server gzipped assets

But the waterfall confirmed that the browser was spending about 30 ms per request on establishing a new connection for each object on the page. I expected to see this per-connection overhead amortized over multiple requests.

While 30 milliseconds might not sound like much, a few reopened connections add up to the 100 ms that Amazon translates to 1% of sales.

Keep-alives

HTTP Keepalives are an HTTP header used to implement persistent connections. Persistent connections reduce the overhead of setting up TCP connections when downloading a page over HTTP. Instead of opening a new connection for each file downloaded, the browser can open one connection and re-use it for as many files as necessary. In addition to saving the ~30 second round-trip for another TCP handshake, it avoids another TCP slow start process for growing the congestion window to take full advantage of available bandwidth.

The server has to know to keep the connection “alive” — to not close the connection after sending the requested file. The client may explicitly request this behavior via the “Connection: keep-alive” header, which most browsers use. But with the end of the connection no longer delimiting the end of file, the client needs another way to determine when to stop waiting for bytes of data when reading the HTTP response.

There are at least two ways the server can communicate how many bytes of data the client must read. One way is with a Content-Length header, which simply tells the client how many bytes of response body to read. But sometimes the server doesn’t know when it sends the headers — at the beginning of the response — how many bytes of data it will eventually send. One example of this is when the response is generated by a CGI script. So another way is to use “chunked” Transfer-Encoding, where the server sends the data in chunks, each of which is preceded by the length of that chunk. That way, the server can buffer a bounded amount of data before sending it to the client and send another chunk later if the CGI script (or whatever source) turns out to produce more data.

Heroku

I expected Heroku to take care of all of these HTTP details for me. After all, that’s what they do — take care of stuff, so I don’t have to.

Heroku does support keep-alives, but only sometimes. Heroku’s router never uses keep-alives between itself and the application, because the latency of establishing a new connection is minimal. But between the browser and their router, their docs say they do support keep-alives with HTTP/1.1 by default. They’re just not completely clear about when they support it.

It makes sense for a high-performance router like Heroku’s to not wait for the application to produce an entire response before starting to forward it to the client. It reduces buffering requirements on the router and reduces overall latency between browser and application to start forwarding the response when it’s only partially received. But it also means that Heroku cannot provide the browser with the file length when it sends response headers unless the length is given by the application, either up-front via Content-Length or incrementially via Transfer-Encoding. (Actually they could theoretially do that, but maybe buffering partial responses to convert them to “chunked” Transfer-Encoding might add complexity or undesirable latency… I’m not sure what their rationale is.)

The heroku_rails_deflate Gem

I expected Rails to take care of sending the right HTTP headers for me. Shouldn’t this “Just Work”?

After some fiddling and comparing response headers of different apps, I discovered that heroku_rails_deflate was interfering with the process. It installed a HerokuRailsDeflate::ServeZippedAssets middleware that was removing the Content-Length header when serving gzipped assets, probably because of ambiguity about whether the Content-Length should be that of the gzipped data or the uncompressed data. But it turns out that the file handler to which it delegates already sets the correct Content-Length. (Rack::File is eventually called by ActionDispatch::Static.)

The issue was identified during a refactor and fixed in a later version, so I upgraded from 0.2.0 to 1.0.3 to enable keep-alives. But this only fixes gzipped assets. The base HTML doesn’t support keep-alives.

Rack::Deflater

It turns out that, in addition to servering the .gz version of CSS and Javascript files, heroku_rails_deflate also installs Rack::Deflater to compress other files.

Compressing HTML is a potentially big win, but unfortunately, Rack::Deflater was indiscriminantly gzipping images as well. This is a waste, because they are already compressed, but it is a small waste. The real problem is that, in the process, it removes their Content-Lengths, causing Heroku to disable keep-alives.

When serving files from disk, the file handler middleware provides a Content-Length field by reading the file size from the filesystem. Rack::Deflater then compresses every response as a stream of unknown length, so it does not know the compressed content length before sending headers.

Removing Rack::Deflater

I decided to turn of Rack::Deflater, at least for images. It turns out that Rack 1.6.0 provides a version of that middleware that allows selective compression, but that would require upgrading rails, which is more than I wanted to do at the time.

So I just disabled the middleware entirely. It means my HTML won’t be gzipped, but it’s generally small enough on my site that enabling keep-alives is a bigger win.

It’s not as easy to remove as config.middleware.delete because the middleware is not yet in place when config/application.rb is loaded. An initializer in config/initializers is likewise too soon. But doing so in an “after_initialize” block is too late; the middleware chain has already been frozen by that time. It turns out that a good time to remove it is in an “initializer” block inside of config/application.rb:

module MyApp
  class Application < Rails::Application
    #
    # ... other configs here ...
    #

    initializer "app.hack" do |app|
      app.config.middleware.delete 'Rack::Deflater'
    end
  end
end

Summary

So in summary, to make Heroku serve my app with keep-alives I had to:

• Upgrade heroku_rails_deflate to ~> 1.0.3
• Remove Rack::Deflater at just the right time

And with that, WebPageTest gives its seal of approval: a big green “A”.