EventMachine and Passenger

In order to fully explain this post, we first need to cover some back story. Originally, Gaug.es was hosted on Heroku. Recently, we moved Gaug.es to RailsMachine (before the great AWS outage luckily), where we are already happily hosting Harmony.

At Heroku, we were running on 1.9.2 and thin. The most common RailsMachine stack is REE 1.8 and Passenger. Sticking with the common stack meant it would be a far easier and faster transition to Railsmachine, so we tweaked a few things and switched.

Heroku, Thin, and EventMachine

While at Heroku, we had been testing using PusherApp for live updating of analytics as they occurred. The pusher gem has two ways to trigger notifications, trigger (net/http) and trigger_async (em-http-request).

Since Heroku runs on thin, we used trigger_async. This meant that sending the PusherApp notifications in the request cycle was fine, as they did not block.

One of the changes when moving to RailsMachine was switching from trigger_async to trigger. Obviously, having an external HTTP request in your request path is less than ideal, but backgrounding it seemed to go against the whole idea of “live”.

Our response times for Gaug.es average around 5ms, so even with 75-100ms for each PusherApp request, we were still in a normally acceptable response time range (not ok with me, but ok for now).

Pusher Conversation

I contacted the fine folks at Pusher and asked if they had any suggestions. One suggestion Martyn mentioned was Thread.new { EM.run }.

Given my lack of experience with threads and event machine, this at first this struck me as dirty/scary and I was not sure if he was serious.

I did a bit of research and discovered he was not only serious, but people we doing it. The AMQP gem even recommends it in the Readme.

Hmmm, This Might Actually Work

After a bit of googling and scouring code on Github I found a few different solutions. I started hacking and got something that was “working” pretty quickly. Quite intrigued I decided to hit up someone smarter than I, Aman Gupta, who maintains the EventMachine and AMQP gems.

He confirmed that it would work and recommended a few tweaks. Yesterday, I pushed it to production and thus far it is working great. Below is the code needed to make the magic happen.

module GaugesEM
  def self.start
    if defined?(PhusionPassenger)
      PhusionPassenger.on_event(:starting_worker_process) do |forked|
        if forked && EM.reactor_running?
          EM.stop
        end
        Thread.new { EM.run }
        die_gracefully_on_signal
      end
    end
  end

  def self.die_gracefully_on_signal
    Signal.trap("INT")  { EM.stop }
    Signal.trap("TERM") { EM.stop }
  end
end

GaugesEM.start

Gaug.es is 100% Sinatra, so I just put this in the file in Gaug.es that works similar to environment.rb or an initializer would in Rails.

There are two key parts. First, if we are running on Passenger and using smart spawning, we need to stop the event machine if it is started. Second, we create a new thread and start the event machine loop.

Now, in the Notification class that we have in Gaug.es, I can do the following to make the Pusher request not block the main request.

EM.next_tick {
  Pusher[channel].trigger_async('hit', doc)
}

The main request carries on as usual and does not wait for the Pusher to request to finish. In the background, event machine is sending all these notifications. Once again, even on Passenger, we now have non-blocking pusher notifications.

Hmm, This Does Work

Since it took me a bit to figure it out, I thought I would post it here for everyone to benefit from and maybe to start some discussion. If you have suggestions or see glaring issues, please let me know.

I have no assumptions that I am wise or that this is perfect, but thus far it is getting the job done with no adverse affects.

Misleading Graph of Proof

Below is a graph of response times for Gaug.es thanks to New Relic. Seriously, where would we be without New Relic! Green is the time spent in external requests. I am sure you can tell at which point I pushed out the event machine integration.

That said, don’t think that all that time is instantly gone. It is still happening, just in a thread in the background without much affect, if any, on our normal response times.

Demo, Plz!

If you are curious about what the live updating looks like currently in Gaug.es, I posted a short video a few weeks back.

Also, if you too are addicted to analytics, you should definitely sign up and try it out. Lots of good stuff coming down the pipe!

Leave a Reply

Your email address will not be published. Required fields are marked *