Monitoring Rails Performance with Munin and a Mongrel

Rails makes things easy on developers—maybe too easy. It’s not uncommon to reference an association while iterating over a collection of objects, resulting in a performance-devouring N+1 queries being executed. Of course, this is easily fixed with eager loading of the association, but you get my point. Rails can be a big gun, and it’s easy to blow off your foot.

Once your application is moved into production, it becomes important to keep an eye on performance over time in order to get a feel for trends and plan capacity intelligently. Numerous Rails performance measuring tools exist, but I find that it helps to correlate performance of your application with simultaneous lower-level performance metrics of your system (CPU and Memory usage, Load Average, etc). Besides that, hey! Pretty Graphs! Enter munin, an open-source, extensible monitoring tool with a number of out-of-the-box plugins that are useful to SysAdmin type folks. There’s a pretty decent (though brief) writeup on howtoforge that explains a bit more about how to go about getting your own munin. Go ahead and check it out — I’ll chill here until you’re back.


That wasn’t too bad, huh? Okay, so at this point you should have a working munin instance. It will take a few minutes of data collection before your graphs look like anything, but going forward, you’ll have historical graphs of several key metrics for managing your server(s). It’s outside the scope of this article, but you can also set up munin to monitor multiple servers on your network, and/or alert you when critical threshold levels are passed.

Well, knowing CPU usage is great, but it would also be nice to have some idea what the average user experience is like on your site. Does your Rails app perform differently at different times of day? How long does it typically take for your app to render a page? Is the majority of the time spent in the database, or rendering? Is the response time about the same between production deployment versions? I wanted answers to these questions too, so I came up with a small ruby program that watches your Rails log in realtime, and when munin asks, provides summary information about the performance of your application. Below, I’ll go over the code section by section; for those who want it now, scroll to the bottom of the article to find the download link.

The basis of the solution is a Ruby array: we stuff values into it, then compute the average of all the values and clear the array each time munin pings us. Every time a value is added, we also check it against the maximum already seen, so we can report the maximum response time in addition to the average. We keep three of these objects around, one each for DB, Rendering, and Total response times. There is also a mode that lets you look at the current value without consuming it — useful for peeking inside without affecting the data that munin will ultimately see. Here’s the class that implements this functionality:

class Accumulator
def initialize
@values = Array.new()
@max = 0
end
def add(value)
@values << value
@max = value if value > @max
end
def average(read_only=false)
return_value = if @values.length == 0
nil
else
@values.inject(0) {|sum,value| sum + value } / @values.length
end
@values = Array.new() unless read_only

return_value
end

def max(read_only=false)
return_value = @max
@max = 0 unless read_only
return_value
end
end

In the next section of the code, we build our accumulators, and begin tailing the logfile to extract performance numbers. This requires the file-tail gem, available from rubyforge. Note that in my setup, this file resides in a subdirectory under lib in RAILS_ROOT. If you choose to place this file elsewhere, you’ll have to adjust the path to the logfile accordingly. Another thing to note: in our environment, the load balancer continually pings a “heartbeat” action on each node to make sure it is still responsive. As we will be hitting this action repeatedly, it is engineered to be as lightweight as possible. Therefore, any numbers from it are pretty meaningless to our overall statistics, so we don’t want to include them. To keep these numbers from skewing our results, we define an IGNORE_PATTERNS regexp (earlier in the code). If the request matches a pattern we want to ignore, its statistics are not collected.

LOGFILE = File.join(File.dirname(FILE), ‘..’, ‘..’, ‘log’, “#{RAILS_ENV}.log”)
$response_data = { :total => Accumulator.new(),
:rendering => Accumulator.new(),
:db => Accumulator.new() }

Thread.abort_on_exception = true
logtail = Thread.new do
File::Tail::Logfile.tail(LOGFILE) do |line|
if line =~ /^Completed in /
parts = line.split(/s+|s+/)
resp = parts.pop
requested_url = resp[/http://[^]]*/]
next if requested_url =~ IGNORE_PATTERNS
parts.each do |part|
part.gsub!(/Completed in/, “total”)
type, time, pct = part.split(/s+/)
type = type.gsub(/:/,’’).downcase.to_sym

$response_data[type].add(time.to_f)
end
end
end
end

So now we have a thread busy gathering our data—how can we expose the data to a munin plugin? There are multiple ways to do this, but I chose to use a small HTTP server listening to requests from the local machine only. We could build such a thing in Rails, but we really don’t need about 90% of the features Rails has to offer. Since we’re running Rails as a Mongrel cluster, we already have a perfect tool at our disposal for writing small HTTP request handlers in Ruby: Mongrel. Here are a couple of pages about how to get started writing Mongrel handlers —it’s pretty straightforward. Here’s our handler:

class ResponseTimeHandler < Mongrel::HttpHandler
def initialize(method)
@method = method
end
def process(request, response)
response.start(200) do |head,out|
debug = Mongrel::HttpRequest.query_parse(request.params[“QUERY_STRING”]).has_key? “debug”
head[“Content-Type”] = “text/plain”
output = $response_data.map do |k,v|
value = v.send(@method, debug)
formatted = value.nil? ? ‘U’ : sprintf(‘%.5f’, value)

“#{k}.value #{formatted}”
end.join(“n”)
output << “n”
out.write output
end
end
end

h = Mongrel::HttpServer.new(“127.0.0.1”, PORT)
h.register(“/avg_response_time”, ResponseTimeHandler.new(:average))
h.register(“/max_response_time”, ResponseTimeHandler.new(:max))
h.run.join

The handler is generic so that it can call an arbitrary method on our collection of data arrays, so we can set up one URI for the average, and one for the maximum. With this in place, we can write a simple munin plugin script that uses Net::HTTP to query our Mongrel to get at the performance data.

Basically, a munin plugin has two usage scenarios. When called with the single argument “config”, it should output information about itself in a format that Munin understands. This includes how to label the chart, scaling information, how many series will be included on the chart, etc. When called with no arguments, the plugin should output the current values of each series. For more information on writing your own munin plugin, start with the HowToWritePlugins munin wiki page. And now, our plugin script:

#!/usr/bin/env ruby

  1. munin plugin to render rails response time graphs
  2. link to /etc/munin/plugins/avg_response_time and /etc/munin/plugins/max_response_time

require ‘open-uri’
PORT = ENV[‘PORT’] || “8888”

def config
title = File.basename($0).split(‘_’).map{|s| s.capitalize }.join(’ ’)
config=<<END_CONFIG
graph_title #{title}
graph_vlabel response time
graph_category rails
total.label total
rendering.label rendering
db.label db
END_CONFIG
puts config
end

def get_data(read_only=false)
qs = read_only ? ‘?debug’ : ’’
puts open(“http://127.0.0.1:#{PORT}/#{File.basename($0)}#{qs}”).read
end

case ARGV.first
when ‘config’
config
when ‘debug’
get_data(true)
else
get_data
end

The script will examine the name with which you linked it in to the munin plugins directory to determine which URI to query. I have also added a debug mode that will show you the current values, so you’re not consuming any data that munin needs to see for an accurate graph. The final piece is a small Daemons wrapper script to control the main log-tailing process, and you should be set. Make sure to restart munin-node so it will notice the new plugins, and after a while, you’ll see something like this:

It is worth noting that the numbers from the Rails log might not be 100% accurate, and this won’t replace the results that you can get from seriously profiling your application. Also, the information you are getting is a bit generic — all actions are lumped together, so there is not a lot of information about the cause of the performance problem. But, for insight into your production application performance, this setup should at least give you some indications about how well your baby is playing in the interwebs.

Download rails_log_monitor.rb

Download rails_response_time

Focus of the day.

I am writing this to make sure I stay focused today. There are currently four projects in my head… MySpyder.net sorry, but not you today. That’s three left. Then there is MunchAway which I need to put on hold unil Monday, where I should Ajaxify the login. Ok, that leaves the “BOOK” and Digital-Seed.

For the book I should

1) write a script to generate the whole model and controllers using generate scaffolding…that will let me rerun it a couple of times until we get it right. I.e. rake scaffold:migrate [VERSION=n] allowing to add and remove stuff…this should also take care of nested resource.

2) Write some initial rSpecs to ensure that the model is right

For DigitalSeed I need

1) integrate the LayoutManager functionality into the base application

2) Refine the Widget drap&drop to visually indicate where the dragged item will appear if dropped.

The Book will be for tonight once the kids and wife are asleep…so let’s get started with Digital-Seed!

Unobtrusive Javascript with Lowpro and Ruby On Rails

Check out http://www.danwebb.net/lowpro for more info on LowPro, a very elegant approach to do Unobtrusive Javascript with Ruby On Rails. Find hereafter a small example of how to add a custom behavior to link.

The View

<%= javascript_include_tag ‘prototype’, ‘lowpro’, ‘remote’, ‘application’ >

    < for watch_result in watch_results %>
    <li>
    <%= link_to watch_result.created_at.to_s(:db),
    diff_watch_result_url(
    watch, watch_result),
    {:id => dom_id(watch_result) }
    %>

    <% end %>


The Javascript

LoadWatchResult = Remote.Link({
onLoading : function() {
$(‘watch_result_difference’).innerHTML=‘’;
$(’watch_result_difference’).addClassName(‘pleaseWait’);
},
onComplete : function(e) {
var source = Event.element(e);
$(‘watch_result_difference’).removeClassName(‘pleaseWait’);
$$(‘div#result_list ul a.active’).each(function (e) {e.removeClassName(‘active’)});
source.addClassName(‘active’);
}
});

Event.addBehavior({
‘#result_list ul li a’: LoadWatchResult
});

The ‘remote.js’ provides additional behaviors creatde by Dan Wedb as part of LowPro (http://svn.danwebb.net/external/lowpro/trunk/behaviours/). The LoadWatchResult behavior we created in this example transforms a ‘standard’ link_to to a link_to_remote with additional behavior on the onLoading and onComplete of the remote call. The view stays clean.

Enjoy!
Daniel

#68 OpenID Authentication

Ever wonder how to implement OpenID authentication in your Rails app? This episode will show you how to add it to a site with an existing authentication system.

#68 OpenID Authentication

Ever wonder how to implement OpenID authentication in your Rails app? This episode will show you how to add it to a site with an existing authentication system.

Weblog update

Hey all I wanted to give you an update:

Three of our active contributors are moving on: Aaron, Eddie, and Val. They plan to stay active in the Rails community, as well as new areas like Facebook. You can follow their work at blog.hungrymachine.com. It has been a great pleasure to work with them and we wish them the best.

Weblog update

Hey all I wanted to give you an update:

Three of our active contributors are moving on: Aaron, Eddie, and Val. They plan to stay active in the Rails community, as well as new areas like Facebook. You can follow their work at blog.hungrymachine.com. It has been a great pleasure to work with them and we wish them the best.

eRubyCon – Bruce Tate – Ruby on Rails Podcast

Robert Stevenson interviews Bruce Tate about Changing the Present and Ruby in the enterprise.
From eRubyCon in Columbus, OH.

eRubyCon – Bruce Tate – Ruby on Rails Podcast

Robert Stevenson interviews Bruce Tate about Changing the Present and Ruby in the enterprise.
From eRubyCon in Columbus, OH.

#67 restful_authentication

Need multiple user authentication? If so, the restful_authentication plugin is a great way to go. It will generate some basic authentication code for you which is good starting point to your authentication system. Watch this episode for details.

#67 restful_authentication

Need multiple user authentication? If so, the restful_authentication plugin is a great way to go. It will generate some basic authentication code for you which is good starting point to your authentication system. Watch this episode for details.

Testing the Right Stuff

        I’m going to take a slightly different tack here, and review some of the unit tests in rails itself.  They show up two common anti patterns, spurious assertions  and coupling your tests to the implementation.


Perhaps the biggest benefit of a suite of unit tests is that they can provide a safety net, preventing you from accidentally adding new bugs or introducing regressions of old bugs.  With a large codebase, the unit tests can also help new developers understand your intent, though they’re <a href="http://fishbowl.pastiche.org/2006/10/08/comment_your_performance_hacks">no substitute for comments</a>.  However if you’re not careful with what gets included in your test cases, you can end up with a liability.


<h2> Be careful what you assert</h2>


Whenever you add an assertion to your test suite you’re sending a signal to future developers that the behaviour you’re asserting is both <strong>intentional</strong> and <strong>desired</strong>.  Future developers who try to refactor your code will see a failing <!--more--> and either give up, or waste time trying to figure out if the assertion is ‘real’ or whether it was merely added because that’s what the code happened to do at present.


For an example, take the <a href="http://dev.rubyonrails.org/browser/trunk/activerecord/test/associations_test.rb?rev=7314#L1464">test_hatbm_attribute_access_and_respond_to from associations_test.rb</a> , especially  the assertions that the project responds to access_level= and joined_on=.  Because of the current implementation of respond_do?, those assertions pass.  But should they?


In reality while those values will get stored in the object, they’ll never be written back to the database.  This is a surprising result for some developers, and removing those accessor methods would go a long way to helping avoid some frustrating moments.


<h2> Mock and Stub with care</h2>


Mock object frameworks like flexmock and mocha make it really easy to test how your code interacts with another system or a third party library.  However you should make sure that the thing that you’re mocking doesn’t merely reflect the <strong>current</strong> implementation of a method.   To take a case from rails,  take a look at <a href="http://dev.rubyonrails.org/browser/trunk/actionpack/test/controller/routing_test.rb?rev=7314#L198">setup_for_named_route in routing_test.rb</a>.


It takes the seemingly sensible approach of building a stubbed-out implementation of url_for instead of trying to build a full implementation into the test cases.  The stubbed version of url_for simply returns the arguments it was passed, this makes it extremely easy to work with and to test.


The problem is not with stubbing out the method, but in the way it is used in all the named route test cases.  Take <a href="http://dev.rubyonrails.org/browser/trunk/actionpack/test/controller/routing_test.rb?rev=7314#L191">test_named_route_with_nested_controller</a>.
1
2
3
4
5
6
def test_named_route_with_nested_controller
  rs.add_named_route :users, 'admin/user', :controller => '/admin/user', :action => 'index'
  x = setup_for_named_route.new
  assert_equal({:controller => '/admin/user', :action => 'index', :use_route => :users, :only_path => false},
  x.send(:users_url))
end
The strange hash value you see in the assertion is the result of the named route method calling url_for, and returning that.  The <a href="http://dev.rubyonrails.org/browser/trunk/actionpack/lib/action_controller/routing.rb?rev=7314#L1169">current implementation of the named route helpers</a> does this, but what if you wanted to <a href="http://git.koziarski.com/?p=.git;a=shortlog;h=fast_path_method">implement a new version of named routes which completely avoids the costly call to url_for</a>?  Every single named route test fails, even though applications which use those methods will work fine.


In this situation you have two options, you could make your tests depend on the full implementation of url_for.  This would probably slow down your test cases, and require a lot more setup code, but because the return values are correct you’re not likely to impede future refactoring.


The other option is to use different stubs for every test case.  Leaving you with something like this:
1
2
3
4
5
6
7
def test_named_route_with_nested_controller
  rs.add_named_route :users, 'admin/user', :controller => '/admin/user', :action => 'index'
  generated_url = "http://test.named.routes/admin/user"
  x = setup_for_named_route.new
  x.stubs(:url_for).returns(generated_url)
  assert_equal(generated_url,  x.send(:users_url))
end
Doing this for each and every test case is going to be quite time consuming and make your test cases extremely verbose.  As with all things in software you’ll have to make a judgement call on this trade off and make a choice between coupling or verbosity.


Whatever approach you choose, just remember that misleading test ‘failures’ can slow down refactoring, and end up <strong>reducing</strong> your ability to respond to change.  As satisfying as ‘100% coverage’ or 2:1 ratios may be, don’t blindly add assertions or mock objects just to satisfy a tool.   Every line in your test cases should be there for a reason, and should be placed there with just as much care as you’d use for a line of application code.

Testing the Right Stuff

I’m going to take a slightly different tack here, and review some of the unit tests in rails itself. They show up two common anti patterns, spurious assertions and coupling your tests to the implementation.

Perhaps the biggest benefit of a suite of unit tests is that they can provide a safety net, preventing you from accidentally adding new bugs or introducing regressions of old bugs. With a large codebase, the unit tests can also help new developers understand your intent, though they’re no substitute for comments. However if you’re not careful with what gets included in your test cases, you can end up with a liability.

Be careful what you assert

Whenever you add an assertion to your test suite you’re sending a signal to future developers that the behaviour you’re asserting is both intentional and desired. Future developers who try to refactor your code will see a failing test, and either give up, or waste time trying to figure out if the assertion is ‘real’ or whether it was merely added because that’s what the code happened to do at present.

For an example, take the test_hatbm_attribute_access_and_respond_to from associations_test.rb , especially the assertions that the project responds to access_level= and joined_on=. Because of the current implementation of respond_do?, those assertions pass. But should they?

In reality while those values will get stored in the object, they’ll never be written back to the database. This is a surprising result for some developers, and removing those accessor methods would go a long way to helping avoid some frustrating moments.

Mock and Stub with care

Mock object frameworks like flexmock and mocha make it really easy to test how your code interacts with another system or a third party library. However you should make sure that the thing that you’re mocking doesn’t merely reflect the current implementation of a method. To take a case from rails, take a look at setup_for_named_route in routing_test.rb.

It takes the seemingly sensible approach of building a stubbed-out implementation of url_for instead of trying to build a full implementation into the test cases. The stubbed version of url_for simply returns the arguments it was passed, this makes it extremely easy to work with and to test.

The problem is not with stubbing out the method, but in the way it is used in all the named route test cases. Take test_named_route_with_nested_controller.

1
2
3
4
5
6
def test_named_route_with_nested_controller
  rs.add_named_route :users, 'admin/user', :controller => '/admin/user', :action => 'index'
  x = setup_for_named_route.new
  assert_equal({:controller => '/admin/user', :action => 'index', :use_route => :users, :only_path => false},
  x.send(:users_url))
end

The strange hash value you see in the assertion is the result of the named route method calling url_for, and returning that. The current implementation of the named route helpers does this, but what if you wanted to implement a new version of named routes which completely avoids the costly call to url_for? Every single named route test fails, even though applications which use those methods will work fine.

In this situation you have two options, you could make your tests depend on the full implementation of url_for. This would probably slow down your test cases, and require a lot more setup code, but because the return values are correct you’re not likely to impede future refactoring.

The other option is to use different stubs for every test case. Leaving you with something like this:

1
2
3
4
5
6
7
def test_named_route_with_nested_controller
  rs.add_named_route :users, 'admin/user', :controller => '/admin/user', :action => 'index'
  generated_url = "http://test.named.routes/admin/user"
  x = setup_for_named_route.new
  x.stubs(:url_for).returns(generated_url)
  assert_equal(generated_url,  x.send(:users_url))
end

Doing this for each and every test case is going to be quite time consuming and make your test cases extremely verbose. As with all things in software you’ll have to make a judgement call on this trade off and make a choice between coupling or verbosity.

Whatever approach you choose, just remember that misleading test ‘failures’ can slow down refactoring, and end up reducing your ability to respond to change. As satisfying as ‘100% coverage’ or 2:1 ratios may be, don’t blindly add assertions or mock objects just to satisfy a tool. Every line in your test cases should be there for a reason, and should be placed there with just as much care as you’d use for a line of application code.

#66 Custom Rake Tasks

Rake is one of those tools that you don’t realize how powerful it is until you start using it. In this episode you will learn how to create custom rake tasks and improve them by using rake features.

#66 Custom Rake Tasks

Rake is one of those tools that you don’t realize how powerful it is until you start using it. In this episode you will learn how to create custom rake tasks and improve them by using rake features.

FragmentFu – Fun with Fragments

First, please read the Advanced Rails Caching.. On the Edge for a primer on ESI, caching, and fragments.
To get started developing with ESI, I’m going to walk through a simple tutorial building an ESI enabled Rails application.
Get Mongrel-ESI
Find the latest mongrel ESI here: http://code.google.com/p/mongrel-esi/downloads/list


wget http://mongrel-esi.googlecode.com/files/mongrel-esi-0.0.5.gem

(Does anyone have a better solution of ruby projects hosted on Google Code? )

Install Mongrel ESI


sudo gem install mongrel-esi-0.0.5.gem

Create a new rails application (Currently using rails 1.2.3)


rails fragment_fu_demo

create
create app/controllers
create app/helpers
create app/models
.....

cd fragment_fu_demo


Create a home page for your application

Delete the public/index.html page


rm public/index.html

Edit the config/routes.rb and uncomment the following line:


map.connect '', :controller => "welcome"

Create a welcome controller


./script/generate controller welcome index

Install the FragmentFu plugin

   
./script/plugin install http://mongrel-esi.googlecode.com/svn/trunk/plugin/fragment_fu

Start

Continue reading “FragmentFu – Fun with Fragments”

FragmentFu – Fun with Fragments

First, please read the Advanced Rails Caching.. On the Edge for a primer on ESI, caching, and fragments.
To get started developing with ESI, I’m going to walk through a simple tutorial building an ESI enabled Rails application.
Get Mongrel-ESI
Find the latest mongrel ESI here: http://code.google.com/p/mongrel-esi/downloads/list


wget http://mongrel-esi.googlecode.com/files/mongrel-esi-0.0.5.gem

(Does anyone have a better solution of ruby projects hosted on Google Code? )

Install Mongrel ESI


sudo gem install mongrel-esi-0.0.5.gem

Create a new rails application (Currently using rails 1.2.3)


rails fragment_fu_demo

create
create app/controllers
create app/helpers
create app/models
.....

cd fragment_fu_demo


Create a home page for your application

Delete the public/index.html page


rm public/index.html

Edit the config/routes.rb and uncomment the following line:


map.connect '', :controller => "welcome"

Create a welcome controller


./script/generate controller welcome index

Install the FragmentFu plugin

   
./script/plugin install http://mongrel-esi.googlecode.com/svn/trunk/plugin/fragment_fu

Start your application

 
./script/server

visit http://localhost:3000 in your browser. You should see a “Welcome#index” page.

Start MongrelEsi
From the command line, run:


mongrel_esi start
** Starting Mongrel listening at 0.0.0.0:2000

visit http://localhost:2000 in your browser. You should see a “Welcome#index” page.

Create a fragment to cache
Edit the app/controllers/welcome_controller.rb
Add an action called now

 
def now
render :text => "#{Time.now} is #{Time.now.usec} nano-seconds"
end

Edit the app/views/welcome/index.html.erb and replace with the following:


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now" %></p>

Page without ESI parsing
visit http://localhost:3000 and you should see: “Time:” in the browser. If you view the source, you’ll see


<h1>Welcome</h1>
<p><esi:include src="/welcome/now" max-age="0"/></p>

This is the simplest of esi tags, and the default max-age is 0, which means do not cache.

Page with ESI Parsing

visit http://localhost:2000 and you’ll see:

Welcome
Date: Mon Aug 06 00:00:00 -0400 2007 which is 62972ns

Caching modules
Edit the index.html.erb and replace with the following:


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now" %></p>
<h2>What time is it again?</h2>
<p><%= render :esi => "/welcome/now"%></p>

If you refresh http://localhost:2000, you’ll notice they nanoseconds are different

Lets add a ttl to first call


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now", :ttl => 45.seconds %></p>
<h2>What time is it again?</h2>
<p><%= render :esi => "/welcome/now"%></p>

If you refresh http://localhost:2000, you’ll notice they are now the same. Refresh again in the next 45 seconds, and it will not change.

The next tutorial will cover a small TODO application, with inline invalidation and exception handling. Coming Soon!

Advanced Rails Caching.. on the Edge

When trying to scale a portal to millions of users with complex personalization, we had to rethink the application design. We needed partial page cacheability, and we wanted that close to the Edge… and recently the idea has been catching on. First with Components Are the New Black, and the idea of Nginx, SSI, and Memcache, what I’d like to briefly describe below is a partial page caching strategy that has worked well for the past year.

First, we use a technology called ESI. ESI stands for Edge Side Includes, which is a simple markup language for describing dynamic assembly of applications. At the core, its similar to SSI, but its a more versatile spec that has been accepted/implemented by a half dozen cache servers, both open source (Squid, Mongrel-ESI) and commercial (Oracle Web Cache, Akamai, among others). It also includes

Twitter

Continue reading “Advanced Rails Caching.. on the Edge”

Advanced Rails Caching.. on the Edge

When trying to scale a portal to millions of users with complex personalization, we had to rethink the application design. We needed partial page cacheability, and we wanted that close to the Edge… and recently the idea has been catching on. First with Components Are the New Black, and the idea of Nginx, SSI, and Memcache, what I’d like to briefly describe below is a partial page caching strategy that has worked well for the past year.

First, we use a technology called ESI. ESI stands for Edge Side Includes, which is a simple markup language for describing dynamic assembly of applications. At the core, its similar to SSI, but its a more versatile spec that has been accepted/implemented by a half dozen cache servers, both open source (Squid, Mongrel-ESI) and commercial (Oracle Web Cache, Akamai, among others). It also includes an invalidation protocol, exception handling on the edge, and a few other features.

The ESI W3C specification has been out for 6+ years, so these ideas are not new, but ESI is a diamond in the rough. No one seems to be using it.

A simple example of ESI in your rails application is including a common header. If the header is static, using a shared layout, or rendering a shared partial across applications, could be sufficient. But if you use the common idiom of a sign-in bar, like “Welcome Bob – Signout”, on a page you want to fully cache, or a common header you want to share across multiple applications, then you may consider another approach.

If you leverage ESI, you can put the below in your application layout.

<div id="header">
<esi:include src="/header" max-age="300"/>
</div>

The <esi:include> tag will inject the response from /header into the page and cache that fragment for 300 seconds. You can also add a variety of options, request header parameters (which can be used to personalize the request) along with a dozen other optional parameters, some of which I will outline below. This is simple SSI (Server Side Includes) .

But if we take it a step further, lets look at how we can apply it to a site like Twitter.

Twitter

If you look at the image above, the majority of the page can be fully cached. I’ve outlined two green boxes, which appear to be dynamic content, which can have different TTLs. Using ESI, your markup could be:

<div id="latest">
<esi:include src="/latest" max-age="5"/>
</div>
...
<esi:include src="/featured" max-age="3600"/>

The ESI enabled cache server would parse this markup, make 2 separate HTTP requests (/latest, and /featured) and cache those with their corresponding TTLs. You can furthermore cache the wrapping template with a Surrogate-Control header, which tells the cache server to keep a cached copy of the template. A request to this page 4 seconds later would make 0 requests to your rails infrastructure. 8 seconds later would only hit the /latest, returning the rest of the page from cache. You can also envision a application pool of servers just to handle /latest, but I’ll get into sharding applications via ESI in a later article.

Exception Handling

If you take this a step further, you can also define timeouts and exception behavior. As defined below, If the /latest request takes more than 1s, The cache server will give up, and retrieve the static snippet defined in the except block from your static web host.

<esi:try>
<esi:attempt>
<esi:include src="/latest" max-age="5" timeout="1"/>
</esi:attempt>
<esi:except>
<esi:include src="http://static.foo.com/latest" max-age="5" timeout="1"/>
</esi:except>
</esi:try>

We call these “Sorry” modules, because they often say “Sorry, this feature is having trouble”, but the rest of the page may surface meaningful content. Even better, write out more meaningful content to disk once a day and serve that from Apache.


Invalidation

The other benefit of ESI is Invalidation support. There are various mechanisms to invalidate content, but my favorite is Inline Invalidation. Consider the common rails idiom of updating data. You post to a controller which redirects to a view of that data. Since the HTTP redirect (301) bubbles all the way back to the browser, any content in the body of the redirect can be parsed by the ESI server. Therefore you can put the invalidation xml is the redirect response body and not conditionally dirty your view logic.

<esi:invalidate>
<?xml version="1.0"?>
<!DOCTYPE INVALIDATION SYSTEM "internal:///WCSinvalidation.dtd">
<INVALIDATION VERSION="WCS-1.1">
<OBJECT>
<BASICSELECTOR URI="/foo/bar/baz"/>
<ACTION REMOVALTTL="0"/>
</OBJECT>
</INVALIDATION>
</esi:invalidate>

The above is a simple single URL example, but the specification supports regex, which would work well for restful resources (e.g. /people/#{person.id}/*), among other things.


FragmentFu

FragmentFu is a plugin that enables ESI support in rails applications. At this point it is an extraction from our internal plugin for modules, caching, and ESI. Its in active extraction/development, but I wanted to involve the community in gathering ideas, suggestions and comments.

Some sample snippets:

<% render :esi => widget_url(1) %>

<%= render :esi => latest_url, :ttl => 5.seconds, :timeout => 1.second %>

<%= render :esi => featured_url, :ttl => 10.hours, :except => '/some/static/path' %>

We’ve also toyed around with the idea of fragment respond_to’s. Since ESI supports adding request header parameters, we can mimic Prototype’s
X-Requested-With behavior to implement:

def latest
...
respond_to |wants| do
wants.html { do something }
wants.fragment { do something }
end
end


How to get started

Mongrel-ESI is a great starting place. It was built in-house and released a few months ago on Google Code. It supports a large subset of ESI, ESI Invalidation, and its a great tool for developing applications that utilize ESI. You can grab Mongrel-ESI, download the Oracle Web Cache Standalone or even Squid 3.0… and when you’re site gets really big, its time to give Akamai a call. 🙂

#65 Stopping Spam with Akismet

The Railscasts site has been getting a lot of comment spam in the past, but no longer. In this episode I will show you how I solved this problem by using the Akismet web service.