Dentaku – a calculator for Ruby

I’m currently working on a project that requires the use of formulas or expressions to implement business rules (the original system is an Excel workbook), and I needed a way to implement some of the same functionality. My first thought was to model the formulas in ActiveRecord, clause by clause, using something like `acts_as_nested_set` to handle hierarchies and grouping. However, I’ve done something similar before, and was never quite satisfied with the result — it always felt “heavy” — so I decided to try a different approach this time.

The result is Dentaku, a gem that parses and evaluates Excel-like formulas, allowing the use of named variables that will be substituted for real values at run time. So far, the result values will be either boolean or numeric, but strings will probably be added soon.

To illustrate how it works, let me set up the following (quite contrived) hypothetical situation: You’re building an upvote system for a Reddit-type site, but the customers do not want to encode all the logic for the system in Ruby — they need to be able to manage all the rules on the fly. However, they are willing to be limited to only using a few predefined attributes of the user as all variable inputs to the rules.

The initial ruleset is something like this: A vote from a “normal” user is worth one point for each week the user has been with the site. A vote from an “admin” user is worth 100 points. A vote from a “limited” user is worth one point, and a vote from a “blacklisted” user is worth zero points. However, users can earn achievements, which update a “bonus” number for the user, and any normal or limited user’s vote points are augmented with the bonus, but only after the user has been with the site for at least five weeks. (Hey, I promised contrived, right?) So how could we accomplish this with Dentaku?

We could create the following rules:


    +-------------------------------+----------------------+  
    | conditions                    | points               |  
    +-------------------------------+----------------------+  
    | admin                         | 100                  |  
    | normal and age_in_weeks < 5   | age_in_weeks         |  
    | limited and age_in_weeks < 5  | 1                    |  
    | normal and age_in_weeks >= 5  | age_in_weeks + bonus |  
    | limited and age_in_weeks >= 5 | 1 + bonus            |  
    | blacklisted                   | 0                    |  
    +-------------------------------+----------------------+  

Then, given a hash representing the user like:

    user = {
      :admin        => false,
      :normal       => true,
      :limited      => false,
      :blacklisted  => false,
      :age_in_weeks => 10,
      :bonus        => 8
    }

we could calculate the points for a vote by the represented user like so:

    def calculate_points_for(user, rules)
      calculator = Dentaku::Calculator.new

      rules.each do |rule|
        if calculator.evaluate(rule.conditions, user)
          return calculator.evaluate(rule.points, user)
        end
      end

      # no rules matched, default to zero
      0
    end

So that’s what Dentaku is about — I hope someone else finds it useful. You can check it out, fork it, etc at the Dentaku Github page, or just `gem install dentaku` and start playing!

Termistat : a status bar for your terminal

When running background processes that produce detail logging, it’s often difficult to strike the right balance between providing overall status information and details about the current step in the process. It’s helpful to be able to see “tail-like” information at the detail level to monitor and debug your processes; however, it’s also helpful to be able to know summary information, such as the overall progress through the entire task. You can intersperse “record 1 of n” lines in your output, but they are easy to miss in all the noise.

In order to be able to display both types of information concurrently, I built a simple gem called termistat, which allows you to display a status bar for summary information at the top of your terminal in addition to the original detailed output. It was meant to be a whyday contribution, but I didn’t quite finish it in time to be released on whyday…oh well.

Here’s a screenshot of termistat in action:
termistat screenshot

Termistat requires the ffi-ncurses gem (which requires the ncurses library to be on your system), and has a configuration DSL to customize the appearance somewhat. Check it out and let me know if you have any ideas for improvement!

Termistat : a status bar for your terminal

When running background processes that produce detail logging, it’s often difficult to strike the right balance between providing overall status information and details about the current step in the process. It’s helpful to be able to see “tail-like” information at the detail level to monitor and debug your processes; however, it’s also helpful to be able to know summary information, such as the overall progress through the entire task. You can intersperse “record 1 of n” lines in your output, but they are easy to miss in all the noise.

In order to be able to display both types of information concurrently, I built a simple gem called termistat, which allows you to display a status bar for summary information at the top of your terminal in addition to the original detailed output. It was meant to be a whyday contribution, but I didn’t quite finish it in time to be released on whyday…oh well.

Here’s a screenshot of termistat in action:
termistat screenshot

Termistat requires the ffi-ncurses gem (which requires the ncurses library to be on your system), and has a configuration DSL to customize the appearance somewhat. Check it out and let me know if you have any ideas for improvement!

Making CRUD less “Cruddy”, one step at a time

One of the great “new” features of Rails (as of 2.3) is accepts_nested_attributes_for, allowing you to build cross-model CRUD forms without “cruddying” your controller. There are some great examples out there about how to do this, but I’d like to walk thorough a particular use case — managing the “join” records in a has_many :through relationship.

Consider the following database schema:

class Villain < ActiveRecord::Base
  has_many :gifts
  has_many :super_powers, :through => :gifts
end

class Gift < ActiveRecord::Base
  belongs_to :villain
  belongs_to :super_power

  validates_uniqueness_of :super_power, :scope => :villain_id
end

class SuperPower < ActiveRecord::Base
  has_many :gifts
  has_many :villains, :through => :gifts
end

In our dataset, there are a relatively small number of super powers which we wish to present as a list of checkboxes on the villain management form. Checking/unchecking the boxes will manage the gift records for that villain, effectively managing the list of super powers available to the baddy.

To get started, we need to add accepts_nested_attributes_for :gifts to the Villain class — piece of cake. To complete the implementation, we need to change the params hash that our form generates. Let’s review the cases that we need to support and the associated params hash format needed to implement the correct functionality.

The first case is a super power record that is not currently associated with the villain. Here, the UI should display an unchecked checkbox. If we check it and submit the form, a gift record should be created linking the villain with the super power, making this bad guy that much badder. Here is an example of the params hash we should be sending to accomplish this:

  {
    'villain' => {
      'name' => 'Lex Luthor',
      ...
      'gifts_attributes' => {
        1 => { 'super_power_id' => 5 },
        2 => { 'super_power_id' => 7 },
        ...
      }
    }
  }

The alternate case is a super power this villain already possesses. In this instance, the UI should display a checked checkbox, and if we uncheck it, the existing gift record should be deleted, diminishing the villain’s capacity for evil. And our params hash needs to look like:

  {
    'villain' => {
      'name' => 'Two-Face',
      ...
      'gifts_attributes' => {
        1 => { 'id' => 101, '_delete' => true },
        ...
      }
    }
  }

Note that the keys for the gifts_attributes hash are arbitrary; we can use any scheme to generate unique keys for the hash.

So how can we craft a form that sends the params hash that Rails wants to see? Here’s my implementation:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%- if gift = @villain.gifts.find_by_super_power_id(super_power.id) -%>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][id]", gift.id %>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][_delete]", false, true %>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][_delete]", true %>
      <%- else -%>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][super_power_id]", super_power.id %>
      <%- end -%>

      <%= super_power.name %>
    </label><br />
  <%- end -%>

If the gift is detected, the villain has the super power, and we handle our second case from above, using the checkbox / hidden field hack Rails employs in the check_box helper method to make sure a value is sent whether or not the checkbox is checked. The else block handles the other case, setting up our params hash to create the gift if the checkbox is checked.

This works, but we probably don’t want to copy and paste that code everywhere we use this pattern. How can we reuse this in a DRY fashion? Here’s a helper method that encapsulates the logic:

  def has_join_relationship(model, join_collection_name, related_item, collection_index, options={})
    returning "" do |output|
      relationship_name = options[:relationship_name] || related_item.class.table_name.singularize + "_id"
      tag_prefix = "#{ model.class.class_name.underscore }[#{ join_collection_name }_attributes][#{ collection_index }]"

      if join_item = model.send(join_collection_name).find(:first, :conditions => { relationship_name => related_item.id })
        output << hidden_field_tag("#{ tag_prefix }[id]", related_item.id)
        output << check_box_tag("#{ tag_prefix }[_delete]", false, true)
        output << hidden_field_tag("#{ tag_prefix }[_delete]", true)
      else
        output << check_box_tag("#{ tag_prefix }[#{ relationship_name }]", related_item.id, false)
      end
    end
  end

Drop that in a helper, and then your form code becomes:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%= has_join_relationship(@villain, :gifts, super_power, index) %>
      <%= super_power.name %>
    </label><br />
  <%- end -%>

Much nicer … although I’m not sold on the name has_join_relationship. Any suggestions?

Making CRUD less "Cruddy", one step at a time

One of the great “new” features of Rails (as of 2.3) is accepts_nested_attributes_for, allowing you to build cross-model CRUD forms without “cruddying” your controller. There are some great examples out there about how to do this, but I’d like to walk thorough a particular use case—managing the “join” records in a has_many :through relationship.

Consider the following database schema:

class Villain < ActiveRecord::Base
  has_many :gifts
  has_many :super_powers, :through => :gifts
end

class Gift < ActiveRecord::Base
  belongs_to :villain
  belongs_to :super_power

  validates_uniqueness_of :super_power, :scope => :villain_id
end

class SuperPower < ActiveRecord::Base
  has_many :gifts
  has_many :villains, :through => :gifts
end

In our dataset, there are a relatively small number of super powers which we wish to present as a list of checkboxes on the villain management form. Checking/unchecking the boxes will manage the gift records for that villain, effectively managing the list of super powers available to the baddy.

To get started, we need to add accepts_nested_attributes_for :gifts to the Villain class—piece of cake. To complete the implementation, we need to change the params hash that our form generates. Let’s review the cases that we need to support and the associated params hash format needed to implement the correct functionality.

The first case is a super power record that is not currently associated with the villain. Here, the UI should display an unchecked checkbox. If we check it and submit the form, a gift record should be created linking the villain with the super power, making this bad guy that much badder. Here is an example of the params hash we should be sending to accomplish this:

  {
    'villain' => {
      'name' => 'Lex Luthor',
      ...
      'gifts_attributes' => {
        1 => { 'super_power_id' => 5 },
        2 => { 'super_power_id' => 7 },
        ...
      }
    }
  }

The alternate case is a super power this villain already possesses. In this instance, the UI should display a checked checkbox, and if we uncheck it, the existing gift record should be deleted, diminishing the villain’s capacity for evil. And our params hash needs to look like:

  {
    'villain' => {
      'name' => 'Two-Face',
      ...
      'gifts_attributes' => {
        1 => { 'id' => 101, '_delete' => true },
        ...
      }
    }
  }

Note that the keys for the gifts_attributes hash are arbitrary; we can use any scheme to generate unique keys for the hash.

So how can we craft a form that sends the params hash that Rails wants to see? Here’s my implementation:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%- if gift = @villain.gifts.find_by_super_power_id(super_power.id) -%>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][id]", gift.id %>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][_delete]", false, true %>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][_delete]", true %>
      <%- else -%>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][super_power_id]", super_power.id %>
      <%- end -%>

      <%= super_power.name %>
    </label><br />
  <%- end -%>

If the gift is detected, the villain has the super power, and we handle our second case from above, using the checkbox / hidden field hack Rails employs in the check_box helper method to make sure a value is sent whether or not the checkbox is checked. The else block handles the other case, setting up our params hash to create the gift if the checkbox is checked.

This works, but we probably don’t want to copy and paste that code everywhere we use this pattern. How can we reuse this in a DRY fashion? Here’s a helper method that encapsulates the logic:

  def has_join_relationship(model, join_collection_name, related_item, collection_index, options={})
    returning "" do |output|
      relationship_name = options[:relationship_name] || related_item.class.table_name.singularize + "_id"
      tag_prefix = "#{ model.class.class_name.underscore }[#{ join_collection_name }_attributes][#{ collection_index }]"

      if join_item = model.send(join_collection_name).find(:first, :conditions => { relationship_name => related_item.id })
        output << hidden_field_tag("#{ tag_prefix }[id]", related_item.id)
        output << check_box_tag("#{ tag_prefix }[_delete]", false, true)
        output << hidden_field_tag("#{ tag_prefix }[_delete]", true)
      else
        output << check_box_tag("#{ tag_prefix }[#{ relationship_name }]", related_item.id, false)
      end
    end
  end

Drop that in a helper, and then your form code becomes:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%= has_join_relationship(@villain, :gifts, super_power, index) %>
      <%= super_power.name %>
    </label><br />
  <%- end -%>

Much nicer … although I’m not sold on the name has_join_relationship. Any suggestions?

Cucumber, meet Routes

I’ve been loving Rails BDD with Cucumber for the past year or so—it helps me focus on the next required step to build a feature in my application, and better focus equals better development velocity. However, one thing I found tedious in starting with Cucumber was defining route matchers in paths.rb.

The stock path_to method is implemented as a case statement, allowing you to add a case for each path you want to recognize. This works, but leaves you feeling a bit un-DRY since you’re basically duplicating information in your routes.rb file.

Here’s a quick hack to paths.rb that lets you leverage your existing routes:

change

    else
      raise "Can't find mapping from \"#{page_name}\" to a path.\n" +
        "Now, go and add a mapping in #{__FILE__}"
    end

to

    else
      begin
        page_name =~ /the (.*) page/
        path_components = $1.split(/\s+/)
        self.send(path_components.push('path').join('_').to_sym)
      rescue Object => e
        raise "Can't find mapping from \"#{page_name}\" to a path.\n" +
          "Now, go and add a mapping in #{__FILE__}"
      end
    end

In your Cucumber steps, you can now use any named route that does not require a parameter. For example, users_path would become “the users page”, and new_product_path would become “the new product page”. As you add new resources, at least the index and new options should work out of the box—no further edits to paths.rb required!

UPDATE:

w00t!

This is now baked into cucumber-rails!

Import your MacHeist serials to AppShelf

If you have purchased the latest MacHeist nano bundle, you might have noticed that there is no option this time to export as an AppShelf file. I felt a little guilty spamming my twitter followers to get my three free bonus apps (Airburst Extreme, Tracks, and Burning Monkey Solitaire), so to atone, I’m sharing a script that will translate your reciept into an AppShelf import file. Just save your receipt page from the browser, then pass the filename to this script as an argument:

RUBYOPT=rubygems ruby generate-appshelf.import.rb MacHeist-Serial.html

The script will create an import file named “nano-bundle-3.appshelf” in your current directory. Enjoy!

RMagick (from source) on Snow Leopard

After the release of 10.5, I published an article about building RMagick from source on Leopard. I won’t rehash the why, you can read the original article for that. My clean install necessitated updating the RMagick script, so here’s what worked for me to install from source on Snow Leopard! For the impatient, here’s the download link: rmagick-build.sh

First, we start with installing wget, as it seems to be a bit more clever than curl about dealing with mirrors, etc. Then, we compile and install each prerequisite package. Finally, we install the gem.

All the links in the script worked for me, but, depending on your location, network, conditions, etc, your mileage may vary. Enjoy!

#!/bin/sh

# install wget, which is cleverer than curl
curl -O http://ftp.gnu.org/gnu/wget/wget-1.11.tar.gz
tar zxvf wget-1.11.tar.gz 
cd wget-1.11
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

# prerequisite packages
wget http://nongnu.askapache.com/freetype/freetype-2.3.9.tar.gz
tar zxvf freetype-2.3.9.tar.gz
cd freetype-2.3.9
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget http://superb-west.dl.sourceforge.net/sourceforge/libpng/libpng-1.2.39.tar.gz
tar zxvf libpng-1.2.39.tar.gz
cd libpng-1.2.39
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
tar xzvf jpegsrc.v6b.tar.gz
cd jpeg-6b
ln -s `which glibtool` ./libtool
export MACOSX_DEPLOYMENT_TARGET=10.6
./configure --enable-shared --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget ftp://ftp.remotesensing.org/libtiff/tiff-3.9.1.tar.gz
tar xzvf tiff-3.9.1.tar.gz
cd tiff-3.9.1
./configure --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget http://superb-west.dl.sourceforge.net/sourceforge/wvware/libwmf-0.2.8.4.tar.gz
tar xzvf libwmf-0.2.8.4.tar.gz
cd libwmf-0.2.8.4
make clean
./configure
make
sudo make install
cd /usr/local/src

wget http://www.littlecms.com/lcms-1.17.tar.gz
tar xzvf lcms-1.17.tar.gz
cd lcms-1.17
make clean
./configure
make
sudo make install
cd /usr/local/src

wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/gs870/ghostscript-8.70.tar.gz
tar zxvf ghostscript-8.70.tar.gz
cd ghostscript-8.70
./configure  --prefix=/usr/local
make
sudo make install
cd /usr/local/src

wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/gs860/ghostscript-fonts-std-8.11.tar.gz
tar zxvf ghostscript-fonts-std-8.11.tar.gz
sudo mv fonts /usr/local/share/ghostscript

# Image Magick
wget ftp://ftp.fifi.org/pub/ImageMagick/ImageMagick.tar.gz
tar xzvf ImageMagick.tar.gz
cd `ls | grep ImageMagick-`
export CPPFLAGS=-I/usr/local/include
export LDFLAGS=-L/usr/local/lib
./configure --prefix=/usr/local --disable-static --with-modules --without-perl --without-magick-plus-plus --with-quantum-depth=8 --with-gs-font-dir=/usr/local/share/ghostscript/fonts --disable-openmp
make
sudo make install
cd /usr/local/src

# RMagick
sudo gem install rmagick

UPDATE There is a bug with libgomp that breaks the convert utility (See comments below). the --disable-openmp configure option has been added to the script to fix this.

UPDATE 2 A new patchlevel of ImageMagick has been released that supersedes the original one referenced in this script, and the original has been removed from the server. Thanks to Sebastian for this update that will grab the latest release.

Introducing Hashdown

If your database is normalized, you will almost always end up with small tables (often referred to as reference data or lookup tables) which provide a set of possible values for a particular attribute. (e.g. Currency, Category, etc.) A common pattern that emerges in many applications is accessing these records by a symbolic name (as opposed to by id) for purposes of clarity when reading the code. In C (and friends), the database ids can be mapped to the positions of an enum datatype. I recently released the hashdown plugin that provides hash-like access for reference data records, and also adds some dropdown option list generation support, since this data is often used to populate select list in forms.

As an example of what hashdown does, suppose we have the following model:

class CardType < ActiveRecord::Base
end

with the following data:

+----+-------+------------------+
| id | code  | name             |
+====+=======+==================+
| 1  | visa  | Visa             |
| 2  | mc    | MasterCard       |
| 3  | disc  | Discover         |
| 4  | amex  | American Express |
+----+-------+------------------+

By adding the following line to the model:

class CardType < ActiveRecord::Base
  finder :code
end

You get the functionality of a hash-like square-bracket accessor for the model that will let you do something like:

@order.card_type = CardType[:visa]

The underlying implementation is similar to:

def CardType < ActiveRecord::Base
  def self.[](value)
    find_by_code(value)
  end
end

…except it adds a caching layer to boost performance by preventing repeated database access.

Adding the following directive:

def CardType < ActiveRecord::Base
  selectable
end

to the model gives you a class method called select_options that can be used to populate a select list like this:

<%= form.select :card_type_id, CardType.select_options %>

produces:

<input type="select" name="order[card_type_id]" id="order_card_type_id">
  <option value="1">Visa</option>
  <option value="2">MasterCard</option>
  <option value="3">Discover</option>
  <option value="4">American Express</option>
</input>

By default, this will use the id attribute as the submitted value of the option and call a display_name method (if it exists) for the displayed value of the option, falling back to the name method/attribute. Each of these can be overridden by passing a symbol attribute / method name, or a lambda that will be executed to generate the value. For (a contrived) example:

<%= form.select :card_type_id, CardType.select_options(:key => :code, :value => lambda{|card_type| card_type.name.reverse }) %>

produces:

<input type="select" name="order[card_type_id]" id="order_card_type_id">
  <option value="visa">asiV</option>
  <option value="mc">draCretsaM</option>
  <option value="disc">revocsiD</option>
  <option value="amex">sserpxE naciremA</option>
</input>

Again, the select_option results are cached for better performance.

This is a pretty small plugin that I’m using to DRY up some code in a current project I’m working on. Let me know if you have feature requests (or fork and patch it on GitHub!)

RailsConf 2009 Day Two

Day Two got off to a good start. Engine Yard did a promotional pitch—the speakers could have been a bit more polished, but it was interesting stuff about their one-button-deployment, and overall not bad for an advertisement.

Next up was Chris Wanstrath. He started with a lead in regarding how to become a famous Rails developer—focusing on yourself, your blog readership numbers, your twitter follower count, etc. Later, he talked about how he went from being an unemployed college dropout to co-founder of the very successful GitHub, due to sharing code. His point was that in his eyes, it’s better to focus more on the community: share code, contribute to open source projects, even write documentation for existing projects. Being a good developer trumps being a famous developer. The complete text of the talk is online here .

For the first session of the day, it was a tough call between Rack/Sinatra and Metric Fu. I finally went with:

Using metric_fu to Make Your Rails Code Better – Jake Scruggs

The central theme of this talk was how to use automated code analysis to direct you on where to spend your refactoring cycles. He used Carlin’s law (Anyone going slower than me is stupid; anyone going faster than me is crazy), but applied to programming. As your programming skills change over time, you see the same code differently.

He touched on coverage as a baseline that you should be doing as a part of your code analysis, then went on to complexity analysis, reviewing two tools available to analyze the “complexity” of your code: Flog and Saikuro. Flog examines your code (flog -g app for a Rails app) and gives you a (somewhat arbitrary) numeric range measuring the relative complexity of your code. Basically, 0-10 is awesome (and practically unatainable), 11-20 is okay for real world methods, and it goes downhill from there. If you have 200+ complexity scores, refactor immediately! Flog is somewhat opinionated about what is good/bad or more/less complex, but generally does a good job in helping you avoid the “icebergs”.

Saikuro gives a more concrete result—the “score” is the number of branches through a method, including tertiary operators, foo if/unless bar, etc. This is a plus over Flog, and usually indicates about how many tests you should have (one per branch). The downside to Saikuro is that it does not pick up on dynamically defined methods, where Flog does.

Next, we walked through a refactoring example, where Jake showed using high Flog scores as a hit list of where to refactor next. He also mentioned that better readability trumps lower complexity scores, one thing to keep in mind—as being generated by automated tools, the scores should be taken as a guide, not a law. A good point was brought up during Q&A: there is currently no way to “flag” a high-scoring method as acceptable, so if you have a justifiably complex method that you choose to live with (e.g. for readability), then you’ll have to live with the flogging you will receive. I’m sure patches would be welcome if someone wanted to fix this!

On to code smell and Reek and Roodi, tools to identify smells (overly large methods, etc.) Reek tends to warn over smaller issues than Roodi, and can indicate false positives. Roodi generally tends to have fewer complaints—if it warns about something, it should probably be fixed!

Next up was Flay, which detects non-DRY-ness in code, anything from strict copy-n-paste to functionally identical blocks with different variable names to do..end blocks matching curly-brace blocks.

Also covered was a way to track source control churn. At this point, you’re probably thinking “How can I keep up with all this?” Luckily, there’s metric_fu, a way to wrap all this up into one package and get all this code analysis goodness in your project. Install the gem, then run rake metrics:all. For more info, installation instructions, etc., see: http://metric-fu.rubyforge.org/ Looking forward to adding this bag of tricks to our CI toolset.

Rails 3: Step off of the Golden Path – Matt Aimonetti

Matt started off with a history of programming languages and how Ruby came to be, including some of Matz’ core philosophies embodied in the language. Moving along to Rails, he talked about the growth of Rails and the desire for increased performance and options that led to the split between Rails and Merb. This led us in to the discussion of the current and future state of affairs for Rails 3.

Currently, as DHH mentioned in the opening keynote, there is no official release for Rails 3. However, much work has been done, and a direction / ideas are emerging that will be implemented once an official release is ready. These include:

  • improved performance
  • increased modularity
  • agnosticism
  • public api
  • mountable apps

Matt emphasized that there will be no drastic changes, and by default, rails app will generate a very similar application to what you would get today under 2.x. However, there will no longer be the idea of “the one true Rails way” of building an app—the framework will be less opinionated. However, you should go through a process of justification to see if you really need something different than the default stack.

Some of the options you will be able to choose from:

  • JavaScript frameworks, including jQuery, YUI, ExtJS, MooTools, Prototype, or the ability to write your own, and plug it in.
  • Different templating engines: HAML, ERb (this is already doable in Rails)
  • Different ORMs: ActiveRecord, DataMapper, SEQUEL, Hibernate, non-RDBMS stores like CouchDB, Tokyo Cabinet, etc.

At this point, Matt gave a demo of some of the nicer features in DataMapper, contrasted with ActiveRecord:

  • DataMapper re-uses existing Ruby object for both sides of a has_many / belongs_to relationship. In other words, if I load parent and child records from the database, and look at parent.object_id as compared to child.parent.object_id, under DataMapper, these will point to the same object automatically, while with ActiveRecord, these will be separate objects. (Note that inverse_of was recently checked into rails, which enables this in ActiveRecord as well )
  • DataMapper does automatic lazy loading as well as strategic eager loading, so in this scenario:
@parent = Parent.find(12345)
@parent.children.each do |child|
  puts child.name
end

ActiveRecord would need some hints (:include => :children) to be added to the original query to avoid the N+1 iteration problem, where DataMapper is clever enough to figure that out and generate 2 SQL queries for you automatically.

  • The ability to have multiple repositories (which looks like it means databases), and a copy method on models to clone data from one database to another—one use case would be an automatic archive or backup process that copies data generated within the last week to a backup database.
  • Query Path, allowing more flexibility in SQL condition generation (WHERE name LIKEfoo‘)
  • one potential gotcha that was mentioned: DataMapper does not support STI and Polymorphic associations as well as ActiveRecord does

Finally, he highlighted some options that would be available for even further customization, such as defining your own:

  • file structure
  • router DSL
  • request handling
    But he voiced the opinion that the vast majority of Rails apps will not need anything like this—make sure your need justifies coloring outside the lines.

All in all a good presentation, maybe a bit much focus on DataMapper specifically. However, I personally enjoyed the DataMapper bits, and might have to try it out on a project, if it’s a fit.

Art of the Ruby Proxy for Scale, Performance, and Monitoring – Ilya Grigorik

I skipped out on the afternoon sessions, so my next talk was Ilya’s—never disappointing. Ilya spoke about EM-Proxy, his event machine based proxy. He gave good example code of how EM proxy could be used to implement transparent and intercepting proxies.

It started with an itch at PostRank, Ilya’s blog aggregation solution. An effective staging environment should closely resemble the production environment—the problem was that their production environment spanned nearly 80 (virtual) servers on EC2’s cloud. Spinning up that many servers just as a staging environment was an expensive proposition. Also, simulating production traffic then becomes a challenge, as you end up trying to store production logs and “replay” them into the staging environment. The way that they chose to solve the problem was to separate a group of the servers into a staging app server pool, set up a proxy that would transparently (to the end user) intercept incoming requests, send them to both the production and staging pools simultaneously, then return only the production response to the user, using the staging response internally for benchmarking, testing output, etc. With this strategy, the more static parts of the system (web servers, load balancers, etc.) can be shared across environments, and the staging environment is testing the part that actually changes (the application servers).

The first example code was a transparent proxy that simply forwarded a request from one port to another. Ilya built on this to show how you could dynamically alter request/response data on the fly. Finally, he built up to the original scenario: duplexing a single request across two (or multiple) backend servers, but returning only a specific response. As he mentioned in the talk, one strategy for servicing a specific request as fast as possible might be to send the request to all machines in your pool, then respond with whichever request completed first.

These examples were centered around HTTP requests, but he went on to show some other examples of how this is not protocol-specific: you are just dealing with data over a socket connection, so as long as you understand the underlying protocol, EM-Proxy could be useful. His examples showed SMTP proxies for accepting/rejecting incoming mail by email address and implementing a spam filter by forwarding the incoming mail to Defensio before passing it along to your real SMTP server. The final example was pretty clever: an implementation of EM-Proxy to reduce the memory overhead of beanstalkd by selectively delaying queue inserts based on the scheduled execution time—basically buffering far future jobs into the database instead of immediately inserting into the work queue.

Slides are available here: http://bit.ly/ruby-proxy

Another packed day at RailsConf 09, one more to go!

RailsConf 2009 Day One

I’ve just finished my first day of RailsConf 09. Things have gotten started well—I’ve met several interesting people, found some interesting people to follow on twitter. I’ll try to post as much as I can in the way of reviews for those of you attending by blog.

Starting with the DHH Keynote. David’s talk was entitled “Rails 3 and the Real Secret to High Productivity”.

He started off by reviewing some of the so-called “mortal wounds” Rails has experienced:

  • It was deemed as not enterprise-ready, just a lot of hucksters trying to generate interest in Ruby to sell their own books, etc.
  • it encountered an “attack of the clones” in which several other languages/frameworks tried to duplicate (what they felt was) the one “key” feature that was the root cause of Rails’ explosion in success and popularity
  • some early adopters switched back (Derek Sivers, CD Baby)
  • some early adopters encountered serious problems (Twitter), which may or may not have been Rails-specific issues, but were perceived as such
  • some people complained about “feature bloat” (Array#fourteenth, etc.)

The takeaway from this review: There is no one best tool for every job—it’s okay if not everyone chooses Rails. Rails can’t be distilled down to a single feature, it must be viewed as a sum of its components. Emotions or reactions can be quite strong, but are usually not all that important/long-lasting.

Next, he outlined some central ideas of the Rails 3 philosophy:

  • Lock up all the unicorns: This is not a complete rewrite. Many people might look at Rails 3 and be disappointed, expecting more radical changes.
  • No “Sacred Cows” allowed: Everything is on the table for possible change in Rails 3. That being said, it’s important to note that any changes causing backward compatibility issues will need a very good justification to be accepted/implemented.
  • “Have it your way”: The default application generated under Rails 3 will rock, but you can ask for things to be done a different way and override the defaults. This seems like a natural extension of the templates now available in Rails 2.

He also covered some of the progress made to date in Rails 3. It sounds like things are not going as fast as was hoped, but there is code available now in GitHub—just no officially tagged release.

Plans for Rails 3 include:

  • a new router, with the ability to route requests by subdomains, user-agent strings, etc. also able to route to other rack machinery
  • XSS protection, in which any text output will be html-escaped by default, requiring you to explicitly request raw output if/when you need it.
  • JS will be unobtrusive and framework-agnostic. This is one of the more exciting plans for me. tags will be generated with the HTML 5 validating data-* attributes, which can be used as unobtrusive javascript hooks:

h3. HTML

  <a href="/person/1" data-remote="true" data-method="delete">Delete This Person</a>

h3. JavaScript

  $(document.body).observe("click", function(event) { 
    var element = event.findElement("a['data-remote']"); 
    if (element) { 
      var method = element.readAttribute("data-method") || "get"; 
      new Ajax.Request(element.readAttribute("href"), { method: method }); 
      event.stop(); 
    } 
  });

also, much like the database adapters in ActiveRecord, there might be a “driver” layer between the Rails javascript calls and the underlying javascript framework, enabling easy adoption of jQuery, MooTools, or any alternative framework.
* more agnosticism in ORM choice and generators
* more refactoring of various “crufty” areas of the framework, speed related improvements

Then, near the end, he got to the “secret to high productivity” part. In a nutshell—don’t view the requirements as carved in stone. If you are able to make (slight) changes to the requirements as originally communicated by the business, it may serve their purposes just as well as the original requirements but be drastically easier to implement. As cool as Rails is, no framework is going to match the performance boost this can bring.

Don’t Mock Yourself Out – David Chelimsky

I went to David Chelimsky’s talk on Mocks, but it was a bit too introductory-level. He basically explained good reasons to use mocks—from the title, I was expecting ways to determine if your mocks are becoming too heavy-handed, preventing you from actually testing any of your implementation!

Here are the highlights:

David Started with a review of the difference between mocks and stubs—basically, mock objects care about which methods are or aren’t called during the test, stubs are “dumb” containers to give canned responses to methods. See Martin Fowler’s explanation for more information.

Here are examples of when stubs might be useful:

  • isolation from non-determinism: when dealing with any randomly generated values, your tests face brittleness from non-determinism. Similar for timestamps, date ranges, etc.
  • isolation from external dependencies: Anything that connects to an external resource (e.g. network/db) can benefit from (smart) mocking. You should still test the “real” operation, but for adding permutations/edge case tests, mocks can speed up your test suite by multiple orders of magnitude.
  • polymorphic collaborators : (strategies / mixins/plugins) calling a method on the “main” object ends up delegating to a method somewhere else

Here are examples of when mocks might be useful:

  • testing side effects (e.g. something gets logged)
  • testing caching / memoization—ensure an expensive method gets called exactly once.
  • interface discovery : “outside-in” development : build a mock of something before it exists, use to discover the “ideal” interface, how you will be using this in the real world.

At this point, we started getting to Rails-specific content. David pointed out that if you are doing traditional Unit/Functional/Integration tests, it’s not DRY-some parts of your app (e.g. model validations) are being tested multiple times across your test suite. He recommends to split tests instead into business-facing / developer-facing specs, even using different frameworks where appropriate-for example, RSpec stories are more suited for business-facing specs, since they are close to the natural language of the domain experts. For lower-level, developer-facing tests (e.g. database-level), it probably doesn’t make sense to invest time writing stories—the developer is the domain expert.

A couple of helpful tips—if you care about which template gets rendered, but not necessarily the content (which could change!), you can stub the call to render, and set an expectation that it receives the correct partial name when called. When doing Rails functional testing, you need to set up valid/invalid data to make sure both sides of create/update actions are executed. Instead, you could stub the new/save/etc methods on the object to force the branch you want to test.

Upcoming “stubble” project helps with this.

UI Fundamentals For Programmers – Ryan Singer

Next, I saw Ryan Singer (37 Signals) talk about UI design concepts for developers. This was a high-level talk to help developers with the approach to take to UI/design, not a how-to, recipe style talk.

First of all, Ryan advocates taking a BDD, outside-in approach—including the UI. The UI is the closest thing to the user, so start there and work your way in, implementing backend functionality as you need.

Next, overcome your natural tendency (as a developer) to be terse—you should add ample text to guide your end user through the flow of your site/screens. Don’t rely on implicit meaning in forms/layouts as a guide, be explicit and explain what is expected of the user.

You should form a model (conceptual model, not ActiveRecord model) of the domain problem that makes sense to the customer and is implementable (feasable). Recommended reading: Domain-Driven Design by Eric Evans. (also Tufte for information display in general)

Designers tend to think in “Screens” as the atomic unit of work. This plays nicely into REST/Resource conventions, mapping e.g. displaying a particular object to the “show” action of your resource controller. For multiple related objects, break the screen into multiple areas that each represent a resource.

Be careful with contrast-make sure that you emphasize the most important elements on the page. Watch colors, sizes, etc. Don’t present a visual “buffet line”—aim more for a “gourmet meal”. Less is more.

Every action a user can take on your site has three distinct parts, a beginning, a middle, and an end. Think about where the user is when they need/want to take the action (where they are linked from). For the middle, collect the data for the action, then consider what the user will want/need to do next.

Some rules of thumb: helpers should not generate blocks of HTML (although generating an HTML element is probably ok—think image_tag); Organize your CSS/JS around REST conventions for your application; Treat ERb partials that generate HTML just as you would an image tag; Use helpers to reveal intention.

Smacking Git Around – Scott Chacon

Scott’s talk (aka the Git Firehose) was just as awesome (and fast-paced) as last year. Much too quick to take good notes, but I was able to capture a couple of useful tips & tricks:

First, on the concept of reachability: every commit that is an ancestor of your current commit. Git uses this in the dot-dot syntax for git-log and subtracts the first from the second. So git log origin/master..master gives you everything in your local master not already pushed to the remote repo. Or, after a fetch, reversing it (git log master..origin/master) gives you all commits you just pulled down that haven’t been merged into local master. Handy! A couple of alternate syntaxes: git log branch_a --not branch_b, or git log branch_a ^branch_b. This is also extensible to more than two branches: git log origin/master master ^experiment would give you all commits on origin/master and/or master that are not in experiment.

Second, he illustrated a common problem with diffing between branches in Git. If you imagine a file existing in two branches, in which lines have been added on both branches, and you ask git for a diff, most often you want to know what has been added to the other branch that you don’t have. By default, though, the question git is answering is “how do I make this file look like the other file?”, which includes deleting the lines you have added on the current branch—probably not what you want. In this instance, you can use the triple-dot syntax: git diff master...topic (asking git “what would happen if I merged the two branches?”)

He talked about subtree merging as an alternative to submodules. Basically, the process is:

# add a remote to the project you want to add as a subtree and fetch:

git remote add rails_remote git@github.com:myfork/rails
git fetch rails_remote

# check out a local branch of the other project:

git checkout -b rails_branch rails_remote/master

# return to your project and use git-read-tree to add the subtree, then add and commit:

git checkout master
git read-tree --prefix=vendor/rails -u rails_branch
git add vendor/rails
git commit

This keeps your ability to make changes to the remote project and submit them back upstream:

# make change(s) to in remote project and commit
# go to remote project’s local branch

git checkout rails_branch

# merge your changes in:

git merge -s subtree --no-commit --squash master

This obviates the need for braid/piston! See http://tinyurl.com/braidgit for more info.

Also mentioned was patch staging—simple, but incredibly useful! Say you’ve made multiple changes to the same file, and you want to commit some, but not all. With patch staging (git add -p), you can select the hunks you want to stage for commit — awesome!

More useful info, including how to give Git the ability to diff binary files (kinda, but it can work). Slides are posted at http://tinyurl.com/gitrailsconf09

Quality Code with Cucumber – Alsak Helles√ły

For the last session, I saw Alsak Helles√ły’s most excellent talk on Cucumber. I was totally drained at this point, and took no notes. He gave an example of how to get started with Cucumber, working through a sample scenario, doing outside-in development. So far, I haven’t used RSpec / BDD in anything serious, but I’m looking forward to trying this out—looks pretty interesting.

Ruby Heros / Evening Keynote

The day ended with the Ruby Hero awards, and then the DHH / Tim Ferriss (4 hour workweek) interview/keynote. I think there was probably good content to be had in the keynote about refactoring your lifestyle. Unfortunately, the sound was bad—very muffled/echoy from where I was sitting. Also, the pace was so much different from the firehose of information presented by the technical talks that it was like a panic stop in a formula one racer.

Overall, a good but exhausting day here in Vegas. If you’re here, look me up and say hi, or give me a follow (rubysolo on Twitter).