Analyzing the Subversion logs from the Rails project with mx:OLAPCube

I started playing with Anthony Eden’s ActiveWarehouse and followed his excellent tutorial on how to analyze the Ruby On Rails Svn Commit Log with the ActiveWarehouse framework. Of course this made me want to try to do the same with the new mx:OLAPCube and mx:OLAPDataGrid provided by Flex 3 as part of the DataVisualization components. Let me just say this…I am not done playing with either the Flex OLAPCube nor the Rails ActiveWarehouse framework as these are pretty complex beasts. Both of these frameworks are overlapping and complementary. There are overlapping as both can digest raw data and perform aggregation of that data. They are complementary in the sense that a server side warehouse needs a good visualization front-end. Maybe the OLAPCube and OLAPDataGrid can be this front-end. In my initial trials I haven’t come up with a compelling way to integrate both, but by using some simple SQL I could extract the data from the ActiveWarehouse and pass it to the OLAPCube.

Before going on you may want to read Anthony’s blog and check his presentation on Data Warehouses with ActiveWarehouse. I didn’t find much information on the Flex OLAPCube besides these: Feature_Introductions:OLAPDataGrid on Adobe’s labs, Flex 3: Feature Introduction Video for OLAP Support, and these">Flex examples.

So I create the following sample application. You can try it out here. Note it’s pretty slow, it takes up to a minute to aggregate 10000 values. The Flex team mentioned they didn’t optimize this component yet. I can confirm this. But I may also have messed something up as these are only my initial steps with that component. The application displays the Author dimension with the Author Name as rows and the Time dimension with the Year and Quarter as columns. The facts is the File Change count during that period. Flex calls the “facts” a measure.


Run the applicaiton

To extract the data from the ActiveWarehouse I created this SQL to join the facts table with all the dimensions table. I need to find out if the ActiveWarehouse doesn’t just return this data in xml format by using it’s build-in classes.

def report_as_xml
sql = <<-EOSQL
file_revision_facts.file_changed AS file_changed
JOIN date_dimension as date
ON file_revision_facts.date_id =
JOIN author_dimension as author
ON file_revision_facts.author_id =
date.calendar_year > ‘2005’

@@xml ||= ActiveRecord::Base.connection.select_all(sql).to_xml(:dasherize => false)
render :text => @@xml

In Flex the OLAPCube can be loaded with the XML

var data:ICollectionView = new ArrayCollection(result.records.record); // is Array
cube.dataProvider = data;
cube.addEventListener(CubeEvent.CUBE_COMPLETE, creationCompleteHandler);

Once the cube is loaded you can slice and dice it in many ways by using an OLAPQuery. I still need to figure out all the possibilities which are offered.

private var cubeResult:IOLAPResult;

private function creationCompleteHandler(event:CubeEvent):void
//Cube was created, let’s query it
var query:OLAPQuery = new OLAPQuery;

var yearSet:IOLAPSet = new OLAPSet;

var quarterSet:IOLAPSet = new OLAPSet;

var newTimeSet:IOLAPSet = yearSet.crossJoin(quarterSet);

var authorSet:IOLAPSet = new OLAPSet;

var rowAxis:IOLAPQueryAxis = query.getAxis(OLAPQuery.ROW_AXIS);
var colAxis:IOLAPQueryAxis = query.getAxis(OLAPQuery.COLUMN_AXIS);

var token:AsyncToken = cube.execute(query);
token.addResponder(new AsyncResponder(displayResult, olapFaultHandler));
private function displayResult(result:Object, token:Object=null):void
cubeResult = result as IOLAPResult;

The cube result is the dataProvider of the Cube which in it’s simplests form can be defined as follows:

I’ve then added a change listener for the grid to create the dataProvider for the ColumnChart.

private var chartData:Array;

private function gridSelectionChanged():void {
if (!(olapGrid.selectedItem is OLAPAxisPosition)) return;
var rowIndex:Number = olapGrid.selectedIndex;
var axis:IOLAPQueryAxis = cubeResult.query.getAxis(OLAPQuery.COLUMN_AXIS);
var columnLength:Number = cubeResult.getAxis(OLAPQuery.COLUMN_AXIS).positions.length;
var newChartData:Array = [];
for (var i:int=0;i<columnLength;i++) {
var tuple:OLAPTuple = axis.tuples[i];
var key:String = tuple.explicitMembers.toArray().join(“,”);
if (key.indexOf(“(All)”) > -1) continue; newChartData.push({key:key, value:cubeResult.getCell(rowIndex, i).value});
chartData = newChartData;

This code to extract a time serie for the chart is a little “hairy”. I hope the Flex team has some OLAPCharts on their todo list 😉

This are my first tribulations with both frameworks. Over the next few month I will have to dive more deeply into the possibilities which are offered. Thanks to both teams as this is pretty cool.


Scripting the Leopard Terminal

Hypothetical situation: you’re sitting down with your favorite tasty beverage close at hand for some Rails hacking, and what commands do you run every single time? It’s probably something like this:

cd Projects/KillerApp
mate .
rake log:clear
tail -f log/development.log

[Command – T for new tab]
cd Projects/KillerApp
mongrel_rails start

[Command – T for new tab]
cd Projects/KillerApp
ruby script/console

[Command – T for new tab]
cd Projects/KillerApp


Hmmm. We’re coding DRY, but this bootstrap process doesn’t seem very DRY. This had been bugging me, so I set out on a Google quest to learn to script in Leopard so that I could do something about it. I first looked at AppleScript, since that’s the de facto scripting language of all things Apple. Given a specimen, I could decipher what it was trying to do reasonably well, but going the other way was difficult — starting with a goal, I was generally unsuccessful in trying to express it in AppleScript. This was the best I could come up with:

tell executive SteveJobs
tell developers at Apple
set theScriptingLanguage of OSX to Ruby
end tell
end tell

Which was a good thought, but didn’t actually do much. At RubyConf last year, Laurent Sansonetti talked about RubyOSA, a scripting bridge between Ruby and the Apple Event Manager, which sounds great, because you can code in Ruby and control AppleScript-able applications, like iTunes. So I ran this:

require ‘rbosa’
terminal =‘Terminal’)

Which resulted in this rather discouraging output:

RuntimeError: Can’t get the target bundle signature
from /Library/Ruby/Gems/1.8/gems/rubyosa-0.4.0/lib/rbosa.rb:329:in __scripting_info__&#8217;<br />
from /Library/Ruby/Gems/1.8/gems/rubyosa-0.4.0/lib/rbosa.rb:329:in
from (irb):3

Hmmm. Not exactly useful. Maybe if I did my coding in iTunes, I could use RubyOSA to build some useful automation. Hopefully someone at Apple can fix Terminal — Hint, Hint. Back to Google…

I eventually ran across Matt Mower’s scripting a better ‘cd’ and then some article. Beautiful. Exactly what I wanted. Except I didn’t want to have to switch to iTerm just to use it. I’m fairly happy with the native in Leopard — the tabs are nice. My biggest complaint is that you cannot name individual tabs, which seems like a glaring omission. Hopefully that will be updated soon.

So Matt’s gp command uses another Ruby / Apple event bridge, called Appscript, which “allows you to control scriptable Mac OS X applications using ordinary Ruby scripts”. Sounds cool, and most of the examples feature TextEdit, which isn’t Terminal, but at least is more of a developer application than iTunes, so we seem to be on the right track.

gem install rb-appscript, and let’s play. In IRB:

> require ‘appscript’
> include Appscript
> term = app(‘Terminal’)
=> app(“/Applications/Utilities/”)
=> app(“/Applications/Utilities/”).windows
=> app(“/Applications/Utilities/”).windows.first

Okay, this is a bit strange — it seems to just repeat what you say back to you. After a bit of playing, I understand what’s going on. Some of the methods aren’t executed until you explicitly tell them to execute. For instance:

=> [app(“/Applications/Utilities/”).windows.ID, app(“/Applications/Utilities/”).windows.ID, app(“/Applications/Utilities/”).windows.ID]
=> app(“/Applications/Utilities/”).windows.ID

Why? Most likely, this is for efficiency reasons. With many of these “Event Bridge” solutions, there is a significant “toll” that must be paid to cross the bridge. If we can batch together a string of method invocations and send them across the bridge for a single round trip, it performs much better than multiple round trips would.

At any rate, we can now get a handle on the Terminal application and its already-open windows. The next step is to be able to open new windows and tabs. There is a somewhat useful tool on the rb-appscript download page called ASDictionary, which can examine an application and dump out the objects and methods that it exposes to the bridge. Kinda like rdoc for AppleScript. Running the dictionary against, I found the do_script(command) method, which, when called on the terminal application object, launches a new terminal window and runs the specified UNIX command in it.

> term.do_script(“ls”)
=> app(“/Applications/Utilities/”).windows.ID.tabs1

If you’re following along, you should have a new Terminal window containing a listing of the files and folders in your home directory. Also, notice that the command returns a reference to the first (and only) tab in the new window — we’ll come back to that later. So, new windows, check; now for new tabs.

The Appscript examples show creating new TextEdit documents by executing:

app(‘TextEdit’).documents.end.make(:new => :document)

And the Appscript dictionary of Terminal showed the make method, and window and tab classes, so I figured something like this might work:

> window =
=> app(“/Applications/Utilities/”).windows.ID
> window.make(:new => :tab)
Appscript::CommandError: CommandError
OSERROR: -10000
MESSAGE: Apple event handler failed.
COMMAND: app(“/Applications/Utilities/”).windows.ID.make({:new=>:tab})

from /Library/Ruby/Gems/1.8/gems/rb-appscript-0.4.0/lib/appscript.rb:505:in <em>send_command&#8217;<br />
from /Library/Ruby/Gems/1.8/gems/rb-appscript-0.4.0/lib/appscript.rb:585:in
from (irb):11

Appscript is having none of that. After many frustrating, fruitless attempts to create a new tab, I found a workaround in native AppleScript here. Here’s the Appscript translation:

app(“System Events”).application_processes[
].keystroke(“t”, :using => :command_down)

Well, okay, sending a Command-T keystroke works, but it’s a little disappointing. Anyone who knows how to programatically create a new tab, feel free to chime in on the comments, and I’ll update the script.

do_script(command) also takes an optional parameter specifying options. One of the available options is :in, which tells terminal in which window and tab to run the command. Putting this together, we can run a command in the new tab we just created:

app(‘Terminal’).do_script(“ls”, :in => window.tabs.last.get)

We don’t have a handle to the new tab, because it was created via hackery, so we need a handle to its parent window object so we can get at its tabs. Well, our first command we ran with do_script returned us a handle to the first tab of the window, surely we can get the window from that. Right? Anyone?

> tab = term.do_script(“ls”)
=> app(“/Applications/Utilities/”).windows.ID.tabs1
> tab.window
RuntimeError: Unknown property, element or command: ‘window’
from /Library/Ruby/Gems/1.8/gems/rb-appscript-0.4.0/lib/appscript.rb:591:in `method_missing’
from (irb):11
from :0

(Sigh) Not so much. Time for a hack on a hack, and this one is just embarrassing. Observant readers have probably noticed that the tab referenced shows its parent window id in the string of object references. What if we use that to get a handle to the parent window? Something like this:

> window = eval(“app(”/Applications/Utilities/“).windows.ID”)
=> app(“/Applications/Utilities/”).windows.ID

I’m not proud of it, but it works. Again, if someone knows the “right” way to do this, please let me know—although programatically creating a tab should obviate the need for this hack, too.

Another thing I wanted to change from the original script was that with Matt’s solution, you need to specify the type of project you are opening, either as a command-line argument, or by symlinking the script to a new name with the type embedded. I want the computer to figure all that out for me. Computers are real smart. So my script includes some configurable heuristics for determining project type based on the folder contents. You should be able to specify set of files and folders that defines a project type. For Rails, I used the “skeleton” folders that every Rails app starts with. I haven’t been programming in Erlang very long, so honestly I just guessed at the folders based on some projects I have seen. If you’re a more experienced Erlang programmer, and feel that the project detection or task list should be changed, please let me know.

Matt later enhanced his script to label the iTerm tabs so that you can easily find the tab you need, so naturally I stole incorporated this idea too. It sorta works in Leopard’s Terminal—it does change the name of the Terminal window, but the tab name is unaffected, so you still have to flip through the tabs and watch the window name. This is my biggest request for a feature enhancement for the Terminal, Hint, Hint again, Apple.

So enjoy. You can use the script anywhere — it will look first in your current directory for a matching folder (you only have to specify a non-ambiguous substring of the project folder name), then falls back to the “project root” folder you specify (currently ~/development, ‘cause that’s what I use.). You can configure how deep it should recurse your projects so that the disk seeking time doesn’t eat up all the time you’re saving from not typing the same “cd” commands over and over.

You can download the finished script here. I call it hack, because hack KillerApp flows so nicely as a command. Also, that’s the way I roll. Feel free to send in money, or flattery, or hate mail, I suppose. Thanks to Matt Mower for the inspiration and also to all the other references I’ve linked!

flash.utils.ByteArray compressing 4.1MB to 20K

I am currently preparing a demo using an mx:OLAPCube and OLAPDataGrid which analyze the Rails svn commit log. however I don’t want to deploy a specific server side application as the Cube can load data from XML. So I have an report.xml that is 4.1MB. I created the following AIR application (ZlibCompressor.mxml) that use the standard compression provided by the ByteArray class to compress this file down to 20Kb. The application that consumes this file (UnzipTest.mxml) uses the URLLoader to read this file straight into a ByteArray and uncompress the data. It’s fast!

The key code for compression is the ‘compress’ and ‘uncompress’ method provided by the ByteArray. Note the URLLoader dataFormat is set to “binary”.


<?xml version=“1.0” encoding=“utf-8”?>

import flash.desktop.ClipboardFormats;
import flash.utils.CompressionAlgorithm;
public function onDragIn(event:NativeDragEvent):void {
var transferable:Clipboard = event.clipboard;
if (transferable.hasFormat(ClipboardFormats.FILE_LIST_FORMAT)) {
public function onDrop(event:NativeDragEvent):void {
var fileList:Array = event.clipboard.dataForFormat(ClipboardFormats.FILE_LIST_FORMAT) as Array;
if (fileList.length==0) return;

var inFile:File = fileList0;
var fileStream:FileStream = new FileStream();, FileMode.READ);
var ba:ByteArray = new ByteArray();
fileStream.readBytes(ba, 0, fileStream.bytesAvailable);

var newFileName:String = inFile.nativePath+".zlib";

var outFile:File = new File(newFileName);
fileStream = new FileStream();, FileMode.WRITE);
fileStream.writeBytes(ba, 0, ba.length);


<?xml version=“1.0” encoding=“utf-8”?>

import flash.utils.ByteArray;
private function loadData():void {
var loader:URLLoader = new URLLoader();
loader.dataFormat = “binary”;
loader.addEventListener(Event.COMPLETE, completeHandler);
var request:URLRequest = new URLRequest(“../data/report.xml.zlib”);
private function completeHandler(event:Event):void {
var loader:URLLoader = URLLoader(;
var ba:ByteArray =;
var s:String = ba.toString();
var xml:XML = new XML;

#81 Fixtures in Rails 2.0

In Rails 2.0, fixtures see many improvements which makes them much easier to manage. Watch this episode for details.

#81 Fixtures in Rails 2.0

In Rails 2.0, fixtures see many improvements which makes them much easier to manage. Watch this episode for details.

Acts_as_nested_set ActiveRecord rendered with mx:Tree in Flex.

ActiveRecord: app/models/category.rb

class Category < ActiveRecord::Base

Controller: app/controllers/categories_controller.rb

class CategoriesController < ApplicationController
def index

Flex Application: ActsAsNestedSet.mxml

<?xml version=“1.0” encoding=“utf-8”?>


XML generated by Category.result_to_attributes_xml(Category.root.full_set):

I used the plugin.

Too cool!

UPDATE: The BetterNestedSet plugin doesn’t work out of the box with Rails 2.0 RC1. Thanks Joel for that info. Read more in the comment of this blog entry.

UPDATE2: Thanks Fabien, BetterNestedSet now works with Rails 2.0!

Shure SE210 Earphones (upgrade from E2c)

When I bought my Shure E2c earphones two years ago, I was blown away by the sound quality. I never spend more than $50 for earphones before and after buying the E2c model I would never consider downgrading to anything less. Shure really knows how to create addictive high quality products. The quality of the sound produced by my old E2c earphones was so good that I had to get rid of some albums that I ripped into mp3s long time ago when hard drive space was measured in megabytes rather than in gigabytes.

After two years of slave-like everyday service my Shure E2c earphones started to show signs of wear and it became apparent that they were going to retire soon.

Afraid of being without a good pair of earphones during my upcoming transatlantic flight, I got a new pair of Shure SE210. The design has change slightly compared to the older model, but it still preserves the main features that I loved so much.

I’ve been listening to some music via these earphones for only an hour or so, but I can already tell that SE210 brings “great sound” to a brand new level. I don’t consider myself to be an audiophile, but even my untrained ear can hear the big difference between the two. The new pair fits better as well as feels more comfortable in my ears.

J, lost his mind a couple of weeks ago and bought a pair of the high-end SE530. I don’t even want to know what those sound like. I know that if I tried them, I would be tempted to pay an insane amount of money for them, so I better say away 🙂

Ryan Daigle – Ruby on Rails Podcast

Ryan Daigle talks about his blog, the upcoming Rails 2.0, and freelancing with style.

Ryan Daigle – Ruby on Rails Podcast

Ryan Daigle talks about his blog, the upcoming Rails 2.0, and freelancing with style.

Evil Twin Plugin

Seriously, I think I have something against Rails’ lib directory. We jumped from keeping gems in lib to vendor/gems back in March. Then we jumped from keeping generic Rake tasks in lib/tasks to Sake. Now we’re gonna jump again.

Hacking Plugins

It’s really not that big of a deal, and pretty common—you want to change the behavior of some existing plugin. Maybe you Piston it and commit your changes. Sure. But maybe you just want to leave the original code alone.

A classic approach has been to stick some kind of hack in the lib directory. Issues abound, for sure. First: the load order. Who gets loaded first? Who reloads and who doesn’t? Second: location. You’ve got one bit of code messing with another bit of code in a totally separate location. Third: testing. Are you testing it? Maybe.

None of these things are deal breakers, but we can certainly address them. And we will.

The Evil Twin Plugin

Evil Twin

Here’s the simple solution: create a plugin called whatever_hacks, where whatever is the name of the plugin you’re hacking. That’s it. An evil twin, if you will.

Adding the _hacks suffix ensures it will always be loaded after the target plugin (assuming you haven’t messed with the default plugin load order—alphabetical). Keeping it right next to the target plugin also ensures anyone who peers into vendor/plugins will instantly know tomfoolery is afoot.

You can now build out a tested, hack happy plugin. Or, y’know, just stick it all in init.rb. With caution.

Caution: init.rb

Caution: init.rb does not always do what you expect it to do. It’s loaded in the context of Rails::Plugin in 2.0 and Rails::Initializer in 1.2.5, not Object. Come again? Like this: re-opening existing classes isn’t as straightforward as elsewhere.

=> init.rb

class Hash
puts Hash.inspect

Guess what that prints. Ready?

$ ./script/runner 

That’s right—we didn’t re-open Hash, we created a new Rails::Plugin::Hash class. Any methods we add in there won’t be added to Hash proper.

If we want to grab a real class and stuff some methods in it, we need to use class_eval or module_eval:

=> init.rb

Hash.class_eval do
  def duck_punched?


As expected:

$ ./script/runner 

Doing it this way (class_eval) forces a constant lookup, making Ruby happily run up the chain and find the class or module in question.


Okay, time for a real example. I wanted to change attachment_fu’s ImageScienceProcessor to crop thumbnails before resizing them. As this is a hack I use on all my apps, I also want to keep it out of my models. Hence, attachment_fu_hacks.

=> vendor/plugins/attachment_fu_hacks/init.rb

klass = Technoweenie::AttachmentFu::Processors::ImageScienceProcessor
klass.module_eval do
  # Hacked to use image_science's #cropped_thumbnail method
  def resize_image(img, size)
    # create a dummy temp file to write to
    filename.sub! /gif$/, 'png'
    self.temp_path = write_to_temp_file(filename)
    grab_dimensions = lambda do |img|
      self.width  = img.width  if respond_to?(:width)
      self.height = img.height if respond_to?(:height) temp_path
      callback_with_args :after_resize, img

    size = size.first if size.is_a?(Array) && size.length == 1
    if size.is_a?(Fixnum) || 
        (size.is_a?(Array) && size.first.is_a?(Fixnum))
      if size.is_a?(Fixnum)
        img.cropped_thumbnail(size, &grab_dimensions)
        img.cropped_thumbnail(size.first, &grab_dimensions)
      new_size = [img.width, img.height].dim size.to_s

      img.cropped_thumbnail(new_size.first, &grab_dimensions)

Works like a charm.

When heavysixer wanted to hack acts_as_taggable, he took the same approach: Feel free to follow suit.

#80 Simplify Views with Rails 2.0

Rails 2.0 is almost here. One of the many features in this release is the smarter helper methods which can make your views much cleaner. Learn all about it in this Episode.

#80 Simplify Views with Rails 2.0

Rails 2.0 is almost here. One of the many features in this release is the smarter helper methods which can make your views much cleaner. Learn all about it in this Episode.

Hal Fulton – Ruby on Rails Podcast

Hal Fulton, author of The Ruby Way.
From RubyConf in Charlotte, North Carolina.

Hal Fulton – Ruby on Rails Podcast

Hal Fulton, author of The Ruby Way.
From RubyConf in Charlotte, North Carolina.

#79 Generate Named Routes

This more advanced episode will show you how to dynamically generate named routes by adding a method to the map object.

#79 Generate Named Routes

This more advanced episode will show you how to dynamically generate named routes by adding a method to the map object.

Stuart Halloway – Ruby on Rails Podcast

Robert Stevenson interviews Stuart Halloway at eRubyCon in Ohio.

Stuart Halloway – Ruby on Rails Podcast

Robert Stevenson interviews Stuart Halloway at eRubyCon in Ohio.

RailsLogVisualizer0.7 for AIR beta 2.

I recompiled the RailsLogVisualizer for AIR beta. I added drag&drop of log files to bypass an AIR bug on Leopard. File.browseForOpen doesn’t trigger the Event.SELECT when the file is selected. I haven’t yet tried this version of the application on older versions of OSX or on Windows. Let me know how it works. Also the feedback when loading large log files could be improved, as the application seems to freeze once the progress bar is complete. Just be a little patient as the AVM is working hard for you to compute all these number.

<h3>Install RailsLogVisualizer0.7.air</h3>
<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"
     id="RailsLogVisualizer" width="215" height="138"
  <param name="movie" value="" />
  <param name="FlashVars" value="appUrl=" />
  <embed src="" quality="high" bgcolor="#ffffff"
             width="215" height="138" name="RailsLogVisualizer" align="middle"
             pluginspage=""> </embed>

<h3>Install Manually</h3>
1) Instal Adobe AIR beta 2. (See release notes if previous version was installed)<br/>
<a href="">Download AIR for OSX</a> <a href="">Download AIR for Windows</a><br/>
Learn more on <a href="">AIR</a> <br/>
2) Download and install <a href=""></a> 

For the log file is currently 98Mb and is loaded and process in less than a minute. Here are the loading details:

Loaded 98571986bytes in 28093 milliseconds.
Parsing file. Please Wait this may take some time....
Parsing. Split 1639453entries in 1447 milliseconds.
found:220767 in 1925 milliseconds.
Aggregating data.
    aggregated:220767 in 13426 milliseconds.
    aggregated String :4440464(bytes) in 2790 milliseconds.

Then you can navigation through time and see how many request where processed and drill down in specific action and specific methods. For example, here we can quickly see that for October 99 people signed up, 869 did login, 22 forgot their password.



Enhanced Migrations v1.2.0

The original release of this highly useful plugin marked a turning point in collaborative Rails development by freeing the developer to commit their database migration without fear of having it ignored because of a higher placed migration number. This latest release includes some minor bug fixes plus a useful method for stepping through migrations one at a time without the need for copying and pasting long version numbers.

  1. Fixed bug where an empty migrations_info table would create a non-parseable schema.rb.
  2. Made plugin database independent.
  3. Added capability to step through migrations using VERSION=[previous, next, first and last].
  4. dump_schema_information now returns all migrations, not just latest (credit to François Beausolei).
  5. Added tests which use SQLite and a task (enhanced_migrations:clean_sqlite_db) to help with testing.

As an example of item number three, consider the following situation. Using the enhanced migrations plugin, you’ve just created a migration that adds a new table to

Continue reading “Enhanced Migrations v1.2.0”