Music and programming: interviews with Chad Fowler and Dave Thomas

I’ll be speaking at RailsConf 2009 this year on music and software development (Five musical patterns for programmers). The basic premise is that software development and music actually have quite a bit in common. This may be surprising to some people, who see programming as a cold, rational left-brain sort of thing, like science. But we programmers know that this is not really the case at all.

So as a prelude to my talk, I decided to interview two programmer-musicians on the subject: Chad Fowler and Dave Thomas. Both compose and perform music, and both are noted programmers. Here is the interview.

Rail Spikes: Tell us a little about your background with both programming and music.

Chad Fowler: I started my professional life as a saxophonist in Memphis. I played the Beale street clubs and all the typical Memphis professional musician stuff. Among others, I played for a while with Ann Peebles and her husband Don Bryant with the rhythm section from all the old Hi Records recordings. I did mostly R&B and jazz professionally but I was probably most well known in the Memphis community for making “strange” music. Before playing music professionally, I played guitar in punk bands in high school. I was a fan of punk, heavy metal, hip hop, pop, (new) classical and pretty much everything else. As I immersed myself in the world of jazz, it became quickly clear that the jazz community doesn’t like punk and other less “serious” types of music and has an almost religious negative reaction to jazz musicians who do.

It was almost as if any deviation from the “normal” world of jazz made you a traitor. So I did the natural thing: started a group called The Jazz Traitors, which played music that 1) we loved and 2) offended the jazz community (not necessarily in that order).

I was also very interested in composing “classical” music. I studied with a composer named Kamran Ince, who is still my favorite such composer.

As for programming, I’ve been interested in programming since I was a young child using my commodore 64. I wasn’t really that good at it as a kid but I played around a lot. I didn’t get serious until I picked up programming again as a hobby while I was a professional musician. After a late night gig at a bar, it was relaxing to go home and unwind to some C programming tutorials. I didn’t have a need to program, nor did I have a project in mind (except that I have always loved video games and wanted to learn how they worked). But I got so into it, that I ended up getting a job in computer support because a friend filled out an application for me.

Being the gamer I am, as soon as I started in computer support, I naturally wanted to “level up”. That meant becoming a network administrator. Then a system administrator. Then a programmer, then a designer, then an architect, then a CTO, etc. Now here I am. It’s been fun.

Dave Thomas: There was always a lot of music in our house. My father liked to play the piano and the organ (I learned to solder as he built a Heathkit organ from a kit in the late 60s). My mother liked Broadway musicals. So we’d often experience alternating hours of Chopin and South Pacific. My brother was also musical. I wasn’t particularly, but I enjoyed noodling on the piano, and spent hours just playing with chords and progressions.

I’ve been programming since I was 15 or so.

Rail Spikes: Some developers – yourself included – have suggested a similarity between programming and music composition or performance. How exactly are music and programming similar?

Dave Thomas: I’m not sure, but I think it might be something to do with the discovery of patterns. Both music and code consist of nested sets of variations and repetitions. There’s a rythm to executing code, in the same way there’s a rythm to music. It is never exact, but it’s there. After a while, I found I could imagine the rythm and structure of my programs as they run, in the same way you can pick apart the structure of a piece of music as you listen to it. And, jsut as with music, it takes experience to be able to feel the deeper structures and notice the more extreme variations. But being able to spot them in programs makes coding simpler and more interesting. The basic coding structures—loops, method calls, and so on—provide the framework for composing in the same way that staff and bar lines do for music. Algorithms are like the progressions, and data becomes the notes. And in the same way that good music takes all these things and then surprises you, good code does the same thing. It isn’t mechanical and repetitive: instead it uses the constraints to build something bigger and more interesting.

Chad Fowler: It’s hard for me to put my finger on. There’s something similar in the way I think when I do each.

I think it all boils down to language, though. In all of these cases (including learning actual language), you take a bunch of tokens (notes, sounds, grunts, functions, classes) and combine them into a grammar which you use to express ideas. The way you do that is totally up to you as long as the intended ideas are communicated. With computer programs, they have to do what they’re meant to do. With music, they express or evoke emotions, paint pictures, cause anxiety or whatever.

Some computer programs evoke emotions and cause anxiety as well.

Rail Spikes: Is Ruby development more like improvised jazz or composed classical music?

Chad Fowler: I think it’s both. And I don’t think Ruby is any different in this than other languages. Much of the discussion about the relationship between programming and music focuses on the more obvious idea of programming as composition. It makes sense, since programmers tend to sit and type their ideas into an editor and then eventually execute it. The programs can be checked, tested, refactored, etc. before the actual performance. This is how classical composition works as well.

But the less obvious angle is that in many situations, programming is like performance. In fact, even in music, improvisation is really just real time composition. You don’t get a chance to refactor because your “code” is executed as you write it.

I’ve had this same feeling while debugging production problems, hacking new features on a tight deadline, or sometimes during the initial creation of an application. The same synapses are firing as when I was trying to play Cherokee at 200 beats per minute. Mistakes can’t be erased, so they have to be nuanced into (worst case) insignificant events or (best case) important drivers behind the work.

From a purely development-oriented perspective, TDD is more like improvisation than composition. I think that’s what I like about it. It’s motivating and creative in an exciting, time-sensitive way. You take small steps and see where they lead you. Sure, you can always revert your changes if you paint yourself into a corner but part of the fun and challenge is to not paint yourself into a corner.

One thing jazz musicians like to say is that every wrong note is just a half step away from a right note. TDD is like that. You might take a slightly wrong turn. It’s fun to see if you can course-correct without starting over.

Rail Spikes: Do developers need to be musically inclined? Does it help?

Chad Fowler: Obviously not. Some of the best programmers I know are not musicians. I can’t tell if it helps, but I would guess that developers who are also musicians are different than developers who aren’t. I don’t think that’s because being a musician changes people, though. I think it’s because the people who are both are the kind of people who need to do both.

This usually means they’re “right brain” people. This leads to a way of thinking that changes how they approach programming problems.

I think learning music (or another right brain discipline) is a good way to exercise your mind. So I wouldn’t be surprised if leaning music helps people exercise their thought processes in ways that will benefit their work as programmers (or authors, or lawyers, or doctors or whatever).

I also think, though, that if we were all musicians at heart, we wouldn’t get much done. I rely heavily on my less artsy colleagues to ground me and be sometimes more pragmatic than I am. So I don’t think we all need to be a “right brain” programmer. It would be disasterous if we were.

Dave Thomas: Do they need to be? No. But many of the good ones I know are. I’d guess that density of musicians in software development is many times the population norm. But that means you could also ask the question “Do musicians have to know software development?”

I think the more interesting question is to ask “how can people best express what they enjoy doing?” because both music and software development are outlets for this.

Rail Spikes: What sort of music do you listen to? Any recommendations for Ruby developers looking to expand their musical horizons?

Chad Fowler: As I mentioned earlier, I like all kinds of music (with a few exceptions). Lately I’ve been listening to a lot of instrumental hip hop, such as DJ Qbert and Mixmaster Mike. I’ve also been getting into a genre of electronic music called “electro”, which sounds like the bleeps and bloops that are the soundtrack of my dreams (if a computer is going to generate music I always like it to sound like a computer generated it).

As for recommendations, here are a few ideas for things that most developers probably haven’t listened to:

  • Kamran Ince – He was my composition teacher and, I think, an accessible introduction to the world of “new music”, which is what we call new composed “classical” music. The term “classical” is a widely spread misnomer. It actually refers to music written in the late 18th and early 19th centuries, but most people use it to mean high brow music written for instruments like violins. So whatever you call it, Kamran Ince writes some beautiful instances of it. Specifically check out his chamber music, such as Domes and Arches.
  • Charlie Wood – I have had the pleasure of playing with Charlie on a few occasions. He is a R&B singer/organist/composer from Memphis and writes some of the most intelligent songs you’ll hear. My favorite album of his is “Who I Am”.
  • John Zorn – Zorn has been around for a long time and is a leader in the world of Avant Garde music. He’s also one of the most amazing saxophonists ever. If you’re new to this kind of thing, his Masada quartet (“radical Jewish music”) produces some great stuff that’s accessible to first time listeners. If you’re looking for something to shock your aural taste buds, try Painkiller (metal-tinged noise) or Naked City.

Dave Thomas: I listen to just about anything that’s interesting. My playlist here is very varied, and I try to add new stuff to it farily regularly. I know people who are trained as musicians, and I tend to ask them what they’re listening to. Sometimes that leads to challenges: my ear isn’t as developed as their ears. But often it leads to whole new areas of cool stuff. So I’d recommend everyone should find a friend who knows more than you do about music and ask them to surprise and challenge you. (That advice probably applies to just about everything, thinking about it.) It’s easy to find music that stimulates your lizard brain. Get into the habit of looking for the stuff that engages at a higher level too. And, like everything, have fun with it.

Programming Ruby 1.9, now in print and shipping

Programming Ruby 1.9 is now in stock.

New Heroku Screencast by Remi

It’s been a bit of a blur here at Heroku HQ in the past couple of weeks. However, amidst all the launch activity we did notice a screencast so sweet we thought we’d share it with you. It really covers the whole platform exceptionally well, and we particularly dig how it manages to show off both Rails and Rack app deployment.

Big ups to Remi for putting this together, and way to shame us for not getting any official screencasts together for the new and improved Heroku. 🙂

Best Podcasts on the Net

Need something else (besides the “Rails Envy Podcast #77″:http://www.railsenvy.com/2009/4/29/rails-envy-podcast-episode-077-04-29-2009) to listen to while you’re riding home, or maybe while you travel to Railsconf? I’ve been meaning to write a blog post for a while going over some of the best Podcasts on the internet. You know, the stuff that inspires me to create better audio content.

If you haven’t listened to some of these, I highly recommend you take the time.

“:http://www.wnyc.org/shows/radiolab/

RadioLab is by far the best science related podcast on the net, breaking new ground when it comes to audio and education. Jason and I took a great deal of inspiration from this when creating the Rails Envy Podcast. “HomePage”:http://www.wnyc.org/shows/radiolab/ | “Subscribe on iTunes”:itpc://www.wnyc.org/shows/radiolab/rss?utm_source=rss&utm_medium=hp&utm_campaign=radiolab

Warning: On the itunes feed you will find both the RadioLab podcasts and the hour long shows. I recommend you listen to all the hour long shows before bothering with the podcasts between the stories.

“:http://www.themoth.org/podcast

“*The Moth Podcast*”:http://www.themoth.org/podcast

True stories told live, on stage, without notes. The Moth is an organization that does story competitions and teaches good storytelling. Give it a try, and you’ll get hooked. “HomePage”:http://www.themoth.org/podcast | “Subscribe on iTunes”:http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewPodcast?id=275699983

 

“:http://www.crescentstation.net/?page_id=11
“*Crescent Station*”:http://www.crescentstation.net/?page_id=11

The first Podiobook I’m going to recommend you listen to if you like Sci-Fi (if you’d prefer Horror, see below). Phil Rossi writes, performs, and even composes the sound track to my favorite book on audio. “Think of HP Lovecraft, in space, knocking boots with the slutty love child of Phillip K. Dick and Anne Rice. That’s Crescent.” (Scott Sigler) “Homepage”:http://www.crescentstation.net/?page_id=11 | “iTunes Feed”:http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewPodcast?id=213859296

Phil also just finished a short story called Eden, which is also amazing.

 

“:http://www.scottsigler.com/
*Anything by Scott Sigler*

Scott Sigler is a legend in the Podiobooks world. His writing, acting, and technical prowess over his art form is nothing short of brilliant. In case you haven’t got it yet, this is NOT JUST someone reading their book on audio, this is the author doing all the voices, adding sound effects, and music. It’s like a good book, you won’t be able to stop listening.

Check out:

“:http://www.scottsigler.com/nocturnal

“:http://www.scottsigler.com/earthcore

“:http://www.scottsigler.com/therookie

“The Rookie”:http://www.scottsigler.com/therookie, is awesome if you enjoy watching Football.

“Earthcore”:http://www.scottsigler.com/earthcore is great Sci-Fi/Action/Horror.

“Nocturnal”:http://www.scottsigler.com/nocturnal is a great Horror/Thriller set in present day San Francisco.

 

Lastly, if you’re into Lost I hope you listen to the “Transmission”:http://www.hawaiiup.com/lost/ (”iTunes”:http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewPodcast?id=81066268).

That’s all for now, although there’s a few other goodies in my collection. If you listen to any of these and love them as much as I do, I’d love to hear from you (or talk to you at Railsconf). Also if you know of any great podcasts please post a comment, I’m always looking for more.

Rails Envy Podcast – Episode #077: 04/29/2009

<style>
.post ul { text-indent: none; margin-left: 0px; padding-left: 0px; }
.post ul li { line-height: 150%; margin-bottom: 8px; }
</style>

Get Harvest
Harvest is a web-based time tracking and invoicing application dedicated to helping freelancers and developers easily track hours and get paid faster. Simple and intuitive to use, it takes care of your numbers while you focus on delivering great work. Harvest’s visual reporting shows you the health of your operation at-a-glance.

Use coupon code RAILSENVY428 for $10 off the first month of service with Harvest.

Sponsored by New Relic
The Rails Envy podcast is also brought to you this week by NewRelic. NewRelic provides RPM which is a plugin for rails that allows you to monitor and quickly diagnose problems with your Rails application in real time. They also recently produced Rails Lab, which gives you expert advice on tuning and optimizing your Rails app.

Episode 77.

Subscribe via iTunes – iTunes only link.
Download the podcast ~17:00 mins MP3.
Subscribe to feed via RSS by copying the link to your RSS Reader

Not the post

This is not the post I wanted to write. The post that I wanted to write, that I in fact have mostly written and would have posted days ago if not for this distraction, was about what a great success Golden Gate Ruby Conference was and how proud we are of putting on a top-notch conference that raised the bar in many ways. But I’m the person who is responsible for the technical program at the conference, and with the astounding level of distress over the presentation Matt Aimonetti gave at my conference, it’s clear I need to do something.

First off, I want to apologize. The technical program at GoGaRuCo was my responsibility. I could have done a better job and prevented this from happening. Everyone had the best of intentions and there are good reasons why things happened the way they did, but that doesn’t excuse the lapse. As a first-time conference organizer there was a lot that I had to learn as I went, and this is definitely an important lesson. I haven’t yet figured out the best way to prevent this from happening again, but I’m determined to find a way to do better next time.

And to be clear, I don’t think Matt’s talk was appropriate for a professional conference. If an employee of Google or Apple or Microsoft gave that presentation at a company event, he likely would be fired. I know that many people found the talk informative, creative and entertaining, and I’m not arguing against that. But there were people who found the sexual imagery in the talk objectionable or felt alienated by the atmosphere it created, and that’s not okay. I think there are ways a talk with that title could have been given that would have worked, but the talk that was delivered didn’t.

The most commonly used word to describe people’s negative reactions to Matt’s talk that I’ve read in online discussions is “offended”. From what I know, this is not accurate. I haven’t spoken to any conference attendee who said they were offended by the talk. I’ve heard that people felt insulted, alienated, threatened or unwelcome. I don’t know if that makes a difference to people’s arguments, but I think it is a distinction that matters.

This all has been particularly frustrating to me because we specifically tried to make GoGaRuCo a conference that would be welcoming to women. I looked for women speakers for the program to make the conference more women-friendly, and was so pleased to get Jacqui Maher, who gave one of the highest rated talks of the conference. When we announced Jacqui was speaking, we had several women register right away. We also made arrangements so that a nursing mother had a private space to use her breast pump. Leah and I talked about the number of women who registered often, and did our best to talk to women and get them to attend. And I don’t think doing any of what we did deserves any kind of special credit – I think it should be just part of what you do if you want a quality conference.

3% attendance by women may seem low (and it is), but compared to most other regional Ruby conferences we were way ahead. One of the other regional conf organizers said how he wished he could have more than 1% women at his conference. But after all that, we end up known as the conf with the sexist pr0n talk. Sigh.

There’s no way I’m going to make this go away, defuse the issue, or correct all the wrong assumptions about what really happened and why. But it might help to know more about how this happened.

I’ve mentioned before that we tried an experiment putting together the program. Half the speakers were invited directly. I contacted people I knew could give a great talk, and asked them to do one for GoGaRuCo. Each of those conversations had some back-and-forth about the talk and what I wanted to see at the conference. There were no proposals for me to read or evaluate, so these conversations were important. Then there were talks that were selected by attendee voting. We asked for talk proposals, put the proposals up on a website, and registered attendees voted for the talks they wanted to see. Matt’s talk was selected by this voting process. There was some amount of conversation I had with the selected presenters, but not as much as with the invited talks. After all, I had descriptions of the talks in the proposals so didn’t feel I needed to do anything. I also wanted to respect the results of the voting process. The whole point was for there to be content accepted that I wouldn’t have necessarily chosen myself.

Matt’s original proposal did not include very much detail about the technical content of the talk. I worked with him to expand the proposal to include sufficient detail to include it in the set of proposals. As a sometime editor with some experience with professional detachment, I tried to keep out of the business of telling him what he should be proposing and focused on getting a proposal that would help voters make their choice. I could have refused to post the proposal, or told Matt to clean up the “pr0n” spin if he wanted to have it included, but I didn’t consider that the actual talk would go over the line. To compound things, Matt has told me that the talks we had over grooming his proposal gave him the impression that I was approving doing a talk that included the porn theme. I had a long talk with Matt last night about this, and I can see how he would see our conversations as approval of the content of his talk. That doesn’t mean I think Matt’s talk was the right talk to give or let him off the hook for what he did, but if nothing else did, this makes my role in the matter clear. It also shows where I need to do better.

I have been to many conferences, spoken at a few, and been on program committees. I’ve never known any conference to pre-screen presentations or give speakers guidelines about what content is acceptable. We all assume people know what is appropriate and usually that’s the case. There have certainly been inappropriate talks given before, and I expect there will continue to be so in the future. But we make the same decision over and over, to let speakers express themselves in their own way, and to use their creativity and passion to communicate in a way that is engaging. Usually that gives us good results. Sometimes it doesn’t. I don’t think that censoring presentations or imposing content guidelines would help much, if at all, and I don’t think it would be worth the potential loss of creativity and quality. It’s certainly worth looking at how to improve things, but there don’t appear to be any easy answers.

People are getting really upset over this, and that’s completely understandable. There has been a lot of anger and name calling and finger pointing and arguing in general. I’ve gotten angry myself too. But I think most people are coming at this with good intentions and are representing what they believe in. Maybe that’s naive, I don’t know. I do think that this has turned into an important conversation about women in our technical communities, and I have to hope that something good will come of it in the end.

I’m pretty overwhelmed by this issue and have spent way more time on it than I have to spare right now, but if there’s something you to have to say to me, I want to hear it. I don’t know how soon I’ll be able to respond, but I’ll do my best.

Update: I realize those without some previous knowledge of the Golden Gate Ruby Conference might not realize that the awesome Leah Silber was co-organizer of the conf with me. She did most of the heavy lifting on the logistics and stuff, but the technical program was my responsibility. Just wanted to make sure Leah gets the credit she deserves.

Rails Envy Podcast – Episode #077: 04/29/2009

Get Harvest
Harvest is a web-based time tracking and invoicing application dedicated to helping freelancers and developers easily track hours and get paid faster. Simple and intuitive to use, it takes care of your numbers while you focus on delivering great work. Harvest’s visual reporting shows you the health of your operation at-a-glance.

Use coupon code RAILSENVY428 for $10 off the first month of service with Harvest.

Sponsored by New Relic
The Rails Envy podcast is also brought to you this week by NewRelic. NewRelic provides RPM which is a plugin for rails that allows you to monitor and quickly diagnose problems with your Rails application in real time. They also recently produced Rails Lab, which gives you expert advice on tuning and optimizing your Rails app.

Episode 77.

Subscribe via iTunes – iTunes only link.
Download the podcast ~17:00 mins MP3.
Subscribe to feed via RSS by copying the link to your RSS Reader

Next Update and RailsConf

News and notes

  • The next book update will probably be released on Sunday before I head off for RailsConf. Definitely in there will be a longish chapter on Shoulda and a bit on how to handle legacy code. Hopefully one other chapter as yet to be determined.
  • Speaking of RailsConf, I’ve also gotten a Birds of a Feather session on the schedule. The title is “Testing In Rails, What’s New, What’s Now, and What’s Missing”, and it’ll be at 9 PM Wednesday night in Pavilion 9 – 10. Come by for what I hope will be an interesting discussion of everything new in Rails Testing land.
  • Also speaking of RailsConf, follow @railsrx or @noelrap on Twitter for news about giveaways. There will be at least two sets, one of which will be based on attendance at my talk, one of which will not. Free books, and some kind of merch for potential winners who already have a license for the book.
  • Finally, speaking of RailsConf, I’d love to meet anybody who is reading the book or this blog. I’m probably the worst networker in the known universe, but please do come up and say hi if you see me around.

More speakers on RailsConf Vegas

RailsConf is going down next week in Vegas. I’m sure it’s going to be a great show and can’t wait to get out there. A few of the presenters have been blogging about the presentations that they’re doing.

Alexander Dymo is doing a session on Advanced Performance Optimization of Rails Applications, Jason Rudolph has one on testing JavaScript in Rails projects, Pat Maddox is talking about dealing with legacy Rails projects, and finally Sean Schofield just has five good reasons to attend the conference.

Check out the entire schedule for the conference and signup if you haven’t already. See you in Vegas!

GoGaRuCo – Ruby on Rails Podcast

Rich Kilmer talks about MacRuby and Aaron Quint discusses Sinatra. From GoGaRuCo in San Francisco.

Also Mentioned

GoGaRuCo – Ruby on Rails Podcast

Rich Kilmer talks about MacRuby and Aaron Quint discusses Sinatra. From GoGaRuCo in San Francisco.

Also Mentioned

#159 More on Cucumber

There is a lot more to Cucumber than I showed in an earlier episode. See how to refactor complex scenarios in this episode.

#159 More on Cucumber

There is a lot more to Cucumber than I showed in an earlier episode. See how to refactor complex scenarios in this episode.

This Week in Edge Rails

April 18, 2009 – April 24, 2009

Edge Rails has been undergoing major surgery for the past week, as the core team gets ready for a Rails 3.0 alpha release at RailsConf. We saw about 50 commits to the master branch in GitHub this week. Here’s a quick overview of the recent changes.

Rails 2.3.x Changes

Before digging into the changes on the master branch, there were a few things committed to the 2-3-stable branch. If you’re running on 2.3 edge, these are ready for you; they’re also ported to the master branch already.

  • A couple of bugs involving associations with hash conditions have been fixed, notable to make sure that :dependent => :delete will work.
  • The PostgreSQL Active Record adapter now does the right thing if you use tables in non-default schemas. You need to set the table name in your model class, of course: set_table_name 'other_schema.customers' commit
  • Also in the PostgreSQL adapter, a couple of bugs related to wrong quoting of names with capital letters are fixed.

ActionView::Path Refactoring

One major chunk of change in Rails 3 this week comes from the continued work to refactor Action Pack. This time, ActionView::Path was the target. Changes in commit include decoupling ActionView::Path from Action Controller and Action Mailer, which gives us two major benefits. First, consolidating similar code in one place makes it easier to understand and maintain. Second, by abstracting this stuff and giving it an API, we’ll make it possible for other components to participate in the controller layer, beyond mailers and traditional controllers.

There’s also some work here to set up for the future. The plan is to decouple templates from the file system, and to decouple localization from ActionView. Stay tuned!

Pluggable JSON Backends

You may recall that recently Rails went to pluggable XML support. This week, thanks to work from Rick Olson, we have pluggable JSON. This means that you can replace the default YAML-based JSON support with the JSON gem:


ActiveSupport::JSON.backend = "JSONGem" 

As part of this change, Rails now uses ActiveSupport::JSON.encode() wherever it needs JSON. This replaces using #to_json, and Rails team is recommending that you do the same. If you do choose to use #to_json, you need to be aware that some libraries might override it. You can use this pattern to make sure that you’re getting Rails’ own definition of #to_json:


gem 'json'
# JSON gem loaded, which overwrites to_json
ActiveSupport::JSON.backend = "JSONGem" 

class ActiveRecord::Base
  # replace the gem's to_json with Rails' own to_json
  alias to_json rails_to_json
end

Active Support à la carte

In Rails 2.x, require "active_support" pulls in all core extensions at once. In Rails 3, require "active_support" only makes the extensions available; you have to explicitly require those you wish to use. Many of this week’s changes were concerned with breaking up Active Support so you can take just what you need and nothing more.

The end result is that it’s easy to cherry-pick features from Active Support without feeling like you’ve invited a portly gentleman for a piggyback ride. As an added bonus, all core extension documentation is now consolidated in one place: the core class. You don’t have to go poking around every extension module to find a method.

Internally, the core extensions have been reorganized so they can be directly required without assuming all of Active Support is available. Want inflections like "car".pluralize? require "active_support/core_ext/string/inflections" and you’ve got them. Note: the implementation and organization will change as we settle on the best way to provide core extensions.

Other Edge Changes

  • The Rails extension to Pathname was dropped, since it was only used in one spot in the code. commit
  • Support for the DRb cache store has been removed. commit
  • The sqlite Adapter joins the PostgreSQL adapter in supporting microsecond resolution. commit

Commercial Launch

When Adam, Orion, and I started Heroku two years ago, we had no idea how much new technology we would have to build to realize our vision of an instant platform for Ruby that just works.

Luckily, we were able to attract an amazing team to work on this problem with us, and the team has really shaped Heroku into the offering it is today. We’re currently by far the fastest and easiest deployment platform for Ruby, and we’ve gotten great feedback on our provisionless hosting architecture.

We have over 25,000 apps running on the platform today, and many of our users have been asking for pricing and paid services for some time now. So today we’re pleased to announce that we are officially out of beta and available for commercial use.

Detailed pricing information is now available.

Dynos

As discussed previously, our system is built around a dyno grid that allows us to deploy your app just by pushing your code, and to scale it up and down instantly. Your app runs inside this grid in any number of dynos.

The number of dynos you run directly affects the concurrency and therefore the performance of your app, so our pricing is based around this same concept. The first dyno for an app is always free, and additional dynos will be billed at $0.05 per dyno-hour.

Play with the dyno slider on our pricing page to estimate your bill.

You can scale your dynos up and down instantly at any time via the web interface, our API, or our command-line gem.

Database

At the database layer, we have two categories of options. We offer a shared database cluster in three storage sizes. The smallest size is free, and you can upgrade/downgrade between options instantly at any time.

For more serious needs, we also offer three dedicated database plans, ranging from small to large in compute and storage levels. These can also be changed at any time, but resizing requires a few moments of downtime.

Add-ons

We’re also officially launching some add-on features. Some of these features, like backups and more frequent cron tasks are paid features. Others, like custom domains and SSL are free, but require you to enter your billing information as an abuse protection measure (more info).

There is a lot more to come – you can get a preview by checking out the beta features in the add-ons section.

Free Service

It’s important to us to continue to provide a free service tier, useful enough to get started with Heroku. This is still available, and is the default for new apps with a single free dyno and free shared database.

This free tier is well suited for rapid-prototyping, staging, and testing purposes, as well as actually running lightweight apps.

Pricing and paid-features are being rolled out in phases across our user base. It may take a few days for commercial service to be activated for all accounts.

The official press release is available here.

Now With Navigation and Charted Archives

In which I discuss some recent updates to the site.

Every couple weeks, I sit down and try to think of things that will make this site better. Rather than post each time I come up with something, I typically wait until I have a few to share and then post about it. This is another one of those posts.

Charted Archives

A few weeks ago, I thought it would be cool to show my archives by month in a bar chart. My thought process was that someone new to the site could see the graph and very easily get a feel for the frequency of my posting. Below is a little screenshot for archival purposes and those in a feed reader. You can also just scroll towards the bottom of any page to see it live.

Archives

Navigation and Search

I’ve always put a lot of emphasis on my footer. From there you can get to pretty much anywhere on the site, but I thought it was high time that I finally put some navigation at the top. Then I thought, while I was at it, I might as well add some new content, so there are now two new pages to peruse.

The first page is dedicated to all my projects. It lists those that I care about, along with a little history and such. I think sometimes our Github profiles and a websites get too disconnected, so I think this will help connect the two a bit.

The second page I am simply referring to as “dude”: http://railstips.org/dude/. Whenever I find something new and fall in love with it, I’m always telling my friends, “Dude, you have to check this out.” That theory along with the success of my setup and software post, led me to believe I should have a page here at RailsTips dedicated solely to the things that make me say “dude!”.

The other tweak I did is I moved the search box to the top of the page from the sidebar. The reason for the move is that I think the search box is easily overlooked in the sidebar, but should do fine up at the top. It also helps balance the content above the navigation a bit.

Again, for archival purposes, I’m including a picture of the nav and search tweaks.

Navigation and Search

The only other addition worth noting is that I’ve switched from sections/categories to tags for articles. More a semantic change than anything, but I thought I would mention it. One of the next tweaks I’ll do is add a tag page where you can view all the tags listed alphabetically to get an idea for the topics covered here.

That is all for now. Just thought I would mention the updates for those who read the posts from their feed reader and for those that might be new to the site. I hope they make it even easier to find and discover the content I’ve been posting.

Uploading Files

        Anyone who’s built a rails application that deals with large file uploads probably has a few horror stories to tell about it.  While some people love to overstate the issues for their own purposes, it’s still something that can be quite challenging to do well.


<h2>What’s the Problem?</h2>


As I mentioned in the article on File Downloads, your rails processes are a scarce resource.  You need them to be free to handle your applications’ requests, if they’re all busy, your users will be left waiting.  When we optimised the download processes we made sure that we used our webservers instead of tying up a rails process to spoon feed the file out over the network to your users.  Dealing with uploads has a similar problem.


When a browser uploads a file, it encodes the contents in a format called ‘multipart mime’ (it’s the same format that gets used when you <!--more--> an email attachment).  In order for your application to do something with that file, rails has to undo this encoding.  To do this requires reading the huge request body, and matching each line against a few regular expressions.  This can be incredibly slow and use a huge amount of <span class="caps">CPU</span> and memory.


While this parsing is happening your rails process is busy, and can’t handle other requests.  Pity the poor user stuck behind a request which contains a 100M upload!


<h2>What’s <strong>not</strong> the problem?</h2>


Some people seem to think that the File Upload problem with rails is that the entire process is blocked while the browser <strong>sends</strong> the encoded body to you.  This isn’t not true, and hasn’t been for a long time. Whether you’re using nginx + mongrel, apache + mongrel or apache + passenger, your web server buffers the entire request before rails locks itself for processing.  So no matter how slow a user’s connection is, your application isn’t locked while they upload their file.


<h2>What can you do?</h2>


There are a number of unattractive options to work around this slow multipart-parsing.  The most common I’ve seen is to send uploads to a non-rails process such as a <span class="caps">CGI</span> script or a merb/mongrel/rack application.  <span class="caps">CGI</span> scripts have the obvious disadvantage that you need to write a script simple enough to start up quickly and featured enough to process your uploads.  Doing it in rack leaves you relying on ruby’s threading to handle parallelism.  <a href="http://www.igvita.com/2008/11/13/concurrency-is-a-myth-in-ruby/">This is probably not what you want</a> and your throughput is probably much lower than it would be without that upload being processed.


<h2>What else can you do?</h2>


Because neither of these options were acceptable Pratik Naik and I have built a <a href="http://modporter.com/">Mod Porter</a> an Apache module that does the heavy lifting for your file uploads. All of the hard stuff is done by <a href="http://httpd.apache.org/apreq/">libapreq</a> though, so you don’t have to worry about using C code written by two ruby programmers!


Porter is essentially the inverse of X-SendFile.  It parses the multipart post in C inside your apache process and writes the files to disk.  Once that work is done it changes the request to look like a regular form <span class="caps">POST</span> which contains pointers to the temp files on disk.  To maintain system security it also signs the modified parameters so people can’t attack your system like those old <span class="caps">PHP</span> apps.


This means that your rails processes don’t have to deal with anything more than a regular form post which is nice and fast. In addition to the apache module, Porter also includes a Rails Plugin which hides all of this from you.  It makes an upload handled by Porter, look just like a regular Rails Upload.


<h2>How fast is it?</h2>


The speed of upload parsing isn’t particularly relevant, the reduced locking is far more important.  Your user’s internet connection is much more important for the round-trip upload performance than your upload handler’s parser.


Having said all that, Porter runs <strong>significantly</strong> faster than the equivalent pure-ruby parsing code.  Depending on the size and number of uploads we’ve seen response times between 30 and 200 times as fast.  That’s not just compared to rails’ upload parser, it’s that much faster than every other ruby mime parser we tried.


<h2>Isn’t this just like the Nginx module?</h2>


Kinda.  We’ve been thinking about this module ever since we started using lighttpd’s X-SendFile header.  When I saw the nginx module get released I decided to start planning the Apache equivalent. Porter is completely transparent to your application, you don’t need a special form action, and you don’t need to tell Porter what form fields to pass through to the web application.  This means you can use porter in production, and mongrel or thin in development, without any changes to your application.


The biggest improvement from this is that you don’t need to change your nginx config every time you add a new input to a form, or a new file upload to your application.  This is extremely tedious and error prone, especially when making these changes involves a support ticket with your hosting provider.  The major goal we have with Porter is to make sure it always ‘Just Works’, so you can put a file upload into any form without having to worry about your web server.


<h2>Getting Started</h2>


Porter is still beta software, so you’re <strong>strongly</strong> advised to test it first, but you already knew that.   The porter website has the <a href="http://modporter.com/install.html">installation instructions</a>.  Once you’ve got that done you’ll need to add the rails plugin, and configure them to share a nice secure secret.  Then, hopefully, your application will Just Work but your uploads will be much less painful.


If you have any issues getting it running, leave us a note on the <a href="http://github.com/actionrails/modporter/issues">git hub issues page</a>.

Uploading Files

Anyone who’s built a rails application that deals with large file uploads probably has a few horror stories to tell about it. While some people love to overstate the issues for their own purposes, it’s still something that can be quite challenging to do well.

What’s the Problem?

As I mentioned in the article on File Downloads, your rails processes are a scarce resource. You need them to be free to handle your applications’ requests, if they’re all busy, your users will be left waiting. When we optimised the download processes we made sure that we used our webservers instead of tying up a rails process to spoon feed the file out over the network to your users. Dealing with uploads has a similar problem.

When a browser uploads a file, it encodes the contents in a format called ‘multipart mime’ (it’s the same format that gets used when you send an email attachment). In order for your application to do something with that file, rails has to undo this encoding. To do this requires reading the huge request body, and matching each line against a few regular expressions. This can be incredibly slow and use a huge amount of CPU and memory.

While this parsing is happening your rails process is busy, and can’t handle other requests. Pity the poor user stuck behind a request which contains a 100M upload!

What’s not the problem?

Some people seem to think that the File Upload problem with rails is that the entire process is blocked while the browser sends the encoded body to you. This isn’t not true, and hasn’t been for a long time. Whether you’re using nginx + mongrel, apache + mongrel or apache + passenger, your web server buffers the entire request before rails locks itself for processing. So no matter how slow a user’s connection is, your application isn’t locked while they upload their file.

What can you do?

There are a number of unattractive options to work around this slow multipart-parsing. The most common I’ve seen is to send uploads to a non-rails process such as a CGI script or a merb/mongrel/rack application. CGI scripts have the obvious disadvantage that you need to write a script simple enough to start up quickly and featured enough to process your uploads. Doing it in rack leaves you relying on ruby’s threading to handle parallelism. This is probably not what you want and your throughput is probably much lower than it would be without that upload being processed.

What else can you do?

Because neither of these options were acceptable Pratik Naik and I have built a Mod Porter an Apache module that does the heavy lifting for your file uploads. All of the hard stuff is done by libapreq though, so you don’t have to worry about using C code written by two ruby programmers!

Porter is essentially the inverse of X-SendFile. It parses the multipart post in C inside your apache process and writes the files to disk. Once that work is done it changes the request to look like a regular form POST which contains pointers to the temp files on disk. To maintain system security it also signs the modified parameters so people can’t attack your system like those old PHP apps.

This means that your rails processes don’t have to deal with anything more than a regular form post which is nice and fast. In addition to the apache module, Porter also includes a Rails Plugin which hides all of this from you. It makes an upload handled by Porter, look just like a regular Rails Upload.

How fast is it?

The speed of upload parsing isn’t particularly relevant, the reduced locking is far more important. Your user’s internet connection is much more important for the round-trip upload performance than your upload handler’s parser.

Having said all that, Porter runs significantly faster than the equivalent pure-ruby parsing code. Depending on the size and number of uploads we’ve seen response times between 30 and 200 times as fast. That’s not just compared to rails’ upload parser, it’s that much faster than every other ruby mime parser we tried.

Isn’t this just like the Nginx module?

Kinda. We’ve been thinking about this module ever since we started using lighttpd’s X-SendFile header. When I saw the nginx module get released I decided to start planning the Apache equivalent. Porter is completely transparent to your application, you don’t need a special form action, and you don’t need to tell Porter what form fields to pass through to the web application. This means you can use porter in production, and mongrel or thin in development, without any changes to your application.

The biggest improvement from this is that you don’t need to change your nginx config every time you add a new input to a form, or a new file upload to your application. This is extremely tedious and error prone, especially when making these changes involves a support ticket with your hosting provider. The major goal we have with Porter is to make sure it always ‘Just Works’, so you can put a file upload into any form without having to worry about your web server.

Getting Started

Porter is still beta software, so you’re strongly advised to test it first, but you already knew that. The porter website has the installation instructions. Once you’ve got that done you’ll need to add the rails plugin, and configure them to share a nice secure secret. Then, hopefully, your application will Just Work but your uploads will be much less painful.

If you have any issues getting it running, leave us a note on the git hub issues page.

Domain-Driven Design Using Naked Objects, now in Beta

Happy Earth Day! In honor of a back-to-nature approach, it’s time to get naked—with Naked Objects, the heart of domain-driven design

Rails Envy Podcast – Episode #076: 04/22/2009

Episode 76. You’ll say wow every time!

Subscribe via iTunes – iTunes only link.
Download the podcast ~17:00 mins MP3.
Subscribe to feed via RSS by copying the link to your RSS Reader


Sponsored by Hashrocket
The Rails Envy podcast is brought to you this week by Integrum. Integrum is a team of dedicated Rails developers who emphasize business value, speed to market, and open communitcation to build remarkable Rails applications. They build software that you’ve been told is impossible.

Sponsored by New Relic
The Rails Envy podcast is also brought to you this week by NewRelic. NewRelic provides RPM which is a plugin for rails that allows you to monitor and quickly diagnose problems with your Rails application in real time. They also recently produced Rails Lab, which gives you expert advice on tuning and optimizing your Rails app.