Matz Named 2011 Free Software Award Winner

We are pleased to announce that Yukihiro "Matz" Matsumoto, the creator of Ruby and Heroku’s Chief Ruby Architect, has received the 2011 annual Advancement of Free Software Award. Presented by Richard Stallman and on behalf of the Free Software Foundation, the award is given each year to those who have greatly contributed to the freedom of software.

This is great news for Matz, who has dedicated over 20 years to the development of free software including the creation of the Ruby programming language. While writing Ruby he chose to focus on programmer happiness and productivity, and the result has been extraordinary. The popularity of the language, helped in part by the Rails framework, has encouraged a new generation of free software advocates who together have released more than 36,000 freely downloadable Ruby libraries known as Gems.

This award is also a huge win for Ruby. It’s a recognition of Ruby’s importance, will bring more people into the community, and encourages more developers to share their work. If you’d like to contribute to Ruby, visit the ruby core page where you can learn how to submit a patch or file a bug report.

We’re very proud of the Ruby community as a whole, and for Matz and all of his accomplishments. So please join us today, along with the Free Software Foundation, in thanking him for all of his hard work.

The Rails View: Create a Beautiful and Maintainable User Experience, now in print

The Rails View: Create a Beautiful and Maintainable User Experience, now in print and shipping.

#336 Copycopter

Copycopter provides a nice interface that clients can use to edit the text in a Rails application. Learn how to deploy a Copycopter server using Heroku and integrate it in a Rails application through I18n.

That Vulnerable Feeling

Peter and Jason talk about the GitHub security vulnerability, handling unsanitized data, and bring you the latest news and project releases.

Blurtt launches in the app store

I first met Jeanette Cajide early in 2011. She was a founder at Blurtt – a company that was then focused on delivering a web based photo-upload service, permitting users to send printed postcards of said photos to friends via mail.

A few things were quickly obvious.

 

1. Blurtt is a great name, but not for sending postcards.

2. Mobile is the right place to develop what Blurtt could be.

3. Blurtt should let people Blurtt things out…..

4. Combining images with text produces a very powerful means of communicating emotion-rich messages.

5. Jeanette is a force of nature and simply refuses to fail.

 

So, Blurtt became archimedes labs second acceleration company, after Tomer Kagan’s Quixey. The 12 months since have been great. Jeanette weathered many storms, and succeeded in building a set of relationships that allowed her to ship her product today, with significantly less than $100,000 spent. You can get it from the app store now.

I couldn’t be more thrilled for Jeanette, Laura, Kuba and the team at Dialexa in Texas…..Congrats

I know that the team is planning lots of new features. Can’t wait to see where this journey takes us.

 

 

 

The dRuby Book: Distributed and Parallel Computing with Ruby, now in print

The dRuby Book: Distributed and Parallel Computing with Ruby, now in print and shipping.

#334 Compass & CSS Sprites

Compass improves the Sass experience by providing useful mixins, functions, and more. You will also learn how to make CSS sprites with it in this episode.

Shifting to the client again

This is my take on the current shift to rich, in-browser JavaScript apps.

Looking back over a few decades, this is the progression of how applications have been built:

  1. mainframes and dumb terminals
  2. minicomputers and smart terminals
  3. networked workstations
  4. workstations and shared code/data repositories
  5. web apps and static HTML
  6. web services and rich browser apps

Translation: The main body of the application code lives on the:

  1. server
  2. server + client
  3. client
  4. server + client
  5. server
  6. server + client

The server/client pendulum swings back and forth. The next logical step is apps that run on the client using standard services. Just give it a few more years…

Java Template Apps on Heroku

Learning a new language or framework can be both fun and rewarding. But tutorials only get you so far: one of the easiest ways to get started is by copying an existing sample app.

Today we’re introducing template-based app creation for Java on Heroku. To try it out, go to www.heroku.com/java and click Create App on one of the four templates at the bottom of the page.

In seconds, you’ll have your own copy of the app deployed and running on Heroku. You can then clone the app and start editing it in Eclipse or other development environment.

We’ve put together four Java app templates that represent the most popular and most exciting ways to build Java apps today. But Heroku is not limited to these frameworks. Heroku runs 100% open-standards Java, so you can run any Java application and use any framework. You can even use no framework at all, such as running stand-alone Java processes that are not web applications.

Deploy your first Java template app on Heroku today!

Introducing key/value data storage in Heroku Postgres

From the Heroku Postgres Blog:

"One of the great strengths of PostgreSQL is extensibility. Just as the JVM has become more than a way to just run Java—spawning languages such as Clojure and Scala—PostgreSQL has become more than just a home to relational data and the SQL language. Our first officially supported Postgres extension, hstore, enables you to build better apps faster without sacrificing the power, reliability, and flexibility of the underlying PostgreSQL storage engine. By using hstore, you will be able to leverage the flexibility and agility of schema-less data stores in existing environments…"

→ Read More

Check out the Heroku Postgres Blog to read more.

Build Awesome Command-Line Applications in Ruby: now in print

Build Awesome Command-Line Applications in Ruby: now in print

#332 Refinery CMS Basics

If you need to quickly create an informational site that can be easily edited, consider using a content management system. Here I show how to build a site using Refinery CMS.

Ruby Programming 32nd Batch: Registrations now open

Registrations are now open for RubyLearning’s popular Ruby programming course. This is an intensive, online course for beginners that helps you get started with Ruby programming.

Here is what Sandra Randall (Butler), a participant who just graduated, has to say – “You kindly offered me the opportunity to join your Ruby course. I’m new to development and found the course, even though basic for programmers, a little tricky for me. I managed to complete all of the assessments and really learnt a lot. Thank you very much for the opportunity. It has really given me the push I needed to learn Ruby and I’m currently treading my way through both the pickaxe and Agile Development books and enjoying it. I’ve recently been offered a position as a Junior Systems Developer at a local Software house in South Africa – all thanks to the push you gave me which gave me the motivation and drive to get going.”

What’s Ruby?

Ruby

According to http://www.ruby-lang.org/en/ – “Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. Ruby’s elegant syntax is natural to read and easy to write.”

Yukihiro Matsumoto, the creator of Ruby, in an interview says –

I believe people want to express themselves when they program. They don’t want to fight with the language. Programming languages must feel natural to programmers. I tried to make people enjoy programming and concentrate on the fun and creative part of programming when they use Ruby.

What Will I Learn?

In the Ruby programming course, you will learn the essential features of Ruby that you will end up using every day. You will also be introduced to Git, GitHub, HTTP concepts, RubyGems, Rack and Heroku.

Depending on participation levels, we throw a Ruby coding challenge in the mix, appropriate for the level we are at. We have been known to give out a prize or two for the ‘best’ solution.

Who’s It For?

A beginner with some knowledge of programming..

You can read what past participants have to say about the course.

Mentors

Satish Talim, Michael Kohl, Satoshi Asakawa, Victor Goff III and others from the RubyLearning team.

Dates

The course starts on Saturday, 14th Apr. 2012 and runs for seven weeks.

RubyLearning’s IRC Channel

Most of the mentors and students hang out at RubyLearning’s IRC (irc.freenode.net) channel (#rubylearning.org) for both technical and non-technical discussions. Everyone benefits with the active discussions on Ruby with the mentors.

How do I register and pay the course fees?

  • The course is based on the The Ultimate Guide to Ruby Programming eBook. This book is normally priced at US$ 19.95 and we are discounting it US$ 10.00 by combining it in the Course+eBook option below.
  • You can pay either by Paypal or send cash via Western Union Money Transfer or by bank transfer (if you are in India). The fees collected helps RubyLearning maintain the site, this Ruby course, the Ruby eBook, and provide quality content to you.
  • Once you pay the fees below, register on the RubyLearning.org site and send us your name and registered email id while creating an account at RubyLearning.org to satish [at] rubylearning [dot] com
  • We will enroll you into the course. If you have purchased the eBook at the time of registration, we will personally email you the eBook within 24 hours.

You can pay the Course Fees by selecting one of the three options from the drop-down menu below. Please select your option and then click on the “Add to Cart” button.

Register

At the end of this course you should have all the knowledge to explore the wonderful world of Ruby on your own.

Here are some details on how the course works:

Important:

Once the course starts, you can login and start with the lessons any day and time and post your queries in the forum under the relevant lessons. Someone shall always be there to answer them. Just to set the expectations correctly, there is no real-time ‘webcasting’.

Methodology:

  • The Mentors shall give you URL’s of pages and sometimes some extra notes; you need to read through. Read the pre-class reading material at a convenient time of your choice – the dates mentioned are just for your guideline. While reading, please make a note of all your doubts, queries, questions, clarifications, comments about the lesson and after you have completed all the pages, post these on the forum under the relevant lesson. There could be some questions that relate to something that has not been mentioned or discussed by the mentors thus far; you could post the same too. Please remember that with every post, do mention the operating system of your computer.
  • The mentor shall highlight the important points that you need to remember for that day’s session.
  • There could be exercises every day. Please do them.
  • Participate in the forum for asking and answering questions or starting discussions. Share knowledge, and exchange ideas among yourselves during the course period. Participants are strongly encouraged to post technical questions, interesting articles, tools, sample programs or anything that is relevant to the class / lesson. Please do not post a simple "Thank you" note or "Hello" message to the forum. Please be aware that these messages are considered noises by people subscribed to the forum.

Outline of Work Expectations:

  1. Most of the days, you will have exercises to solve. These are there to help you assimilate whatever you have learned till then.
  2. Some days may have some extra assignments / food for thought articles / programs
  3. Above all, do take part in the relevant forums. Past participants will confirm that they learned the best by active participation.

Some Commonly Asked Questions

  • Qs. Is there any specific time when I need to be online?
    Ans. No. You need not be online at a specific time of the day.
  • Qs. Is it important for me to take part in the course forums?
    Ans. YES. You must Participate in the forum(s) for asking and answering questions or starting discussions. Share knowledge, and exchange ideas among yourselves (participants) during the course period. Participants are strongly encouraged to post technical questions, interesting articles, tools, sample programs or anything that is relevant to the class / lesson. Past participants will confirm that they learned the best by active participation.
  • Qs. How much time do I need to spend online for a course, in a day?
    Ans. This will vary from person to person. All depends upon your comfort level and the amount of time you want to spend on a particular lesson or task.
  • Qs. Is there any specific set time for feedback (e.g., any mentor responds to me within 24 hours?)
    Ans. Normally somebody should answer your query / question within 24 hours.
  • Qs. What happens if nobody answers my questions / queries?
    Ans. Normally, that will not happen. In case you feel that your question / query is not answered, then please post the same in the thread – “Any UnAnswered Questions / Queries”.
  • Qs. What happens to the class (or forums) after a course is over? Can you keep it open for a few more days so that students can complete and discuss too?
    Ans. The course and its forum is open for a month after the last day of the course.

Remember, the idea is to have fun learning Ruby.

Technorati Tags: , , ,


(Powered by LaunchBit)

The Heroku Toolbelt

The Heroku Toolbelt is a package of the Heroku CLI, Foreman, and Git — all the tools you need to get started using Heroku at the command line. The Toolbelt is available as a native installer for OS X, Windows, and Debian/Ubuntu Linux.

The Toolbelt has been available since last fall as part of our polyglot platform. Since then it’s matured substantially with a huge amount of user testing, and now even has a shiny new landing page. Ruby developers can continue to use gem install heroku, but developers in other languages (Python, Java, Clojure, etc) will probably prefer not to have to install Ruby and RubyGems to use Heroku.

The installer won’t trample your existing install of Git if you have one. Similarly, although the Heroku CLI uses Ruby under the hood, the Toolbelt packaging isolates all of its libraries so it will not interfere with an existing Ruby setup.

The entire Toolbelt is open source. File an issue or, better yet, send a pull request if you see ways that it can be improved.

What is docrails?

Over the years I have seen some confusion about what is exactly
docrails
and how it relates to the documentation of Ruby on Rails.

This post explains everything you want to know about this aspect of the project.

What is docrails?

docrails is a branch of Ruby on Rails with
public write access where anyone can push doc fixes.

If you see a typo, you’d like to correct a factual error, complement some
existing documentation, add a useful example… before docrails existed you had
to open a pull request (or the equivalent in those days) and follow the ordinary
workflow to get it accepted. docrails allows you to clone the repo, edit, and
push. Done!

ZOMG, that’s awesome! Tell me more!

Changes to the code base need review before they are pushed. Each individual new
feature or bug fix needs the perspective and responsability of core team
members to take a decision about it.

Documentation fixes, though, are much more likely to be fine as they are. So,
docrails has a public write policy to ease the workflow for contributors.

All commits have to be reviewed anyway, so docrails needs the same effort from
Rails committers than going through pull requests, please everyone give big
props to Vijay Dev who is nowadays in charge
of this time consuming task.

The point of docrails is to provide a way to contribute to the Rails documentation
that is fast and easy for contributors.

But wait, I am editing some separate thing?

docrails is a separate branch because it has a different access policy, but
you are editing the actual Ruby on Rails documentation.

Every few days, once all new commits are reviewed docrails is merged into
master, and master is merged into docrails. Also, very important edits may be
cherry-picked into stable branches at the discrection of who merges.

What is allowed in docrails?

You can freely push changes to any RDoc, guides, and READMEs.

No code can be touched at all. That’s a hard rule. No matter how insignificant,
not even a one character typo in a string literal.

CHANGELOGs cannot be edited either.

Is docrails a documentation project?

No, Ruby on Rails has no documentation project. Treating documentation as a
separate aspect of the project would be similar to treating testing as an
external part of the project.

Documentation is an integral part of the development of Ruby on Rails.
Contributing a feature or bug fix means contributing its code, test coverage,
and documentation.

I am preparing a pull request, should I document later via docrails?

No, docrails is meant only for quick doc fixes.

Pull requests should be complete: code, tests, and docs. If a pull request lacks
any of those in general it won’t be accepted as is.

Also, updating docs does not only mean that you edit the RDoc next to the code
you are touching. Often the change needs grepping the project tree to find
instances of what the pull request is about, to update examples, revise
explanations affected by your change, or writing new documentation.

Tidbit: run ack -a to have guides included in the search.

I made a doc fix, when is it going to be online?

Rails releases are a complete set. The documentation itself is part of the
release. The fix is going to be online in the stable API or
guides websites when
the branch that contains the fix gets released.

Edits merged into master are always online in the edge API
and edge guides,
which are regenerated after every push to master. Thus, edits done via docrails
are online in the edge docs website after the next docrails/master cross-merge.

Can I open pull requests for documentation fixes in Ruby on Rails?

Absolutely. Specially if you are unsure about the fix. But if you feel confident
just push to docrails.

Please do not open pull requests in docrails.

Note that docrails has no issues tab. The reason is docrails is not a project,
as explained above, only a way to bypass pull requests. Documentation issues are
Ruby on Rails issues and belong to the Ruby on Rails project just as any other
kind of issue.

Does Ruby on Rails has documenters?

No, documentation comes with each push to master. Everyone documents Rails.

The only exception is guide authors. Guide authors take the task to write an
entire new guide about a certain topic, and they are allowed to push early
drafts to docrails for convenience (only guides in the public index are
considered to be published).

That’s for new guides. Once published, guides maintenance happens in master as
everything else.

Deploying with JRuby now in beta; PragPub magazine

Deploying with JRuby now in beta; PragPub magazine for March; Arduino updates to 1.0

#330 Better Sass with Bourbon

If you are tired of the browser vendor prefixes in CSS, check out Bourbon. It provides Sass mixins and functions to make CSS more convenient.

Misleading Title About Queueing

I don’t know about you, but I find it super frustrating when people blog about cool stuff at the beginning of a project, but then as it grows, they either don’t take the time to teach or they get all protective about what they are doing.

I am going to do my best to continue to discuss the strategies we are using to grow Gauges. I hope you find them useful and, by all means, if you have tips or ideas, hit me. Without any further ado…

March 1st of last year (2011), we launched Gauges. March 1st of this year (a few days ago), we finally switched to a queue for track requests. Yes, for one full year, we did all report generation in the track request.

1. In the Beginning

My goal for Gauges in the beginning was realtime. I wanted data to be so freakin’ up-to-date that it blew people’s minds. What I’ve realized over the past year of talking to customers is that sometimes Gauges is so realtime, it is too realtime.

That is definitely not to say that we are going to work on slowing Gauges down. More what it means, is that my priorities are shifting. As more and more websites use Gauges to track, availability moves more and more to the front of my mind.

Gut Detects Issue

A few weeks back, with much help from friends (Brandon Keepers, Jesse Newland, Kyle Banker, Eric Lindvall, and the top notch dudes at Fastest Forward), I started digging into some performance issues that were getting increasingly worse. They weren’t bad yet, but I had this gut feeling they would be soon.

My gut was right. Our disk io utilization on our primary database doubled from January to February, which was also our biggest growth in terms of number of track requests. If we doubled again from February to March, it was not going to be pretty.

Back to the Beginning

From the beginning, Gauges built all tracking reports on the fly in the track request. When a track came in, Gauges did a few queries and then performed around 5-10 updates.

When you are small, this is fine, but as growth happens, updating live during a track request can become an issue. I had no way to throttle traffic to the database. This meant if we had enough large sites start tracking at once, most likely our primary database would say uncle.

As you can guess, if your primary says uncle, you start losing tracking data. In my mind, priority number one is now to never lose tracking data. In order to do this effectively, I felt we were finally at the point where we needed to separate tracking from reporting.

2. Availability Takes Front Seat

My goal is for tracking to never be down. If, occasionally, you can’t get to your reporting data, or if, occasionally, your data gets behind for a few minutes, I will survive. If, however, tracking requests start getting tossed to the wayside while the primary screams for help, I will not.

I talked with some friends and found Kestrel to be very highly recommended, particularly by Eric (linked above). He swore by it, and was pushing it harder than we needed to, so I decided to give it a try.

A few hours later, my lacking JVM skills (Kestrel is Scala) were bearing their head big time. I still had not figured out how to build or run the darn thing. I posted to the mailing list, where someone quickly pointed out that Kestrel defaults to /var for logging, data, etc. and, unfortunately, spits out no error on startup about lacking permissions on OSX. One sudo !! later and I was in business.

3. Kestrel

Before I get too far a long with this fairy tail, let’s talk about Kestrel — what is it and why did I pick it?

Kestrel is a simple, distributed message queue, based on Blaine Cook’s starling. Here are a few great paragraphs from the readme:

Each server handles a set of reliable, ordered message queues. When you put a cluster of these servers together, with no cross communication, and pick a server at random whenever you do a set or get, you end up with a reliable, loosely ordered message queue.

In many situations, loose ordering is sufficient. Dropping the requirement on cross communication makes it horizontally scale to infinity and beyond: no multicast, no clustering, no “elections”, no coordination at all. No talking! Shhh!

It features the memcached protocol, is durable (journaled), has fanout queues, item expiration, and even supports transactional reads.

My favorite thing about Kestrel? It is simple, soooo simple. Sound too good to be true? Probably is, but the honeymoon has been great so far.

Now that we’ve covered what Kestrel is and that it is amazing, let’s talk about how I rolled it out.

4. Architecture

Here is the general idea. The app writes track requests to the tracking service. Workers process off those track requests and generate the reports in the primary database.

After the primary database writes, we send the information through a pusher proxy process, which sends it off to pusher.com, the service that provides all the live web socket goodness that is in Gauges. Below is a helpful sketch:

That probably all makes sense, but remember that we weren’t starting from scratch. We already had servers setup that were tracking requests and I needed to ensure that was uninterrupted.

5. Rollout

Brandon and I have been on a tiny classes and services kick of late. What I am about to say may sound heretical, but we’ve felt that we need a few more layers in our apps. We’ve started using Gauges as a test bed for this stuff, while also spending a lot of time reading about clean code and design patterns.

We decided to create a tiny standardization around exposing services and choosing which one gets used in which environment. Brandon took the standardization and moved it into a gem where we could start trying stuff and share it with others. It isn’t much now, but we haven’t needed it to be.

Declaring Services

We created a Registry class for Gauges, which defined the various pieces we would use for Kestrel. It looked something like this:

class Registry
  include Morphine

  register :track_service do
    KestrelTrackService.new(kestrel_client, track_config['queue'])
  end

  register :track_processor do
    KestrelTrackProcessor.new(blocking_kestrel_client, track_config['queue'])
  end
end

We then store an instance of this register in Gauges.app. We probably should have named it Gauges.registry, but we can worry about that later.

At this point, what we did probably seems pointless. The kestrel track service and processor look something like this:

class KestrelTrackService
  def initialize(client, queue)
    @client = client
    @queue  = queue
  end

  def record(attrs)
    @client.set(@queue, MessagePack.pack(attrs))
  end
end

class KestrelTrackProcessor
  def initialize(client, queue)
    @client = client
    @queue = queue
  end

  def run
    loop { process }
  end

  def process
    record @client.get(@queue)
  end

  def record(data)
    Hit.record(MessagePack.unpack(data))
  end
end

The processor uses a blocking kestrel client, which is just a decorator of the vanilla kestrel client. As you can see, all we are doing is wrapping the kestrel-client and making it send the data to the right place.

Using Services

We then used the track_service in our TrackApp like this:

class TrackApp < Sinatra::Base
  get '/track.gif' do
    # stuff
    Gauges.app.track_service.record(track_attrs)
    # more stuff
  end
end

Then, in our track_processor.rb process, we started the processor like so:

Gauges.app.track_processor.run

Like any good programmer, I knew that we couldn’t just push this to production and cross our fingers. Instead, I wanted to roll it out to work like normal, but also push track requests to kestrel. This would allow me to see kestrel receiving jobs.

On top of that, I also wanted to deploy the track processors to pop track requests off. At this point, I didn’t want them to actually process those track requests and write to the database, I just wanted to make sure the whole system was wired up correctly and stuff was flowing through it.

Another important piece was seeing how many track request we could store in memory with Kestrel, based on our configuration, and how it performed when it used up all the allocated memory and started going to disk.

Service Magic

The extra layer around tracking and processing proved to be super helpful. Note that the above examples used the new Kestrel system, but that I wanted to push this out and go through a verification process first. First, to do the verification process, we created a real-time track service:

class RealtimeTrackService
  def record(attrs)
    Hit.record(attrs)
  end
end

This would allow us to change the track_service in the registry to perform as it currently was in production. Now, we have two services that know how to record track requests in a particular way. What I needed next was to use both of these services at the same time so I created a multi track service:

class MultiTrackService
  include Enumerable

  def initialize(*services)
    @services = services
  end

  def record(attrs)
    each { |service| service.record(attrs) }
  end

  def each
    @services.each do |service|
      yield service
    end
  end
end

This multi track services allowed me to record to both services for a single track request. The updated registry looked something like this:

class Registry
  include Morphine

  register :track_service do
    which = track_config.fetch(:service, :realtime)
    send("#{which}_track_service")
  end

  register :multi_track_service do
    MultiTrackService.new(realtime_track_service, kestrel_track_service)
  end

  register :realtime_track_service do
    RealtimeTrackService.new
  end

  register :kestrel_track_service do
    KestrelTrackService.new(kestrel_client, track_config['queue'])
  end
end

Note that now, track_service selects which service to use based on the config. All I had to do was update the config to use “multi” as the track service and we were performing realtime track requests while queueing them in Kestrel at the same time.

The only thing left was to beef up failure around the Kestrel service so that it was limited in how it could affect production. For this, I chose to catch failures, log them, and move on as if they didn’t happen.

class KestrelTrackService

  def initialize(client, queue, options={})
    @client = client
    @queue  = queue
    @logger = options.fetch(:logger, Logger.new(STDOUT))
  end

  def record(attrs)
    begin
      @client.set(@queue, MessagePack.pack(attrs))
    rescue => e
      log_failure(attrs, e)
      :error
    end
  end

  private

  def log_failure(attrs, exception)
    @logger.info "attrs: #{attrs.inspect}  exception: #{exception.inspect}"
  end
end

I also had a lot of instrumentation in the various track services, so that I could verify counts at a later point. These verifications counts would prove whether or not things were working. I left that out as it doesn’t help the article, but you definitely want to verify things when you roll them out.

Now that the track service was ready to go, I needed a way to ensure that messages would flow through the track processors without actually modifying data. I used a similar technique as above. I created a new processor, aptly titled NoopTrackProcessor.

class NoopTrackProcessor < KestrelTrackProcessor
  def record(data)
    # don't actually record
    # instead  just run verification
  end
end

The noop track processor just inherits from the kestrel track processor and overrides the record method to run verification instead of generating reports.

Next, I adjusted the registry to allow flipping the processor that is used based on the config.

class Registry
  include Morphine

  register :track_processor do
    which = track_config.fetch(:processor, :noop)
    send("#{which}_track_processor")
  end

  register :kestrel_track_processor do
    KestrelTrackProcessor.new(blocking_kestrel_client, track_config['queue'])
  end

  register :noop_track_processor do
    NoopTrackProcessor.new(blocking_kestrel_client, track_config['queue'])
  end
end

With those changes in place, I could now set the track service to multi, the track processor to noop, and I was good to deploy. So I did. And it was wonderful.

6. Verification

For the first few hours, I ran the multi track service and turned off the track processors. This created the effect of queueing and never dequeueing. The point was to see how many messages kestrel could hold in memory and how it performed once messages started going to disk.

I used scout realtime to watch things during the evening while enjoying some of my favorite TV shows. A few hours later and almost 530k track requests later, Kestrel hit disk and hummed along like nothing happened.

Now that I had a better handle of Kestrel, I turned the track processors back on. Within a few minutes they had popped all the messages off. Remember, at this point, I was still just noop’ing in the track processors. All reports were still being built in the track request.

I let the multi track service and noop track processors run through the night and by morning, when I checked my graphs, I felt pretty confident. I removed the error suppression from the kestrel service and flipped both track service and track processor to kestrel in the config.

One more deploy and we were queueing all track requests in Kestrel and popping them off in the track processors after which, the reports were updated in the primary database. This meant our track request now performed a single Kestrel set, instead of several queries and updates. As you would expect, response times dropped like a rock.

It is pretty obvious when Kestrel was rolled out as the graph went perfectly flat and dropped to ~4ms response times. BOOM.

You might say, yeah, your track requests are now fast, but your track processors are doing the same work that the app was doing before. You would be correct. Sometimes growing is just about moving slowness into a more manageable place, until you have time to fix it.

This change did not just move slowness to a different place though. It separated tracking and reporting. We can now turn the track processors off, make adjustments to the database, turn them back on, and instantly, they start working through the back log of track requests queued up while the database was down. No tracking data lost.

I only showed you a handful of things that we instrumented to verify things were working. Another key metric for us, since we aim to be as close to realtime as possible, is the amount of time that it takes to go from queued to processing.

Based on the numbers, it takes us around 500ms right now. I believe as long as we keep that number under a second, most people will have no clue that we aren’t doing everything live.

7. Conclusion

By no means are we where I want us to be availability-wise, but at least we are one more step in the right direction. Hopefully this article gives you a better idea how to roll things out into production safely. Layers are good. Whether you are using Rails, Sinatra, or some other language entirely, layer services so that you can easily change them.

Also, we are now a few days in and Kestrel is a beast. Much thanks to Robey for writing it and Twitter for open sourcing it!

Hiring Goto Programmers

Jason and Peter talk about the new release of Rails, the upcoming PATCH verb, and more.

The Mega Ruby News and Release Roundup for February 2012

Oops! I forgot to post the weekly Ruby news updates from Ruby Weekly to Ruby Inside in February so.. here’s a mega roundup of all that was new in the Ruby and Rails worlds in February 2012. I’ll try to keep posting every week from here on – sorry.

Highlights include: a new Ruby 1.9.3 release, REE’s end of life, Spree 1.0, some Rails 4 news, Devise 2.0, a new private gem hosting service.. and that’s just scratching the surface 🙂 Enjoy! (And don’t forget to subscribe to Ruby Weekly if you want to receive something like this every week via e-mail or The Ruby Show if you want it in podcast/audio form.)

Headlines

Ruby 1.9.3-p125 Released
Patchlevel 125 of Ruby 1.9.3 is the latest production release of MRI. It adds LLVM/clang support (ideal for OS X Lion users), GCC 4.7 support, and includes security fixes in the OpenSSL extension.

Ruby Enterprise Edition 1.8.7-2012.02 released; End of Life Imminent
Phusion has unveiled the latest release of REE which is based on Ruby 1.8.7-p358 and RubyGems 1.8.15 and is compatible with XCode 4 and OS X Lion. However, REE is being slowly retired and no Ruby 1.9 version is forthcoming for several reasons.

Full Draft of ‘Rails Tutorial’ 2nd Edition is Now Available
Michael Hartl, author of ‘Rails Tutorial’, has finished a draft of a new Rails 3.2-focused edition of the popular book and Web site. This is a popular way to learn Rails and it’s only getting better!

Spree 1.0 Released (Rails-based E-commerce System)
Spree is almost certainly the most popular, fully featured Rails-based e-commerce system and its creators are proud to announce the release of version 1.0.0. This follows the $1.5M seed funding of Spree’s parent company in October 2011.

RubyMine 4.0 (Ruby IDE) Released
RubyMine is a popular Ruby and Rails IDE by JetBrains (the folks behind IntelliJ IDEA). A focus has been put on improving its performance and UI, but it now also supports all of Rails 3.2 features, including CoffeeScript compilation right from the IDE.

Refinery CMS 2.0.0 Released
Refinery CMS is a Rails based CMS which in its new 2.0.0 incarnation is now fully Rails 3.2 and asset pipeline compliant. This post sums up some of the changes. Congrats to them.

For Rails 4, PATCH Is The New Primary HTTP Method for Updates
Xavier Noria of the Rails core team shares a new development that’s in edge Rails (and due to be released in Rails 4): switching to using HTTP’s ‘PATCH’ verb for making partial updates to resources.

British Government’s New Portal Built on Ruby, Rails and Sinatra
The British Government has launched the beta of ‘gov.uk’, a new attempt to centralize government Web sites. Ruby is not a focus of this article but it’s revealed that Ruby makes up ‘most of the application code’ with a mixture of Rails and Sinatra.

Reading

The History of Rubinius
A beautifully presented magazine article by Evan Phoenix about the background to Rubinius’ creation and its development all the way up to 1.0. Not much tech stuff in here but a truly heartwarming story about the little Ruby implementation that could.

The Joke Is On Us: How Ruby 1.9 Supports the Goto Statement
Pat Shaughnessy is back with a dive into Ruby’s support for the concept of “goto” (a la your favorite pre 1990s programming languages). Did you realize Ruby has a hidden feature to support GOTOs and labels? Neither did I. Wow!

The Pry Ecosystem (the awesome alternative to IRB)
Pry is a popular alternative to IRB and in this post, its creator John Mair looks at several of the additional plugins people have developed for it.

Five Favorite ‘Hidden’ Features in Rails 3.2
Jose Valim shows off five features of Rails 3.2 that he particularly digs. An enjoyable list; I didn’t know of any of these.

How To Build a Naive Bayes Classifier
Want to do spam detection, classification, language detection or similar? Bayes classification may be for you. This post walks through how it works before producing a Ruby implementation.

An Introduction to Ruboto: Ruby + Android
Marc Berszick presents a practical introduction to Ruboto, technology that allows you to run Ruby scripts on Android devices.

A Chat with Nick Quaranto About RubyGems.org Internals
An interesting and extremely visual interview with Nick Quaranto of RubyGems.org about how the site works and how gems are stored and distributed. Lots of interesting stuff in here.

Using Rails 3.2 With Ruby 1.9.3 on Heroku
Daniel Kehoe has updated his Rails on Heroku tutorial to support Rails 3.2 and Ruby 1.9.3. Want to get a new Rails app up and running on Heroku? This is a fine place to start.

37signals: Code Statistics for Basecamp Next
37signals’ Basecamp was the app from which Rails was initially extracted, so it’s interesting to see how 37signals are using Rails to build the next, fully rewritten version. The real value here is in the comments where DHH answers many questions about their stack.

Building A Unix Shell in Ruby
The first article in a series where ‘Working with Unix Processes’ author Jesse Storimer implements a Unix shell in pure Ruby code.

Is Ruby Interpreted or Compiled?
Pat Shaughnessy returns with yet another great article diving into the world of Ruby intepreters and execution. This time he looks at the compilers in Rubinius and JRuby and what they’re producing.

Rails Went Off The Rails: Why I’m Rebuilding Archaeopteryx In CoffeeScript
The never-dull Giles Bowkett explains why he thinks Rails is old and busted and picks quite a few Ruby related scabs along the way. Sure, it’s dramatic, but you might enjoy it.

Ruby Patterns from GitHub’s Codebase
Zach Holman of GitHub presents some things that the developers at GitHub have done to help with the maintainability and reliability of their Ruby apps. Key takeaways are their ‘bootstrap’ script and using TomDoc for documentation.

Let’s Write a Gem
Stephen Ball presents a two part series on building a Ruby library and gem from scratch including best practices like BDD.

Work-Stealing and Recursive Partitioning with Fork/Join in JRuby
The Fork/Join framework in JDK7 implements a clever work-stealing technique for parallel execution. Ilya Grigorik explains what it does and shows off some JRuby-based examples.

Watching & Listening

Ruby Trick Shots: 24 Ruby Tips and Tricks in a Screencast
Over the years I’ve collected 100+ bite-sized Ruby tips and tricks that I’ve seen surprise other Rubyists. I have bigger plans for them but decided to record a video showing off 24 of them. Enjoy these.. Ruby trick shots!

Watch Steve Klabnik Hacking on RubyGems
Steve Klabnik has shared a 50 minute video of him working on a pull request for the RubyGems project. It’s not a focused or narrated video but if watching a master at work sounds interesting to you, check it out.

Ruby Freelancers — A New Podcast on the Business of Ruby Dev
Ruby Freelancers is a new podcast focusing on the art of being a freelance Ruby or Rails developer. Things kick off with Charles Max Wood, Eric Davis, Evan Light, and Jeff Schoolcraft at the helm.

Play by Play with Aaron ‘Tenderlove’ Patterson
PeepCode has released the latest in its ‘play by play’ series with a two hour peep over the shoulder of popular Rubyist Aaron Patterson’s shoulder. It costs a little money but it has gotten an excellent reaction so far.

Twitter Bootstrap Basics by RailsCasts
If you’re not yet sick of hearing about Twitter Bootstrap or want to learn how to work with it using Rails, Ryan Bates’ latest screencast is a great place to start.

Code

Devise 2.0 Released: The Flexible Auth System Grows Up
Devise 2.0 is here but it’s aimed at deprecations rather than new features. Nonetheless, it adds Rails 3.2 support and supports e-mail reconfirmation. Beware though, it now only supports Rails 3.1 and above.

Sidekiq: Efficient Resque-compatible Message Processing for Rails 3
Got lots of Resque processes running at once? Sidekiq offers ‘simple, efficient message processing’ for Rails 3 applications and due to its actor-based concurrency, a single Sidekiq process could do the work of many Resque ones.

Virtus 0.2.0: Attributes on Steroids for Plain Old Ruby Objects
Virtus is a partial extraction of the DataMapper Property API with which you can extend your Ruby objects with attributes that require data type coercions. Handier than it sounds; see the README examples.

AlterEgo: Hosted Two Factor Authentication for your Web App
A gem to access AlterEgo, a two factor authentication service built by MailChimp, the folks I use to send Ruby Weekly each week 🙂

Prickle: A Simple DSL Extending Capybara
Capybara has become the de facto Ruby acceptance test framework for web apps and Prickle takes things a step further with extra methods for finding elements, performing actions, and more.

Newman: A Microframework for Email-based Apps
Still in a highly experimental state, Newman is a micro-framework aiming to do for email-based applications what Rack and Sinatra have done for web apps. Gregory Brown leads the project.

Heads Up: A Rails and JS Powered Desktop Heads Up Display
Heads Up is a simple HUD showing your calendar items for today, your unread emails and custom notes. The interesting part is it’s an OS X app, yet it’s built on Rails, JavaScript (using Spine), Rack::Offline and MacGap.

A Patched Ruby 1.9.3-p0 for A 30% Faster Rails Boot
A somewhat adhoc set of patches to boost the performance of Ruby 1.9.3-p0 have been floating around on GitHub this week. Tread with caution but it may appeal to you. Hopefully the ideas will make their way into MRI proper.

Plymouth: Start Pry in the Context of a Failed Test
Pry is an awesome alternative to the IRB Ruby console and Plymouth will automatically bring up a Pry session when a test fails in your suite, putting you right into the context of the failure. Supports Bacon, Minitest and RSpec.

SimpleForm 2.0: Rails Form Creator, Now With Twitter Bootstrap Support
SimpleForm is a popular DSL for creating forms in Rails apps and version 2 is now out. It includes a new wrapper API to create custom input stacks and also includes some integration features for Twitter Bootstrap.

UnixUtils: Like FileUtils But For Using Unix Tools
Like FileUtils, but provides access to zip, unzip, bzip2, bunzip2, tar, untar, sed, du, md5sum, shasum, cut, head, tail, wc, curl, etc. The README explains why you wouldn’t want to just spawn these yourself.

Ruby Jobs of the Month

Miscellaneous

Ruby Reloaded #5 Coming in March, Sign Up to be Notified
No big announcement yet but you can now sign up to be notified when Ruby Reloaded 5 is taking place (sometime in March) as well as get a discount. If you don’t know what Ruby Reloaded even is (it’s an online course I run), you can read about that too! 🙂

Flowstone 2: A Ruby-based Graphical Programming Environment
Flowstone is a commercial Ruby-based graphical development environment aimed at the robotics and device control field. Certainly looks interesting from the screenshots and a restricted free version is available to download (Windows only).

Ruby and Rails Documentation Apps for the iPhone and iPad
Pandalab has put together two free, reasonable looking iPad and iPhone apps for looking up Ruby 1.9 and Rails 3 documentation (note: they offer an in app purchase to remove their ad). If only I’d bought my iPad into the office today to try them out..

Gemfury: A Private Gem Hosting Service in the Cloud
Gemfury is a hosted service for private and custom gems. Upload your gems, enable Gemfury as a source, and you can securely deploy any gem to any host.

Events