Heroku Postgres – GIS Support Now Available

Today we're announcing geospatial support for Heroku Postgres with PostGIS 2.0.

Heroku Postgres is increasingly enabling rich use cases – adding services from key/value datatype in hstore, querying across postgres databases with dblink, and now adding rich geospatial functionality. Adding PostGIS within your Postgres database reduces the number of services you need to add to your stack, reducing complexity and allowing you to build in location-based functionality into apps faster.

Get started integrating location into your apps today by provisioning your Heroku Postgres database and exploring the functionality of PostGIS 2.0 or read more about it on the Heroku Postgres Blog.

RailsConf 2013

Developers are worthy of great experiences and at Heroku we aim to help improve this. Whether its making it easier to prepare your application for production on Heroku, not having to worry about security updates in your database, or getting notified of the latest rails vulnerability we want to make the world better for developers. This extends beyond the Heroku platform as well. For instance, our Ruby Task Force contributes back to projects like Ruby on Rails, Bundler, Code Triage, and Rails Girls.

Likewise, we aim to do everything we can to make your RailsConf experience better. Whether it is relaxing with some sake or getting performance advice for your application, we have a slew of activities planned for the conference:

  • Performance Bar – Over the last month, we have been helping customers with Bamboo to Cedar migrations, visibility tooling, and scaling. We are bringing this white glove service to RailsConf. Heroku Engineers will be at our booth to help with your PostgreSQL database, better introspection, and any other issue to help improve your app for deployment. The Performance Bar will be open during Expo Hall hours on Tuesday and Wednesday at our booth. Feel free to walk up and get one-on-one help from a Heroku Engineer or preschedule an appointment by emailing tammy@heroku.com.
  • Performance Workshop – We're partnering with our good friends, JumpStart Lab, to run a free Performance Workshop on Wednesday evening at 6:30pm in room D136 at the Portland Convention Center. Get hands on performance tips and then test your knowledge afterwards in an application performance tuning contest. We'll hand out prizes for the fastest times. It's free to sign up! If you can't make it to RailsConf, you can check out the livestream.
  • Sake Happy Hour – On Tuesday and Wednesday from 3-4 pm we'll be hosting a Sake Happy Hour at our booth complete with a limited supply of Heroku shot glasses each day. Come celebrate and talk shop with Heroku Engineers.
  • Sessions – There will be two talk sessions given by Heroku Engineers.

We're looking forward to being a part of your RailsConf experience. If you have feedback on how we can improve further please reach out to us.

Introducing Production Check

Entering production is a key transition in your app’s lifecycle; it signals that your app will be delivering value to end users. You are no longer optimizing for testing—you are optimizing for performance and reliability, and there are new factors to consider at this stage.

Production Check

Today we’re announcing Production Check, an enhancement to the Heroku Dashboard that helps ensure that your app is ready to go to production. Production Check tests your app’s configuration against a set of optional—but highly recommended—criteria. It makes it easy to ensure that your app’s configuration lends itself to maximum uptime. Moreover, it ensures that you have tools available for understanding and monitoring the factors that contribute to uptime.

How to Check Your App

To run Production Check, click the “Run Production Check” link in the header for any app in the Heroku Dashboard.

Heroku Production Check

Production Check will run a series of tests on your app that we recommend for maintaining and monitoring availability. Each check includes useful links to the Heroku Dev
and other related resources. Let’s look at a few of the areas it focuses on:

DNS Configuration

Previously, it was recommended to point your app's DNS directly to Heroku's three provided IP addresses. Now, the Cedar stack allows you to point your app to app-name.herokuapp.com or a *.herokussl.com domain rather than a legacy IP address. This ensures that in the event of an infrastructure-level issue, core components can be replaced without requiring you to make changes to your apps.

Dyno Redundancy

Although an app may not require the resources of more than a single dyno, we recommend running at least 2 dynos for a production app. One reason for this is that dynos restart roughly every 24 hours, and dyno redundancy helps ensure that you don't experience excessive queueing or dropped requests while a single dyno restarts. Additionally, dynos and the processes running on them can occasionally crash, and having another dyno helps ensure that you don't have downtime while that dyno recovers. Finally, single web dynos sleep after an hour of
, and having a second dyno means that all dynos will stay awake.

App Monitoring

Inevitably, performance issues and downtime occur. It’s important to have tools available that monitor how often and why it happens to your app. New Relic is a great tool for getting immediate, specific visibility into your app’s performance and availability. Not sure why a specific action takes so long in your app? New Relic can help you identify the specific line of code responsible for
the slowness.

Log Monitoring

While New Relic gives you visibility into factors that contribute to downtime and performance issues, it’s important to have log-monitoring tools like Papertrail to monitor for events more specific to your app—for example, you might want to be alerted if there’s a sudden spike in failed credit card authorizations. While Heroku’s log for your app only goes back 1500 lines (which for a large production site may only represent seconds of activity), these add-ons also allow you to search days or weeks back through your app’s logs, allowing you to understand the context in which an error occurred.

Let Us Know What You Think

Heroku defines a “production app” as an app with 2 or more dynos and a production-tier Postgres database (if one is present). While far from exhaustive, the tests Production Check provides give additional guidance in getting you and your app prepared to handle the challenges your app will face as it moves into production.

With Production Check, you can save time in getting apps ready for launch day while learning best practices on Heroku. Go to the Heroku Dashboard now and give it a try.

If you have questions or comments about Production Check, email us at dashboard-feedback@heroku.com.

Introducing the Europe Region, Now Available in Public Beta

Today we’re happy to announce Heroku’s Europe region, available in public beta. With more than 3 million apps running on our platform from developers all over the globe, it's not surprising that we've had high demand for Heroku in more regions of the world. After collaborating closely with customers during private beta, we're now ready to offer Heroku services in Europe to all customers as part of a public beta. The Europe region runs Heroku applications from datacenters located in Europe, offering improved performance for users in that region.

One Heroku, Two Continents

The Europe region offers all the features of the existing US region. Both regions run apps on infrastructure dedicated to each region, but are managed by the same unified Heroku interface you already know. This provides powerful global app management with isolated, geolocated infrastructure:

Faster Apps

The physical proximity of the Europe region to European end-users means reduced latency, often resulting in a dramatic improvement in app responsiveness to those users. We’ve observed performance improvements of 100ms per request or more for European end-users:

Lower Latency in Europe

A Familiar Workflow

All Heroku users can now create and deploy apps to the Europe region:

$ heroku create --region eu
$ git push heroku master

Once the app is created, you can interact with it just like any other Heroku app.

Easy App Migration with Heroku Fork

To ease the process of migrating existing applications to the Europe region, we created heroku fork, a new addition to the Heroku CLI. Heroku fork copies an app’s Heroku Postgres data and config vars, and re-provisions all its add-ons. If you have an existing app you’d like to run in the Europe region, check that you’ve got the most recent Heroku toolbelt installed and use Heroku fork to create a running copy of your app in the new region:

$ heroku fork --region eu
Creating fork myapp-332... done
Copying slug... done
Adding newrelic:professional... done
Copying config vars... done
Fork complete, view it at http://myapp-332.herokuapp.com/

Note: heroku fork will not move any domains or scale your app past a single dyno, so you're free to decide when your app will be made available to your customers. For more information about this powerful new feature, see the Dev Center article on heroku fork.


More than 60 add-ons are currently available in the Europe region, with more on the way. To ensure that your app has fast access to its data wherever it’s deployed, Heroku automatically provisions latency-sensitive add-ons in your app’s region. Many add-ons are already available to apps in the Europe region, including:

  1. Heroku Postgres

  2. MongoLab

  3. Memcachier

  4. openredis

  5. New Relic

  6. Websolr

  7. Scheduler

  8. Deploy Hooks

  9. SendGrid

  10. Airbrake

  11. Papertrail

  12. ClearDB

To discover which add-ons are available in the Europe region, search for 'europe' on the addons homepage or use the new --region flag in the heroku CLI:

$ heroku update
$ heroku addons:list --region eu

Customer Success

Several Heroku customers have already deployed their production Heroku apps to the Europe region for improved performance. For example, top Swedish television network TV4 is currently running its video on demand service TV4 Play out of Heroku’s Europe region. TV4 CTO Per Åström says:

Deploying our app closer to our users in Heroku's Europe region gave us a 150ms improvement in web performance. Based on this win for our users, we’re moving all of our apps to the Europe region.

Digital agencies and other customers delivering user-facing mobile and social apps have also benefited from improved performance in the Europe region. For example, app development company Betapond COO Conor Ryan says:

Deploying our Facebook apps live to the Heroku cloud in the EU was pain-free and as easy as staging. We're delighted to see an average of 11% improvement in page load times since making the switch. Consumers don't tolerate waiting, particularly on mobile devices, so Heroku's EU contribution to a snappy user experience is a big boost for Betapond and our customers.

Safe Harbor Compliance is Coming Soon

Heroku is not yet a registered participant in the Safe Harbor program. We’ve laid the groundwork for becoming Safe Harbor certified and expect to have it soon. The Europe region public beta is designed to let you build high-performance apps for European users. It does not currently address data residency or jurisdiction concerns. You should assume that some portions of your app and its data will be in, or pass through,
datacenters located in the US.

We Want Your Feedback

If you have comments or questions about the Heroku Europe region, email us at eu-beta@heroku.com.
To receive email updates about the Heroku Europe region as it progresses to general availability, subscribe to the beta mailing list below.

Additional Resources

Ruby Programming 42nd Batch: Registrations now open

Send to Kindle

Registrations are now open for RubyLearning’s popular Ruby programming course. This is an intensive, online course for beginners that helps you get started with Ruby programming.

Course Fee

  • US$ 49.95 for the first 5 participants
  • US$ 63.95 for the next 10 participants
  • US$ 79.95 after the first 15 participants

Please pay by clicking on the PayPal button Paypal

Download ‘Advice for Ruby Beginners’ as a .zip file.

Here is what Demetris Demetriou, a participant who just graduated, has to say – “When I joined this course I was sceptical about how useful this course would be for me instead of reading material and watching videos on YouTube and thus saving money. After the course started I realised how valuable this course was. In the past I had read many Ruby books over and over, but never got into really getting practical with it and never had confidence in it. Lots of theory but couldn’t use it. I feel that the exercises in this course and the support, monitoring from our mentor Victor, made the huge difference that all books in the past didn’t. It wasn’t about reading lots of books, but simply few things and get practical and understand them well. I feel I learnt a lot and I’m coming back for more to rubylearning.org Thanks a lot Victor and Satish and all the other Rubyists who gave us today’s Ruby.”

What’s Ruby?


According to http://www.ruby-lang.org/en/ – “Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. Ruby’s elegant syntax is natural to read and easy to write.”

Yukihiro Matsumoto, the creator of Ruby, in an interview says –

I believe people want to express themselves when they program. They don’t want to fight with the language. Programming languages must feel natural to programmers. I tried to make people enjoy programming and concentrate on the fun and creative part of programming when they use Ruby.

What Will I Learn?

In the Ruby programming course, you will learn the essential features of Ruby that you will end up using every day. You will also be introduced to Git, GitHub, HTTP concepts, RubyGems, Rack and Heroku.

Depending on participation levels, we throw a Ruby coding challenge in the mix, right for the level we are at. We have been known to give out a prize or two for the ‘best’ solution.

Who’s It For?

A beginner with some knowledge of programming.

You can read what past participants have to say about the course.


Satish Talim, Michael Kohl, Satoshi Asakawa, Victor Goff III and others from the RubyLearning team.


The course starts on Saturday, 18th May 2013 and runs for seven weeks.

RubyLearning’s IRC Channel

Most of the mentors and students hang out at RubyLearning’s IRC (irc.freenode.net) channel (#rubylearning.org) for both technical and non-technical discussions. Everyone benefits with the active discussions on Ruby with the mentors.

How do I register and pay the course fees?

  • The course is based on the The Ultimate Guide to Ruby Programming eBook. This book is priced at US$ 9.95
  • You can pay either by Paypal or send cash via Western Union Money Transfer or by bank transfer (if you are in India). The fees collected helps RubyLearning maintain the site, this Ruby course, the Ruby eBook, and provide quality content to you.

The Course Fee is:

  • US$ 49.95 for the first 5 participants
  • US$ 63.95 for the next 10 participants
  • US$ 79.95 after the first 15 participants

Please pay by clicking on the PayPal button Paypal

At the end of this course you should have all the knowledge to explore the wonderful world of Ruby on your own.

Here are some details on how the course works:


Once the course starts, you can login and start with the lessons any day and time and post your queries in the forum under the relevant lessons. Someone shall always be there to answer them. Just to set the expectations correctly, there is no real-time ‘webcasting’.


  • The Mentors shall give you URL’s of pages and sometimes some extra notes; you need to read through. Read the pre-class reading material at a convenient time of your choice – the dates mentioned are just for your guideline. While reading, please make a note of all your doubts, queries, questions, clarifications, comments about the lesson and after you have completed all the pages, post these on the forum under the relevant lesson. There could be some questions that relate to something that has not been mentioned or discussed by the mentors thus far; you could post the same too. Please remember that with every post, do mention the operating system of your computer.
  • The mentor shall highlight the important points that you need to remember for that day’s session.
  • There could be exercises every day. Please do them.
  • Participate in the forum for asking and answering questions or starting discussions. Share knowledge, and exchange ideas among yourselves during the course period. Participants are strongly encouraged to post technical questions, interesting articles, tools, sample programs or anything that is relevant to the class / lesson. Please do not post a simple "Thank you" note or "Hello" message to the forum. Please be aware that these messages are considered noises by people subscribed to the forum.

Outline of Work Expectations:

  1. Most of the days, you will have exercises to solve. These are there to help you assimilate whatever you have learned till then.
  2. Some days may have some extra assignments / food for thought articles / programs
  3. Above all, do take part in the relevant forums. Past participants will confirm that they learned the best by active participation.

Some Commonly Asked Questions

  • Qs. Is there any specific time when I need to be online?
    Ans. No. You need not be online at a specific time of the day.
  • Qs. Is it important for me to take part in the course forums?
    Ans. YES. You must Participate in the forum(s) for asking and answering questions or starting discussions. Share knowledge, and exchange ideas among yourselves (participants) during the course period. Participants are strongly encouraged to post technical questions, interesting articles, tools, sample programs or anything that is relevant to the class / lesson. Past participants will confirm that they learned the best by active participation.
  • Qs. How much time do I need to spend online for a course, in a day?
    Ans. This will vary from person to person. All depends upon your comfort level and the amount of time you want to spend on a particular lesson or task.
  • Qs. Is there any specific set time for feedback (e.g., any mentor responds to me within 24 hours?)
    Ans. Normally somebody should answer your query / question within 24 hours.
  • Qs. What happens if nobody answers my questions / queries?
    Ans. Normally, that will not happen. In case you feel that your question / query is not answered, then please post the same in the thread – “Any UnAnswered Questions / Queries”.
  • Qs. What happens to the class (or forums) after a course is over? Can you keep it open for a few more days so that students can complete and discuss too?
    Ans. The course and its forum is open for a month after the last day of the course.

Remember, the idea is to have fun learning Ruby.

Technorati Tags: , , ,

(Powered by LaunchBit)

Expanded HTTP Method Support

HTTP and its secure variant, HTTPS, are the protocols used by every web
client to talk to web applications. A key part of the protocol is the HTTP
. Traditionally, web applications used a very limited set of HTTP
methods. It is common for application servers, proxies and routers (such as the
Heroku HTTP router) to prevent unknown methods from being used. This unnecessary
constraint of the Heroku HTTP router has increasingly become a limitation to
developers building modern web applications.

In the past, if you tried to use an unsupported HTTP method in your Heroku
application, clients would receive a 405 METHOD_NOT_ALLOWED error.

As of today, that's no longer the case. The Heroku routers now accept any HTTP
, allowing you to use newer methods that have recently
gained adoption, such as PATCH.

Why is this important?

Many new web frameworks use recently introduced HTTP methods such as PATCH or
even custom methods that are not yet standardized. Ruby on Rails is one such
framework. Rails has constantly evolved since its birth to always incorporate
the latest development techniques. One such evolution is happening in Rails
where the framework uses the PATCH HTTP method to perform
partial updates of resources

Before Heroku removed restrictions on HTTP method, it was not possible to
leverage this new functionality in Rails 4. Now that any method is supported,
you can take full advantage of PATCH support in Rails 4.


For the curious, a little more background is in order.

HTTP is extensible

A huge factor in HTTP's popularity has been its extensibility. HTTP is a simple
protocol that can be used for many different purposes. For instance, the
HTTP/1.1 Spec defines the following set of methods: OPTIONS, GET,

But the spec also states that new methods can be added. In fact, a number of
other RFCs have standardized additional HTTP methods, often for specific
applications such as version control or calendaring. Yet in most cases,
regardless of the underlying application, developers can leverage the same
supporting infrastructure: load balancers, proxies, caching proxies; all
designed for HTTP.


A recent extension to HTTP is RFC5789, which introduced the PATCH
method. This method is particularly useful for web application developers, as it
standardizes a common way to make partial updates to resources. Rails 4 and many
other libraries now make use of it.

We can quickly demonstrate how it is used by deploying a basic Sinatra app to
Heroku using the quick start guide. Replace the code in
app.rb with:

require 'sinatra'
require 'json'

class App < Sinatra::Base

  # fake persistent data.
  DATA = { "a" => 1, "b" => 2, "c" => 3 }

  put '/data' do
    # put replaces the existing resource so we simply return the new resource
    new_data = JSON.parse(request.body.read)

  patch '/data' do
    # patch 'patches' the existing resource
    new_data = DATA.merge(JSON.parse(request.body.read))


Once your app is deployed, perform a PATCH and PUT request with curl:

$ curl -X PATCH http://shiny-object-1234.herokuapp.com/data -d '{"a": 2, "b": 3}'
$ curl -X PUT http://shiny-object-1234.herokuapp.com/data -d '{"a": 2, "b": 3}'

The PATCH logic merges the new values into the existing data structure and
returns the whole data structure, whereas PUT simply replaces the old data
structure with a new one. While this example doesn't persist the changes, it
demonstrates the semantic difference between these two HTTP methods.

Other HTTP methods

PATCH is just one example of how a new HTTP method was defined, standardized
and successfully achieved widespread use. Because Heroku no longer imposes any
restrictions on HTTP methods, future extensions will work seamlessly without
requiring explicit support. This also means that the Heroku platform can be used
as a proving ground for new, non-standardized HTTP methods.

Why restrict HTTP methods in the first place?

The original motivation for application servers, proxies and HTTP routers to
restrict HTTP methods was that these intermediaries often performed more
functions than just passing through requests. For example, if an intermediary
wants to retry a failed request, it needs to understand whether a retry is safe.
POST and PATCH requests are generally not safe to retry while GET, PUT, and

Heroku's routers pass requests straight through to the application, without
special handling based on the particular HTTP method; the routers therefore
handle all requests the same, regardless of which method was used. This results
in maximum flexibility for the application developer to use any HTTP method and
fully control the semantics of each request.

For more on HTTP routing visit our Dev Center article.

A FREE Git and GitHub Course – 8th batch

Send to Kindle

A FREE Git and GitHub Course – 8th batch

Registrations are now open for RubyLearning’s popular Git and GitHub course. This is an introductory but an intensive, online course for beginners. Here you’ll learn the essential features of both Git and GitHub that you’ll end up using every day.

What’s Git and GitHub?

Git is an open source version control system designed to handle very large projects with speed and efficiency, but just as well suited for small personal repositories (collection of resources that can be accessed to retrieve information); it is especially popular in the open source community. With GitHub you can host your public and private projects there and use it to collaborate on projects.

Ruby Master, Josh Susser1 in an interview with RubyLearning said:

First off, get an account on GitHub. (If you haven’t learned git yet, get going on that too – it’s the future for open source SCM.) Then do some exploring around the various projects and see where you think you can jump in and contribute something. Pick a project that is currently under development and has an active community. You’ll have enough going on that you don’t want to be championing a project just yet.

Get involved in the Ruby community. Join GitHub

What Will I Learn?

The course topics in brief are:

  • What’s Version Control
  • What’s Git
  • Downloading and Installing Git
  • Create your SSH Key
  • Introduce yourself to Git
  • Add some additional Git settings
  • What’s GitHub?
  • Set up your GitHub account
  • Follow a Friend
  • Watch projects
  • Creating a new repository
  • Deleting and renaming repositories
  • Fork a repository
  • Push changes to a repository
  • Clone a public project
  • Add collaborators to a project
  • Collaborate with other users
  • Send a pull request
  • Merge changes from a pull request
  • Use project wikis
  • Create and delete branches and tags
  • Create GitHub pages
  • Gist
  • Exercises

Who’s It For?

The Git and GitHub course is a starting point for people new to Git and GitHub, to learn it as quickly and easily as possible.


The course starts on 4th May 2013 and runs for a week.

The first seven batches were a run-away success. So hurry, registrations have started.

Is the course really free?

A lot of effort and time goes into building such a course and we would really love that you pay at least US$ 5 for the course. Since this is a “Pay if you Like” course, you are under no obligation to pay anything at all and hence the course would be free for you. For those who contribute US$ 5 or more, we shall email them a copy of the book (.pdf) “Using Git & GitHub eBook” – the course is based on this book.

How do I register?

  • For the “Free” participants, register on the RubyLearning.org site and send us your name and registered email address while creating an account at RubyLearning.org to satish [at] rubylearning [dot] org We will enroll you into the course which normally takes place within 48 hours.
  • For those who want to contribute, you can pay either by Paypal or send cash via Western Union Money Transfer or by bank transfer (if you are in India). The fees collected helps RubyLearning maintain the site, the various courses and provide quality content to you.
  • Once you pay the fees below, register on the RubyLearning.org site and send us your name and registered email address while creating an account at RubyLearning.org to satish [at] rubylearning [dot] org We will enroll you into the course which normally takes place within 48 hours.

After you click the “Pay Now” button below, you will be taken to a webpage as shown below.

Please enter the amount you want to pay for the course in the “Item price” field and click on the “Update” link. You can safely pay via PayPal.

You can now pay the Course Fees by clicking on the “Pay Now” button below.

Technorati Tags: ,

(Powered by LaunchBit)

Status Update

It’s been well over a month since the last official update to Master Space and Time with JavaScript, and since I was hoping to get updates out more-or-less weekly, it’s probably a good idea to check in and let you know what’s going on. (Could be worse, though, I’m still hoping to post my top 10 books I read list. From 2011.)

All of the MSTWJS customers out there have been either very patient in waiting for the next update, or you are completely disengaged. Personally, I’m choosing to assume patient.

I am continuing to work on the book, it’s just slowed down quite a bit. The next update will most likely be sometime after RailsConf — ideally sometime in the week or so following. So by, say, May 10th. I post this date publicly to increase the chance that I’ll actually hit it.

There are a few reasons why the book’s progress has slowed.

To some extent, it’s a deliberate slowdown so I don’t have to rewrite the thing a dozen times. Yes, Ember is in an API freeze, but they are still adding new things that preserve compatibility, and Ember-data is decidedly not in an API freeze. Just in the last week or so, integration testing tools are starting to emerge — see this discussion for details. Honestly, the fact that I had to throw my hands up over integration testing in the last update was very disappointing, and I’d very much like to get that working in the example.

There’s also been some fumbling about what I want to cover in the rest of the book and how I want to get there. This one, I think I’m getting a handle on.

I also got busy. For example, I’ve been starting this weekly sort-of-screecast series for Table XI called XI to eye. I’ve never done anything like it, but I’m pleased with how this is going. There are five so far, you can see them all at http://www.tablexi.com/blog/category/xi-to-eye. Please do check them out.

I’m also doing two sessions at RailsConf — a normal session comparing rich client MVC with the 37Signals Basecamp approach and an intro track session on testing complex systems. More on those next week, but I’m excited for both of them.

There’s also been some laziness, and some lack of momentum caused by the combination of the previous points.

Still, hoping that this will move a little more rapidly in May — we’re now coming on the two-year anniversary of me starting the project (though I suppose I’d be done if I hadn’t decided to pick up Ember as a topic), and I’m certainly ready to move on. (I have some idea what my upcoming writing projects will be, just not sure which one I want to tackle next.

Thanks for your patience, if you are enjoying the book more will be coming — there will also be another errata catch up on the first three books. Please do help spread the word, or maybe buy it yourself.

Let Nunes Do It

In a moment of either genius or delirium I decided to name my newest project after myself. Why? Well, here is the story whether you want to know or not.

Why Nunes?

Naming is always the hardest part of a project. Originally, it was named Railsd. The idea of the gem is automatically subscribe to all of the valuable Rails instrumentation events and send them to statsd in a sane way, thus Railsd was born.

After working on it a bit, I realized that the project was just an easy way to send Rails instrumentation events to any service that supports counters and timers. With a few tweaks, I made Railsd support InstrumentalApp, a favorite service of mine, in addition to Statsd.

Thus came the dilemma. No longer did the (already terrible) name Railsd make sense. As I sat and thought about what to name it, I remembered joking one time about naming a project after myself, so that every time anyone used it they had no choice but to think about me. Thus Nunes was born.

Lest you think that I just wanted to name it Nunes only so that you think of me, here is a bit more detail. Personally, I attempt to instrument everything I can. Be it code, the steps I take, or the calories I consume, I want to know what is going on. I have also noticed that which is automatically instrumented is the easiest to instrument.

I love tracking data so deeply that I want to instrument your code. Really, I do. I want to clone your repo, inject a whole bunch of instrumentation and deploy it to production, so you can know exactly what is going on. I want to sit over your shoulder and look at the graphs with you. Ooooooh, aren’t those some pretty graphs!

But I don’t work for you, or with you, so that would be weird.

Instead, I give you Nunes. I give you Nunes as a reminder that I want to instrument everything and you should too. I give you Nunes so that instrumenting is so easy that you will feel foolish not using it, at least a start. Go ahead, the first metric is free! Yep, I want you to have that first hit and get addicted, like me.

Using Nunes

I love instrumenting things. Nunes loves instrumenting things. To get started, just add Nunes to your gemfile:

# be sure to think of me when you do :)
gem "nunes"

Once you have nunes in your bundle (be sure to think of bundling me up with a big hug), you just need to tell nunes to subscribe to all the fancy events and provide him with somewhere to send all the glorious metrics:

# yep, think of me here too
require 'nunes'

# for statsd
statsd = Statsd.new(...)
Nunes.subscribe(statsd) # ooh, ooh, think of me!

# for instrumental
I = Instrument::Agent.new(...)
Nunes.subscribe(I) # one moooore tiiiime!

With just those couple of lines, you get a whole lot of goodness. Out of the box, Nunes will subscribe to the following Rails instrumentation events:

  • process_action.action_controller
  • render_template.action_view
  • render_partial.action_view
  • deliver.action_mailer
  • receive.action_mailer
  • sql.active_record
  • cache_read.active_support
  • cache_generate.active_support
  • cache_fetch_hit.active_support
  • cache_write.active_support
  • cache_delete.active_support
  • cache_exist?.active_support

Thanks to all the wonderful information those events provide, you will instantly get some of these counter metrics:

  • action_controller.status.200
  • action_controller.format.html
  • action_controller.exception.RuntimeError – where RuntimeError is the class of any exceptions that occur while processing a controller’s action.
  • active_support.cache_hit
  • active_support.cache_miss

And these timer metrics:

  • action_controller.runtime
  • action_controller.view_runtime
  • action_controller.db_runtime
  • action_controller.posts.index.runtime – where posts is the controller and index is the action
  • action_view.app.views.posts.index.html.erb – where app.views.posts.index.html.erb is the path of the view file
  • action_view.app.views.posts._post.html.erb – I can even do partials! woot woot!
  • action_mailer.deliver.post_mailer – where post_mailer is the name of the mailer
  • action_mailer.receive.post_mailer – where post_mailer is the name of the mailer
  • active_record.sql
  • active_record.sql.select – also supported are insert, update, delete, transaction_begin and transaction_commit
  • active_support.cache_read
  • active_support.cache_generate
  • active_support.cache_fetch
  • active_support.cache_fetch_hit
  • active_support.cache_write
  • active_support.cache_delete
  • active_support.cache_exist

But Wait, There is More!

In addition to doing all that work for you out of the box, Nunes will also help you wrap your own code with instrumentation. I know, I know, sounds too good to be true.

class User < ActiveRecord::Base
  extend Nunes::Instrumentable # OH HAI IT IS ME, NUNES

  # wrap save and instrument the timing of it
  instrument_method_time :save

This will instrument the timing of the User instance method save. What that means is when you do this:

# the nerve of me to name a user nunes
user = User.new(name: "NUNES!")

An event named instrument_method_time.nunes will be generated, which in turn is subscribed to and sent to whatever you used to send instrumentation to (statsd, instrumental, etc.). The metric name will default to “class.method”. For the example above, the metric name would be user.save. No fear, you can customize this.

class User < ActiveRecord::Base
  extend Nunes::Instrumentable # never

  # wrap save and instrument the timing of it
  instrument_method_time :save, 'crazy_town.save'

Passing a string as the second argument sets the name of the metric. You can also customize the name using a Hash as the second argument.

class User < ActiveRecord::Base
  extend Nunes::Instrumentable # gonna

  # wrap save and instrument the timing of it
  instrument_method_time :save, name: 'crazy_town.save'

In addition to name, you can also pass a payload that will get sent along with the generated event.

class User < ActiveRecord::Base
  extend Nunes::Instrumentable # give nunes up

  # wrap save and instrument the timing of it
  instrument_method_time :save, payload: {pay: "loading"}

If you subscribe to the event on your own, say to log some things, you’ll get a key named :pay with a value of "loading" in the event’s payload. Pretty neat, eh?


I hope you find Nunes useful and that each time you use it, you think of me and how much I want to instrument your code for you, but am not able to. Go forth and instrument!

P.S. If you have ideas for Nunes, create an issue and start some chatter. Let’s make Nunes even better!

Heroku Postgres – Version 9.2 now Default

Over a year ago we began working with the community as to how we could help to make Postgres better. Much of this came to fruition with PostgreSQL version 9.2 and three months ago we released support of Postgres 9.2 into GA. PostgreSQL version 9.2 is now the new default when provisioning a Heroku Postgres database.

You can read more about the powerful features in this version over on the Heroku Postgres blog.

Empowering Change: Programming Literacy for All

There has never been a better time to be a programmer. Every day more and more gadgets get connected or over-clocked. Programming is so prevalent that it often goes unnoticed in our daily lives. Whether we're scripting out social presence with IFTTT, or doing taxes with Excel, automation and programming has become an inescapable part of the modern world.

Heroku believes that to invest in our future, we must invest in programming literacy. While we're waiting for recursion to be a staple in our children’s classrooms, we can work on continuing and higher education today.

Heroku engineers are given opportunities and encouragement to be part of this movement. They’ve done so through supporting and participating in a number of groups including Hungry academy, Rails Girls, PyLadies, and more.

As a Heroku engineer I had a recent opportunity to teach a class in Ruby on Rails at the University of Texas in Austin. While nothing beats an in-classroom experience, it's not modular or scalable. In an effort to further scale programming literacy, we’ve been working to make this content available for everyone. After many re-takes, re-writes and hours of editing, we are happy to provide you with over 40 hours of video, lectures, exercises and quizes for free: Heroku Presents: UT on Rails.

The course will take a brand new developer up through the ranks, until they can build and deploy a fully functional website. If you or someone you know is interested in learning web programming, it's a great opportunity.

Staying Connected

At Heroku, we are encouraged to get involved with our respective communities. While we dogfood our product on a daily basis, there is no substitute for seeing how other developers actually work with it.

Developers don't have to teach a full length course to get involved with their communities. We've had developers help out at a number of community sponsored events that focus on getting more people engaged with technology. One of them, Rails Girls, has been so successful that we've had engineers participate at events in over 6 different countries.

Terence Lee at Rails Girls Amsterdam, photos by: Konstantin Haase

By staying connected, we help a new generation of programmers while making our own products better. It's a win-win situation.

Feedback Cycles

Learning is a feedback loop. You take an action, see the result, learn a lesson. The smaller the loop, the less time from action to result: the quicker you learn, the faster you advance. For students, using Heroku means they can spend less time worrying about their production environments and more focusing on their application logic.

Millions of application developers choose Heroku to get their features to market as quickly as possible. It's that same speed of delivery and deployment that also makes Heroku a natural choice for beginners.

The Next Generation

As we move further into an ever-connected society, developers and companies must do their part to promote programming and technology in the classroom. While the demand for tech jobs seems to be constantly increasing, where will tomorrow's senior developers come from? Perhaps they will come from programs like Let’s Learn Python at this years PyCon or through intensive focus courses like those offered by gSchool, or perhaps from general education courses. Wherever the next generation of programmers comes from, we want to set them up to succeed.

Heroku has always had a free tier which is perfect for prototyping and learning. Putting the right tools in a student’s hands can empower them to succeed. If you are picking up web development or know someone who is, consider introducing them to this Rails course and volunteering to help mentor them. The reward is far greater than the effort.

If you're teaching technology or programming and are using Heroku, we want to get to know you better. Please reach out and contact us, we're interested in your story. Help us invest in programming literacy, and make the future a better place.

See you at LessConf and RailsConf?

Hey all,

It&#8217;s been a while (most of my blogging is over on the <a href="http://blog.planetargon.com/">Planet Argon blog</a>)... but I&#8217;m hoping to have some technical-related posts coming in the near future.

If you&#8217;ll be at <a href="http://lessconf.com/">LessConf</a> in Florida this week, I&#8217;ll be there. Do I owe you a drink?

Also, if you&#8217;re coming to visit Portland for <a href="http://www.railsconf.com/">RailsConf 2013</a> and will arrive the weekend before, you should <a href="http://planetargon.com/railsconf-hike">join us on a hike</a>.  If you can&#8217;t make it, be sure to say hello at the conference!

Hope all is well!



See you at LessConf and RailsConf?

Hey all,

It’s been a while (most of my blogging is over on the Planet Argon blog)… but I’m hoping to have some technical-related posts coming in the near future.

If you’ll be at LessConf in Florida this week, I’ll be there. Do I owe you a drink?

Also, if you’re coming to visit Portland for RailsConf 2013 and will arrive the weekend before, you should join us on a hike. If you can’t make it, be sure to say hello at the conference!

Hope all is well!


2X Dynos in Public Beta

A dyno, the unit of computing power on Heroku, is a lightweight container running a single user-specified command. Today we’re announcing a dyno with twice the capacity: 2X dynos.

2X Dynos

Existing dynos are now called 1X dynos. They come with 512MB of memory and 1x CPU share. They cost $0.05/hr. 2X dynos are exactly what their name implies: 1GB of memory and twice the CPU share for $0.10/hr. To support the growth of current and future apps on the platform, you can now control your dyno resources on two axes: size and quantity.

Let’s try them out.

Getting started with 2X dynos

Using the Heroku Toolbelt, resize your dynos with the resize command:

$ heroku ps:resize web=2X worker=1X
Resizing dynos and restarting specified processes... done
web dynos now 2X ($0.10/dyno-hour)
worker dynos now 1X ($0.05/dyno-hour)

To view the dyno size of a process type, use the ps command:

$ heroku ps
=== web (2X): `bundle exec unicorn -p $PORT -c ./config/unicorn.rb`
web.1: up 2013/03/27 14:27:58 (~ 6h ago)
web.2: up 2013/03/27 14:47:04 (~ 6h ago)
web.3: up 2013/03/27 15:08:23 (~ 5h ago)

=== worker (1X): `bundle exec rake worker:job`
worker.1: up 2013/03/27 14:39:04 (~ 6h ago)
worker.2: up 2013/03/27 15:08:24 (~ 5h ago)
worker.3: up 2013/03/27 14:30:55 (~ 6h ago)

Using the Dashboard use the app’s resources page:

Dashboard Resources Page

For full instructions, see the Dev Center article.

Use Cases

You can already scale horizontally, adding more dynos with heroku ps:scale. When you want to scale vertically instead, use larger dynos with heroku ps:resize.

1. Concurrency for Rails with Unicorn – The first use case is increasing concurrency on single-threaded Rails apps using Unicorn. Due to the improved in-dyno queuing efficiency that results from increasing the number of Unicorn workers, doubling the workers in 2X dynos while halving the dyno count often results in better performance.

2. JVM Languages – The modern JVM has an explicit memory model designed for multi-threaded concurrency, and has many frameworks explicitly designed to take advantage of this property. Utilizing more threads requires more memory for both the thread stacks and for objects created by these threads. The JVM is fully capable of taking advantage of the vertical scale in a 2X dyno.

3. Memory-intensive background jobs – image processing and geospatial processing often need larger dynos. If you’ve gotten an R14 out-of-memory error and this is not a leak, 2X dynos might be for you.

Horizontal vs. Vertical Scale

2X dynos give you a new knob to turn: vertical scale. Vertical scaling is appealing when the consolidation of compute power and memory increases overall performance. This can translate into better app performance at the same cost, and possibly lower costs if the set of dynos can be reduced by more than half.

It's impossible to make blanket statements about app performance due to each app's unique performance characteristics. I/O-bound apps may not benefit from an increase in memory and compute, while process-heavy apps would. The best way to determine in what configuration your app runs best is to measure it. Heroku provides several tools to help you understand your application's behavior.

Use log2viz to understand how much memory and CPU is being consumed by your current app/dyno configuration. If the memory usage or web dyno activity indicators are close to their limits, 2X dynos may provide much-needed increase in resources. The latest New Relic add-on accurately displays request queue times and can be used to visualize the impact a higher-concurrency configuration has on your app. Use 2X dynos to double your in-dyno Unicorn workers and mitigate inconsistent and variable queue times.


Through the duration of the beta period, 2X dynos will cost the same as 1X dynos, that is $0.05 per hour. Once generally available they will cost $0.10 per hour.

If you’re looking to improve concurrency on Rails apps, making use of JVMs or other memory-hungry tasks, 2X dynos have a lot of potential to make apps faster. Give them a try.


How can I measure how much memory my dynos are using?

The log-runtime-metrics Labs feature provides logged metrics that can be consumed and inspected with an add-on that consumes logs such as Papertrail, or displayed by a visualization tool like log2viz.

How do 2X dynos affect the 750 free dyno hours?

2X dynos consume twice as many free dyno-hours per hour as 1X dynos. Example: A 2X one dyno app will run for free for 375 hours compared to 750 hours for a 1X one dyno app.

Do single 2X dyno apps idle?

Yes. Idling affects single 1X and 2X dyno apps alike.

Can dyno size be configured on a per process-type basis?

Yes. Dyno size can be applied on a per process-type basis. For example: heroku ps:resize web=2X worker=1X worker2=2X

Can I get 4X (or larger) dynos?

We are exploring the option. If you have an app that may need these, send us a note.

Heroku Postgres Databases Patched

This post originally appeared on the postgres blog. We are also posting it in full here because we believe the content is so important.

Data is one of the most valuable assets of any company. As a database-as-a-service provider, one of our biggest responsibilities is ensuring your data is kept safe. A few weeks ago, one of the worst security vulnerabilities to date in PostgreSQL was discovered. To address this issue, Heroku deployed a point release upgrade across the entire Heroku Postgres service earlier this week. This resulted in a period of database unavailability, typically with a duration of less than one minute. Every database running on Heroku Postgres is now appropriately patched and is unaffected by the vulnerability.

PostgreSQL Vulnerability Details

The PostgreSQL project has provided official detail on CVE-2013-1899.

Several weeks ago there was a responsible disclosure of a serious security vulnerability within PostgreSQL by Mitsumasa Kondo and Kyotaro Horiguchi. The vulnerability allows unauthenticated remote users to use the ‘postmaster‘ process to write data to any accessible file, including critical internal database files.

The vulnerability was fixed and then committed to the PostgreSQL’s private git repository, but only after updates to anonymously accessible copies were disabled. Updated versions of PostgreSQL were released today to most large packaging repositories, as well as source code and installers.

Heroku Postgres Patching

The Heroku Postgres team worked with the PostgreSQL community to ensure we would be able to rapidly apply this patch. However, due to the nature of the issue, and aiming to mitigate risk for others, we were not able to discuss specifics until now. Our goal — in addition to ensuring your data was safe — was to continue monitoring this upgrade as it was deployed, providing early feedback to the community should bugs be found, and not jeopardizing in any way the coordinated public disclosure process stewarded by the PostgreSQL community. Most importantly, the PostgreSQL source code that included the patch was held in the utmost secrecy. In addition, the deployment plan was reviewed by PostgreSQL community members in advance.

Once the source code was released to the PostgreSQL packagers—of which a member of the Heroku Postgres staff is a part of—we began applying this patch to all Heroku Postgres databases, with the first updates starting on Monday. As of Wednesday at 6:30 PM PDT, all Heroku Postgres databases had been upgraded to their appropriate point release and were no longer vulnerable to CVE-2013-1899.


We realize that having no control over a maintenance window, however brief, is among the worst possible experiences. We are very sorry. Two reasons prevented us from working with you to schedule the security update. First, we prioritize ensuring your data is safe above all else, as a result making sure that every database was patched before this exploit was weaponized was paramount. Secondly, this was the first time we've had to deal with a security update of this scale, and have no machinery in place to schedule upgrades of this sort. Spending time to build such machinery would have prevented us from having every database patched in time. We will continue to work on improving our process around such maintenance to provide a better experience in the future.

As of late Wednesday all Heroku Postgres databases were upgraded and no longer at risk of CVE-2013-1899. No further action is required on your part to ensure your data remains safe.

#412 Fast Rails Commands

Rails commands, such as generators, migrations, and tests, have a tendency to be slow because they need to load the Rails app each time. Here I show three tools to make this faster: Zeus, Spring, and Commands.

Ruby Matrix, the Forgotten Library

Send to Kindle

Ruby Matrix, the Forgotten Library

This guest post is contributed by Matthew Kirk, who is a partner at Modulus 7, specializing in software development and strategy. The basis of his career has been around utilizing science to improve businesses. He has spoken at technology conferences around the world and in his spare time, he enjoys traveling and adding to his 2000+ vinyl record collection.

Matthew Kirk Remember matrices from math class? No not the movie, but the rectangular array of numbers. While you might not see it often, Ruby has a matrix implementation that is well tested and allows you to accomplish tough calculations quickly.

While I won’t be able to teach you everything there is to be known about matrices, we will cover how to use matrices within Ruby as well as some quirks and their major selling points. By the end of this I hope that you delve deeper into learning about matrices and use them in your next project.

What are matrices?

A matrix according to Wikipedia is a rectangular array of numbers. Used heavily in math, matrices are all over languages like R and Matlab. They can be a great way to store numerical data and simplify many difficult and tedious problems. Instead of solving systems of equations matrices can simplify these into one equation.

In terms of how Ruby implements matrices, Ruby stores all matrix rows into one big array. The only requirement is that the arrays are of the same dimension. So for each row that is added to a matrix each one must be of the same size.

Just like arrays, matrices are zero indexed meaning that the first row is index 0 and the first column is index 0. Unlike arrays though you have to have two indexes to get to an element:

Making matrices:

Some quirks:

The Matrix library has some quirks. The Matrix class allows non-numerical data to go into itself. This could be useful for storing things like symbols in a more x, y format but render most of the matrix functions useless.

For instance:

Another quirk to be aware of: Matrix[*rows] does not copy the rows objects but instead points to it. To avoid this use Matrix.rows(rows) or Matrix.columns(columns). Implementation wise Matrix[*rows] calls the function Matrix.rows(rows, copy = false).

Iterating over matrices:

How do you iterate over a matrix? Most likely you would think that matrices read left to right top to bottom. And that’s true. But there are other cases as well.

In total there are 7 ways to iterate over a Matrix in ruby:

  • :all This reads left to right top to bottom. This is the default case when you type matrix.each
  • :diagonal: This only reads the diagonal elements or row index == column index
  • :off_diagonal: This will read everything not on the diagonal or row index != column index
  • :lower: This reads the lower triangle of the matrix or row index <= column index
  • :strict_lower: this is more strict and reads only row index < column index
  • :strict_upper: this is a strict upper triangle and is row index > column index
  • :upper: row index >= column index

An example:

Example: Parabola with matrix:

Imagine you want to fit a curve through three points. If you remember math class you might remember that you can do this by fitting a quadratic. For instance lets say we want a line that goes through (1,2) (3, 5.5) and (6, 6). To solve this we would write the equations:

Math equation

The way to solve this would usually involve lots of algebra and substitutions. While it’s easy to solve this in the case where we already know the numbers it is difficult to come up with a general solution (try it I dare you).

Instead of worrying about solving this using non-matrix algebra we can solve it using matrix algebra. The first step is to rewrite the above system into this form:

Math equation

To make it even easier to solve we would rewrite this as Ax = b. To solve this we would take the inverse of A and then multiply both sides by that. Which would yield:

Math equation

Now that we know this, we can easily solve this using Ruby with the following formula.

While it’s close it won’t be correct unless you use Rational. Ruby’s matrix library graciously utilizes functions that preserve precision. You would expect most libraries to convert to floats but ruby does not.

For instance you can change the above function call to:

Whenever possible try to preserve the precision!

The general case, fitting an n-power polynomial to n points:

Up above we only fit this curve to 3 points. But what about 4 or 15 points? This would be quite simple to do and would only require a little bit of modification:


While you might not use matrices every day, they can be useful to solve problems involving systems of equations. Ruby has a robust matrix library that can be useful in finding solutions to these types of problems. Next time you want to fit a curve keep in mind matrices might be the best way to go!

For more information about matrices I recommend reading Wikipedia articles. There are lots of math professors who spend hours updating them tediously. If they are too confusing, think about picking up a book on matrix algebra like Matrix Computations.

Feel free to ask questions and give feedback in the comments section of this post. Thanks!

Technorati Tags: , , ,

(Powered by LaunchBit)

Routing and Web Performance on Heroku: a FAQ

Hi. I'm Adam Wiggins, cofounder and CTO of Heroku.

Heroku has been my life’s work. Millions of apps depend on us, and I take that responsibility very personally.

Recently, Heroku has faced criticism from the hacker community about how our HTTP router works, and about web performance on the platform in general. I’ve read all the public discussions, and have spent a lot of time over the past month talking with our customers about this subject.

The concerns I've heard from you span past, present, and future.

The past: some customers have hit serious problems with poor web performance and insufficient visibility on their apps, and have been left very frustrated as a result. What happened here? The present: how do you know if your app is affected, and if so what should you do? And the future: what is Heroku doing about this? Is Heroku a good place to run and scale an app over the long term?

To answer these questions, we’ve written a FAQ, found below. It covers what happened, why the router works the way that it does, whether your app is affected by excessive queue time, and what the solution is.

As to the future, here’s what we’re doing. We’re ramping up hands-on migration assistance for all users running on our older stack, Bamboo, or running a non-concurrent backend on our new stack, Cedar. (See the FAQ for why this is the fix.) We’re adding new features such as 2X dynos to make it easier to run concurrent backends for large Rails apps. And we're making performance and visibility a bigger area of product attention, starting with some tools we've already released in the last month.

If you have a question not answered by this FAQ, post it as a comment here, on Hacker News, or on Twitter. I’ll attempt to answer all such questions posted in the next 24 hours.

To all our customers who experienced real pain from this: we're truly sorry. After reading this FAQ, I hope you feel we're taking every reasonable step to set things right, but if not, please let us know.



Q. Is Heroku’s router broken?

A. No. While hundreds of pages could be written on this topic, we’ll address some of this in Routing technology. Summary: the current version of the router was designed to provide the optimum combination of uptime, throughput, and support for modern concurrent backends. It works as designed.

Q. So what’s this whole thing about then?

A. Since early 2011, high-volume Rails apps that run on Heroku and use single-threaded web servers sometimes experienced severe tail latencies and poor utilization of web backends (dynos). Lack of visibility into app performance, including incorrect queue time reporting prior to the New Relic update in February 2013, made diagnosing these latencies (by customers, and even by Heroku’s own support team) very difficult.

Q. What types of apps are affected?

A. Rails apps running on Thin, with six or more dynos, and serving 1k reqs/min or more are the most likely to be affected. The impact becomes more pronounced as such apps use more dynos, serve more traffic, or have large request time variances.

Q. How can I tell if my app is affected?

A. Add the free version of New Relic (heroku addons:add newrelic) and install the latest version of the newrelic_rpm gem, then watch your queue time. Average queue times above 40ms are usually indicative of a problem.

Some apps with lower request volume may be affected if they have extremely high request time variances (e.g., HTTP requests lasting 10+ seconds) or make callbacks like this OAuth example.

Q. What’s the fix?

A. Switch to a concurrent web backend like Unicorn or Puma on JRuby, which allows the dyno to manage its own request queue and avoid blocking on long requests.

This requires that your app be on our most current stack, Cedar.

Q. Can you give me some help with this?

A. Certainly. We’ve already emailed all customers with apps running on Thin with more than six dynos with self-migration instructions, and a way to reach us for direct assistance.

If you haven’t received the email and want help making the switch, contact us for migrating to Cedar or migrating to Unicorn.

Routing technology

Q. Why does the router work the way that it does?

A. The Cedar router was built with two goals in mind: (1) to support the new world of concurrent web backends which have become the standard in Ruby and all other language communities; and (2) to handle the throughput and availability needs of high-traffic apps.

Read detailed documentation of Heroku’s HTTP routing.

Q. Even with concurrent web backends, wouldn’t a single global request queue still use web dynos more efficiently?

A. Probably, but it comes with trade-offs for availability and performance. The Heroku router favors availability, stateless horizontal scaling, and low latency through individual routing nodes. Per-app global request queues require a sacrifice on one or more of these fronts. See Kyle Kingsbury’s post on the CAP theorem implications for global request queueing.

After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that beats the simplicity and robustness of random routing to web backends that can support multiple concurrent connections.

Q. So does that mean you aren’t working on improving HTTP performance?

A. Not at all. We're always looking for new ways to make HTTP requests on Heroku faster, more reliable, and more efficient. For example, we’ve been experimenting with backpressure routing for web dynos to signal to the router that they are overloaded.

You, our customers, have told us that it’s not routing algorithms you ultimately care about, but rather overall web performance. You want to serve HTTP requests as quickly as possible, for fast page loads or API calls for your users. And you want to be able to quickly and easily diagnose performance problems.

Performance and visibility are what matters, and that’s what we’ll work on. This will include ongoing improvements to dynos, the router, visibility tools, and our docs.


Q. Did the Bamboo router degrade?

A. Yes. Our older router was built and designed during the early years of Heroku to support the Aspen and later the Bamboo stack. These stacks did not support concurrent backends, and thus the router was designed with a per-app global request queue. This worked as designed originally, but then degraded slowly over the course of the next two years.

Q. Were the docs wrong?

A. Yes, for Bamboo. They were correct when written, but fell out of date starting in early 2011. Until February 2013, the documentation described the Bamboo router only sending one connection at a time to any given web dyno.

Q. Why didn’t you update Bamboo docs in 2011?

A. At the time, our entire product and engineering team was focused on our new product, Cedar. Being so focused on the future meant that we slipped on stewardship of our existing product.

Q. Was the "How It Works" section of the Heroku website wrong?

A. Yes. Similar to the docs, How It Works section of our website described the router as tracking which dynos were tied up by long HTTP requests. This was accurate when written, but gradually fell out of date in early 2011. Unlike the docs, we completely rewrote the homepage in June of 2011 and it no longer referenced tracking of long requests.

Q. Was the queue time metric in New Relic wrong?

A. Yes, for the same 2011—2013 period from previous questions. The metric was transmitted to the New Relic instrumentation in the app via a set of HTTP headers set by the Heroku router. The root cause was the same as the Bamboo router degradation: the code didn't change, but scaling out the router nodes caused the data to become increasingly inaccurate and eventually useless. With New Relic's help, we fixed this in February 2013 by calculating queue time using a different method.

Q. Why didn’t Heroku take action on this until Rap Genius went public?

A. We’re sorry that we didn’t take action on this based on the customer complaints via support tickets and other channels sooner. We didn’t understand the magnitude of the confusion and frustration caused by the out-of-date Bamboo docs, incorrect queue time information in New Relic, and the general lack of visibility into web performance on the platform. The huge response to the Rap Genius post showed us that this touched a nerve in our community.

The Future

Q. What are we doing to make things right from here forward?

A. We’ve been working with many of our customers to get their queue times down, get them accurate visibility into their app’s performance, and make sure their app is fast and running on the right number of dynos. So far, the results are good.

Q. What about everyone else?

A. If we haven’t been in touch yet, here’s what we’re doing for you:

  • Migration assistance: We’ll give you hands-on help migrating to a concurrent backend, either individually or in online workshops. This includes the move to Cedar if you’re still on Bamboo. If you’re running a multi-dyno app on a non-concurrent backend and haven’t received an email, drop us a line about Thin to Unicorn or Bamboo to Cedar.
  • 2X dynos: We’re fast-tracking the launch of 2X dynos, to provide double the memory and allow for double (or more) Unicorn concurrency for large Rails apps. This is already available in private beta in use by several hundred customers, and will be available in public beta shortly.
  • New visibility tools: We’re putting more focus on bringing you new performance visibility features, such as the log2viz dashboard, CPU and memory use logging, and HTTP request IDs. We’ll be working to do much more on this front to make sure that you can diagnose performance problems when they happen and know what to do about it.

Want something else not mentioned here? Let us know.