Running Rails on Heroku Update

On February 16th, we published a blog post outlining five specific and immediate actions we would take to improve our Rails customers' experience with Heroku. We want to provide you with an update on where these things stand. As a reminder, here’s what we committed to do:

  1. Improve our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
  2. Remove incorrect and confusing metrics reported by Heroku or partner services like New Relic
  3. Add metrics that let customers determine queuing impact on application response times
  4. Provide additional tools that developers can use to augment our latency and queuing metrics
  5. Work to better support concurrent-request Rails apps on Cedar

We have resolved the first two items:

  1. Improving Documentation: We’ve updated our Dev Center docs and website to more accurately describe how routing occurs on Heroku (e.g., HTTP Routing on Bamboo and How Routing Works on the Bamboo Stack).

  2. Removing Incorrect Metrics: We’ve worked with New Relic to release an updated version of their monitoring tools, which we documented in Better Queuing Metrics With Updated New Relic Add-on.

We are working on the remaining three items in partnership with our early beta users:

  1. Adding New Metrics: We are improving the data available through our logs in two ways. First, we’re adding a transaction ID to every request log. Second, we are working on application middleware to provide additional metrics in customer logs.
  2. Providing Additional Measurement Tools: We have created a data visualization tool that analyzes log data in real-time and helps observe an app’s performance across a few key Heroku-specific metrics. This tool is currently being beta tested.
  3. Improving Cedar Support: Unicorn is now the recommended Rails app server. We are currently beta testing larger dynos to support additional unicorn processes.

Fulfilling these commitments is Heroku’s number one priority. We have dedicated engineering and product teams focused on improving the performance of Rails apps on Heroku.

We’ll detail the availability of our improvements in future blog posts. If you would like to beta test any of these new features, please drop us a note and let us know which features you are interested in testing.

Bamboo Routing Performance

Yesterday, one of our customers let us know about significant performance issues they have experienced on Heroku. They raised an important issue and I want to let our community know about it. In short, Ruby on Rails apps running on Bamboo have experienced a degradation in performance over the past 3 years as we have scaled.

We failed to explain how our product works. We failed to help our customers scale. We failed our community at large. I want to personally apologize, and commit to resolving this issue.

Our goal is to make Heroku the best platform for all developers. In this case, we did not succeed. But we will make it right. Here’s what we are working on now:

  • Posting an in-depth technical review tomorrow
  • Quickly providing more visibility into your app’s queue of web requests
  • Improving our documentation and website to accurately reflect our product
  • Giving you tools to understand and improve the performance of your apps
  • Working closely with our customers to develop long-term solutions

I am committing to listening to you, acting quickly to meet your needs and making sure Heroku is a platform that you trust for all of your applications. If you have additional concerns, please let me know. My email address is oren.teich@heroku.com.

Oren Teich
GM, Heroku

Cross-Site Request Forgery Vulnerability Resolution

On Friday January 18, security researcher Benjamin Manns notified Heroku of a security vulnerability related to our add-ons program. At a high level, the vulnerability could have resulted in disclosing our Cross-Site Request Forgery tokens (these tokens are used to prevent browser hijacking) to third parties.

We quickly addressed the vulnerability and on Sunday, we deployed a patch to remediate the issue. We also reviewed our code for related vulnerabilities and conducted a review of our audit logs to determine the impact of the vulnerability. We found no instances of this issue being exploited.

We wish to thank Mr. Manns for his work and commitment to responsible disclosure. You can access his write up here: http://www.benmanns.com/posts/security-vulnerability-found-in-heroku-and-rails-form-tag/

We would also like to reaffirm our commitment to the security and integrity of our customers’ data and code. Nothing is more important to us.

Oren Teich, Chief Operating Officer

Password Hijacking Security Vulnerability and Response

Heroku recently learned of and resolved a security vulnerability. We want to report this to you, describe how we responded to the incident, and reiterate our commitment to constantly improving the security and integrity of your data and source code.

On December 19, 2012, security researcher Stephen Sclafani notified us of an issue in our account creation system. Using a maliciously-crafted HTTP request, an attacker could change the password of a pre-existing Heroku user account, and thus gain control of it. This attack would not disclose the pre-existing password to the attacker (those are stored internally as non-recoverable bcrypt hashes).

Upon receiving notification, our engineering and security staff engaged with Mr. Sclafani. We developed and deployed a preliminary patch to production on December 20. While we were deploying the patch, Mr. Sclafani also discovered a related issue in the password reset flow that could be used to reset the passwords of a certain subset of users at random. A preliminary patch for this was also developed and deployed on December 20.

After deploying these patches, we conducted a thorough and comprehensive audit of our internal logs. We found no evidence that these vulnerabilities were exploited prior to Mr. Sclafani’s research on December 19, either by him or any other third parties. Due to the nature of the vulnerability, any customer whose account was compromised would have found both their existing password and API key invalidated, and would have had to initiate a password reset.

While both Mr. Sclafani and Heroku endeavoured to use test accounts exclusively, a very small number of customer account passwords were reset during the incident. We have contacted the impacted customers and advised them to reset their passwords and credentials.

We would like to thank Mr. Sclafani for notifying us of this vulnerability, and giving us ample opportunity to fix it. His description is available at http://stephensclafani.com/2013/01/09/vulnerabilities-in-heroku/. We are extremely grateful to both him and all external security researchers who practice responsible disclosure.

We are confident in the steps we have taken to protect our customers from this vulnerability and will continue to improve our internal processes in order to provide our customers with a trusted cloud platform. We would also like to reaffirm our commitment to the security and integrity of our customers’ data and code. Nothing is more important to us.

Oren Teich, Chief Operating Officer

Hosting San Francisco Rails 3.1 Hackfest

The rails community is making the final push to get 3.1 out and is looking for your help! As part of a worldwide effort over the weekend, Heroku is hosting a local hackfest to help finalize Rails 3.1.

On Saturday, July 23rd from 12pm to 5pm, Heroku will be hosting a gathering for the
Rails 3.1 Hackfest. We’re looking for
people that want to improve things at all levels of the Rails stack – from debugging
to documentation. Come with apps to upgrade to Rails 3.1. We’ll also be working
on getting Rails 3.1 apps running on Heroku’s Celadon Cedar stack. If you haven’t done this yet, don’t miss the opportunity!

The Rails 3.1 Hackfest will be at our San Francisco office:

321 11th St in SOMA

Saturday, July 23rd, 12pm to 5pm

Beer and Pizza will be provided! Make sure to let us know you’re coming so we have enough food and we’ll see you on Saturday.

See the official announcement

Post-mortem on April 21 Outage

On April 21st 2011, Heroku experienced a widespread application outage. We have posted a full post-mortem detailing the causes and steps we are taking to prevent similar outages from happening in the future.

Heroku status always contains our current status. You can follow @herokustatus to follow status updates via twitter.