Introducing Heroku Private Spaces: Private PaaS, delivered as-a-Service

As the world becomes more cloud-centric, and more of our apps and business depend on its capabilities, the trust, control and management of cloud services is more important than ever. Since the first days of Heroku — and Platform-as-a-Service in general — many companies have struggled to balance the impact and success of the cloud with the control offered by traditional software and on-premise infrastructure. Too often that balance tips back towards software, with companies choosing to meet those requirements by building and running their own platforms, inevitably becoming frustrated by the resulting complexity, cost and poor experience.

Today Heroku is introducing Private Spaces, a new Heroku runtime that delivers the best of both worlds; the simplicity and success of the cloud, combined with the network and trust controls historically only available with on premise, behind the firewall deployments. Available today in public beta, Private Spaces is powered by Heroku

dw_regions@2x
dw_outboundIP@2x

Continue reading “Introducing Heroku Private Spaces: Private PaaS, delivered as-a-Service”

Introducing Heroku Enterprise: New Features for Teams

Apps have transformed how we do almost everything. The ubiquity of mobile devices with millions of available apps mean that today, anyone can pull up an app in seconds to check their car engine, turn on the lights at home or ride a scooter across town.

So far, the companies behind these apps have mostly been startups, many of which use Heroku to help them iterate quickly and stay focused on the customer experience instead of wasting time on infrastructure.

Today we’re proud to announce Heroku Enterprise, a new edition of Heroku that helps larger companies take advantage of the same technology recipe while meeting their unique needs for more advanced collaboration, better access controls and enterprise-grade support.

Collaboration at Any Scale

Heroku Enterprise gives you a single organization-wide view of all developers, all applications and all resources. You can empower every single developer to innovate at their own pace while the organization retains complete visibility to the state of all projects: What apps are running in production? Who is collaborating on a particular app? What resources are being consumed by each app? What is the total resource use this month compared to previous months?

screenshot-#(dashboard,-orgs,-demo,-access)

With Heroku Enterprise, teams have a safe way to collaborate openly while still meeting company security standards. When off-boarding developers, all application access is revoked with a single action. Administrators can also audit the multi-factor authentication status of developer accounts to ensure they comply with company standards.

Targeted Access Controls

Every business, large or small, needs to have strong security controls in place. When you are asking customers to entrust you with their personal or business information, you (and they) will want to know who has access to that data and who can deploy code that touches it.

Image 2015-02-18 at 4

Continue reading “Introducing Heroku Enterprise: New Features for Teams”

Introducing the General Availability of Performance Dynos in Europe

Since day one, developers from all over the world have been deploying apps on Heroku, and we’re extremely proud of the strong global community we’ve built. Our European customers in particular have asked for the ability to deploy applications geographically close to their European customer base so they can offer a better user experience with more responsive apps. In 2013 we launched 1X and 2X dynos in Europe to meet this demand. Today we’re pleased to announce the general availability of Performance Dynos in our European region.

The availability of Performance Dynos in Europe provides the flexibility needed to build and run large-scale, high-performance apps. Performance Dynos are highly isolated from workloads in other dynos and contain 12 times the memory of a single Heroku Dyno, resulting in significantly faster and more consistent response times.

Deploying apps in the Heroku EU region can reduce latency by 100ms or more for European users, improving responsiveness and application experience. With the launch of Performance Dynos in Europe, highly scaled apps can reach even greater improvements. We also have 110 Heroku Add-ons available in the EU region to extend application functionality. All these EU region services are managed by the same unified Heroku interface that you are already familiar with, and there is no difference in pricing from our US region.

Growth of Heroku in Europe

Ranging from small EU-based startups to international Fortune 500 enterprises, we’ve seen strong adoption of Heroku in Europe. In 2014, the number of apps deployed in Heroku’s Europe Region increased by nearly 180%. This includes great companies like Toyota Motor Europe and Sweden’s largest commercial television company, TV4.

When the TV4 team moved their first web apps from the US region to the European region, they immediately saw 150 ms improvements in web performance. Today, TV4 deploys all their Heroku apps to the EU region and are taking full advantage of Performance Dynos to deliver the very best user experience to their customers.

It is more important than ever to build responsive apps that react to customer actions in milliseconds. Heroku is committed to sustained investment in Europe and beyond to ensure that you have the geographic deployment options you need to meet this goal.

Additional Resources:

Two-factor Authentication Now Generally Available

Two-factor authentication is a powerful and simple way to greatly enhance security for your Heroku account. It prevents an attacker from accessing your account using a stolen password. After a 4 month beta period, we are now happy to make two-factor authentication generally available.

Turning on two-factor authentication

You can enable and disable two-factor authentication for your Heroku account in the Manage Account section of Dashboard.

Before you turn it on, please read on here to understand the risks of account lock-out. You can also refer to the Dev Center docs for more details.

How two-factor authentication protects you

Without two-factor authentication, an attacker can gain access to your Heroku account by just knowing your password. The most common way attackers get access to passwords is by hijacking email accounts and issuing a password reset request. If you reuse the same password for multiple services, an attacker may also learn your password if one of your other services are compromised and its password database leaks (therefore, never use the same password for multiple services).

After you turn on two-factor authentication, you can only authenticate by providing both the password and a "second factor" code. The second factor code is a code that can only be used once or that expires very quickly (30-60 seconds). You obtain the code from an authenticator app on your mobile device.

Now, it is only possible to access your account by knowing your password and having access to your (unlocked) mobile device.

If you lose your two-factor device

When you enable two-factor authentication, it is critical that you download a set of recovery codes and store them in a safe place. If you lose your mobile device or if it gets wiped, you can authenticate using these recovery codes in place of the two-factor code generated by your device.

If you have enabled two-factor authentication and not saved your recovery codes, go to the accounts page now and download your codes.

If for some reason you have neither your two-factor device nor your recovery codes, there are a few additional ways you may be able to recover.

Your responsibility when turning on two-factor authentication

When you enable two-factor authentication, please understand that

It is possible for you to lock yourself out of your account with no ability to regain access.

It is critical that you download recovery codes and store them in a place where you can access them in case of an emergency.

If you are locked out and none of the recovery methods work for you, there is no guarantee that you can regain access to your account because we may not be able to confirm ownership of the account.

In the future, we may add additional forms of account ownership verification to aid in cases of lock-out, but there is no single solution that fully solves this problem.

Therefore, please do save those recovery codes!

What two-factor authentication does not protect against

Security is a multi-faceted problem and two-factor authentication is not designed to protect against all possible attacks. For example, it will not protect you against malware that gives a remote attacker access to your computer. Two-factor authentication is specifically designed to protect against password leaks. You should continue to follow all other security best practices to achieve maximum protection.

Introducing Heroku DX: The New Heroku Developer Experience

One of our core beliefs at Heroku is that developers do their best work when the development process is as simple, elegant, and conducive to focus and flow as possible. We are grateful for how well many of our contributions to that cause have been received, and today we are making generally available a new set of features that have been inspired by those values.

Collectively, we call these new features Heroku DX–the next evolution in Heroku’s developer experience. Our goal with these new features–Heroku Button, Heroku Dashboard + Metrics and Heroku Postgres DbX–is to make it faster than ever for developers to build, launch and scale applications.

heroku-dx

Heroku Button

Heroku is known for making it easy to get started on a new app. Many tutorials and sample apps contain step-by-step guides for deploying to Heroku because it is one of the easiest platforms to start on. But we wanted to make it even easier. We wanted step 2 to be: “There is no step 2”.



Heroku Button makes this a reality. With a single click you can set up a new app from sample code, complete with environment configuration, Add-ons and the first deploy. Any public GitHub project can be Heroku Button enabled simply by adding an app.json file with some additional metadata. During the beta, we saw over 400 Heroku Buttons created, including examples from Dropbox, Twilio, Uber and Coinbase. As part of general availability, we are launching a real-time view into the Heroku Button community at buttons.heroku.com, which is continually updated with the newest and most popular buttons.

Heroku Dashboard + Metrics

At the center of Heroku DX is an entirely new web Dashboard redesigned from the ground up for improved developer experience. Under the hood, the new Dashboard is written in Ember.js, allowing for faster and smoother interactions. At the surface, a new visual design makes navigating your collection of apps and associated resources and properties simpler. New interaction patterns, such as drag and drop app assignment, simplify previously complex tasks. As a modern web app, the new Dashboard is fully responsive, and works across devices and screen sizes, making it easy to manage and scale your Heroku apps wherever you go.

A new Activity section in Dashboard makes it dramatically easier for developers to collaborate. Every activity on every app shows up in the Dashboard so everyone on the team knows what’s going on. You can even inspect the build output from team members’ builds and help them troubleshoot problems right there in the Dashboard.

dashboard-metrics

With Heroku DX, we are also introducing a significant enhancement to the runtime management of your apps with a new Metrics section in Dashboard that gives you a refreshingly simple and straightforward view on your application’s performance.

Heroku Metrics surfaces the key runtime attributes of your application, including Response Time, Requests / Sec, and CPU, and adds an intelligence layer that automatically surfaces the key recommendations and performance changes. Unified across a single time axis, the relationship between applications metrics is made clear, allowing you to easily see their interactions. The new Metrics feature is available for any application that has been scaled to more than one running dyno.

Heroku Postgres DbX

At the center of many applications running on Heroku is Heroku Postgres, and as part of the Heroku DX launch we are releasing one of our most significant upgrades ever to our Postgres Database-as-a-Service. These new Postgres features–or Heroku Postgres DbX–are designed to advance the experience of developing and managing databases in a way that is similar to how Dashboard enhances the other related parts of your developer experience. At the core of Postgres DbX is a new lineup of database plans, which on average offer 2-3X performance of our previous plans while keeping the same price. Premium plans and above also automatically gain encryption at rest, simplifying many compliance requirements.

Heroku Postgres’ most prominent addition is Performance Analytics. Understanding the operating and behavior of a relational database–in a way that is clear and actionable–typically requires an uncommon set of tools, skills and patience. Performance Analytics reveals one of the key benefits of the “as-a-Service” model as applied to databases; a complex pipeline of configuration, logs, processing and visualization is reduced to a simple and immediate web experience for the developer.

postgres

Learn More

The Dev Center provides detailed technical information on all of the features launched today. You can also learn more by registering for an online session we’ll be delivering on October 8.

Better Queuing Metrics With Updated New Relic Add-On

Today our partner, New Relic, released an update to the Ruby New Relic agent that addresses issues brought up by our customers. The new version corrects how New Relic reports performance metrics for applications running on Heroku. Queueing time is now reported as the total time from when a request enters the router until the application starts processing it. Previous versions of New Relic only reported queueing time in the router. The new approach will result in more accurate queueing metrics that allow you to better understand and tune the performance of your application.

Update, Feb 22:

New Relic has released a similar update for Python. Python developers should update to this latest version to benefit from the improved metrics. JVM language developers do not need to take any action. The current New Relic Java agent already includes the improved queue time metrics.

Install or update the New Relic Ruby Add-on

If you are already using New Relic with your Ruby apps, then simply update your Gemfile to reference the new agent version:

gem "newrelic_rpm", "~> 3.5.7.59"

then run

$ bundle update newrelic_rpm
$ git add Gemfile Gemfile.lock
$ git commit -m 'update new relic agent'
$ git push heroku master

If you are not yet using New Relic, you can learn how to install and configure the add-on on Dev Center.

How It Works

The updated New Relic agent uses an improved strategy for reporting request queue times on Heroku. Prior to this update, New Relic reported request queue time using a value set by the Heroku routing mesh. This only reflected the time a request spent in the router queue and did not properly include time spent waiting in the dyno’s request queue.

Our routing performance update documents our finding that some applications have requests that may spend significant time queued on dynos. To help our customers understand precisely where their applications are being delayed, the updated New Relic agent includes dyno wait time in the total queue time metric. The new queue time is calculated as the difference between the time the Heroku router first gets a request and the time the request begins processing on the dyno. The result is a more accurate picture of how long requests wait in queues.

Clock Skew

The new version of New Relic calculates queue times using two different clocks — the dyno and router clocks. While Heroku servers regularly sync their clocks, it’s common for clocks to drift apart between syncs. This phenomenon is known as clock skew and it can affect the queue time metric collected by New Relic. In our experience, even though clock skew can cause small inaccuracies, the overall trend data displayed by New Relic will still accurately reflect your application’s queue times.

How to Learn More

If you’d like more information on how to install and configure the New Relic add-on, please see the New Relic Dev Center article and the Unicorn specific instructions. For general suggestions on how to improve the performance of your app, check out our performance overview page.

Routing Performance Update

Over the past couple of years Heroku customers have occasionally reported unexplained latency on Heroku. There are many causes of latency—some of them have nothing to do with Heroku—but until this week, we failed to see a common thread among these reports. We now know that our routing and load balancing mechanism on the Bamboo and Cedar stacks created latency issues for our Rails customers, which manifested themselves in several ways, including:

  • Unexplainable, high latencies for some requests
  • Mismatch between reported queuing and service time metrics and the observed reality
  • Discrepancies between documented and observed behaviors

For applications running on the Bamboo stack, the root cause of these issues is the nature of routing on the Bamboo stack coupled with gradual, horizontal expansion of the routing cluster. On the Cedar stack, the root cause is the fact that Cedar is optimized for concurrent request routing, while some frameworks, like Rails, are not concurrent in their default configurations.

We want Heroku to be the best place to build, deploy and scale web and mobile applications. In this case, we’ve fallen short of that promise. We failed to:

  • Properly document how routing works on the Bamboo stack
  • Understand the service degradation being experienced by our customers and take corrective action
  • Identify and correct confusing metrics reported from the routing layer and displayed by third party tools
  • Clearly communicate the product strategy for our routing service
  • Provide customers with an upgrade path from non-concurrent apps on Bamboo to concurrent Rails apps on Cedar
  • Deliver on the Heroku promise of letting you focus on developing apps while we worry about the infrastructure

We are immediately taking the following actions:

  • Improving our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
  • Removing incorrect and confusing metrics reported by Heroku or partner services like New Relic
  • Adding metrics that let customers determine queuing impact on application response times
  • Providing additional tools that developers can use to augment our latency and queuing metrics
  • Working to better support concurrent-request Rails apps on Cedar

The remainder of this blog post explains the technical details and history of our routing infrastructure, the intent behind the decisions we made along the way, the mistakes we made and what we think is the path forward.

How routing works on the Bamboo stack

In 2009, Heroku introduced the Bamboo stack. It supported only one language, one web framework and one embedded webserver. These were: Ruby (MRI 1.8), Rails (2.x) and Thin, respectively.

The Bamboo stack does not support concurrency. On Bamboo, a single process can serve only one request at a time. To support this architecture, Heroku’s HTTP router was designed to queue requests at the router level. This enabled it to efficiently distribute requests to all available dynos.

The Bamboo router never used a global per-application request queue. The router is a clustered service where each node in the cluster maintains its own per-application request queue. This is less efficient than routing with a global request queue, but it is a reasonable compromise as long as the cluster is small.

To see why, let’s look at a simplistic example. In the two diagrams below, requests are coming in through three router nodes and being passed to two dynos. The majority of requests take 50ms, while a rare slow request takes 5000ms. In the first diagram, you can see how a slow request, coming in to Router 1, is passed to Dyno 1. Until Dyno 1 is finished with that request, Router 1 will not send any more requests to that dyno. However, Routers 2 and 3 may still send requests to that dyno.

Meanwhile, as illustrated in the next diagram, because Routers 2 and 3 are not aware that Dyno 1 is busy, they may still queue up one request each for Dyno 1. These requests are delayed until Dyno 1 finishes processing the slow request.

The inefficiency in request routing gets worse as the number of routers increases. This is essentially what’s been happening with Rails apps running on the Bamboo stack. Our routing cluster remained small for most of Bamboo’s history, which masked this inefficiency. However, as the platform grew, it was only a matter of time before we had to scale out and address the associated challenges.

Routing on Cedar

As part of the new Cedar stack, we chose to evolve our router design to achieve the following:

  • Support additional HTTP features like long polling and chunked responses
  • Support multi-threaded and multi-process runtimes like JVM, Node.js, Unicorn and Puma
  • Stateless architecture to optimize for reliability and scalability

Additionally, to meet the scalability requirements of Cedar we chose to remove the queuing logic and switch to random assignment. This new routing design was released exclusively on Cedar and was significantly different from the old design. What’s important to note is we intended customers to get the new routing behavior only when they deployed applications to Cedar.

Degradation of Bamboo routing

In theory, customers who had relied on the behavior of Bamboo routing could continue to use the Bamboo stack until they were ready to migrate to Cedar. Unfortunately that is not what happened. As traffic on Heroku grew, we added new nodes to the routing cluster rendering the per-node request queues less and less efficient, until Bamboo was effectively performing random load balancing.

We did not document this evolution for our customers nor update our reporting to match the changing behavior. As a result, customers were presented with confusing metrics. Specifically, our router logs captured the service time and the depth of the per app request queue and present that to customers, who in turn were relying on these metrics to determine scaling needs. However, as the cluster grew, the time-and-depth metric for an individual router was no longer a relevant way to determine latency in your app.

As a result, customers experienced what was effectively random load balancing applied to their Bamboo applications. This was not caused by an explicit change to the Bamboo routing code. Nor was it related to the new routing logic on Cedar. It was a pure side-effect of the expansion of the routing cluster.

No path for concurrent Rails apps on Cedar

We launched Cedar in beta in May 2011 with support for Node.js and Ruby on Rails. Our documentation recommends the use of Thin, which is a single-threaded, evented web server. In theory, an evented server like Thin can process multiple concurrent requests, but doing this successfully depends on the code you write and the libraries you use. Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn.

Rails apps deployed to Cedar with Thin can rather quickly end up with request queuing problems. Because the Cedar router no longer does any queuing on behalf of the app, requests queued at the dyno must wait until the single Rails process works its way through the queue. Many customers have run into this issue and we failed to take action and provide them with a better approach to deploying Rails apps on Cedar.

Next Steps

To reiterate, here is what we are doing now:

  • Improving our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
  • Removing incorrect and confusing metrics reported by Heroku or partner services like New Relic
  • Adding metrics that let customers determine queuing impact on application response times
  • Providing additional tools that developers can use to augment our latency and queuing metrics
  • Working to better support concurrent-request Rails apps on Cedar

If you have thoughts or questions, please comment below or reach out to me directly at jesperj@heroku.com.

Java Template Apps on Heroku

Learning a new language or framework can be both fun and rewarding. But tutorials only get you so far: one of the easiest ways to get started is by copying an existing sample app.

Today we’re introducing template-based app creation for Java on Heroku. To try it out, go to www.heroku.com/java and click Create App on one of the four templates at the bottom of the page.

In seconds, you’ll have your own copy of the app deployed and running on Heroku. You can then clone the app and start editing it in Eclipse or other development environment.

We’ve put together four Java app templates that represent the most popular and most exciting ways to build Java apps today. But Heroku is not limited to these frameworks. Heroku runs 100% open-standards Java, so you can run any Java application and use any framework. You can even use no framework at all, such as running stand-alone Java processes that are not web applications.

Deploy your first Java template app on Heroku today!

Deploy Grails Applications on Heroku

We’re happy to announce the public beta of Grails application deployment on Heroku with support for Grails 1.3.7 and 2.0 provided by the open source Heroku Grails buildpack.

Grails is a high-productivity web application framework for the JVM based on the Groovy programming language and featuring many similarities with Rails. Since its inception in 2006, the framework has enjoyed broad adoption in the Java community as it combines the strengths of the JVM and richness of the Java platform with the productivity benefits of modern frameworks like Rails.

Today the Grails team announced Grails 2.0, the latest incarnation of the framework. It features numerous large improvements including an overhauled command line tool, faster and more reliable reloads, and static asset support. Details are covered in the What’s New section of the Grails docs.

The release includes the Grails Heroku plugin that provides simple commands to set up your Grails app with Heroku add-on services like Postgres, Memcached, Redis, MongoDB from MongoLabs or MongoHQ and RabbitMQ.

Deploying a Grails app on Heroku

Create a new Grails project:

$ grails createApp HelloWorld
| Created Grails Application at /Users/jjoergensen/dev/HelloWorld
$ cd HelloWorld

Commit to Git:

$ grails integrate-with --git
$ git init
Initialized empty Git repository in /Users/jjoergensen/dev/HelloWorld/.git/
$ git add .
$ git commit -m init
[master (root-commit) bd0f36b] init
 58 files changed, 2788 insertions(+), 0 deletions(-)
 create mode 100644 .classpath
 create mode 100644 .gitignore
 create mode 100644 .project
...

Create Heroku Cedar app:

$ heroku create --stack cedar
Creating smooth-night-8061... done, stack is cedar
http://smooth-night-8061.herokuapp.com/ | git@heroku.com:smooth-night-8061.git
Git remote heroku added

Deploy to Heroku:

$ git push heroku master
Counting objects: 73, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (69/69), done.
Writing objects: 100% (73/73), 97.82 KiB, done.
Total 73 (delta 2), reused 0 (delta 0)

-----> Heroku receiving push
-----> Grails app detected
-----> Grails 2.0.0 app detected
-----> Installing Grails 2.0.0..... done
-----> executing grails -plain-output -Divy.default.ivy.user.dir=/app/tmp/repo.git/.cache war

       |Loading Grails 2.0.0.
       ...
       |Done creating WAR target/HelloWorld-0.1.war
-----> No server directory found. Adding jetty-runner 7.5.4.v20111024 automatically.
-----> Discovering process types
       Procfile declares types  -> (none)
       Default types for Grails -> web
-----> Compiled slug size is 30.6MB
-----> Launching... done, v3
       http://smooth-night-8061.herokuapp.com deployed to Heroku

Learn More