Extend the Web Forward

If we want to move the web forward, we must increase our ability as web developers to extend it with new features.

For years, we’ve grabbed the browsers extension points with two hands, not waiting for the browser vendors to gift us with new features. We built selector engines, a better DOM API, cross-domain requests, cross-frame APIs.

When the browser has good extension points (or any extension points, really), we live in a virtuous cycle:

  • Web developers build new APIs ourselves, based on use-cases we have
  • We compete with each other, refining our libraries to meet use cases we didn’t think of
  • The process of competition makes the libraries converge towards each other, focusing the competition on sharp use-case distinctions
  • Common primitives emerge, which browser vendors can implement. This improves performance and shrinks the amount of library code necessary.
  • Rinse, repeat.

We’ve seen this time and time again. When it works, it brings us querySelectorAll, the template element, and Object.observe.

The Sad Truth

The sad truth is that while some areas of the browser are extremely extensible, other areas are nearly impossible to extend.

Some examples include the behavior and lifecycle of custom element in HTML, the CSS syntax, and the way that the browser loads an HTML document in the first place. This makes it hard to extend HTML, CSS, or build libraries that support interesting offline capabilities.

And even in some places that support extensibility, library developers have to completely rewrite systems that already exist. For example, John Resig had to rewrite the selector engine from scratch just to add a few additional pseudo-properties, and there is still no way add custom pseudo-properties to querySelectorAll.

Declarative vs. Imperative

A lot of people see this as a desire to write everything using low-level JavaScript APIs, forever.

No.

If things are working well, JavaScript library authors write new declarative APIs that the browser can roll in. Nobody wants to write everything using low-level calls to canvas, but we’re happy that canvas lets us express low-level things that we can evolve and refine.

The alternative, that web authors are stuck with only the declarative APIs that standards bodies have invented, is too limiting, and breaks the virtuous cycle that allows web developers to invent and iterate on new high-level features for the browser.

In short, we want to extend the web forward with new high-level APIs, but that means we need extension points we can use.

Explaining the Magic

If we want to let web authors extend the web forward, the best way to do that is to explain existing and new high-level forms in terms of low-level APIs.

A good example of in-progress work along these lines in Web Components, which explains how elements work in terms of APIs that are exposed to JavaScript. This means that if a new custom element becomes popular, it’s a short hop to implementing it natively, because the JavaScript implementation is not a parallel universe; it’s implemented in terms of the same concepts as native elements.

That doesn’t necessarily mean that browsers will simply rubber-stamp popular components, but by giving library authors the the tools to make components with native-like interfaces, it will be easy for vendors to synthesize web developer zeitgeist into something standard.

Another example is offline support. Right now, we have the much-derided AppCache, which is a declarative-only API that makes it possible to display an HTML page, along with its assets, even if the browser is offline.

AppCache is not implemented in terms of a low-level JavaScript API, so when web developers discovered problems with it, we had no way to extend or modify it to meet our needs. This also meant that we had no way to show the browser vendors what kinds of solutions would work for us.

Instead, we ended up with years of stagnation, philosophical disagreements and deadlock between web developers and specification editors, and no way to move forward.

What we need instead is something like Alex Russell’s proposal that allows applications to install JavaScript code in the cache that intercepts HTTP requests from the page and can fulfill them, even when the app is offline. With an API like this, the current AppCache could be written as a library!

Something like Jonas Sicking’s app cache manifest is a great companion proposal, giving us a nice starting point for a high-level API. But this time if the high-level API doesn’t work, we can fix it by using the low-level API to tweak and improve the manifest.

We can extend the web forward.

Extensions != Rewriting

It’s important to note that web developers don’t want a high level API and then a cliff into the low-level API.

Today, while you can implement custom elements or extend the selector engine, you can only do this by rewriting large chunks of the stack alongside the feature you want.

Real extensibility means an architecture that lets you tweak, not rewrite. For example, it would be possible to add custom rules to CSS by writing a full selector engine and application engine, and apply rules via .style as the DOM changes. With mutation observers, this might even be feasible. In fact, this is how some of the most devious hacks in the platform today (like the Polymer Shadow DOM polyfill) actually work.

That kind of “extensibility” doesn’t fit the bill. It doesn’t compose well with other extensions, defeats the browser’s ability to do performance work on unrelated parts of the stack (because the entire stack had to be rewritten), and is too hard to provide meaningful iteration.

Browser implementers are often wary of providing extension points that can be performance footguns. The biggest footgun is using libraries that rewrite the entire stack in JavaScript, and whole-stack-rewriting strategies are the tactic du jour today. For performance, we have little to lose and much to gain by making extensions more granular.

Extend the Web Forward

So what do we gain from a more extensible web? I’ll let Dave Herman, a member of TC39, answer that for me.

  • When you design new APIs, you are forced to think about how the existing system can express most of the semantics. This cleanly separates what new power is genuinely needed and what isn’t. This prevents cluttering the semantics with unnecessary new magic
  • Avoiding new magic avoids new security surface area
  • Avoiding new magic avoids new complexity (and therefore bugs) in implementation
  • Avoiding new magic makes more of the new APIs polyfillable
  • Being more polyfillable means people can ramp up faster, leading to faster adoption and evolution of the platform
  • Avoiding new magic means that optimizations in the engines can focus on the stable core, which affects more of new APIs as they are added. This leads to better performance with less implementation effort
  • Avoiding new magic means less developer education required; people can understand new APIs more easily when they come out, because they build off of known concepts
  • This means that the underlying platform gets fleshed out to be expressive enough to prototype new ideas. Library authors can experiment with new features and create more cowpaths to fill the Web API pipeline

All this, and more! There’s something for everybody!

Implementors and web developers: let’s work together to extend the web forward!

I’m Running to Reform the W3C’s TAG

Elections for the W3C’s Technical Architecture Group are underway, and I’m running!

There are nine candidates for four open seats. Among the nine candidates, Alex Russell, Anne van Kesteren, Peter Linss, and Marcos Cáceres are running on a reform platform. What is the TAG, and what do I mean by reform?

What is the TAG?

According to the TAG’s charter, it has several roles:

  • to document and build consensus around principles of Web architecture
  • to interpret and clarify these principles when necessary
  • to resolve issues involving general Web architecture brought to the TAG
  • to help coordinate cross-technology architecture developments inside and outside W3C

As Alex has said before, the existing web architecture needs reform that would make it more layered. We should be able to explain the declarative parts of the spec (like markup) in terms of lower level primitives that compose well and that developers can use for other purposes.

And the W3C must coordinate much more closely with TC39, the (very active) committee that is designing the future of JavaScript. As a member of both TC39 and the W3C, I believe that it is vital that as we build the future of the web platform, both organizations work closely together to ensure that the future is both structurally coherent and pleasant for developers of the web platform to use.

Developers

I am running as a full-time developer on the web platform to bring that perspective to the TAG.

For the past several years, I have lobbied for more developer involvement in the standards process through the jQuery organization. This year, the jQuery Foundation joined both the W3C and ECMA, giving web developers direct representatives in the consensus-building process of building the future.

Many web developers take a very cynical attitude towards the standards process, still burned from the flames of the first browser wars. As a group, web developers also have a very pragmatic perspective: because we can’t use new features in the short-term, it’s very costly to take an early interest in standards that aren’t even done yet.

Of course, as a group, we developers don’t hesitate to complain about standards that didn’t turn out the way we would like.

(The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHOULD”, “SHOULD NOT”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC2119.)

The W3C and its working groups MUST continue to evangelize to developers about the importance of participating early and often. We MUST help more developers understand the virtues of broad solutions and looking beyond specific present-day scenarios. And we MUST evolve to think of web developers not simply as “authors” of content, but as sophisticated developers on the most popular software development platform ever conceived.

Layering

When working with Tom Dale on Ember.js, we often joke that our APIs are layered, like a delicious cake.

What we mean by layering is that our high-level features are built upon publicly exposed lower-level primitives. This gives us the freedom to experiment with easy-to-use concise APIs, while making it possible for people with slightly different needs to still make use of our hard implementation work. In many cases, such as in our data abstraction, we have multiple layers, making it possible for people to implement their requirements at the appropriate level of abstraction.

It can be tempting to build primitives and leave it up to third parties to build the higher level APIs. It can also be tempting to build higher level APIs only for particular scenarios, to quickly solve a problem.

Both approaches are prevalent on the web platform. Specs like IndexedDB are built at a very low level of abstraction, leaving it up to library authors to build a higher level of abstraction. In contrast, features like App Cache are built at a high level of abstraction, for a particular use-case, with no lower level primitives to use if a user’s exact requirements do not match the assumptions of the specification.

Alex’s effort on this topic is focused on Web Components and Shadow DOM, an effort to explain the semantics of existing tags in terms of lower-level primitives. These primitives allow web developers to create new kinds of elements that can have a similar level of sophistication to the built-in elements. Eventually, it should be possible to describe how existing elements work in terms of these new primitives.

Here’s another example a layer deeper: many parts of the DOM API have magic behavior that are extremely difficult to explain in terms of the exposed API of ECMAScript 3. For example, the innerHTML property has side-effects, and ES3 does not provide a mechanism for declaring setters. The ECMAScript 5 specification provides some additional primitives that make it possible to explain more of the existing DOM behavior in terms of JavaScript. While designing ECMAScript 6, the committee has repeatedly discussed how certain new features could help explain more of the DOM API.

Today, the web platform inherits a large number of existing specifications designed at one of the ends of the layering spectrum. I would like to see the TAG make an explicit effort to describe how the working groups can reform existing APIs to have better layering semantics, and to encourage them to build new specifications with layering in mind.

TC39 and JavaScript

Today, developers of the web platform increasingly use JavaScript to develop full-blown applications that compete with their native counterparts.

This has led to a renaissance in JavaScript implementations, and more focus on the ECMAScript specification itself by TC39. It is important that the evolution of JavaScript and the DOM APIs take one another into consideration, so that developers perceive them as harmonious, rather than awkward and ungainly.

Any developer who has worked with NodeList and related APIs knows that the discrepancies between DOM Array-likes and JavaScript Arrays cause pain. Alex has talked before about how standardizing subclassing of built-in object would improve this situation. This would allow the W3C to explicitly subclass Array for its Array-like constructs in a well-understood, compatible way. That proposal will be strongest if it is championed by active members of both TC39 and the HTML working group.

Similarly, TC39 has worked tirelessly on a proposal for loading JavaScript in an environment-agnostic way (the “modules” proposal). That proposal, especially the aspects that could impact the network stack, would be stronger with the direct involvement of an interested member of a relevant W3C working group.

As the web’s development picks up pace, the W3C cannot see itself as an organization that interacts with ECMA at the periphery. It must see itself as a close partner with TC39 in the development and evolution of the web platform.

Progress

If that (and Alex’s similar post) sounds like progress to you, I’d appreciate your organization’s vote. My fellow reformers Alex Russell, Anne van Kesteren, Peter Linss and Marcos Cáceres are also running for reform.

AC reps for each organization can vote here and have 4 votes to allocate in this election. Voting closes near the end of the month, and it’s also holiday season, so if you work at a member organization and aren’t the AC rep, please, find out who that person in your organization is and make sure they vote.

As Alex said:

The TAG can’t fix the web or the W3C, but I believe that with the right people involved it can do a lot more to help the well-intentioned people who are hard at work in the WGs to build in smarter ways that pay all of us back in the long run.

Follow Me to Google+

I wrote my first post on this blog in January 2007.

In 2007, this blog was the easiest way I had to write my thoughts down for people who cared to read them. I wrote long posts and short post (but mostly long posts). I wrote deeply technical posts. I wrote proposals. I wrote introductory posts.

I did not post often.

In 2012, there are many more ways to write and reach an audience. I write whimsically on Twitter. I write personally on Facebook. More and more, I find that I write casually on Google+.

Without the 140-character constraint of Twitter, I can start writing and stop when I reach the end of a thought. Unlike the long-form nature of my blog, I find myself writing often, whenever something is on my mind. If you’re interested in reading that sort of thing, follow my Google+ profile. Because I never remember to include one person’s Google+ account in my reading rotation, I made it easy: plus.yehudakatz.com.

I’ll keep posting long-form pieces here. To keep things simple, I’ll always link to them from Google+. If you follow me there, I’ll make sure you always know when I post something, wherever that happens to be. If you care, join me on Google+.

Tokaido Status Update: Implementation Details

Hey guys!

Since my last update, Tokaido was fully funded, and I’ve been hard at work planning, researching and working on Tokaido.

So far, we have a working binary build of Ruby, but no setup chrome. Because the binary build already exists, Terence Lee was able to experiment with it at a recent Rails Girls event, with great success:

Great thanks to Terence to put together a simple installer script that we could use to test whether the core build worked on a wide variety of OSX systems.

One thing that I mentioned in my original proposal was a desire to work closely with others working on related projects. Very soon after my project was announced, I teamed up with Michal Papis of the rvm team to make the core statically compiled distribution something that would work outside of the GUI part of Tokaido.

We decided to use the sm scripting framework to build Tokaido, to make it easy to share code between rvm2, Tokaido, and the Unix Rails Installer. The majority of the work I have done so far has been in researching how to properly build a portable Ruby, and working with Michal to build the solution in terms of the sm framework. The rest of this blog post discusses the results of that research, for those interested.

The discussion in this blog post is specific to Mac OSX.

Portable Build

The hardest technical part of the project is creating a portable binary build of Ruby that can be moved around to various machines. What do I mean by that?

When you compile Ruby using the normal ./configure && make, the resulting binary is not portable between machines for a number of reasons.

Hard-Coded Paths

By default, the compiled Ruby comes with a binary .bundle file for each compiled part of the standard library. For example, Aaron Patterson wrote the psych library as a C library. When you compile Ruby, you get psych.bundle as part of the distribution. When a Ruby program invokes require "psych", the system’s dynamic loader will load in psych.bundle.

By default, Ruby hard-codes the path to the Ruby dynamic library (libruby.1.9.1.dylib) into psych.bundle. Since Psych uses C functions from Ruby, this path is used by the dynamic loader to make sure that the require Ruby dependency is available. We can use a tool called otool to see all of the dependencies for a particular library:

$ otool -L psych.bundle 
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.3.0/psych.bundle:
	/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby.1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1)
	/Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

The second line in the output references the libruby.1.9.1.dylib using an absolute path on my local machine. If I take these binaries and give them to you, the linker won’t be able to find libruby and won’t load Psych.

In addition to the problem with the compiled .bundle files, the compiler also hardcodes the paths in the Ruby binary itself:

$ otool -L `which ruby`
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/bin/ruby:
	/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby.1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

Finally, the location of the standard library is hardcoded into the Ruby binary:

$ strings /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby.1.9.1.dylib | grep rvm
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/x86_64-darwin11.3.0
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby/1.9.1
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin11.3.0
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.3.0

Fortunately, the C Ruby folks know about this problem, and include an (undocumented?) flag that you can pass to ./configure, --enable-load-relative. This flag fixes the problems with hardcoded paths:

Instead of creating a separate libruby.1.9.1.dylib that the ruby executable links to, this flag includes the compiled binary code inside of the ruby executable.

$ ./configure --enable-load-relative --prefix=/Users/wycats/Code/ruby/build
... snip ...
$ make && make install
... snip ...
$ otool -L build/bin/ruby                                                  
build/bin/ruby:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

You can see that Ruby still links against a few system dynamic libraries. These dynamic libraries are extremely stable in OSX, and aren’t a problem for binary distributions.

In order to enable compilation of native extensions, this build of Ruby distributes an archive file instead of a dylib. As we will see later, the OSX linker knows how to automatically handle this.

This flag also affects psych.bundle:

$ otool -L build/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle                      
build/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle:
	/usr/local/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

External Dependencies

In addition to the general problem of hardcoded paths, there’s another issue lurking in the above otool output for Psych. After eliminating the hardcoded path to a local Ruby, we are still left with a link to /usr/local/lib/libyaml-0.2.dylib. Unfortunately, libyaml doesn’t come with OSX, so if I take this distribution of Ruby and hand it off to a fresh system, Psych will fail to find libyaml at runtime and fail to load.

A number of the .bundle files that Ruby ships with have similar external dependencies. In general, these dependencies ship with OSX, but some, like openssl, may not last more than another release or two. In addition, the specific versions of these dependencies shipped with OSX may change over time, possibly resulting in different behavior on different systems.

In general, we can eliminate these problems by including the binaries we need into the .bundle files, instead of asking the operating system’s dynamic loader to find them at runtime.

The OSX linker’s (ld) behavior in this respect is interesting:

  • The linker starts with a list of paths to search for libraries
  • When compiling a program, it may need a particular dependency (psych needs libyaml)
  • It searches through the path for that library. Both libyaml.dylib and libyaml.a will suffice.
  • If the linker finds a .dylib first, it will dynamically link that dependency.
  • If the linker finds a .a first, it will statically link that dependency. By statically linking, we mean that it simply includes the binary into the outputted compiled file
  • If the linker finds a directory containing both a .a and a .dylib, it will dynamically link the dependency

In this case, our goal is to get the linker to statically link libyaml. In order to do this, we will need to build a libyaml.a and get the directory containing that file to the front of the linker’s path.

In the case of libyaml, getting a .a looks like this:

$ wget http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
... snip ...
$ tar -xzf yaml-0.1.4.tar.gz
$ cd yaml-0.1.4 
$ ./configure --disable-shared
... snip ...
$ make
... snip ...
$ otool -L src/.libs/libyaml.a
Archive : src/.libs/libyaml.a
src/.libs/libyaml.a(api.o):
src/.libs/libyaml.a(reader.o):
src/.libs/libyaml.a(scanner.o):
src/.libs/libyaml.a(parser.o):
src/.libs/libyaml.a(loader.o):
src/.libs/libyaml.a(writer.o):
src/.libs/libyaml.a(emitter.o):
src/.libs/libyaml.a(dumper.o):

We now have a libyaml.a. Note that the configure flag for getting a .a for a given library is not particularly standardized. Three popular ones: --static, --enable-static, --disable-shared.

Next, we need to move libyaml.a into a directory with any other .a files we want to use and pass them to the compilation process:

$ LDFLAGS="-L/Users/wycats/Code/ruby/deps" ./configure --enable-load-relative --prefix=/Users/wycats/Code/ruby/build
... snip ...
$ make && make install
... snip ...
$ otool -L build/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundlebuild/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

And voila! We now have a psych.bundle that does not depend on libyaml.dylib. Instead, libyaml is now included in psych.bundle itself. This moves us a step closer to having a portable Ruby build.

We will want to repeat this process for every part of the Ruby standard library with external dependencies (openssl, readline, and zlib are some others). Even though libyaml is the only library that does not ship with OSX, eliminating external dependencies on the operating system insulates our build from changes that Apple makes in the future. Of the dependencies, OpenSSL is the most problematic, as it has already been deprecated in Lion.

The sm Framework

This is where the sm (scripting management) framework comes into play. The goal of the sm framework is to encapsulate solutions to these concerns into reusable libraries. In particular, it abstracts the idea of downloading and compiling a package, and common requirements, like static compilation.

For example, let’s take a look at the libyaml library.

The first important file here is config/defaults:

version=0.1.4
base_url=http://pyyaml.org/download/libyaml
configure_flag_static=--disable-shared

This specifies the current version, the URL to download the tarball from, and importantly for us, the configure flag that libyaml expects in order to build a .a file. We added that third line because we needed it for Tokaido. This satisfies one of the major goals of the project: to get as much of the code as possible into shared code instead of code that is specific to Tokaido.

The other important file in the libyaml library is shell/functions:

#!/bin/sh
 
libyaml_prefetch()
{
  package define \
    file "yaml-${package_version}.${archive_format}" \
    dir "yaml-${package_version}"
}
 
libyaml_preconfigure()
{
  os is darwin || autoreconf -is --force > autoreconf.log 2>&1 ||
    __sm.package.error "Autoreconf of ${package_name} ${package_version} failed! " "$PWD/autoreconf.log"
}

The sm framework defines a series of steps that a package install goes through:

# preinstall
#
#   prefetch
# fetch
#   postgetch
#   preextract
# extract
#   prepatch
# patch
#   preconfigure
# configure
#   postconfigure
#   prebuild
# build
#   preinstall
# install
#   preactivate
# activate
#   postactivate
#
# postinstall

The indented functions above are user-defined. Functions like fetch and configure are defined by sm.

In our case, the libyaml library defines two of those steps: prefetch and preconfigure. The prefetch function allows us to provide extra information to the fetch method, which specifically allows the prefetch to override the package_file (${package_file:="${package_name}-${package_version}.${archive_format}"}). In our case, even though the package name is libyaml, we want to download the file yaml-1.1.4.tar.gz.

The openssl library is somewhat more complicated. As with libyaml, we needed to teach sm how to install openssl statically. You can check out the commit to see how easy that was.

The great thing about getting this stuff into sm is that there is now a shared resource to answer questions like “how do you statically build openssl”. The work I did with Michal to improve the information for the libaries that Ruby depends on can now be used by anyone else trying to build Ruby (or anything else with those dependencies for that matter).

Tokaido is an sm library

Tokaido itself is an sm library! This means that the core Tokaido build will work on Linux, so it can be used to create a standalone distribution of Ruby for Linux, and maybe even the core of a Tokaido for Linux!

The Tokaido package implements a lot of the sm hooks, so check it out to learn more about what you can do using sm’s package API.

In my next post, I’ll talk about the architecture of the Tokaido UI component.

Tokaido Status Update: Implementation Details

Hey guys!

Since my last update, Tokaido was fully funded, and I’ve been hard at work planning, researching and working on Tokaido.

So far, we have a working binary build of Ruby, but no setup chrome. Because the binary build already exists, Terence Lee was able to experiment with it at a recent Rails Girls event, with great success:

Great thanks to Terence to put together a simple installer script that we could use to test whether the core build worked on a wide variety of OSX systems.

One thing that I mentioned in my original proposal was a desire to work closely with others working on related projects. Very soon after my project was announced, I teamed up with Michal Papis of the rvm team to make the core statically compiled distribution something that would work outside of the GUI part of Tokaido.

We decided to use the sm scripting framework to build Tokaido, to make it easy to share code between rvm2, Tokaido, and the Unix Rails Installer. The majority of the work I have done so far has been in researching how to properly build a portable Ruby, and working with Michal to build the solution in terms of the sm framework. The rest of this blog post discusses the results of that research, for those interested.

The discussion in this blog post is specific to Mac OSX.

Portable Build

The hardest technical part of the project is creating a portable binary build of Ruby that can be moved around to various machines. What do I mean by that?

When you compile Ruby using the normal ./configure && make, the resulting binary is not portable between machines for a number of reasons.

Hard-Coded Paths

By default, the compiled Ruby comes with a binary .bundle file for each compiled part of the standard library. For example, Aaron Patterson wrote the psych library as a C library. When you compile Ruby, you get psych.bundle as part of the distribution. When a Ruby program invokes require "psych", the system’s dynamic loader will load in psych.bundle.

By default, Ruby hard-codes the path to the Ruby dynamic library (libruby.1.9.1.dylib) into psych.bundle. Since Psych uses C functions from Ruby, this path is used by the dynamic loader to make sure that the require Ruby dependency is available. We can use a tool called otool to see all of the dependencies for a particular library:

$ otool -L psych.bundle 
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.3.0/psych.bundle:
    /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby.1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1)
    /Users/wycats/.rvm/usr/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

The second line in the output references the libruby.1.9.1.dylib using an absolute path on my local machine. If I take these binaries and give them to you, the linker won’t be able to find libruby and won’t load Psych.

In addition to the problem with the compiled .bundle files, the compiler also hardcodes the paths in the Ruby binary itself:

$ otool -L `which ruby`
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/bin/ruby:
    /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby.1.9.1.dylib (compatibility version 1.9.1, current version 1.9.1)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

Finally, the location of the standard library is hardcoded into the Ruby binary:

$ strings /Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/libruby.1.9.1.dylib | grep rvm
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/x86_64-darwin11.3.0
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby/1.9.1
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin11.3.0
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1
/Users/wycats/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.3.0

Fortunately, the C Ruby folks know about this problem, and include an (undocumented?) flag that you can pass to ./configure, --enable-load-relative. This flag fixes the problems with hardcoded paths:

Instead of creating a separate libruby.1.9.1.dylib that the ruby executable links to, this flag includes the compiled binary code inside of the ruby executable.

$ ./configure --enable-load-relative --prefix=/Users/wycats/Code/ruby/build
... snip ...
$ make && make install
... snip ...
$ otool -L build/bin/ruby                                                  
build/bin/ruby:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

You can see that Ruby still links against a few system dynamic libraries. These dynamic libraries are extremely stable in OSX, and aren’t a problem for binary distributions.

In order to enable compilation of native extensions, this build of Ruby distributes an archive file instead of a dylib. As we will see later, the OSX linker knows how to automatically handle this.

This flag also affects psych.bundle:

$ otool -L build/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle                      
build/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle:
    /usr/local/lib/libyaml-0.2.dylib (compatibility version 3.0.0, current version 3.2.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

External Dependencies

In addition to the general problem of hardcoded paths, there’s another issue lurking in the above otool output for Psych. After eliminating the hardcoded path to a local Ruby, we are still left with a link to /usr/local/lib/libyaml-0.2.dylib. Unfortunately, libyaml doesn’t come with OSX, so if I take this distribution of Ruby and hand it off to a fresh system, Psych will fail to find libyaml at runtime and fail to load.

A number of the .bundle files that Ruby ships with have similar external dependencies. In general, these dependencies ship with OSX, but some, like openssl, may not last more than another release or two. In addition, the specific versions of these dependencies shipped with OSX may change over time, possibly resulting in different behavior on different systems.

In general, we can eliminate these problems by including the binaries we need into the .bundle files, instead of asking the operating system’s dynamic loader to find them at runtime.

The OSX linker’s (ld) behavior in this respect is interesting:

  • The linker starts with a list of paths to search for libraries
  • When compiling a program, it may need a particular dependency (psych needs libyaml)
  • It searches through the path for that library. Both libyaml.dylib and libyaml.a will suffice.
  • If the linker finds a .dylib first, it will dynamically link that dependency.
  • If the linker finds a .a first, it will statically link that dependency. By statically linking, we mean that it simply includes the binary into the outputted compiled file
  • If the linker finds a directory containing both a .a and a .dylib, it will dynamically link the dependency

In this case, our goal is to get the linker to statically link libyaml. In order to do this, we will need to build a libyaml.a and get the directory containing that file to the front of the linker’s path.

In the case of libyaml, getting a .a looks like this:

$ wget http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
... snip ...
$ tar -xzf yaml-0.1.4.tar.gz
$ cd yaml-0.1.4 
$ ./configure --disable-shared
... snip ...
$ make
... snip ...
$ otool -L src/.libs/libyaml.a
Archive : src/.libs/libyaml.a
src/.libs/libyaml.a(api.o):
src/.libs/libyaml.a(reader.o):
src/.libs/libyaml.a(scanner.o):
src/.libs/libyaml.a(parser.o):
src/.libs/libyaml.a(loader.o):
src/.libs/libyaml.a(writer.o):
src/.libs/libyaml.a(emitter.o):
src/.libs/libyaml.a(dumper.o):

We now have a libyaml.a. Note that the configure flag for getting a .a for a given library is not particularly standardized. Three popular ones: --static, --enable-static, --disable-shared.

Next, we need to move libyaml.a into a directory with any other .a files we want to use and pass them to the compilation process:

$ LDFLAGS="-L/Users/wycats/Code/ruby/deps" ./configure --enable-load-relative --prefix=/Users/wycats/Code/ruby/build
... snip ...
$ make && make install
... snip ...
$ otool -L build/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundlebuild/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

And voila! We now have a psych.bundle that does not depend on libyaml.dylib. Instead, libyaml is now included in psych.bundle itself. This moves us a step closer to having a portable Ruby build.

We will want to repeat this process for every part of the Ruby standard library with external dependencies (openssl, readline, and zlib are some others). Even though libyaml is the only library that does not ship with OSX, eliminating external dependencies on the operating system insulates our build from changes that Apple makes in the future. Of the dependencies, OpenSSL is the most problematic, as it has already been deprecated in Lion.

The sm Framework

This is where the sm (scripting management) framework comes into play. The goal of the sm framework is to encapsulate solutions to these concerns into reusable libraries. In particular, it abstracts the idea of downloading and compiling a package, and common requirements, like static compilation.

For example, let’s take a look at the libyaml library.

The first important file here is config/defaults:

version=0.1.4
base_url=http://pyyaml.org/download/libyaml
configure_flag_static=--disable-shared

This specifies the current version, the URL to download the tarball from, and importantly for us, the configure flag that libyaml expects in order to build a .a file. We added that third line because we needed it for Tokaido. This satisfies one of the major goals of the project: to get as much of the code as possible into shared code instead of code that is specific to Tokaido.

The other important file in the libyaml library is shell/functions:

#!/bin/sh
 
libyaml_prefetch()
{
  package define \
    file "yaml-${package_version}.${archive_format}" \
    dir "yaml-${package_version}"
}
 
libyaml_preconfigure()
{
  os is darwin || autoreconf -is --force > autoreconf.log 2>&1 ||
    __sm.package.error "Autoreconf of ${package_name} ${package_version} failed! " "$PWD/autoreconf.log"
}

The sm framework defines a series of steps that a package install goes through:

# preinstall
#
#   prefetch
# fetch
#   postgetch
#   preextract
# extract
#   prepatch
# patch
#   preconfigure
# configure
#   postconfigure
#   prebuild
# build
#   preinstall
# install
#   preactivate
# activate
#   postactivate
#
# postinstall

The indented functions above are user-defined. Functions like fetch and configure are defined by sm.

In our case, the libyaml library defines two of those steps: prefetch and preconfigure. The prefetch function allows us to provide extra information to the fetch method, which specifically allows the prefetch to override the package_file (${package_file:="${package_name}-${package_version}.${archive_format}"}). In our case, even though the package name is libyaml, we want to download the file yaml-1.1.4.tar.gz.

The openssl library is somewhat more complicated. As with libyaml, we needed to teach sm how to install openssl statically. You can check out the commit to see how easy that was.

The great thing about getting this stuff into sm is that there is now a shared resource to answer questions like “how do you statically build openssl”. The work I did with Michal to improve the information for the libaries that Ruby depends on can now be used by anyone else trying to build Ruby (or anything else with those dependencies for that matter).

Tokaido is an sm library

Tokaido itself is an sm library! This means that the core Tokaido build will work on Linux, so it can be used to create a standalone distribution of Ruby for Linux, and maybe even the core of a Tokaido for Linux!

The Tokaido package implements a lot of the sm hooks, so check it out to learn more about what you can do using sm’s package API.

In my next post, I’ll talk about the architecture of the Tokaido UI component.

Tokaido: My Hopes and Dreams

A few weeks ago, I started a kickstarter project to fund work on a project to make a long-term, sustainable binary build of Ruby. The outpouring of support was great, and I have far exceeded my original funding goal. First, I’d like to thank everyone in the community who contributed large and small donations. This kickstarter couldn’t have been as successful as it has been without the hundreds (650 at latest count!) of individual donations by interested Rubyists.

In this post, I want to talk about what my goals are for this project, and why I think it will be a tool that everyone, myself included, will use.

What is Tokaido

Precompiled, Static Ruby

At its core, Tokaido is a binary distribution of Ruby without any external dependencies on your system. This means that Ruby itself, as well as all of the compiled elements of the standard library, will come in a self-contained directory with no additional requirements.

The binary will also have no hardcoded paths, so it will be possible to move the directory anywhere on the file system and have it continue to work as long as its bin directory is on the $PATH. This is an important goal, as Tokaido will ship as a downloadable .app which should work wherever it is dropped on the file system.

Precompiled Binary Gems

Tokaido will come with all of the necessary gems to use Rails and common Rails extensions. Because some of these gems have external dependencies (for example, nokogori depends on libxml2), Tokaido will come with precompiled versions of these gems built for OSX that do not depend on external libraries (they will be statically compiled, not dynamically linked).

As part of this project, I plan to work with people who ship common native extensions to help them build and ship binary versions of their gems for OSX without external dependencies. Luis Lavena has been doing amazing work with rake-compiler to make this process easy on gem developers, and Wayne Seguin and Michał Papis have been doing great work with sm, which makes it easy to precompile the needed dependencies for inclusion in the precompiled gems. These tools will be essential in the effort to make dependency-free precompiled gems a standard part of the OSX Ruby ecosystem.

I anticipate that gem authors will, in general, start distributing precompiled binary versions of their gems. If, by the time I ship the first version of Tokaido, some important gems do not yet ship as precompiled binaries, Tokaido will bootstrap the process by including the binaries.

Terminal-Based Workflow

Tokaido does not aim to replace the Terminal as the main way to work with a Rails app. It will ship an isolated environment with no external dependencies designed for use in the Terminal. The application UI will supplement, rather than replace, the normal application workflow. This is a crucial part of the overall goal to make Tokaido an excellent tool for both experienced Rails developers, intermediate Rails developers, and totally new Rails developers.

Because many gems come with executables, and Tokaido couldn’t abstract every possible executable even if it wanted to, it is essential that new developers get used to using the Terminal as early as possible in their Rails experience, but in a way that minimized unnecessary errors.

Extras: Code and App Health

A number of really great tools exist to help Rails applications remain healthy:

  • bundle outdated lets you know when new versions of gems are available
  • rails_best_practices helps find common mistakes in Rails applications
  • reek, flog, flay, roodi and other metrics tools help identify Ruby smells and prioritizes areas where you can reduce your technical debt
  • simplecov does coverage analysis on your test suite to identify areas lacking good coverage
  • bullet identifies N+1 queries and unused eager loading
  • ActiveSupport::Notifications provides introspection information about Rails applications

The Tokaido UI will attempt to centralize this useful information into a health check, and help you drill in if you want to do more with the information. Because they are so useful, it will encourage you to use these tools in new application, and provide immediate rewards if you do.

Extras: yourapp.dev

The pow dev server and PassengerPane make it possible to avoid needing to manually load a Rails server and gives you a friendly alias for your application at yourapp.dev instead of having to type in a port on localhost.

Tokaido will come with integrated support for this, so every app you manage through Tokaido will have a aliased development server out of the box.

Extras: rails:// (or ruby://) Protocol Handler

Tokaido will come with a protocol handler that will allow .gem files hosted on rubygems.org or other locations to be added to an app managed by Tokaido. It may also allow web sites to host installation templates that could execute against a maintained application. This will require coordination with rubygems.org, and the specifics of it may change over time, but the basic idea is to enable communication between web sites and the Tokaido app.

This idea came from a number of community members, and the specifics (especially around security and the protocol specification) definitely need fleshing out, but it seems extremely promising.

Extras: Ruby Toolbox Integration?

TODO

Goals of Tokaido

Eliminate Failure Scenarios

The primary goal of Tokaido is to build a distribution of Ruby that eliminates the main failure scenarios that people encounter when trying to install Ruby. In general, things fail because of conflicts with the existing system environment (or the user’s environment). These failures do not happen to everyone, but when they happen, they are extremely difficult to recover from. This has happened to me personally and to people I have tried to get started with Ruby.

This sort of thing does not necessarily happen on every installation, but once you start going down the rabbit hole, it can be very difficult to find your way out. The environment difficulties can be caused by anything from an old MacPorts installation to a mistaken attempt to install something to the system once upon a time to something failing during a previous step in the installation process.

It also may not be enough to install a precompiled Ruby into the system, because the system environment itself may be corrupted with things like erroneous executables on the $PATH or bad dynamic libraries that get linked during native compilation. Also, later installations may do damage to the system-installed Ruby. Tokaido is a standalone environment that is loaded on top of an existing session, and therefore minimizes the possible damage that load order or subsequent installs can do.

Precompile Everything

In order to eliminate a certain class of failure scenarios, Tokaido will ship with a precompiled Ruby, which will eliminate the possibility of compilation errors when installing Ruby. This precompiled Ruby will also come with all of the dependencies it needs, like zlib, yaml and others, instead of allowing the system to try to find them at runtime. This will eliminate the possibility that changes to the system environment will cause a normally working version of Ruby to fail in some scenarios.

As above, Tokaido will also use this technique for commonly used native gems. For example, Tokaido will ship with a precompiled version of Nokogiri that comes with libxml, instead of relying on the system’s copy of libxml (incidentally, the system’s libxml has occasionally been subtly broken, necessitating installation via homebrew). I expect that this will happen because

Use Tokaido Myself

Since Tokaido does not fundamentally alter a developer’s relationship with Ruby, I expect to be able to start using it for day-to-day Rails development. Some of the additional extras, like app health and built-in server support, are things I already do manually and will enjoy having as part of a larger tool. I’m also extremely interested in ideas for other extras that can add value for experienced developers without altering how people work with Ruby today.

Integration Testing

One of the coolest, unheralded things about the Rails project is the integration suite for Rails put together by Sam Ruby. It essentially transcribes the Agile Web Development With Rails book into a series of very complete integration tests for Rails. When refactoring Rails for Rails 3, this suite was extremely useful, and helped keep the number of unintentional changes to Rails to a tiny number. It also kept us honest about intentional changes.

This suite is perfect for Tokaido, as it tests every aspect of Rails, which is itself touches a wide swath of Ruby itself. To start, Tokaido will use this suite for integration testing.

Collaboration

There are several other projects, notably rvm, trying to solve problems in a similar space. Especially when it comes to precompilation, there is no reason not to work together on the core infrastructure that powers these efforts. I have already started to work with Michał Papis, who also works on sm and rvm on shared work that can be useful for both projects. He has been teaching me all about the work that folks associated with rvm have done to simplify things like downloading and compiling libz.a, a prerequisite for precompiled Rubies.

I will also work with authors of native gems to start shipping precompiled binary gems for OSX. I have already started working with Aaron Patterson, who works on the sqlite and nokogiri gems, and have started reaching out to several others.

I am very interested in working together with anyone working on a similar area so that the tools we are all working on can be shared between projects. This is a core mission of the Tokaido project, and something that the extra funding I got will allow me to prioritize.

Migration to “System”

Also through Michał Papis, I learned about an awesome new (still experimental) feature in rvm called mount that will mount an external Ruby into the rvm system. I will make sure that the Tokaido ruby can be added to an rvm installation. This will mean that if someone wants to migrate from Tokaido to a more advanced rvm setup, they can take their Ruby environment with them with no trouble.

I would be open to other approaches to migrating a Tokaido environment to the system as well. The goal would be to seamlessly migrate an environment to the system without introducing opportunities for failure during that process.

Looking Forward

I’m really excited to the work I’ve been doing to prepare for this project, and looking forward to shipping a great environment for OSX Ruby users. I have also really enjoyed reaching out to others working in similar areas and the prospect of collaborating with so many smart people on a shared goal.

Thanks so much!

Tokaido: My Hopes and Dreams

A few weeks ago, I started a kickstarter project to fund work on a project to make a long-term, sustainable binary build of Ruby. The outpouring of support was great, and I have far exceeded my original funding goal. First, I’d like to thank everyone in the community who contributed large and small donations. This kickstarter couldn’t have been as successful as it has been without the hundreds (650 at latest count!) of individual donations by interested Rubyists.

In this post, I want to talk about what my goals are for this project, and why I think it will be a tool that everyone, myself included, will use.

What is Tokaido

The name “Tokaido” (東海道 in Japanese) comes from the Tōkaidō Shinkansen bullet train line in Japan.

Precompiled, Static Ruby

At its core, Tokaido is a binary distribution of Ruby without any external dependencies on your system. This means that Ruby itself, as well as all of the compiled elements of the standard library, will come in a self-contained directory with no additional requirements.

The binary will also have no hardcoded paths, so it will be possible to move the directory anywhere on the file system and have it continue to work as long as its bin directory is on the $PATH. This is an important goal, as Tokaido will ship as a downloadable .app which should work wherever it is dropped on the file system.

Precompiled Binary Gems

Tokaido will come with all of the necessary gems to use Rails and common Rails extensions. Because some of these gems have external dependencies (for example, nokogori depends on libxml2), Tokaido will come with precompiled versions of these gems built for OSX that do not depend on external libraries (they will be statically compiled, not dynamically linked).

As part of this project, I plan to work with people who ship common native extensions to help them build and ship binary versions of their gems for OSX without external dependencies. Luis Lavena has been doing amazing work with rake-compiler to make this process easy on gem developers, and Wayne E. Seguin and Michał Papis have been doing great work with sm, which makes it easy to precompile the needed dependencies for inclusion in the precompiled gems. These tools will be essential in the effort to make dependency-free precompiled gems a standard part of the OSX Ruby ecosystem.

I anticipate that gem authors will, in general, start distributing precompiled binary versions of their gems. If, by the time I ship the first version of Tokaido, some important gems do not yet ship as precompiled binaries, Tokaido will bootstrap the process by including the binaries.

Terminal-Based Workflow

Tokaido does not aim to replace the Terminal as the main way to work with a Rails app. It will ship an isolated environment with no external dependencies designed for use in the Terminal. The application UI will supplement, rather than replace, the normal application workflow. This is a crucial part of the overall goal to make Tokaido an excellent tool for both experienced Rails developers, intermediate Rails developers, and totally new Rails developers.

Because many gems come with executables, and Tokaido couldn’t abstract every possible executable even if it wanted to, it is essential that new developers get used to using the Terminal as early as possible in their Rails experience, but in a way that minimizes unnecessary errors.

Extras: Code and App Health

A number of really great tools exist to help Rails applications remain healthy:

  • bundle outdated lets you know when new versions of gems are available
  • rails_best_practices helps find common mistakes in Rails applications
  • reek, flog, flay, roodi and other metrics tools help identify Ruby smells and prioritizes areas where you can reduce your technical debt
  • simplecov does coverage analysis on your test suite to identify areas lacking good coverage
  • bullet identifies N+1 queries and unused eager loading
  • ActiveSupport::Notifications provides introspection information about Rails applications

The Tokaido UI will attempt to centralize this useful information into a health check, and help you drill in if you want to do more with the information. Because they are so useful, it will encourage you to use these tools in new application, and provide immediate rewards if you do.

Extras: yourapp.dev

The pow dev server and PassengerPane make it possible to avoid needing to manually load a Rails server and gives you a friendly alias for your application at yourapp.dev instead of having to type in a port on localhost.

Tokaido will come with integrated support for this, so every app you manage through Tokaido will have a aliased development server out of the box.

Extras: rails:// (or ruby://) Protocol Handler

Tokaido will come with a protocol handler that will allow .gem files hosted on rubygems.org or other locations to be added to an app managed by Tokaido. It may also allow web sites to host installation templates that could execute against a maintained application. This will require coordination with rubygems.org, and the specifics of it may change over time, but the basic idea is to enable communication between web sites and the Tokaido app.

This idea came from a number of community members, and the specifics (especially around security and the protocol specification) definitely need fleshing out, but it seems extremely promising.

Extras: Ruby Toolbox Integration?

Ruby Toolbox has emerged as an amazing directory of Ruby libraries, categorized by type. It also has up-to-date information on Github activity and other useful indicators when evaluating tools. Several people have asked for integration with Ruby Toolbox, and assuming the author is willing, Tokaido will make it easy to use the information in Ruby Toolbox to bootstrap an app and add new functionality as your app grows.

Finding the right gem for the job is easy if you know what you’re looking for, but even experienced developers find the Ruby Toolbox useful when researching tools in an unfamiliar area.

Goals of Tokaido

Eliminate Failure Scenarios

The primary goal of Tokaido is to build a distribution of Ruby that eliminates the main failure scenarios that people encounter when trying to install Ruby. In general, things fail because of conflicts with the existing system environment (or the user’s environment). These failures do not happen to everyone, but when they happen, they are extremely difficult to recover from. This has happened to me personally and to people I have tried to get started with Ruby.

This sort of thing does not necessarily happen on every installation, but once you start going down the rabbit hole, it can be very difficult to find your way out. The environment difficulties can be caused by anything from an old MacPorts installation to a mistaken attempt to install something to the system once upon a time to something failing during a previous step in the installation process.

It also may not be enough to install a precompiled Ruby into the system, because the system environment itself may be corrupted with things like erroneous executables on the $PATH or bad dynamic libraries that get linked during native compilation. Also, later installations may do damage to the system-installed Ruby. Tokaido is a standalone environment that is loaded on top of an existing session, and therefore minimizes the possible damage that load order or subsequent installs can do.

Precompile Everything

In order to eliminate a certain class of failure scenarios, Tokaido will ship with a precompiled Ruby, which will eliminate the possibility of compilation errors when installing Ruby. This precompiled Ruby will also come with all of the dependencies it needs, like zlib, yaml and others, instead of allowing the system to try to find them at runtime. This will eliminate the possibility that changes to the system environment will cause a normally working version of Ruby to fail in some scenarios.

As above, Tokaido will also use this technique for commonly used native gems. For example, Tokaido will ship with a precompiled version of Nokogiri that comes with libxml, instead of relying on the system’s copy of libxml (incidentally, the system’s libxml has occasionally been subtly broken, necessitating installation via homebrew). I expect that this will happen because gem authors will start shipping precompiled versions of their gems for OSX. If there are still a few common gems straggling by the time Tokaido ships, we’ll bootstrap the process by shipping with binary gems we compile ourselves.

Use Tokaido Myself

Since Tokaido does not fundamentally alter a developer’s relationship with Ruby, I expect to be able to start using it for day-to-day Rails development. Some of the additional extras, like app health and built-in server support, are things I already do manually and will enjoy having as part of a larger tool. I’m also extremely interested in ideas for other extras that can add value for experienced developers without altering how people work with Ruby today.

Integration Testing

One of the coolest, unheralded things about the Rails project is the integration suite for Rails put together by Sam Ruby. It essentially transcribes the Agile Web Development With Rails book into a series of very complete integration tests for Rails. When refactoring Rails for Rails 3, this suite was extremely useful, and helped keep the number of unintentional changes to Rails to a tiny number. It also kept us honest about intentional changes.

This suite is perfect for Tokaido, as it tests every aspect of Rails, which is itself touches a wide swath of Ruby itself. To start, Tokaido will use this suite for integration testing.

Collaboration

There are several other projects, notably rvm, trying to solve problems in a similar space. Especially when it comes to precompilation, there is no reason not to work together on the core infrastructure that powers these efforts. I have already started to work with Michał Papis, who also works on sm and rvm on shared work that can be useful for both projects. He has been teaching me all about the work that folks associated with rvm have done to simplify things like downloading and compiling libz.a, a prerequisite for precompiled Rubies.

I will also work with authors of native gems to start shipping precompiled binary gems for OSX. I have already started working with Aaron Patterson, who works on the sqlite and nokogiri gems, and have started reaching out to several others.

I am very interested in working together with anyone working on a similar area so that the tools we are all working on can be shared between projects. This is a core mission of the Tokaido project, and something that the extra funding I got will allow me to prioritize.

Migration to “System”

Also through Michał Papis, I learned about an awesome new (still experimental) feature in rvm called mount that will mount an external Ruby into the rvm system. I will make sure that the Tokaido ruby can be added to an rvm installation. This will mean that if someone wants to migrate from Tokaido to a more advanced rvm setup, they can take their Ruby environment with them with no trouble.

I would be open to other approaches to migrating a Tokaido environment to the system as well. The goal would be to seamlessly migrate an environment to the system without introducing opportunities for failure during that process.

Looking Forward

I’m really excited to the work I’ve been doing to prepare for this project, and looking forward to shipping a great environment for OSX Ruby users. I have also really enjoyed reaching out to others working in similar areas and the prospect of collaborating with so many smart people on a shared goal.

Thanks so much!

JavaScript Needs Blocks

While reading Hacker News posts about JavaScript, I often come across the misconception that Ruby’s blocks are essentially equivalent to JavaScript’s “first class functions”. Because the ability to pass functions around, especially when you can create them anonymously, is extremely powerful, the fact that both JavaScript and Ruby have a mechanism to do so makes it natural to assume equivalence.

In fact, when people talk about why Ruby’s blocks are different from Python‘s functions, they usually talk about anonymity, something that Ruby and JavaScript share, but Python does not have. At first glance, a Ruby block is an “anonymous function” (or colloquially, a “closure”) just as a JavaScript function is one.

This impression, which I admittedly shared in my early days as a Ruby/JavaScript developer, misses an important subtlety that turns out to have large implications. This subtlety is often referred to as “Tennet’s Correspondence Principle”. In short, Tennet’s Correspondence Principle says:

“For a given expression expr, lambda expr should be equivalent.”

This is also known as the principle of abstraction, because it means that it is easy to refactor common code into methods that take a block. For instance, consider the common case of file resource management. Imagine that the block form of File.open didn’t exist in Ruby, and you saw a lot of the following in your code:

begin
  f = File.open(filename, "r")
  # do something with f
ensure
  f.close
end

In general, when you see some code that has the same beginning and end, but a different middle, it is natural to refactor it into a method that takes a block. You would write a method like this:

def read_file(filename)
  f = File.open(filename, "r")
  yield f
ensure
  f.close
end

And you’d refactor instances of the pattern in your code with:

read_file(filename) do |f|
  # do something with f
end

In order for this strategy to work, it’s important that the code inside the block look the same after refactoring as before. We can restate the correspondence principle in this case as:

# do something with f

should be equivalent to:

do
  # do something with
end

At first glance, it looks like this is true in Ruby and JavaScript. For instance, let’s say that what you’re doing with the file is printing its mtime. You can easily refactor the equivalent in JavaScript:

try {
  // imaginary JS file API
  var f = File.open(filename, "r");
  sys.print(f.mtime);
} finally {
  f.close();
}

Into this:

read_file(function(f) {
  sys.print(f.mtime);
});

In fact, cases like this, which are in fact quite elegant, give people the mistaken impression that Ruby and JavaScript have a roughly equivalent ability to refactor common functionality into anonymous functions.

However, consider a slightly more complicated example, first in Ruby. We’ll write a simple class that calculates a File’s mtime and retrieves its body:

class FileInfo
  def initialize(filename)
    @name = filename
  end
 
  # calculate the File's +mtime+
  def mtime
    f = File.open(@name, "r")
    mtime = mtime_for(f)
    return "too old" if mtime < (Time.now - 1000)
    puts "recent!"
    mtime
  ensure
    f.close
  end
 
  # retrieve that file's +body+
  def body
    f = File.open(@name, "r")
    f.read
  ensure
    f.close
  end
 
  # a helper method to retrieve the mtime of a file
  def mtime_for(f)
    File.mtime(f)
  end
end

We can easily refactor this code using blocks:

class FileInfo
  def initialize(filename)
    @name = filename
  end
 
  # refactor the common file management code into a method
  # that takes a block
  def mtime
    with_file do |f|
      mtime = mtime_for(f)
      return "too old" if mtime < (Time.now - 1000)
      puts "recent!"
      mtime
    end
  end
 
  def body
    with_file { |f| f.read }
  end
 
  def mtime_for(f)
    File.mtime(f)
  end
 
private
  # this method opens a file, calls a block with it, and
  # ensures that the file is closed once the block has
  # finished executing.
  def with_file
    f = File.open(@name, "r")
    yield f
  ensure
    f.close
  end
end

Again, the important thing to note here is that we could move the code into a block without changing it. Unfortunately, this same case does not work in JavaScript. Let’s first write the equivalent FileInfo class in JavaScript.

// constructor for the FileInfo class
FileInfo = function(filename) {
  this.name = filename;
};
 
FileInfo.prototype = {
  // retrieve the file's mtime
  mtime: function() {
    try {
      var f = File.open(this.name, "r");
      var mtime = this.mtimeFor(f);
      if (mtime < new Date() - 1000) {
        return "too old";
      }
      sys.print(mtime);
    } finally {
      f.close();
    }
  },
 
  // retrieve the file's body
  body: function() {
    try {
      var f = File.open(this.name, "r");
      return f.read();
    } finally {
      f.close();
    }
  },
 
  // a helper method to retrieve the mtime of a file
  mtimeFor: function(f) {
    return File.mtime(f);
  }
};

If we try to convert the repeated code into a method that takes a function, the mtime method will look something like:

function() {
  // refactor the common file management code into a method
  // that takes a block
  this.withFile(function(f) {
    var mtime = this.mtimeFor(f);
    if (mtime < new Date() - 1000) {
      return "too old";
    }
    sys.print(mtime);
  });
}

There are two very common problems here. First, this has changed contexts. We can fix this by allowing a binding as a second parameter, but it means that we need to make sure that every time we refactor to a lambda we make sure to accept a binding parameter and pass it in. The var self = this pattern emerged in JavaScript primarily because of the lack of correspondence.

This is annoying, but not deadly. More problematic is the fact that return has changed meaning. Instead of returning from the outer function, it returns from the inner one.

This is the right time for JavaScript lovers (and I write this as a sometimes JavaScript lover myself) to argue that return behaves exactly as intended, and this behavior is simpler and more elegant than the Ruby behavior. That may be true, but it doesn’t alter the fact that this behavior breaks the correspondence principle, with very real consequences.

Instead of effortlessly refactoring code with the same start and end into a function taking a function, JavaScript library authors need to consider the fact that consumers of their APIs will often need to perform some gymnastics when dealing with nested functions. In my experience as an author and consumer of JavaScript libraries, this leads to many cases where it’s just too much bother to provide a nice block-based API.

In order to have a language with return (and possibly super and other similar keywords) that satisfies the correspondence principle, the language must, like Ruby and Smalltalk before it, have a function lambda and a block lambda. Keywords like return always return from the function lambda, even inside of block lambdas nested inside. At first glance, this appears a bit inelegant, and language partisans often accuse Ruby of unnecessarily having two types of “callables”, in my experience as an author of large libraries in both Ruby and JavaScript, it results in more elegant abstractions in the end.

Iterators and Callbacks

It’s worth noting that block lambdas only make sense for functions that take functions and invoke them immediately. In this context, keywords like return, super and Ruby’s yield make sense. These cases include iterators, mutex synchronization and resource management (like the block form of File.open).

In contrast, when functions are used as callbacks, those keywords no longer make sense. What does it mean to return from a function that has already returned? In these cases, typically involving callbacks, function lambdas make a lot of sense. In my view, this explains why JavaScript feels so elegant for evented code that involves a lot of callbacks, but somewhat clunky for the iterator case, and Ruby feels so elegant for the iterator case and somewhat more clunky for the evented case. In Ruby’s case, (again in my opinion), this clunkiness is more from the massively pervasive use of blocks for synchronous code than a real deficiency in its structures.

Because of these concerns, the ECMA working group responsible for ECMAScript, TC39, is considering adding block lambdas to the language. This would mean that the above example could be refactored to:

FileInfo = function(name) {
  this.name = name;
};
 
FileInfo.prototype = {
  mtime: function() {
    // use the proposed block syntax, `{ |args| }`.
    this.withFile { |f|
      // in block lambdas, +this+ is unchanged
      var mtime = this.mtimeFor(f);
      if (mtime < new Date() - 1000) {
        // block lambdas return from their nearest function
        return "too old";
      }
      sys.print(mtime);
    }
  },
 
  body: function() {
    this.withFile { |f| f.read(); }
  },
 
  mtimeFor: function(f) {
    return File.mtime(f);
  },
 
  withFile: function(block) {
    try {
      var f = File.open(this.name, "r");
      block.call(f);
    } finally {
      f.close();
    }
  }
};

Note that a parallel proposal, which replaces function-scoped var with block-scoped let, will almost certainly be accepted by TC39, which would slightly, but not substantively, change this example. Also note block lambdas automatically return their last statement.

Our experience with Smalltalk and Ruby show that people do not need to understand the SCARY correspondence principle for a language that satisfies it to yield the desired results. I love the fact that the concept of “iterator” is not built into the language, but is instead a consequence of natural block semantics. This gives Ruby a rich, broadly useful set of built-in iterators, and language users commonly build custom ones. As a JavaScript practitioner, I often run into situations where using a for loop is significantly more straight-forward than using forEach, always because of the lack of correspondence between the code inside a built-in for loop and the code inside the function passed to forEach.

For the reasons described above, I strongly approve of the block lambda proposal and hope it is adopted.

Amber.js (formerly SproutCore 2.0) is now Ember.js

After we announced Amber.js last week, a number of people brought Amber Smalltalk, a Smalltalk implementation written in JavaScript, to our attention. After some communication with the folks behind Amber Smalltalk, we started a discussion on Hacker News about what we should do.

Most people told us to stick with Amber.js, but a sizable minority told us to come up with a different name. After thinking about it, we didn’t feel good about the conflict and decided to choose a new name.

Henceforth, the project formerly known as SproutCore 2.0 will be known as Ember.js. Our new website is up at http://www.emberjs.com

(and yes, we know this is pretty ridiculous)

Announcing Amber.js

A little over a year ago, I got my first serious glimpse at SproutCore, the JavaScript framework Apple used to build MobileMe (now iCloud). At the time, I had worked extensively with jQuery and Rails on client-side projects, and I had never found the arguments for the “solutions for big apps” very compelling. At the time, most of the arguments (at least within the jQuery community) focused on bringing more object orientation to JavaScript, but I never felt that they offered the layers of abstraction you really want to manage complexity.

When I first started to play with SproutCore, I realized that the bindings and computed properties were what gave it its real power. Bindings and computed properties provide a clean mechanism for building the layers of abstractions that improve the structure of large applications.

But even before I got involved in SproutCore, I had an epiphany one day when playing with Mustache.js. Because Mustache.js was a declarative way of describing a translation from a piece of JSON to HTML, it seemed to me that there was enough information in the template to also update the template when the underlying data changed. Unfortunately, Mustache.js itself lacked the power to implement this idea, and I was still lacking a robust enough observer library.

Not wanting to build an observer library in isolation (and believing that jQuery’s data support would work in a pinch), I started working on the first problem: building a template engine powerful enough to build automatically updating templates. The kernel of the idea for Handlebars (helpers and block helpers as the core primitives) came out of a discussion with Carl Lerche back when we were still at Engine Yard, and I got to work.

When I met SproutCore, I realized that it provided a more powerful observer library than anything I was considering at the time for the data-binding aspect of Handlebars, and that SproutCore’s biggest weakness was the lack of a good templating solution in its view layer. I also rapidly became convinced that bindings and computed properties were a significantly better abstraction, and allowed for hiding much more complexity, than manually binding observers.

After some months of retooling SproutCore with Tom Dale to take advantage of an auto-updating templating solution that fit cleanly into SproutCore’s binding model, we reached a crossroads. SproutCore itself was built from the ground up to provide a desktop-like experience on desktop browsers, and our ultimate plan had started to diverge from the widget-centric focus of many existing users of SproutCore. After a lot of soul-searching, we decided to start from scratch with SproutCore 2.0, taking with us the best, core ideas of SproutCore, but leaving the large, somewhat sprawling codebase behind.

Since early this year, we have worked with several large companies, including ZenDesk, BazaarVoice and LivingSocial, to iterate on the core ideas that we started from to build a powerful framework for building ambitious applications.

Throughout this time, though, we became increasingly convinced that calling what we were building “SproutCore 2.0″ was causing a lot of confusion, because SproutCore 1.x was primarily a native-style widget library, while SproutCore 2.0 was a framework for building web-based applications using HTML and CSS for the presentation layer. This lack of overlap causes serious confusion in the IRC room, mailing list, blog, when searching on Google, etc.

To clear things up, we have decided to name the SproutCore-inspired framework we have been building “Amber.js”. Amber brings a proven MVC architecture to web applications, as well as features that eliminate common boilerplate. If you played with SproutCore and liked the concepts but felt like it was too heavy, give Amber a try. And if you’re a Backbone fan, I think you’ll love how little code you need to write with Amber.

In the next few days, we’ll be launching a new website with examples, documentation, and download links. Stay tuned for further updates soon.

UPDATE: The code for Amber.js is still, as of December 8, hosted at the SproutCore organization. It will be moved and re-namespaced within a few days.

Announcing Amber.js

A little over a year ago, I got my first serious glimpse at SproutCore, the JavaScript framework Apple used to build MobileMe (now iCloud). At the time, I had worked extensively with jQuery and Rails on client-side projects, and I had never found the arguments for the “solutions for big apps” very compelling. At the time, most of the arguments (at least within the jQuery community) focused on bringing more object orientation to JavaScript, but I never felt that they offered the layers of abstraction you really want to manage complexity.

When I first started to play with SproutCore, I realized that the bindings and computed properties were what gave it its real power. Bindings and computed properties provide a clean mechanism for building the layers of abstractions that improve the structure of large applications.

But even before I got involved in SproutCore, I had an epiphany one day when playing with Mustache.js. Because Mustache.js was a declarative way of describing a translation from a piece of JSON to HTML, it seemed to me that there was enough information in the template to also update the template when the underlying data changed. Unfortunately, Mustache.js itself lacked the power to implement this idea, and I was still lacking a robust enough observer library.

Not wanting to build an observer library in isolation (and believing that jQuery’s data support would work in a pinch), I started working on the first problem: building a template engine powerful enough to build automatically updating templates. The kernel of the idea for Handlebars (helpers and block helpers as the core primitives) came out of a discussion with Carl Lerche back when we were still at Engine Yard, and I got to work.

When I met SproutCore, I realized that it provided a more powerful observer library than anything I was considering at the time for the data-binding aspect of Handlebars, and that SproutCore’s biggest weakness was the lack of a good templating solution in its view layer. I also rapidly became convinced that bindings and computed properties were a significantly better abstraction, and allowed for hiding much more complexity, than manually binding observers.

After some months of retooling SproutCore with Tom Dale to take advantage of an auto-updating templating solution that fit cleanly into SproutCore’s binding model, we reached a crossroads. SproutCore itself was built from the ground up to provide a desktop-like experience on desktop browsers, and our ultimate plan had started to diverge from the widget-centric focus of many existing users of SproutCore. After a lot of soul-searching, we decided to start from scratch with SproutCore 2.0, taking with us the best, core ideas of SproutCore, but leaving the large, somewhat sprawling codebase behind.

Since early this year, we have worked with several large companies, including ZenDesk, BazaarVoice and LivingSocial, to iterate on the core ideas that we started from to build a powerful framework for building ambitious applications.

Throughout this time, though, we became increasingly convinced that calling what we were building “SproutCore 2.0″ was causing a lot of confusion, because SproutCore 1.x was primarily a native-style widget library, while SproutCore 2.0 was a framework for building web-based applications using HTML and CSS for the presentation layer. This lack of overlap causes serious confusion in the IRC room, mailing list, blog, when searching on Google, etc.

To clear things up, we have decided to name the SproutCore-inspired framework we have been building (so far called “SproutCore 2.0″) “Amber.js”. Amber brings a proven MVC architecture to web applications, as well as features that eliminate common boilerplate. If you played with SproutCore and liked the concepts but felt like it was too heavy, give Amber a try. And if you’re a Backbone fan, I think you’ll love how little code you need to write with Amber.

In the next few days, we’ll be launching a new website with examples, documentation, and download links. Stay tuned for further updates soon.

UPDATE: The code for Amber.js is still, as of December 8, hosted at the SproutCore organization. It will be moved and re-namespaced within a few days.

Clarifying the Roles of the .gemspec and Gemfile

TL;DR

Although apps and gems look like they share the concept of “dependency”, there are some important differences between them. Gems depend on a name and version range, and intentionally don’t care about where exactly the dependencies come from. Apps have more controlled deployments, and need a guarantee that the exact same code is used on all machines (dev, ci and production).

When developing a gem, use the gemspec method in your Gemfile to avoid duplication. In general, a gem’s Gemfile should contain the Rubygems source and a single gemspec line. Do not check your Gemfile.lock into version control, since it enforces precision that does not exist in the gem command, which is used to install gems in practice. Even if the precision could be enforced, you wouldn’t want it, since it would prevent people from using your library with versions of its dependencies that are different from the ones you used to develop the gem.

When developing an app, check in your Gemfile.lock, since you will use the bundler tool across all machines, and the precision enforced by bundler is extremely desirable for applications.


Since my last post on this topic, a lot of people have asked follow-up questions about the specific roles of the Gemfile, used by bundler, and the .gemspec, used by Rubygems. In short, people seem to feel that there is duplication between the two files during gem development, and end up using yet another tool to reduce the duplication.

Ruby Libraries

Ruby libraries are typically packaged up using the gem format, and distributed using rubygems.org. A gem contains a number of useful pieces of information (this list is not exhaustive):

  • A bunch of metadata, like name, version, description, summary, email address, etc.
  • A list of all the files contained in the package.
  • A list of executables that come with the gem and their location in the gem (usually bin)
  • A list of directories that should be put on the Ruby load path (usually lib)
  • A list of other Ruby libraries that this library needs in order to function (dependencies)

Only the last item, the list of dependencies, overlaps with the purpose of the Gemfile. When a gem lists a dependency, it lists a particular name and a range of version numbers. Importantly, it does not care where the dependency comes from. This makes it easy to internally mirror a Rubygems repository or use a hacked version of a dependency that you install to your system. In short, a Rubygems dependency is a symbolic name. It is not a link to a particular location on the Internet where the code can be found.

Among other things, this characteristic is responsible for the resilience of the overall system.

Also, gem authors choose a range of acceptable versions of their dependencies, and have to operate under the assumption that the versions of the dependencies that the author tested against may change before the gem is used in deployment. This is especially true about dependencies of dependencies.

Ruby Applications

A Ruby application (for instance, a Rails application), has a complex set of requirements that are generally tested against a precise set of third-party code. While the application author may not have cared much (up front), which version of nokogiri the application used, he does care a lot that the version used in development will remain the version used in production. Since the application author controls the deployment environment (unlike the gem author), he does not need to worry about the way that a third-party will use the code. He can insist on precision.

As a result, the application author will want to specify the precise gem servers that the deployment environment should use. This is in contrast to the gem author, who should prefer long-term resilience and mirroring to the brittleness that comes with a short-term guarantee.

The application author often also wants a way to tweak shared community gems for the particular project at hand (or just use the “edge” version of a gem which hasn’t yet been released). Bundler handles this problem by allowing application authors to override a particular gem and version with a git repository or a directory. Because the gem author has specified the gem’s dependencies as a name and version range, the application developer can choose to satisfy that dependency from an alternate source (in this case, git).

Now, the application developer must be able to demand that the gem comes from that particular source. As you can see, there are critical differences between the desires of the gem developer (which are focused around longevity and flexibility) and the app developer (which are focused around absolute guarantees that all of the code, including third-party code, remains precisely the same between environments).

Rubygems and Bundler

The Rubygems libraries, CLI and gemspec API are built around the needs of the gem author and the Rubygems ecosystem. In particular, the gemspec is a standard format for describing all of the information that gets packed with gems then deployed to rubygems.org. For the reasons described above, gems do not store any transient information about where to find dependencies.

On the other hand, bundler is built around the needs of the app developer. The Gemfile does not contain metadata, a list of files in the package, executables exposed by the package, or directories that should be put on the load path. Those things are outside the scope of bundler, which focuses on making it easy to guarantee that all of the same code (including third-party code) is used across machines. It does, however, contain a list of dependencies, including information about where to find them.

Git “Gems”

For the reasons described above, it can sometimes be useful for application developers to declare a dependency on a gem which has not yet been released. When bundler fetches that gem, it uses the .gemspec file found in the git repository’s root to extract its metadata as well as discover additional dependencies declared by the in-development gem. This means that git gems can seamlessly be used by bundler, because their .gemspec allows them to look exactly like normal gems (including, importantly, having more dependencies of its own). The only difference is that the application developer has told bundler exactly where to look for the gem in question.

Gem Development

Active development on gems falls in between these two views of the world, for two reasons:

  • Active development on a gem often involves dependencies that have not yet been released to rubygems.org, so it becomes important to specify the precise location to find those dependencies, during development
  • During gem development, you cannot rely on rubygems to set up an environment with everything on the load path. Instead, you need a hybrid environment, with one gem coming from the file system (the current gem under development), some gems possibly coming from git (where you depend on a gem that hasn’t yet been released), and yet additional gems coming from traditional sources. This is the situation that Rails development is in.

Because gems under active development will eventually become regular gems, they will need to declare all of the regular metadata needed by gem build. This information should be declared in a .gemspec, since the Gemfile is an API for declaring dependencies only.

In order to make it easy to use the bundler tool for gem development without duplicating the list of dependencies in both the .gemspec and the Gemfile, bundler has a single directive that you can use in your Gemfile (documented on the bundler site):

source "http://www.rubygems.org"
 
gemspec

The gemspec line tells bundler that it can fine a .gemspec file alongside the Gemfile. When you run bundle install, bundler will find the .gemspec and treat the local directory as a local, unpacked gem. It will find and resolve the dependencies listed in the .gemspec. Bundler’s runtime will add the load paths listed in the .gemspec to the load path, as well as the load paths of any of the gems listed as dependencies (and so on). So you get to use bundler while developing a gem with no duplication.

If you find that you need to develop against a gem that hasn’t yet been released (for instance, Rails often develops against unreleased Rack, Arel, or Mail gems), you can add individual lines in the Gemfile that tell bundler where to find the gems. It will still use the dependencies listed in the .gemspec for dependency resolution, but it now knows exactly where to find the dependency during gem development.

source "http://www.rubygems.org"
 
gemspec
 
# if the .gemspec in this git repo doesn't match the version required by this
# gem's .gemspec, bundler will print an error
gem "rack", :git => "git://github.com/rack/rack.git"

You wouldn’t want to include this information in the .gemspec, which will eventually be released to Rubygems after the in-development gem has also been released. Again, this makes the overall system more resilient, as it doesn’t depend on transient external URLs. This information is purely something used to set up a complete environment during development, which requires some precision.

This is also why we recommend that people do not check in their Gemfile.lock files in gem development. That file is emitted by bundler, and guarantees that all gems, including dependencies of dependencies, remain the same. However, when doing gem development, you want to know immediately when some change in the overall ecosystem breaks your setup. While you may want to insist on using a particular gem from its git location, you do not want to hardcode your development to a very specific set of gems, only to find out later that a gem released after you ran bundle install, but compatible with your version range, doesn’t work with your code.

It’s not a way to guarantee that your code works on all versions that you specified in your .gemspec, but the incentive to lock down the exact versions of each gem simply doesn’t exist when the users of your code will get it via gem install.

Update

Just to clarify, you should check in your Gemfile.lock in applications, and not in gems. The Gemfile.lock remembers the exact versions and sources of every piece of third-party code that you use. For applications, you want this. When developing a gem, this can obscure issues that will occur because gems are deployed (using the gem command) without the benefit of bundler.

What’s Wrong with “HTML5″

In the past year or so, the term “HTML5″ has increasingly been picked up by the tech press as the successor to “DHTML”, “Web 2.0″ or “Ajax”. When used by the tech press, it is becoming a generic term for “the next generation of web technology”, except that the term “HTML5″ is less precise than even that.

Consider the first paragraph of an article about HTML published this week:

HTML5 is the hot topic nowadays. Everyone from Apple to Google and everyone else in between have shown their support for the standard. Word has it that HTML5 is the Adobe Flash-killer. It seems that the World Wide Web Consortium [W3C] — which is the main international standards organization for the World Wide Web — doesn’t agree. If anything, the W3C doesn’t think that HTML5 is “ready for production yet.”

The problem with HTML5 appears to be that it currently lacks a video codec. In addition, digital rights management [DRM] is also not supported in HTML5, which obviously makes a problem for various companies.

My engineer friends have all but given up on pushing back against the “HTML5″ moniker. After all, it’s just a term for the tech press. Everyone who knows anything understands that it means just as little as requests for “Ajaxy animations” a few years back, right? I had started to agree with this line of reasoning. It’s true: there’s no point in being pedantic on this front for the sake of being pedantic.

Unfortunately, the term, and therefore the way the technology is understood by tech writers, is causing some fairly serious problems.

First, keep in mind that unlike “Ajax” or “Web 2.0″, which were pretty benign, vague terms, HTML5 sounds like a technology. It can have “beta” versions, and one day it will be “complete” and “ready for production”. Take a look at this snippet from an InfoWorld article:

His advice on HTML5 was endorsed by industry analyst Al Hilwa of IDC.

“HTML 5 is at various stages of implementation right now through the Web browsers. If you look at the various browsers, most of the aggressive implementations are in the beta versions,” Hilwa said. “IE9 (Internet Explorer 9), for example, is not expected to go production until close to mid-next year. That is the point when most enterprises will begin to consider adopting this new generation of browsers.”

And because HTML5 is portrayed as a “Flash killer”, whether it is “complete” or in “beta” sounds really relevant. In comparison, “Web 2.0″ was never portrayed as a technology, but rather the ushering in of a new era of web technologies which would allow people to build new kinds of applications.

The truth is, the “completion” of HTML5 is absolutely irrelevant. At some point, the W3C will approve the HTML5 spec, which will mean nothing about the overall availability of individual features. At some point, the other related specs (like Web Storage and Web Sockets) will be approved, and also mean nothing about the overall availability of individual features.

In response to this conundrum, most of my friends immediately throw out the idea of getting more specific with the tech press. And they’re right. The tech press is not going to understand Web Sockets or local storage. They’re having trouble even grasping HTML5 video.

The thing is, the core problem isn’t that the name is too fuzzy. It’s what the name implies. HTML5 sounds like a technology which goes through a beta period and is finally complete. Instead, what the tech press calls “HTML5″ is really a continual process of improving the web browsers that people use. And honestly, that’s how I’d like to see the tech press cover us. Not as group of people working towards a singular milestone that will change the web as we know it, but as a group that has gotten our groove back.

Tech reporters: please stop talking about the current state of web technologies as an event (the approval of the HTML5 spec). There are interesting stories happening all the time, like Scribd, YouTube, and Gmail leveraging newer web technologies to improve their products. Little guys are doing the same every day. Not everything is about whether “Flash is dead yet”. It’s worth talking about the tradeoffs that guys like Hulu make, which prevent them from moving to web technologies. But make no mistake: large swaths of the web will be using next-generation browser features well before the last guy does. The process of getting there is an interesting story, and one you should be covering.

Here’s to the Next 3 Years

I officially joined Engine Yard on January 1, 2008, about a week before we announced our Series A funding, becoming its twenty-second employee. I was Engine Yard’s very first “Engineering” hire, and I would spend the next year working on Ezra’s Merb project, finally releasing Merb 1.0 at MerbCamp that October.

When I joined Engine Yard, I had already been working on the jQuery project for a couple years, having started visualjquery.com at a time in jQuery’s history before jQuery had version numbers. I learned a lot from John Resig about building and motivating open source communities, and I did my first professional work writing jQuery in Action.

The Merb project was my first full-time open source work, and I met some of my earliest open source friends while working on it. When I first met Carl, he was working on a rewrite of the Merb router, the one part of Merb I was still scared to work on. I met Andy Delcambre, who now works on Engine Yard’s AppCloud, after he volunteered to help with Merb.

I met Loren Segal, well-known for the YARD documentation tool, while working to give Merb documentation I could be proud of. And I became the first Github user outside of Chris, PJ and Tom in order to get the Merb project on git. If you’re paying attention, that made me Github user nil.id, and was the source of some hilarious bugs.

In Hampton Catlin’s first Ruby survey in 2008, about one in eight respondents said that they preferred Merb, and by the end of the year, the Rails and Merb teams were coming to blows. During this same time, Engine Yard had taken on a Series B round of funding (this time for $15 million), and had grown from a small company of a couple dozen to a company pushing 100. We had Sales and Marketing departments. We also had an Engineering department, which had spent a large part of the year on our first fully automated product, Engine Yard Solo.

On a personal level, I started getting heavily involved in Open Source other than Merb, especially DataMapper and Rubinius. I started speaking at conferences, starting with a talk on DataMapper at MountainWest RubyConf in 2008. Engine Yard also threw its first Hackfest at that event, an event we would repeat in 2009 and again in 2010. This event, and the work our great community marketing, Open Source and support teams would do over the next several years elevated Engine Yard’s reputation as a team of experts that would keep your app running no matter what.

As Engine Yard grew, a lot of people asked us why we were supporting Merb when so much of our business depended on the success of the Rails community. By late 2008, it was probably the most common question I got when speaking at conferences. In truth, the best I could tell was that Engine Yard was worried about the future of Rails, and wanted to make sure that Ruby remained successful. What we didn’t realize at the time was that the one full-time employee (me) that Engine Yard had on Merb was more than the total number of worldwide full-time developers on Rails. In short, Rails was a fully volunteer project, and that had something to do with some of the issues that drove people to Merb in the first place.

At the end of 2008, Tom Mornini, the CTO of Engine Yard, recognized that if we could join forces (and this was definitely a big if), we could help shore up Rails into the future. I remember the first time Tom asked me about it in the stairwell of our South Park office. I thought he was off the wall to even suggest it. After all, it didn’t seem like the teams could get along at all. But the Merb team had adopted so much of the Rails philosophy that at its core, the only question that remained was how aggressive the Rails project would get to clean things up.

In the years since Ezra first announced Merb, Rails had gotten thread-safe, Rack support, and some improvements to the internals. After hashing out the details (in some very contentious conversations), we decided to give it a try, announcing we would merge the projects on December 23, 2008. That announcement quite literally changed the trajectory of my life. In the almost two years since that announcement, I have worked harder than I have ever worked before, and am more proud of the work I have done than I have ever been before.

Perhaps more importantly, I have worked with some of the best open source developers in the business without whom Rails 3 would have been a pale shadow of itself. Including me, Rails 3 brought on eight new committers. The Rails 3 project did more than improve the quality of the codebase: it reinvigorated the community of developers actively contributing. And by moving toward a model that allowed Rails to have more dependencies, Rails became the center of a thriving Ruby community, where anyone can reuse the code that makes Rails tick for their non-Rails projects.

In the summer of 2009, José Valim became a Google Summer of Code student dedicated to rewriting the Rails generators so that users of DataMapper, RSpec, and Haml could use the standard Rails generators. A lot of people thought it was too optimistic to do in a summer, but José finished the job well before the end of the summer, giving him time to continue his contributions.

Around the same time, Mikel Lindsaar decided to write a ground-up mail implementation, releasing the imaginatively named mail gem in October 2009. This project, which involved writing a parser that could parse the gigabytes of mail in the Trec and Enron email databases, strained credulity, and would enable us to give ActionMailer the facelift it needed for Rails 3.0.

Santiago Pastorino showed up out of nowhere, with a level of pluck that is rarely seen in the Ruby community. Despite a noticeable language barrier, Santiago went from a warning fixer to a Rails committer. This month, he became an official member of the Rails core team. While working on Rails, I learned a lot from a lot of people, but Santiago’s raw determination and skill should stand out as an example to the Ruby community.

Everyone knows Aaron Patterson, and I was just as surprised as everyone else when he started the long, dedicated task of improving the performance of the Arel library. He already has quite the reputation in the Ruby community, and even has commit access to Ruby itself, so I’m personally gratified to work with him on Rails. If you haven’t yet watched it, watch his Worst Ideas Ever presentation from RubyConf last year. Be prepared to laugh.

Xavier Noria has been a one-man documentation machine, helping to grow and maintain the Rails guides, and constantly reminds everyone that documentation is just as essential to the codebase as tests. Without him, Rails 3 might have been great internally, but nobody would know it. Documentation folks often get little love in open source projects, but to me, the work Xavier does is key to our success.

More recently, I’ve been extremely impressed by Piotr Sarnacki’s work (for Ruby Summer of Code) on finally delivering on the promise of truly mountable engines. His work is already merged into Rails master, and will ship with Rails 3.1. As with the rest of the guys I’ve talked about, he took on a pretty complicated and entangled task, and managed to get more done in less time than people expected. Keep an eye on him!

In short, the Rails 3 project was about more than code; it was about reshaping the Rails community. I’m proud to have been a part of growing the Rails ecosystem, and humbled by the number of people who have worked so hard for so little.

Now that we released Rails 3, I have had an opportunity to think about what to do next. I could, of course, continue full-time work on Rails for Engine Yard, which has been a fantastic place to do Open Source for the past several years. I am still extremely interested in open web technologies, and want to do my part to help the open web succeed. Having robust, mature server side frameworks that are easy for anyone to use is certainly one large piece of the puzzle, and I’ve really enjoyed helping to bring the core of Rails up to date with modern web technologies.

But as Avi Bryant pointed out last year at DjangoCon, the technology of the open web is shifting toward the client side. That doesn’t change the need for robust, mature server-side frameworks. Even the richest web applications get and sync their data with servers, mostly over the HTTP protocol. And although a lot of people have opined that rich clients can simply wire directly into something like CouchDB or MongoDB, applications with rich clients tend to have the same kinds of complex needs (like authorization) as more tranditional document-based applications have. I see Rails remaining an excellent server-side solution even as clients become richer over the next few years.

There’s still a lot to do for Rails in the future, and I want to be a part of that. On the other hand, the field of rich client development is just getting starting, with new mobile devices like the iPhone, Android and iPad giving us an almost blank canvas to work on. Unfortunately, a lot of the really big bets in this space are on selling licenses to either entire libraries or the development tools needed to efficiently work with the tools.

What the open web needs is a robust, easy to use framework with development tools that are trivial to get started with and awesome to work with. And both the library and the developer tools need to be offered for free, like Rails or jQuery. Since 2008, I have poured my soul into helping to build an amazing open source Ruby web framework and the ecosystem that powers it. Now, I want to bring that same energy and enthusiasm to building great tools that leverage the power of the modern open web and a free and open source community around them.

That’s why I was so excited when I saw that Charles Jolley, who worked on Apple’s Mobile Me product, announced that he was leaving Apple to form a new company dedicated to those same principles. Charles had helped build Mobile Me on SproutCore, a free and open source framework, and had managed to make sure that Apple continued to contribute back to the open source project under the MIT license.

In his blog post announcing that he was leaving Apple, Charles said “SproutCore is now and will always be totally free and open source. I think this business of charging for a commercial license is not an effective way to grow a project. Sure you make a little cash, but at what expense to the community? My goal is to make SproutCore and all of the developer tools that surround it totally free to everyone. All I ask is that you participate in the community somehow to make things a little better for those who come after you.”

This philosophy closely mirrored my thinking on the future of client-side development, and seeing a mature client-side framework open a permissive open source philosophy really excited me. Since the beginning of 2010, I was starting to think that if I wanted a permissively licensed client-side toolkit, I was going to have to build it and its community myself. I already knew about SproutCore–early versions of its build tools were built on Merb, and had met Charles a few times. After reading his blog post, I emailed him to find out what he was thinking about doing and he told me about his plans for SproutCore.

The SproutCore library itself, today, is definitely recovering from having been developed almost exclusively inside Apple. It comes with a number of amazing low-level primitives, and you can build a large, complex application on it. That said, it never had a big enough new user community to get a very friendly public API. Thankfully, it’s much, much easier to build a great public API than it is to build the core of a web application toolkit. Starting this week, I will be joining Charles’ company, Strobe, and will get right to work on the next generation of the SproutCore tools.

I have to say, I’m excited about what comes next.

Postscript

Because you’re probably wondering, I will still be active in the Rails community, and will be doing work for Engine Yard on Rails 3.1. I plan to use Rails 3 and Engine Yard for the server-side elements of the projects I will be doing, and really want the new Rails 3.1 features I talked about at Windy City Rails. I will also be helping Engine Yard start a new Technical Advisory Board, which will help provide community feedback for its cloud products, and guidance about what open source efforts could use the Engine Yard bump. I forsee being part of both the Rails and Engine Yard family for years to come.

Automatic Flushing: The Rails 3.1 Plan

preamble: this post explains, in some detail, how we will implement a nice performance boost for Rails developers. Understanding the details might help gain the full benefits of the optimization, but you will gain some benefits even if you have no idea how it works.

As you’ve probably seen, DHH announced that we’d be looking at flushing in Rails 3.1 to improve the client-side performance of typical Rails applications.

The most obvious solution, and one that already exists in plugin form, is to allow a layout to have a new flush method, which would immediately flush the contents of the layout to the browser. By putting the flush method below the JavaScript and CSS includes, the browser could begin downloading and evaluating those static assets while the server continues building the page.

Unfortunately, this solution has a major problem: it requires a fairly significant change in the current model of how people build applications. In general, for performance optimizations (including client-side optimizations), we like to make the default as fast as possible, without asking people to understand a brand new paradigm, centered around the optimization.

The problem lies in the fact that a Rails layout is essentially a template with a bunch of holes to fill in.

<html>
  <head>
    <title><%= yield :title %></title>
    <%= javascript_include_tag :defaults %>
    <%= yield :extra_javascripts %>
    <%= stylesheet_link_tag :defaults %>
    <%= yield :extra_stylesheets %>
  </head>
  <body>
    <%= yield :sidebar %>
    <%= yield %>
  </body>
</html>

I this simple example, each yield is a slot that is filled in by the template (usually via content_for). In order to achieve this, Rails evaluates the template first, which populates a Hash with each piece of content. Next, it renders the layout, and each yield checks the Hash for that content. In short, because of the way layouts work, Rails renders the template first, and then the layout.

To get around this, one option would be to say that everything before the flush must not use yield, and must be able to run before the template. Unfortunately, it’s somewhat common for people to set up a content_for(:javascripts) in a template, to keep the JavaScript needed for a particular snippet of HTML close to the HTML. This means that not only does the user have to be careful about what can go above and below the flush, he can no longer use content_for for things high up in the template, which is a fairly significant change to the overall design of Rails applications.

For Rails 3.1, we wanted a mostly-compatible solution with the same programmer benefits as the existing model, but with all the benefits of automatic flushing. After a number of very long discussions on the topic, José Valim came up with the idea of using Ruby 1.9 fibers to jump back and forth between the template and layout.

Let’s start by taking a look at a very simplified version of the current Rails rendering pipeline. First, we set up a Buffer object purely for logging purposes, so we can see what’s happening as we push things onto the buffer.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
    end
 
    def <<(value)
      super
 
      puts "#{@name} is pushing #{value.inspect}"
    end
  end
end

Next, we create a simple version of ActionView::Base. We implement the content_for method simply, to print out a bit of logging information and stash the value into the @content_for Hash. Note that the real version is pretty similar, with some added logic for capturing the value of the block from ERB.

module Basic
  class ViewContext
    def initialize
      @buffer      = Buffer.new(:main, self)
      @content_for = {}
    end
 
    def content_for(name, value = nil)
      value = yield if block_given?
      puts "Setting #{name} to #{value.inspect}"
      @content_for[name] = value
    end
 
    def read_content(name)
      @content_for[name]
    end
  end
end

Next, we create a number of methods on the ViewContext that look like compiled ERB templates. In real life, the ERB (or Haml) compiler would define these methods.

module Basic
  class ViewContext
    def layout
      @buffer << "<html><head>"
      @buffer << yield(:javascripts).to_s
      @buffer << yield(:stylesheets).to_s
      @buffer << "</head><body>"
      @buffer << yield.to_s
      @buffer << yield(:not_existant).to_s
      @buffer << "</body></html>"
      @buffer
    end
 
    def template
      buffer =  Buffer.new(:template, self)
      content_for(:javascripts) do
        "<script src='application.js'></script>"
      end
      content_for(:stylesheets) do
        "<link href='application.css' rel='stylesheet' />"
      end
      puts "Making a SQL call"
      sleep 1 # Emulate a slow SQL call
      buffer << "Hello world!"
      content_for(:body, buffer)
    end
  end
end

Finally, we define the basic rendering logic:

module Basic
  class ViewContext
    def render
      template
      layout { |value| read_content(value || :body) }
    end
  end
end

As you can see, we first render the template, which will fill up the @content_for Hash, and then call the layout method, with a block which pulls the value from that Hash. This is how yield :javascripts in a layout works.

Unfortunately, this means that the entire template must be rendered first, including the (fake) slow SQL query. We’d prefer to flush the buffer after the JavaScripts and CSS are determined, but before the SQL query is made. Unfortunately, that requires running half of the template method, then continuing with the layout method, retaining the ability to resume the template method later.

You can think of the way that templates are currently rendered (in Rails 2.x and 3.0) like this:

flush.001.png

Unfortunately, this makes it very hard to get any more performance juice out without asking the end-developer to make some hard choices. The solution we came up with is to use Ruby 1.9 fibers to allow the rendering to jump back and forth between the template and layout.

flush.002.png

Instead of starting with the template and only rendering the layout when ready, we’ll start with the layout, and jump over to the template when a yield is called. Once the content_for that piece is provided by the template, we can jump back to the layout, flush, and continue rendering. As we need more pieces, we can jump back and forth between the template and layout, flushing as we fill in the holes specified by the yield statements.

The implementation is mostly straight-forward:

require "fiber"
 
module Fibered
  class ViewContext < Basic::ViewContext
    def initialize
      super
      @waiting_for = nil
      @fiber       = nil
    end
 
    def content_for(name, value = nil)
      super
      @fiber.resume if @waiting_for == name
    end
 
    def read_content(name)
      content = super
      return content if content
 
      begin
        @waiting_for = name
        Fiber.yield
      ensure
        @waiting_for = nil
      end
 
      super
    end
 
    def layout
      @fiber = Fiber.new do
        super
      end
      @fiber.resume
      @buffer
    end
 
    def render
      layout { |value| read_content(value || :body) }
      template
      @fiber.resume while @fiber.alive?
      @buffer
    end
  end
end

For our fibered implementation, we’ll inherit from Basic::ViewContext, because we want to be able to use the same templates as we used in the original implementation. We update the content_for, read_content, layout and render methods to be fiber-aware. Let’s take them one at a time.

def layout
  @fiber = Fiber.new do
    super
  end
  @fiber.resume
  @buffer
end

First, we wrap the original implementation of layout in a Fiber, and start it right away. Next, we modify the read_content method to become Fiber-aware:

def read_content(name)
  content = super
  return content if content
 
  begin
    @waiting_for = name
    Fiber.yield
  ensure
    @waiting_for = nil
  end
 
  super
end

If the @content_for Hash already has the content, return it right away. Otherwise, say that we’re waiting for the key in question, and yield out of the Fiber. We modify the render method so that the layout is rendered first, followed by the template. As a result, yielding out of the layout will start the template’s rendering.

def render
  layout { |value| read_content(value || :body) }
  template
  @fiber.resume while @fiber.alive?
  @buffer
end

Next, modify the content_for method so that when the content we’re waiting for is provided, we jump back into the layout.

def content_for(name, value = nil)
  super
  @fiber.resume if @waiting_for == name
end

With this setup, the layout and template will ping-pong back and forth, with the layout requesting data, and the template rendering only as far as it needs to go to provide the data requested.

Finally, let’s update the Buffer to take our fibered implementation into consideration.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
      @fibered = context.fibered?
    end
 
    def <<(value)
      super
 
      if @fibered
        puts "Flushing #{value.inspect}" if @fibered
      else
        puts "#{@name} is pushing #{value.inspect}"
      end
    end
  end
 
  class ViewContext
    def fibered?
      false
    end
  end
end
 
module Fibered
  class ViewContext
    def fibered?
      true
    end
  end
end

Now that we’re rendering the layout in order, we can flush as we go, instead of being forced to wait for the entire template to render before we can start flushing.

It’s worth mentioning that optimal flushing performance will be based on the order of the content_for in your template. If you run your queries first, then put the expensive template rendering, and only finally do the content_for(:javascript) at the end, the flushing behavior will look like this:

flush.003.png

Instead of flushing quickly, before the SQL call, things are barely better than they are in Rails 2.3, when the entire template must be rendered before the first flush. Because things are no worse, even in the worst-case scenario, we can make this the default behavior. Most people will see some benefit from it, and people interested in the best performance can order their content_for blocks so they cause the most beneficial flushing.

Even for people willing to put in the effort, this API is better than forcing a manual flush, because you can still put your content_for blocks alongside the templates that they are related to.

Look for this feature in Rails 3.1!

Small Caveat

For the purposes of this simplified example, I assumed that content_for can only be run once, immediately setting the value in the @content_for Hash. However, in some cases, people want to accumulate a String for a particular value. Obviously, we won’t be able to flush until the full String for that value is accumulated.

As a result, we’ll be adding a new API (likely called provide), which will behave exactly the same as content_for, but without the ability to accumulate. In the vast majority of cases, people will want to use provide (for instance, provide :sidebar), and get all the benefits of autoflushing. In a few cases, people will want to be able to continue accumulating a String, and will still be able to use content_for, but the template will not return control to the layout when that happens.

Also note that this (fairly detailed) explanation is not something that you will need to understand as a Rails developer. Instead, you will continue to go about your development as before, using provide if you have just a single piece of content to add, and content_for if you have multiple pieces of content to add, and Rails will automatically optimize flushing for you as well as we can.

My Common Git Workflow

A recent post that was highly ranked on Hacker News complained about common git workflows causing him serious pain. While I won’t get into the merit of his user experience complaints, I do want to talk about his specific use-case and how I personally work with it in git.

Best I can tell, Mike Taylor (the guy in the post) either tried to figure out a standard git workflow on his own, or he followed poor instructions that tried to bootstrap someone from an svn background while intentionally leaving out important information. In any event, I’ll step through my personal workflow for his scenario, contrasting with subversion as I go.

Cloning the Repository

The very first step when working with a repository is to clone it. In subversion, this is accomplished via svn checkout svn://url/to/repo/trunk. This retrieves the most recent revision of the trunk branch of the repository.

In git, this is accomplished via git clone git://url/to/repo (the http protocol is also possible). This retrieves the entire repository, including other branches and tags.

Making the Change

In both git and subversion, you make the change using a normal text editor.

After Making the Change

In git, you make a local commit, marking the difference between the most recent pulled version (master) and the changes you made. In subversion, the normal workflow does not involve making a change, but apparently some people make manual diffs in order to have a local copy of the changes before updating from the remote. Here’s an example comment from the Hacker News post:

I’ll tell you what happens when I use svn and there’s been an upstream change: I never update my local tree with local modifications. Instead, I extract all my local changes into a diff, then I update my local tree, and then I merge my diff back into the updated tree and commit.

When I need three-way merging, which isn’t often – usually patch can resync simple things like line offsets – it’s handled by a file comparison tool. I have a simple script which handles this

My personal process for making the commit in git almost always involves the gitx GUI, which lets me see the changes for each individual file, select the files (or chunks in the files) to commit, and then commit the whole thing. I sometimes break up the changes into several granular commits, if appropriate.

Updating from the remote

Now that we have our local changes, the next step is to update from the remote. In subversion, you would run svn up. Here, subversion will apply a merge strategy to attempt to merge the remote changes with the local ones that you made. If a merge was unsuccessful, subversion will tell you that a conflict has occurred. If you did not manually save off a diff file, there is no way to get back to the status from before you made the change.

In git, you would run git pull. By default, git applies the “recursive” strategy, which tries to merge your current files with the remote files at the most recent revision. As with subversion, this can result in a conflict. You can also pass the --rebase flag to pull, which is how I usually work. This tells git to stash away your commits, pull the remote changes, and then reapply your changes on top one at a time.

If you use --rebase, you may get a conflict for each of your local commits, which is usually easier to handle than a bunch of conflicts all at once.

I definitely recommend using --rebase which also provides instructions for dealing with conflicts as they arise.

In either case, in my experience, git’s merging capabilities are more advanced than subversion’s. This will result in many fewer cases where conflicts occur.

Resolving Conflicts

From here on, I am assuming you followed my advice and used git pull --rebase.

If a conflict has occurred, you will find that if you run git status, all of the non-conflicting files are already listed as “staged”, while the conflicting files are listed outside the staging area. This means that the non-conflicting files are already considered “added” to the current commit.

To resolve the conflicts, fix up the files listed outside the staging area and git add them. Again, I normally use gitx to move the resolved files into the staging area.

Once you have resolved the conflict, run git rebase --continue. This tells git to use the fixed up changes you just made instead of the original commit it was trying to put on top of the changes you got from the remote.

In subversion, if you got a conflict, subversion will create three files for you: file.mine, file.rOLD, and file.rNEW. You are responsible for fixing up the conflicts and getting back a working file. Once you are done, you run svn resolved.

NOTE: If you had not used git pull --rebase, but instead did raw git pull, you would fix up the files, add the files using git add or gitx, and the run git commit to seal the deal

Yikes! Something went wrong!

In git, if something goes wrong, you just run git reset --hard, which will bring you back to your last local commit.

In subversion, it’s not always possible unless you manually stored off a diff before you started.

Pushing

Now that you’re in sync with the remote server, you push your changes. In git, you run git push. In subversion, you run svn commit.

One Glossed-Over Difference

Subversion allows you to commit changes even if you haven’t svn uped and there have been changes to the remote, as long as there are no conflicts between your local files and the remote files.

Git never allows you to push changes to the remote if there have been remote changes. I personally prefer the git behavior, but I could see why someone might prefer the subversion behavior. However, I glossed over this difference because every subversion reference I’ve found advises running svn up before a commit, and I personally always did that in my years using subversion.

Comparison

Here’s a workflow comparison between git and subversion:

Operation git svn
Clone a repository git clone git://github.com/rails/rails.git svn checkout http://dev.rubyonrails.org/svn/rails/trunk
Preparing changes git commit (using gitx) nothing or create a manual diff
Update from the remote git pull --rebase svn up
Resolving conflicts git add then git rebase --continue svn resolve
Resolving conflicts without –rebase git add then git commit N/A
Yikes! Rolling back git reset –hard svn up -rOLD then apply diff (only if you manually made a diff first)
Pushing git push svn commit

Note that I am not attempting to provide an exhaustive guide to git here; there are many more git features that are quite useful. Additionally, I personally do a lot of local branching, and prefer to be able to think about git in terms of cheap branches, but the original poster explicitly said that he’d rather not. As a result, I didn’t address that here.

I also don’t believe that thinking of git in terms of subversion is a good idea. That said, the point of this post (and the point of the original poster) is that there are a set of high-level version control operations that you’d expect git to be able to handle in simple cases without a lot of fuss.

The How and Why of Bundler Groups

Since version 0.9, Bundler has had a feature called “groups”. The purpose of this feature is to allow you to specify groups of dependencies which may be used in certain situations, but not in others.

For instance, you may use ActiveMerchant only in production. In this case, you could say:

group :production do
  gem "activemerchant"
end

Specifying groups allows you to do two things. First, you can install the gems in your Gemfile, minus specific groups. For instance, Rails puts mysql and pg in a database group so that if you’re just working on ActionPack, you can bundle install --without db and run the ActionPack tests without having to worry about getting the gems installed.

Second, you can list specific groups to autorequire using Bundler.require. By default, Bundler.require requires all the gems in the default group (which is all the gems that have no explicit group). You can also say Bundler.require(:default, :another_group) to require specific groups.

Note the difference between these operations: bundle install is opt-out, while Bundler.require is opt-in. This is because the common usage of groups is to specify gems for different environments (such as development, test and production) and you shouldn’t need to specify that you want the “development” and “test” gems just to get up and running. On the other hand, you don’t want your test dependencies loaded in development or production.

It is also worth noting that all gems that you installed (i.e. not the ones that you excluded at install time with --without) will be available to require. This has no effect unless you actually require them. This means that in development mode, if you explicitly require rspec, it will work.

Rails 3 defaults to mapping groups to environment names, and explicitly autorequiring the implicit default group and the group named the same as the current environment. For example, in development mode, Rails will require the default group and the development group. The code that does this is in your application.rb:

Bundler.require(:default, Rails.env) if defined?(Bundler)

Consistency

In order to ensure consistency across all environments, bundler resolves the dependencies of your application using the gems listed in all groups, even if you specify --without. This means that while you can skip installing the gems listed in the production group by saying --without production, bundler will still download and examine the gems in order to properly resolve all dependencies.

As a result, the dependencies you install in development mode and test with will be compatible with the gems in other environments. In essence, this policy ensures that if your tests pass and run in development, your app will not fail to run in production because the dependencies resolved differently.

Multiple Inconsistent Configurations

Sometimes, especially when developing gems for wider use, you want to test your code against multiple incompatible configurations. At first glance, you might think that you could use groups for this case, but as described above, groups are designed for cases where all of the gems are compatible, but you don’t always want to have to install them in all situations.

Instead, use multiple Gemfiles, one for each incompatible configuration. When installing, do bundle install --gemfile Gemfile.rails2. This will tell Bundler to use Gemfile.rails2 rather than the default Gemfile. As in all cases in Bundler, you can also specify this option globally with an environment variable (BUNDLE_GEMFILE).

Ruby 1.9 Encodings: A Primer and the Solution for Rails

UPDATE: The DataObjects drivers, which are used in DataMapper, are now updated to honor default_internal. Let’s keep this moving.

Since Ruby 1.9 announced support for encodings, there has been a flurry of activity to make existing libraries encoding aware, and a tornado of confusion as users of Ruby and Rails have tried to make sense of it.

In this post, I will lay out the most common problems people have had, and what we can do as a community to put these issues to bed in time for Ruby 1.9.2 final.

A Quick Tour

I’m going to simplify some of this, but the broad strokes are essentially correct.

Before we begin, many of you are probably wonder what exactly an “encoding” is. For me, getting a handle on this was an important part of helping me understand the possible solution space.

On disk, all Strings are stored as a sequence of bytes. An encoding simply specifies how to take those bytes and convert them into “codepoints”. In some languages, such as English, a “codepoint” is exactly equivalent to “a character”. In most other languages, there is not a one-to-one correspondence. For example, a German codepoint might specify that the next codepoint should get an ümlaut.

The list of English characters represented by the first seven bits of ASCII (characters 0 through 127 in “ASCII-7″) have the same representation in many (but not all) encodings. This means that if you only use English characters, the on-disk representation of the characters will often be exactly the same regardless of the source encoding.

However, once you start to use other characters, the bytes on disk mean different things in different encodings. Have you ever seen a page on the Internet filled with something like “Führer”? That is the consequence of the bytes of “Führer” stored as UTF-8 being interpreted as Latin-1.

You can trivially see this problem using Ruby 1.9′s encoding support by running this program:

# encoding: UTF-8
 
puts "hello ümlaut".force_encoding("ISO-8859-1").encode("UTF-8")
 
# Output
# hello ümlat

First, we create a String (“hello ümlaut”) in the UTF-8 encoding. Next, we tell Ruby that the String is actually Latin-1. It’s not, so an attempt to read the characters will interpret the raw bytes of the “ü” as though they were Latin-1 bytes. We ask Ruby to give us that interpretation of the data in UTF-8 via encode and print it out.

We can see that while the bytes for “hello ” and “mlat” were identical in both UTF-8 and Latin-1, the bytes for “ü” in UTF-8 mean “ü” in Latin-1.

Note that while force_encoding simply tags the String with a different encoding, encode converts the bytes of one encoding into the equivalent bytes of the second. As a result, while force_encoding should almost never be used unless you know for sure that the bytes actually represent the characters you want in the target encoding, encode is relatively safe to use to convert a String into the encoding you want.

You’ve probably also seen the reverse problem, where bytes encoded in Latin-1 ended up inside a page encoded in UTF-8.

# encoding: ISO-8859-1
 
puts "hello ümlaut".force_encoding("UTF-8")
 
# Output
# hello ?mlat

Here, the sequence of bytes that represents an “ü” in Latin-1 could not be recognized in UTF-8, so they were replaced with a “?”. Note that puts will always simply write out the bytes to your terminal, and the terminal’s encoding will determine how they are interpreted. The examples in this post are all outputted to a terminal using UTF-8 encoding.

As you can imagine, this presents quite the issue when concatenating two Strings of different encodings. Simply smashing together the raw bytes of the two Strings can result in output that is incomprehensible in either encoding. To make matters worse, it’s not always possible to represent all of the characters in one encoding in another. For instance, the characters of the Emoji encoding cannot be represented in the ISO-8859-1 encoding (or even in a standardized way onto the UTF-8 encoding).

As a result, when you attempt to concatenate two Strings of different encodings in Ruby 1.9, Ruby displays an error.

# encoding: UTF-8
 
puts "hello ümlaut".encode("ISO-8859-1") + "hello ümlaut"
 
# Output
# incompatible character encodings: ISO-8859-1 and UTF-8 (Encoding::CompatibilityError)

Because it’s extremely tricky for Ruby to be sure that it can make a lossless conversion from one encoding to another (Ruby supports almost 100 different encodings), the Ruby core team has decided to raise an exception if two Strings in different encodings are concatenated together.

There is one exception to this rule. If the bytes in one of the two Strings are all under 127 (and therefore valid characters in ASCII-7), and both encodings are compatible with ASCII-7 (meaning that the bytes of ASCII-7 represent exactly the same characters in the other encoding), Ruby will make the conversion without complaining.

# encoding: UTF-8
 
puts "hello umlat".encode("ISO-8859-1") + "hello ümlaut"
 
# Output
# hello umlathello ümlaut

Since Ruby does not allow characters outside of the ASCII-7 range in source files without a declared encoding, this exception eliminates a large number of potential problems that Ruby’s strict concatenation rules might have introduced.

Binary Strings

By default, Strings with no encoding in Ruby are tagged with the ASCII-8BIT encoding, which is an alias for BINARY. Essentially, this is an encoding that simply means “raw bytes here”.

In general, code in Rails applications should not encounter BINARY strings, except for Strings created in source files without encodings. However, since these Strings will virtually always fall under the ASCII-7 exception, Ruby programmers should never have to deal with incompatible encoding exceptions where one of the two encodings is ASCII-8BIT (i.e. BINARY).

That said, almost all of the encoding problems reported by users in the Rails bug tracker involved ASCII-8BIT Strings. How did this happen?

There are two reasons for this.

The first reason is that early on, database drivers generally didn’t properly tag Strings they retrieved from the database with the proper encoding. This involves a manual mapping from the database’s encoding names to Ruby’s encoding names. As a result, it was extremely common from database drivers to return Strings with characters outside of the ASCII-7 range (because the original content was encoded in the database as UTF-8 or ISO-8859-1/Latin-1).

When attempting to concatenate that content onto another UTF-8 string (such as the buffer in an ERB template), Ruby would raise an incompatible encoding exception.

# encoding: UTF-8
 
puts "hello ümlaut" + "hello ümlaut".force_encoding("BINARY")
 
# Output
# incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)

This is essentially identical to the scenario many people encountered. A UTF-8 String was presented to Ruby as a BINARY String, since the database driver didn’t tag it. When attempting to concatenate it onto UTF-8, Ruby had no way to do so reliably, so it raised an exception.

One reason that many people didn’t encounter this problem was that either the contents of the template or the text from the database were entirely in the ASCII-7 subset of their character set. As a result, Ruby would not complain. This is deceptive, because if they made a small change to their template, or if a user happened to enter non-ASCII-7 data (for instance, they got their first user named José), they would suddenly start seeing an incompatible encoding exception.

When people see this incompatible encoding exception, one common reaction is to call force_encoding("UTF-8") on the BINARY data. This will work great for Strings whose bytes actually are encoded in UTF-8. However, if people whose Strings were encoded in ISO-8859-5 (Russian) followed this instruction, they would end up with scrambled output.

Additionally, it’s impossible to simply encode the data, since Ruby doesn’t actually know the source encoding. In essence, a crucial piece of information has been lost at the database driver level.

Unfortunately, this means that well-meaning people who have solved their problem by force_encoding their Strings to UTF-8 (because the bytes actually did represent UTF-8 characters) become baffled when their solution doesn’t work for someone working on a Russian website.

Thankfully, this situation is now mostly solved. There are updates for all database drivers that map the encodings from the database to a Ruby encoding, which means that UTF-8 text from the database will be UTF-8 Strings in Ruby, and Latin-1 text from the database will be ISO-8859-1 Strings in Ruby.

Unfortunately, there is a second large source of BINARY Strings in Ruby. Specifically, data received from the web in the form of URL encoded POST bodies often do not specify the content-type of the content sent from forms.

In many cases, browsers send POST bodies in the encoding of the original document, but not always. In addition, some browsers say that they’re sending content as ISO-8859-1 but actually send it in Windows-1251. There is a long thread on the Rack tracker about this, but the bottom line is that it’s extremely difficult to determine the encoding of a POST body sent from the web.

As a result, Rack handlers send the raw bytes through as BINARY (which is reasonable, since handlers shouldn’t be in the business of trying to wade through this problem) and no middleware exists (yet) to properly tag the String with the correct encoding.

This means that if the stars align, the raw bytes are UTF-8, end up in a UTF-8 database, and end up coming back out again tagged as UTF-8. If the stars do not align, the text might actually be encoded in ISO-8859-1, get put into a UTF-8 database, and come out tagged as UTF-8 (and we know what happens when ISO-8859-1 data is mistakenly tagged as UTF-8).

In this case, because the ISO-8859-1 data is improperly tagged as UTF-8, Ruby happily concatenates it with other UTF-8 Strings, and hilarity ensues.

Because English characters have the same byte representation in all commonly used encodings, this problem is not as common as you might imagine. Unfortunately, this simply means that people who do encounter it are baffled and find it hard to get help. Additionally, this problem doesn’t manifest itself as a hard error. it can go unnoticed and dismissed as a minor annoyance if the number of non-ASCII-7 characters are low.

In order to properly solve this problem for Ruby 1.9, we need a very good heuristic for properly determining the encoding of web-sent POST bodies. There are some promising avenues that will get it right 99.9% of the time, and we need to package them into up a middleware that will tag Strings correctly.

Incompatible Encodings

If you’ve been paying attention, you’ve probably noticed that while the database drivers have solved one problem, they actually introduced another one.

Imagine that you’re using a MySQL database encoded in ISO-8859-1 (or ISO-8859-5, popular for Russian applications, or any other non-UTF-8 encoding). Now that the String coming back from the database is properly tagged as ISO-8859-1, Ruby will refuse to concatenate it onto the ERB buffer (which is encoded in UTF-8). Even if we solved this problem for ERB, it could be trivially reintroduced in other parts of the application through regular concatenation (+, concat, or even String interpolation).

Again, this problem is somewhat mitigated due to the ASCII-7 subset exception, which means that as long as one of the two incompatible Strings uses only English characters, users won’t see any problems. Again, because this “solution” means that the Ruby developer in question still may not understand encodings, this simply defers the problem to some uncertain point in the future when they either add a non-ASCII-7 character to their template or the user submits a non-ASCII-7 String.

The Solution

If you got this far, you’re probably thinking “Holy shit this encoding stuff is crazy. I don’t want to have to know any of this! I just want to write my web app!”

And you’d be correct.

Other languages, such as Java and Python, solve this problem by encodeing every String that enters the language as UTF-8 (or UTF-16). Theoretically, it is possible to represent the characters of every encoding in UTF-8. By doing this, programmers only ever deal with one kind of String, and concatenation happens between UTF-8 Strings.

However, this solution does not work very well for the Japanese community. For a variety of complicated reasons, Japanese encoding, such as SHIFT-JIS, are not considered to losslessly encode into UTF-8. As a result, Ruby has a policy of not attempting to simply encode any inbound String into UTF-8.

This decision is debatable, but the fact is that if Ruby transparently transcoded all content into UTF-8, a large portion of the Ruby community would see invisible lossy changes to their content. That part of the community is willing to put up with incompatible encoding exceptions because properly handling the encodings they regularly deal with is a somewhat manual process.

On the other hand, many Rails applications work mostly with encodings that trivially encode to UTF-8 (such as UTF-8 itself, ASCII, and the ISO-8859-1 family). For this rather large part of the community, having to manually encode Strings to solve incompatible encoding problem feels like a burden that belongs on the machine has been inappropriately shifted onto Rails application developers.

But there is a solution.

By default, Ruby should continue to support Strings of many different encodings, and raise exceptions liberally when a developer attempts to concatenate Strings of different encodings. This would satisfy those with encoding concerns that require manual resolution.

Additionally, you would be able to set a preferred encoding. This would inform drivers at the boundary (such as database drivers) that you would like them to convert any Strings that they tag with an encoding to your preferred encoding immediately. By default, Rails would set this to UTF-8, so Strings that you get back from the database or other external source would always be in UTF-8.

If a String at the boundary could not be converted (for instance, if you set ISO-8859-1 as the preferred encoding, this would happen a lot), you would get an exception as soon as that String entered the system.

In practice, almost all usage of this setting would be to specify UTF-8 as a preferred encoding. From your perspective, if you were dealing in UTF-8, ISO-8859-* and ASCII (most Western developers), you would never have to care about encodings.

Even better, Ruby already has a mechanism that is mostly designed for this purpose. In Ruby 1.9, setting Encoding.default_internal tells Ruby to encode all Strings crossing the barrier via its IO system into that preferred encoding. All we’d need, then, is for maintainers of database drivers to honor this convention as well.

It doesn’t require any changes to Ruby itself, and places the burden squarely on the few people who already need to deal with encodings (because taking data from outside of Ruby, via C, always already requires a tagging step). I have spoken with Aaron Patterson, who has been working on the SQLite3 driver, and he feels that this change is simple enough for maintainers of drivers dealing with external Strings to make it a viable option. He has already patched SQLite3 to make it respect default_internal.

However you feel about Ruby’s solution to expose String encodings directly in the language, you should agree that since we’re stuck with it for the forseeable future, this solution shifts the burden of dealing with it from the unwashed masses (most of whom have no idea what an encoding is) to a few maintainers of C extensions and libraries that deal in binary data. Getting this right as soon as possible will substantially ease the transition from Ruby 1.8 to Ruby 1.9.

Postscript: What Happened in 1.8!?

When people first move to 1.9 and encounter these shenanigans, they often wonder why everything seemed so simple in Ruby 1.8, and yet seemed to work.

There are a few reasons for this.

First, keep in mind that in Ruby 1.8, Strings are simple sequences of bytes. Ruby String operations just concatenate those byte sequences together without any kind of check. This means that concatenating two UTF-8 Strings together will just work, since the combined byte sequence is still valid UTF-8. As long as the client for the Ruby code (such as the browser) is told that the bytes are encoded in UTF-8, all is well. Rails does this by setting the default charset for all documents to UTF-8.

Second, Ruby 1.8 has a “UTF-8″ mode that makes its regular expression engine treat all Strings as UTF-8. In this mode (which is triggered by setting $KCODE = “UTF-8″), the regular expression engine correctly matches a complete UTF-8 character for /./, for instance. Rails sets this global by default, so if you were using Rails, regular expressions respect unicode characters, not raw bytes.

Third, very little non-English content in the wild is actually encoded in ISO-8859-1. If you were expecting to deal with content that was not English, you would probably set your MySQL database to use a UTF-8 encoding. Since Rails sets UTF-8 as the charset of outbound documents, most browsers will in fact return UTF-8 encoded data.

Fourth, the problems caused when an ISO-8859-1 String is accidentally concatenated into a UTF-8 String are not as jarring as the errors produced by Ruby 1.9. Let’s try a little experiment. First, open up a text editor, create a new file, and save it in the ISO-8859-1 encoding.

$KCODE = "UTF-8"
 
iso_8859_1 = "ümlaut"
 
# the byte representation of ümlaut in unicode
utf8 = "\xC3\xBCmlat"
 
puts iso_8859_1
puts utf8
 
puts iso_8859_1 + utf8
puts utf8 + iso_8859_1
 
# Output
# ?mlat
# ümlaut
# ?mlatümlaut
# ümlaut?mlat

If you somehow get ISO-8859-1 encoded content that uses characters outside of the ASCII-7 range, Ruby doesn’t puke. Instead, it simply replaces the unidentified character with a “?”, which can easily go unnoticed in a mostly English site with a few “José”s thrown into the mix. It could also easily be dismissed as a “weird bug that we don’t have time to figure out right now”.

Finally, Rails itself provides a pure-Ruby UTF-8 library that mops up a lot of the remaining issues. Specifically, it provides an alternate String class that can properly handle operations like split, truncate, index, justify and other operations that need to operate on characters, not bytes. It then uses this library internally in helpers like truncate, transparently avoiding a whole other class of issue.

In short, if you’re dealing mostly with English text, and you get unlucky enough the get ISO-8859-1 input from somewhere, the worst case is that you get a “?” instead of a “é”. If you’re dealing with a lot of non-English text, you’re probably being not using ISO-8859-1 sources. In either case, English (ASCII) text is compatible with UTF-8, and Rails provides solid enough pure-Ruby UTF-8 support to get you most of the rest of the way.

That said, anyone dealing with encodings other than UTF-8 and ISO-8859-1 (Japanese and Russian Rubyists) were definitely not in a good place with Ruby 1.8.

Thanks

I want to personally thank Jay Freeman (aka saurik), who in addition to being a general badass, spent about 15 hours with me patiently explaining these issues and working through the Ruby 1.9 source to help fully understand the tradeoffs available.

The Web Doesn’t Suck. Browsers Are Innovating.

This week saw a flurry of back-and-forth about the future of the web platform. In the “web sucks” camp were Sachin Agarwal of Posterous (The Web Sucks. Browsers need to innovate) and Joe Hewitt (Sachin summarized some of his tweets at @joehewitt agrees with me).

Chris Blizzard responded with a few narrow examples of what Firefox has been doing lately (geolocation, orientation, WebGL).

On Hacker News, most of the comments took Sachin to task for his argument that standards don’t matter, pointing people back to the “bad old days” of the browser wars.

In my opinion, both camps kind of miss the point.

Sachin made some very pointed factual claims which are just complete, pure hokum. Close to the top of his post, he says:

Web applications don’t have threading, GPU acceleration, drag and drop, copy and paste of rich media, true offline access, or persistence. Are you kidding me?

Really? Are you kidding me. In fact, browsers have implemented, and the WHAT-WG has been busily standardizing, all of these things for quite a number of years now. Threading? Check. GPU acceleration? Check and check. Drag and drop? Check. Offline access and persistence? Check and check.

Almost universally, these features, which exist in shipping browsers today, were based on experiments conducted years ago by browsers (notably Firefox and Safari, more recently Chrome), and did not wait for approval from the standards bodies to begin implementing.

In fact, large swaths of HTML5 are based on proposals from browsers, who have either already implemented the feature (CSS-based animations and transitions) or who then build the feature without waiting for final approval. And this is transparently obvious to anyone, since HTML5 has not not yet been approved, and some parts are the subject of internal W3C debate, yet the browsers are implementing the parts they like and innovating in other areas.

You can see this explicitly in the features prefixed with -moz or -webkit, because they’re going ahead and implementing features that have not gotten consensus yet.

In 2010, the WHAT-WG is a functioning place to bring a proposal for further review once you’re conducting some experiments or have a working implementation. Nobody is sitting on their hands waiting for final approval of HTML5. Don’t confuse the fact that people are submitting their ideas to the standards bodies with the misguided idea that only approved standards are being implemented.

And in fact, this is basically never how it’s worked. Apple implemented the <canvas> tag and <input type="search" /> in 2004 (here’s a 2004 rant from the editor of the then-brand-new HTML5 spec). Opera and Apple worked on the <video> tag in 2007. The new CSS flexible box model is based on XUL, which Firefox has had for over a decade. HTML5 drag and drop support is based on implementations shipped by the IE team over a decade ago. Firefox has been extending JavaScript for almost its entire history (most recently with JavaScript 1.8).

CSS transition, transforms and animations were built by Apple for the iPhone before they were a spec, and only later ported to the desktop. Firefox did the initial experimentation on WebGL in 2006, back when they were calling it Canvas 3d (here‘s a working add-on by Mozilla developer Vladimir Vukićević).

Google Gears shipped what would become the Application Cache, Web Storage, and Web Workers, proving out the concepts. It also shipped geolocation, which is now part of the HTML5 spec.

@font-face originally shipped in Internet Explorer 5.5. Apple has added new mobile events (touchstart, touchmove, touchend, onorientationchange) with accompanying JavaScript APIs (Apple developer account needed) without any spec activity that I can discern. Firefox, at the same time, added support for hardware accelerometers.

Even Microsoft bucked the standards bodies by shipping its own implementation of the W3C’s “cross-origin-resource-sharing” spec called Cross domain request, and shipped it in IE8.

It’s perfectly understandable that people who haven’t been following along for the past few years might have missed all of this. But the fact remains that browser vendors have been moving at a very rapid clip, implementing all kinds of interesting features, and then sharing them with other browsers for feedback, and, one hopes, consensus.

Some of these implementations suck (I’ve heard some people say not-nice things about WebGL and drag-and-drop, for instance). But that’s quite a bit different from saying that browsers are sitting on their hands waiting for the W3C or WHAT-WG to tell them what to do.