The Facebook Patent License Punishes You For Suing Facebook, But Lets Them Sue You

There’s been a lot of discussion recently about the Facebook patent clause (“PATENTS” in Facebook repositories).

While most of the objections to the license have focused on the patent revocation provisions, most of the defense focuses on the patent grant. This has meant that both sides are talking past each other, and casual readers are getting confused about what this is all about.


The Facebook patent grant comes with a revocation clause. It is meant to protect Facebook from patent lawsuits in general. It applies to Facebook’s patents, and therefore the revocation clause does not apply to Facebook itself (by definition). Your license to Facebook’s patents in order to use the OSS is revoked if:

  • You sue Facebook for patent infringement, or
  • You sue someone for patent infringement for using a Facebook product or service, or
  • You sue someone for using the OSS software

Notably, it does nothing to punish Continue reading “The Facebook Patent License Punishes You For Suing Facebook, But Lets Them Sue You”

The Facebook Patent License Punishes You For Suing Facebook, But Lets Them Sue You

There’s been a lot of discussion recently about the Facebook patent clause ("PATENTS" in Facebook repositories).

While most of the objections to the license have focused on the patent revocation provisions, most of the defense focuses on the patent grant. This has meant that both sides are talking past each other, and casual readers are getting confused about what this is all about.


The Facebook patent grant comes with a revocation clause. It is meant to protect Facebook from patent lawsuits in general. It applies to Facebook’s patents, and therefore the revocation clause does not apply to Facebook itself (by definition). Your license to Facebook’s patents in order to use the OSS is revoked if:

  • You sue Facebook for patent infringement, or
  • You sue someone for patent infringement for using a Facebook product or service, or
  • You sue someone for using the OSS software

Notably, it does nothing to punish Continue reading “The Facebook Patent License Punishes You For Suing Facebook, But Lets Them Sue You”

The Glimmer VM: Boots Fast and Stays Fast

Great web applications boot up fast and stay silky smooth once they’ve started.

In other contexts, applications can choose quick loading or responsiveness once they’ve loaded. Great games can get away with a long loading bar as long as they react instantly once the gamer gets going. In contrast, scripting languages like Ruby, Python or Bash optimize for instant boot, but run their programs more slowly.

To optimize for boot time, scripting languages use interpreters and avoid expensive compilation steps. To optimize for responsiveness, games pre-fill their caches and do as much work up front as they can get away with. The web demands that we do both at the same time: users coming from search results pages must see content within a second on modern devices, but also demand 60fps once the application gets going.

Over the years, web browsers have responded to this requirements with more JIT tiers

Fast Updates in Ember

Continue reading “The Glimmer VM: Boots Fast and Stays Fast”

The Glimmer VM: Boots Fast and Stays Fast

Great web applications boot up fast and stay silky smooth once they’ve started.

In other contexts, applications can choose quick loading or responsiveness once they’ve loaded. Great games can get away with a long loading bar as long as they react instantly once the gamer gets going. In contrast, scripting languages like Ruby, Python or Bash optimize for instant boot, but run their programs more slowly.

To optimize for boot time, scripting languages use interpreters and avoid expensive compilation steps. To optimize for responsiveness, games pre-fill their caches and do as much work up front as they can get away with. The web demands that we do both at the same time: users coming from search results pages must see content within a second on modern devices, but also demand 60fps once the application gets going.

Over the years, web browsers have responded to this requirements with more JIT tiers

Fast Updates in Ember

Continue reading “The Glimmer VM: Boots Fast and Stays Fast”

Why I’m Working on Yarn

(This post is about Yarn, a new JS package manager that was announced today.)

I work with Node and npm packages almost every day, on Tilde’s main app, Skylight, or on one of Ember’s many packages.

Many have remarked upon how fast the npm registry has grown, and it’s hard to imagine working on any of my packages without the npm ecosystem.

I’ve also worked on a couple of application-level package managers (Bundler for Ruby and Cargo for Rust), so it’s no surprise that people have routinely asked me whether I’d consider writing a "bundler for node".

While it’s something I considered idly from time to time, the truth is that for all of the complaints people have about the official client, it does a whole lot that people rely on, and the npm team has done a lot to improve it over the years. I genuinely respect Continue reading “Why I’m Working on Yarn”

Why I’m Working on Yarn

(This post is about Yarn, a new JS package manager that was announced today.)

I work with Node and npm packages almost every day, on Tilde’s main app, Skylight, or on one of Ember’s many packages.

Many have remarked upon how fast the npm registry has grown, and it’s hard to imagine working on any of my packages without the npm ecosystem.

I’ve also worked on a couple of application-level package managers (Bundler for Ruby and Cargo for Rust), so it’s no surprise that people have routinely asked me whether I’d consider writing a “bundler for node”.

While it’s something I considered idly from time to time, the truth is that for all of the complaints people have about the official client, it does a whole lot that people rely on, and the npm team has done a lot to improve it over the years. I genuinely respect Continue reading “Why I’m Working on Yarn”

An Extensible Approach to Browser Security Policy

Alex Russell posted some thoughts today about how he wishes the W3C would architect the next version of the Content Security Policy.

I agree with Alex that designing CSP as a “library” that uses other browser primitives would increase its long-term utility and make it compose better with other platform features.

Alex is advocating the use of extensible web principles in the design of this API, and I wholeheartedly support his approach.

Background

You can skip this section if you already understand CSP.

For the uninitiated, Content Security Policy is a feature that allows web sites to opt into stricter security than what the web platform offers by default. For example, it can restrict which domains to execute scripts from, prevent inline scripts from running altogether, and control which domains the network stack is allowed to make HTTP requests to.

To opt into stricter security using the current version of CSP, a website includes a new header (Content-Security-Policy) which can contain a number of directives.

For example, in order to prevent the browser from making any network requests to cross-domain resources, a server can return this header:

Content-Security-Policy: default-src 'self'

This instructs the browser to restrict all network requests to the current domain. This includes images, stylesheets, and fonts. Essentially, this means that scripts run on your page will be unable to send data to third-party domains, which is a common source of security vulnerabilities.

If you want to allow the browser to make requests to its own domain, plus the Google Ajax CDN, your server can do this:

Content-Security-Policy: default-src 'self' ajax.googleapis.com

Factoring the Network Layer

If you look at what CSP is doing, it’s essentially a syntax for controlling what the network stack is allowed to do.

There are other parts of the web platform that likewise control the network stack, and more all the time. What you’d like is for all of these features to be defined in terms of some lower-level primitive—ideally, one that was also exposed to JavaScript itself for more fine-grained, programmatic tweaks.

Imagine that you had the ability to intercept network requests programmatically, and decide whether to allow the request to continue. You might have an API something like this:

var origin = window.location.origin;
 
page.addEventListener('fetch', function(e) {
  var url = e.request.url;
  if (origin !== url.origin) {
    // block the network request
    e.preventDefault();
  }
 
  // otherwise, allow the network request through
});

You would then be able to describe how the browser interprets CSP in terms of this primitive API.

You could even imagine writing a CSP library purely in JavaScript!

page.addEventListener('fetch', function(e) {
  if (e.type === 'navigate') {
    e.respondWith(networkFetch(url).then(function(response) {
      // extract CSP headers and associate them with e.window.id
      // this is a pseudo-API to keep the implementation simple
      CSP.setup(e.window.id, response);
 
      return response;
    });
  } else {
    if (!CSP.isAllowed(e.window.id, e.request)) {
      e.preventDefault();
    }
  }
});

The semantics of CSP itself can be expressed in pure JavaScript, so these primitives are enough to build the entire system ourselves!

I have to confess, I’ve been hiding something from you. There is already a proposal to provide exactly these network layer hooks. It even has exactly the API I showed above.

The Extensible Web

Extensible web principles give us a very simple path forward.

Continue iterating on the declarative form of the Content Security Policy, but describe it in terms of the same primitives that power the Navigation Controller proposal.

When web developers want to tweak or extend the built-in security features, they can write a library that intercepts requests and applies tweaks to the policy by extending the existing header syntax.

If all goes well, those extensions will feed into the next iteration of CSP, giving us a clean way to let platform users inform the next generation of the platform.

This approach also improves the likelihood that other features that involve the network stack will compose well with CSP, since they will also be written in terms of this lower level primitive.

Many of the benefits that Dave Herman outlined in the closing of my last post are brought into concrete terms in this example.

I hope to write more posts that explore how extensible web principles apply to platform APIs, both new and old.


Fellow web developers, let’s persuade Adam Barth, Dan Veditz, Mike West (the CSP specification editors) to factor the next version of CSP in terms of the new Navigation Controller specification.

Then, we will have the tools we need to extend the web’s security model forward.

An Extensible Approach to Browser Security Policy

Alex Russell posted some thoughts today about how he wishes the W3C would architect the next version of the Content Security Policy.

I agree with Alex that designing CSP as a “library” that uses other browser primitives would increase its long-term utility and make it compose better with other platform features.

Alex is advocating the use of extensible web principles in the design of this API, and I wholeheartedly support his approach.

Background

You can skip this section if you already understand CSP.

For the uninitiated, Content Security Policy is a feature that allows web sites to opt into stricter security than what the web platform offers by default. For example, it can restrict which domains to execute scripts from, prevent inline scripts from running altogether, and control which domains the network stack is allowed to make HTTP requests to.

To opt into stricter security using the current version of Continue reading “An Extensible Approach to Browser Security Policy”

An Extensible Approach to Browser Security Policy

Alex Russell posted some thoughts today about how he wishes the W3C would architect the next version of the Content Security Policy.

I agree with Alex that designing CSP as a “library” that uses other browser primitives would increase its long-term utility and make it compose better with other platform features.

Alex is advocating the use of extensible web principles in the design of this API, and I wholeheartedly support his approach.

Background

You can skip this section if you already understand CSP.

For the uninitiated, Content Security Policy is a feature that allows web sites to opt into stricter security than what the web platform offers by default. For example, it can restrict which domains to execute scripts from, prevent inline scripts from running altogether, and control which domains the network stack is allowed to make HTTP requests to.

To opt into stricter security using the current version of Continue reading “An Extensible Approach to Browser Security Policy”

Extend the Web Forward

If we want to move the web forward, we must increase our ability as web developers to extend it with new features.

For years, we’ve grabbed the browsers extension points with two hands, not waiting for the browser vendors to gift us with new features. We built selector engines, a better DOM API, cross-domain requests, cross-frame APIs.

When the browser has good extension points (or any extension points, really), we live in a virtuous cycle:

  • Web developers build new APIs ourselves, based on use-cases we have
  • We compete with each other, refining our libraries to meet use cases we didn’t think of
  • The process of competition makes the libraries converge towards each other, focusing the competition on sharp use-case distinctions
  • Common primitives emerge, which browser vendors can implement. This improves performance and shrinks the amount of library code necessary.
  • Rinse, repeat.

We’ve seen this time and time again. When it works, it brings us querySelectorAll, the template element, and Object.observe.

The Sad Truth

The sad truth is that while some areas of the browser are extremely extensible, other areas are nearly impossible to extend.

Some examples include the behavior and lifecycle of custom element in HTML, the CSS syntax, and the way that the browser loads an HTML document in the first place. This makes it hard to extend HTML, CSS, or build libraries that support interesting offline capabilities.

And even in some places that support extensibility, library developers have to completely rewrite systems that already exist. For example, John Resig had to rewrite the selector engine from scratch just to add a few additional pseudo-properties, and there is still no way add custom pseudo-properties to querySelectorAll.

Declarative vs. Imperative

A lot of people see this as a desire to write everything using low-level JavaScript APIs, forever.

No.

If things are working well, JavaScript library authors write new declarative APIs that the browser can roll in. Nobody wants to write everything using low-level calls to canvas, but we’re happy that canvas lets us express low-level things that we can evolve and refine.

The alternative, that web authors are stuck with only the declarative APIs that standards bodies have invented, is too limiting, and breaks the virtuous cycle that allows web developers to invent and iterate on new high-level features for the browser.

In short, we want to extend the web forward with new high-level APIs, but that means we need extension points we can use.

Explaining the Magic

If we want to let web authors extend the web forward, the best way to do that is to explain existing and new high-level forms in terms of low-level APIs.

A good example of in-progress work along these lines in Web Components, which explains how elements work in terms of APIs that are exposed to JavaScript. This means that if a new custom element becomes popular, it’s a short hop to implementing it natively, because the JavaScript implementation is not a parallel universe; it’s implemented in terms of the same concepts as native elements.

That doesn’t necessarily mean that browsers will simply rubber-stamp popular components, but by giving library authors the the tools to make components with native-like interfaces, it will be easy for vendors to synthesize web developer zeitgeist into something standard.

Another example is offline support. Right now, we have the much-derided AppCache, which is a declarative-only API that makes it possible to display an HTML page, along with its assets, even if the browser is offline.

AppCache is not implemented in terms of a low-level JavaScript API, so when web developers discovered problems with it, we had no way to extend or modify it to meet our needs. This also meant that we had no way to show the browser vendors what kinds of solutions would work for us.

Instead, we ended up with years of stagnation, philosophical disagreements and deadlock between web developers and specification editors, and no way to move forward.

What we need instead is something like Alex Russell’s proposal that allows applications to install JavaScript code in the cache that intercepts HTTP requests from the page and can fulfill them, even when the app is offline. With an API like this, the current AppCache could be written as a library!

Something like Jonas Sicking’s app cache manifest is a great companion proposal, giving us a nice starting point for a high-level API. But this time if the high-level API doesn’t work, we can fix it by using the low-level API to tweak and improve the manifest.

We can extend the web forward.

Extensions != Rewriting

It’s important to note that web developers don’t want a high level API and then a cliff into the low-level API.

Today, while you can implement custom elements or extend the selector engine, you can only do this by rewriting large chunks of the stack alongside the feature you want.

Real extensibility means an architecture that lets you tweak, not rewrite. For example, it would be possible to add custom rules to CSS by writing a full selector engine and application engine, and apply rules via .style as the DOM changes. With mutation observers, this might even be feasible. In fact, this is how some of the most devious hacks in the platform today (like the Polymer Shadow DOM polyfill) actually work.

That kind of “extensibility” doesn’t fit the bill. It doesn’t compose well with other extensions, defeats the browser’s ability to do performance work on unrelated parts of the stack (because the entire stack had to be rewritten), and is too hard to provide meaningful iteration.

Browser implementers are often wary of providing extension points that can be performance footguns. The biggest footgun is using libraries that rewrite the entire stack in JavaScript, and whole-stack-rewriting strategies are the tactic du jour today. For performance, we have little to lose and much to gain by making extensions more granular.

Extend the Web Forward

So what do we gain from a more extensible web? I’ll let Dave Herman, a member of TC39, answer that for me.

  • When you design new APIs, you are forced to think about how the existing system can express most of the semantics. This cleanly separates what new power is genuinely needed and what isn’t. This prevents cluttering the semantics with unnecessary new magic
  • Avoiding new magic avoids new security surface area
  • Avoiding new magic avoids new complexity (and therefore bugs) in implementation
  • Avoiding new magic makes more of the new APIs polyfillable
  • Being more polyfillable means people can ramp up faster, leading to faster adoption and evolution of the platform
  • Avoiding new magic means that optimizations in the engines can focus on the stable core, which affects more of new APIs as they are added. This leads to better performance with less implementation effort
  • Avoiding new magic means less developer education required; people can understand new APIs more easily when they come out, because they build off of known concepts
  • This means that the underlying platform gets fleshed out to be expressive enough to prototype new ideas. Library authors can experiment with new features and create more cowpaths to fill the Web API pipeline

All this, and more! There’s something for everybody!

Implementors and web developers: let’s work together to extend the web forward!

Extend the Web Forward

If we want to move the web forward, we must increase our ability as web developers to extend it with new features.

For years, we’ve grabbed the browsers extension points with two hands, not waiting for the browser vendors to gift us with new features. We built selector engines, a better DOM API, cross-domain requests, cross-frame APIs.

When the browser has good extension points (or any extension points, really), we live in a virtuous cycle:

  • Web developers build new APIs ourselves, based on use-cases we have
  • We compete with each other, refining our libraries to meet use cases we didn’t think of
  • The process of competition makes the libraries converge towards each other, focusing the competition on sharp use-case distinctions
  • Common primitives emerge, which browser vendors can implement. This improves performance and shrinks the amount of library code necessary.
  • Rinse, repeat.

We’ve seen this time and time again. When it Continue reading “Extend the Web Forward”

Extend the Web Forward

If we want to move the web forward, we must increase our ability as web developers to extend it with new features.

For years, we’ve grabbed the browsers extension points with two hands, not waiting for the browser vendors to gift us with new features. We built selector engines, a better DOM API, cross-domain requests, cross-frame APIs.

When the browser has good extension points (or any extension points, really), we live in a virtuous cycle:

  • Web developers build new APIs ourselves, based on use-cases we have
  • We compete with each other, refining our libraries to meet use cases we didn’t think of
  • The process of competition makes the libraries converge towards each other, focusing the competition on sharp use-case distinctions
  • Common primitives emerge, which browser vendors can implement. This improves performance and shrinks the amount of library code necessary.
  • Rinse, repeat.

We’ve seen this time and time again. When it Continue reading “Extend the Web Forward”

I’m Running to Reform the W3C’s TAG

Elections for the W3C’s Technical Architecture Group are underway, and I’m running!

There are nine candidates for four open seats. Among the nine candidates, Alex Russell, Anne van Kesteren, Peter Linss, and Marcos Cáceres are running on a reform platform. What is the TAG, and what do I mean by reform?

What is the TAG?

According to the TAG’s charter, it has several roles:

  • to document and build consensus around principles of Web architecture
  • to interpret and clarify these principles when necessary
  • to resolve issues involving general Web architecture brought to the TAG
  • to help coordinate cross-technology architecture developments inside and outside W3C

As Alex has said before, the existing web architecture needs reform that would make it more layered. We should be able to explain the declarative parts of the spec (like markup) in terms of lower level primitives that compose well and that developers can use for other purposes.

And the W3C must coordinate much more closely with TC39, the (very active) committee that is designing the future of JavaScript. As a member of both TC39 and the W3C, I believe that it is vital that as we build the future of the web platform, both organizations work closely together to ensure that the future is both structurally coherent and pleasant for developers of the web platform to use.

Developers

I am running as a full-time developer on the web platform to bring that perspective to the TAG.

For the past several years, I have lobbied for more developer involvement in the standards process through the jQuery organization. This year, the jQuery Foundation joined both the W3C and ECMA, giving web developers direct representatives in the consensus-building process of building the future.

Many web developers take a very cynical attitude towards the standards process, still burned from the flames of the first browser wars. As a group, web developers also have a very pragmatic perspective: because we can’t use new features in the short-term, it’s very costly to take an early interest in standards that aren’t even done yet.

Of course, as a group, we developers don’t hesitate to complain about standards that didn’t turn out the way we would like.

(The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHOULD”, “SHOULD NOT”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC2119.)

The W3C and its working groups MUST continue to evangelize to developers about the importance of participating early and often. We MUST help more developers understand the virtues of broad solutions and looking beyond specific present-day scenarios. And we MUST evolve to think of web developers not simply as “authors” of content, but as sophisticated developers on the most popular software development platform ever conceived.

Layering

When working with Tom Dale on Ember.js, we often joke that our APIs are layered, like a delicious cake.

What we mean by layering is that our high-level features are built upon publicly exposed lower-level primitives. This gives us the freedom to experiment with easy-to-use concise APIs, while making it possible for people with slightly different needs to still make use of our hard implementation work. In many cases, such as in our data abstraction, we have multiple layers, making it possible for people to implement their requirements at the appropriate level of abstraction.

It can be tempting to build primitives and leave it up to third parties to build the higher level APIs. It can also be tempting to build higher level APIs only for particular scenarios, to quickly solve a problem.

Both approaches are prevalent on the web platform. Specs like IndexedDB are built at a very low level of abstraction, leaving it up to library authors to build a higher level of abstraction. In contrast, features like App Cache are built at a high level of abstraction, for a particular use-case, with no lower level primitives to use if a user’s exact requirements do not match the assumptions of the specification.

Alex’s effort on this topic is focused on Web Components and Shadow DOM, an effort to explain the semantics of existing tags in terms of lower-level primitives. These primitives allow web developers to create new kinds of elements that can have a similar level of sophistication to the built-in elements. Eventually, it should be possible to describe how existing elements work in terms of these new primitives.

Here’s another example a layer deeper: many parts of the DOM API have magic behavior that are extremely difficult to explain in terms of the exposed API of ECMAScript 3. For example, the innerHTML property has side-effects, and ES3 does not provide a mechanism for declaring setters. The ECMAScript 5 specification provides some additional primitives that make it possible to explain more of the existing DOM behavior in terms of JavaScript. While designing ECMAScript 6, the committee has repeatedly discussed how certain new features could help explain more of the DOM API.

Today, the web platform inherits a large number of existing specifications designed at one of the ends of the layering spectrum. I would like to see the TAG make an explicit effort to describe how the working groups can reform existing APIs to have better layering semantics, and to encourage them to build new specifications with layering in mind.

TC39 and JavaScript

Today, developers of the web platform increasingly use JavaScript to develop full-blown applications that compete with their native counterparts.

This has led to a renaissance in JavaScript implementations, and more focus on the ECMAScript specification itself by TC39. It is important that the evolution of JavaScript and the DOM APIs take one another into consideration, so that developers perceive them as harmonious, rather than awkward and ungainly.

Any developer who has worked with NodeList and related APIs knows that the discrepancies between DOM Array-likes and JavaScript Arrays cause pain. Alex has talked before about how standardizing subclassing of built-in object would improve this situation. This would allow the W3C to explicitly subclass Array for its Array-like constructs in a well-understood, compatible way. That proposal will be strongest if it is championed by active members of both TC39 and the HTML working group.

Similarly, TC39 has worked tirelessly on a proposal for loading JavaScript in an environment-agnostic way (the “modules” proposal). That proposal, especially the aspects that could impact the network stack, would be stronger with the direct involvement of an interested member of a relevant W3C working group.

As the web’s development picks up pace, the W3C cannot see itself as an organization that interacts with ECMA at the periphery. It must see itself as a close partner with TC39 in the development and evolution of the web platform.

Progress

If that (and Alex’s similar post) sounds like progress to you, I’d appreciate your organization’s vote. My fellow reformers Alex Russell, Anne van Kesteren, Peter Linss and Marcos Cáceres are also running for reform.

AC reps for each organization can vote here and have 4 votes to allocate in this election. Voting closes near the end of the month, and it’s also holiday season, so if you work at a member organization and aren’t the AC rep, please, find out who that person in your organization is and make sure they vote.

As Alex said:

The TAG can’t fix the web or the W3C, but I believe that with the right people involved it can do a lot more to help the well-intentioned people who are hard at work in the WGs to build in smarter ways that pay all of us back in the long run.

I’m Running to Reform the W3C’s TAG

Elections for the W3C’s Technical Architecture Group are underway, and I’m running!

There are nine candidates for four open seats. Among the nine candidates, Alex Russell, Anne van Kesteren, Peter Linss, and Marcos Cáceres are running on a reform platform. What is the TAG, and what do I mean by reform?

What is the TAG?

According to the TAG’s charter, it has several roles:

  • to document and build consensus around principles of Web architecture
  • to interpret and clarify these principles when necessary
  • to resolve issues involving general Web architecture brought to the TAG
  • to help coordinate cross-technology architecture developments inside and outside W3C

As Alex has said before, the existing web architecture needs reform that would make it more layered. We should be able to explain the declarative parts of the spec (like markup) in terms of lower level primitives that compose well and that developers can use for Continue reading “I’m Running to Reform the W3C’s TAG”

I’m Running to Reform the W3C’s TAG

Elections for the W3C’s Technical Architecture Group are underway, and I’m running!

There are nine candidates for four open seats. Among the nine candidates, Alex Russell, Anne van Kesteren, Peter Linss, and Marcos Cáceres are running on a reform platform. What is the TAG, and what do I mean by reform?

What is the TAG?

According to the TAG’s charter, it has several roles:

  • to document and build consensus around principles of Web architecture
  • to interpret and clarify these principles when necessary
  • to resolve issues involving general Web architecture brought to the TAG
  • to help coordinate cross-technology architecture developments inside and outside W3C

As Alex has said before, the existing web architecture needs reform that would make it more layered. We should be able to explain the declarative parts of the spec (like markup) in terms of lower level primitives that compose well and that developers can use for Continue reading “I’m Running to Reform the W3C’s TAG”

Follow Me to Google+

I wrote my first post on this blog in January 2007.

In 2007, this blog was the easiest way I had to write my thoughts down for people who cared to read them. I wrote long posts and short post (but mostly long posts). I wrote deeply technical posts. I wrote proposals. I wrote introductory posts.

I did not post often.

In 2012, there are many more ways to write and reach an audience. I write whimsically on Twitter. I write personally on Facebook. More and more, I find that I write casually on Google+.

Without the 140-character constraint of Twitter, I can start writing and stop when I reach the end of a thought. Unlike the long-form nature of my blog, I find myself writing often, whenever something is on my mind. If you’re interested in reading that sort of thing, follow my Google+ profile. Because I never remember to include one person’s Google+ account in my reading rotation, I made it easy: plus.yehudakatz.com.

I’ll keep posting long-form pieces here. To keep things simple, I’ll always link to them from Google+. If you follow me there, I’ll make sure you always know when I post something, wherever that happens to be. If you care, join me on Google+.

Follow Me to Google+

I wrote my first post on this blog in January 2007.

In 2007, this blog was the easiest way I had to write my thoughts down for people who cared to read them. I wrote long posts and short post (but mostly long posts). I wrote deeply technical posts. I wrote proposals. I wrote introductory posts.

I did not post often.

In 2012, there are many more ways to write and reach an audience. I write whimsically on Twitter. I write personally on Facebook. More and more, I find that I write casually on Google+.

Without the 140-character constraint of Twitter, I can start writing and stop when I reach the end of a thought. Unlike the long-form nature of my blog, I find myself writing often, whenever something is on my mind. If you’re interested in reading that sort of thing, follow my Google+ profile. Because I never remember Continue reading “Follow Me to Google+”

Follow Me to Google+

I wrote my first post on this blog in January 2007.

In 2007, this blog was the easiest way I had to write my thoughts down for people who cared to read them. I wrote long posts and short post (but mostly long posts). I wrote deeply technical posts. I wrote proposals. I wrote introductory posts.

I did not post often.

In 2012, there are many more ways to write and reach an audience. I write whimsically on Twitter. I write personally on Facebook. More and more, I find that I write casually on Google+.

Without the 140-character constraint of Twitter, I can start writing and stop when I reach the end of a thought. Unlike the long-form nature of my blog, I find myself writing often, whenever something is on my mind. If you’re interested in reading that sort of thing, follow my Google+ profile. Because I never remember Continue reading “Follow Me to Google+”

August Tokaido Update

It’s been a while since I posted anything on my blog, and I figured I’d catch everyone up on the work I’ve been doing on Tokaido.

Components

Tokaido itself is made up of a number of components, which I am working on in parallel:

  • Ruby binary build, statically compiled
  • A logging and alerting UI
  • Remote Notifications for Rails 3+
  • Integration with Puma Express
  • Integration with code quality tools
  • Resolve bundler issue related to binary builds and deployment to Heroku
  • Work with community members to start shipping binary builds of popular gems (gates on fixing the bundler bug)

A number of people are doing parts of the work to make Tokaido a reality. I specifically want to thank:

  • Michal Papis of the rvm team for using the sm framework to make the Tokaido build maintainable over time and doing the heavy lifting to take the initial spike I did and get a reproducible binary build
  • Terence Lee of Heroku for packaging up these early binary builds for use at several Rails Girls events, with great success!

Ruby Binary Build, Statically Compiled

This is the first work I did, a few months ago, with help from Michal Papis. I detailed the hard parts of making Ruby statically compiled in June’s status update. Since then, Terence Lee (@hone02) has used the binary build at several Rails Girls events, dramatically reducing the time needed for Rails installation on OSX. In addition to the Tokaido binary build, Terence also precompiled a number of gems, and put built a script to download the entire thing as a zip, install it into the user’s home directory, and modify their ~/.profile to include it.

This strategy works well for trainings, but has a number of limitations:

  • Since it relies on precompiled gems already existing in the gem home, the gems cannot be removed or upgraded without requiring a C compiler. The correct solution is what gems authors do for Windows: ship versions of binary gems with specific OSX designations. Currently, binary gems do not interact well with Heroku deploys (or anyone using bundle install --deployment), so we have been working to resolve those issues before foisting a bunch of new binary gems on the world. See below.
  • It relies on modifying all instances of the Terminal, which means that some system edge-cases leak into this solution. It also relies on modifying ~/.profile, which may not work depending on what shell the user is using and what other startup scripts the user is running. For Tokaido, we will have a way to launch a Terminal that reliably injects itself without these problems, and without polluting all instances of Terminal.
  • It is difficult to upgrade components, like patch levels of Ruby or versions of C dependencies like libyaml. Tokaido.app will store its copy of Ruby and gems in a sandbox, loadable from the UI (as I described in the second bullet), which makes it easy for the .app to upgrade patch levels of Ruby or whatever components it needs.

Because I understand that many people WANT the ability to have their Ruby take over their terminal, Tokaido will integrate with rvm (and rbenv, if possible) to mount Tokaido.app as the default version of Ruby in your shell.

A Logging and Alerting UI

The primary UI for Tokaido will be a logging and alerting UI for your Rails application. I expect that you will spend the most time in the “Requests” UI, which will show you a list of the previous requests, and the log of the current request. My goal is to use the logging UI to improve on the typical “tail the log” experience. I’ve been working with Austin Bales, a designer at do.com to develop some mockups, and hope to have something for people to look at in the next few weeks.

In addition to an improved logging experience, Tokaido will alert you when something has gone wrong in your request. This may include exceptions (5xx) or information provided by plugins (bullet can let you know when your code is triggering an N+1 query; see the README for more information). The list of prior requests will highlight requests with issues, and problems in the notifications tray will directly link you to the request where the error occurred.

If everything goes well, the Tokaido UI will replace your logging workflow, without impacting the rest of the tasks you perform from the command-line.

Remote Notifications for Rails 3+

Rails 3 shipped with a new instrumentation API, which provides detailed information about many events that happen inside of Rails. This system is used by Rails’ own logging system, and I would like to use it in Tokaido as well.

The instrumentation API provides all of the information we need, but no built-in way to communicate those notifications across processes. Because Tokaido.app will not run in the same process as the Rails app, I have been working on a gem (remote_notifications) that provides a standard way to send these notifications to another process.

This gem also backports Aaron’s Rails instrumentation work (see his talk at Railsberry for more information) to Rails 3.0 and 3.1, which makes it possible to build a reliable tree from notifications, even on systems with low clock resolution.

It also includes a mechanism for subscribing to notifications sent from another process using the regular notifications API, and pluggable serializers and deserializers. It isn’t quite done yet, but I should have an initial release soon.

Integration with Puma Express

Puma is a new threaded web server built by Evan Phoenix of Rubinius fame. Puma Express manages Puma servers, automatically setting up the DNS (appname.dev) for you. This is similar to the approach used by Pow, but without a Node dependency.

Tokaido will integrate with Puma Express, so you will not need to manually boot and shut down your server. Because Tokaido also takes care of logging, there shouldn’t be any need for dedicated tabs with persistent running processes when you use Tokaido.

You will also be able to install an executable into your system (a la GitX and Textmate) to make it easy to boot up a Tokaido in the context of the current application:

$ tokaido .

This will be especially useful for people using the rvm “mounted” Tokaido.

Integration with Code Quality Tools

By default, Tokaido will integrate with flog, flay and rails-best-practices. It will periodically scan your Rails app for problems and notify you via the app’s notification tray. This mechanism will be extensible, so a future version of Tokaido might integrate with Code Climate or support code quality plugins.

Binary Gems and Heroku

Bundler provides a mechanism for deployment (bundle install --deployment) that specifically rejects deploys that require changes to the Gemfile.lock. When deploying a Gemfile.lock that was built using binary gems from a different platform, this mechanism rejects the deploy. At present, only Windows makes heavy use of binary gems, and Heroku’s solution to date has been to simply remove the Gemfile.lock and re-resolve dependencies.

Unfortunately, this solution eliminates the most important guarantee made by bundler, and is untenable in the long-term. If OS X users started to use binary gems more broadly, this would bring this problem into much wider circulation. Also, because platform-specific gems can contain alternate dependencies (see, for example Nokogiri 1.4.2), it is important that bundle install support alternate platforms more extensively than with a naïve solution.

There are a few possible solutions:

  • Do not use --deployment on Heroku. This would allow bundler to update the Gemfile.lock for the new platform. It would also mean that if a developer didn’t update their Gemfile.lock before deploying, they might run unexpected code. This is a short-term fix, but would probably alleviate most of the symptoms without introducing a lot of new caveats.
  • Fix --deployment to allow changes to the Gemfile.lock, but only in response to a genuine platform change. Unfortunately, because of cases like Nokogiri 1.4.2, this could theoretically result in totally different code running in production.
  • Provide a mechanism for developers to explicitly add a deployment environment: bundle platform add x86_64-linux. This would pre-resolve gems across all available platforms and ensure that the same gem versions could be used everywhere.
  • Improve the bundler resolution mechanism to allow gems with the same name, version and dependencies to be treated the same. Because dependencies can also exhibit this problem (a dependency of some JRuby gem could be another JRuby gem with its own non-standard dependencies), this would need to ensure that the entire subtree was the same.

The last solution is most promising, because the platform-specific gems Tokaido is concerned with are simply precompiled variants of the same gem. In the vast majority of cases, this simply means that the gem author is saving the end-developer the step of using a C compiler, but doesn’t actually change a lot else.

Unfortunately, Rubygems itself doesn’t distinguish between precompiled variants of gems and gems with different dependencies, so we will have to add smarts into bundler to detect the difference. My guess is that we will use a short-term fix like the first option while working on a longer-term fix like the last option.

Work Time

As I said in the original Kickstarter, I planned to take time off from work to work on the project. I have structured that time by working mornings on Tokaido, and shifting client work to the afternoons and evenings.

I don’t have a specific ship-date in mind for Tokaido yet, but you should start seeing more work-product as the project progresses, starting over the next few weeks.

Thanks for your patience. It shall be rewarded!

August Tokaido Update

It’s been a while since I posted anything on my blog, and I figured I’d catch everyone up on the work I’ve been doing on Tokaido.

Components

Tokaido itself is made up of a number of components, which I am working on in parallel:

  • Ruby binary build, statically compiled
  • A logging and alerting UI
  • Remote Notifications for Rails 3+
  • Integration with Puma Express
  • Integration with code quality tools
  • Resolve bundler issue related to binary builds and deployment to Heroku
  • Work with community members to start shipping binary builds of popular gems (gates on fixing the bundler bug)

A number of people are doing parts of the work to make Tokaido a reality. I specifically want to thank:

  • Michal Papis of the rvm team for using the sm framework to make the Tokaido build maintainable over time and doing the heavy lifting to take the initial spike I did and get a reproducible binary build
  • Terence Lee of Heroku for packaging up these early binary builds for use at several Rails Girls events, with great success!

Ruby Binary Build, Statically Compiled

This is the first work I did, a few months ago, with help from Michal Papis. I detailed the hard parts of making Ruby statically compiled in June’s status update. Since then, Terence Lee (@hone02) has used the binary build at several Rails Girls events, dramatically reducing the time needed for Rails installation on OSX. In addition to the Tokaido binary build, Terence also precompiled a number of gems, and put built a script to download the entire thing as a zip, install it into the user’s home directory, and modify their ~/.profile to include it.

This strategy works well for trainings, but has a number of limitations:

  • Since it relies on precompiled gems already existing in the gem home, the gems cannot be removed or upgraded without requiring a C compiler. The correct solution is what gems authors do for Windows: ship versions of binary gems with specific OSX designations. Currently, binary gems do not interact well with Heroku deploys (or anyone using bundle install --deployment), so we have been working to resolve those issues before foisting a bunch of new binary gems on the world. See below.
  • It relies on modifying all instances of the Terminal, which means that some system edge-cases leak into this solution. It also relies on modifying ~/.profile, which may not work depending on what shell the user is using and what other startup scripts the user is running. For Tokaido, we will have a way to launch a Terminal that reliably injects itself without these problems, and without polluting all instances of Terminal.
  • It is difficult to upgrade components, like patch levels of Ruby or versions of C dependencies like libyaml. Tokaido.app will store its copy of Ruby and gems in a sandbox, loadable from the UI (as I described in the second bullet), which makes it easy for the .app to upgrade patch levels of Ruby or whatever components it needs.

Because I understand that many people WANT the ability to have their Ruby take over their terminal, Tokaido will integrate with rvm (and rbenv, if possible) to mount Tokaido.app as the default version of Ruby in your shell.

A Logging and Alerting UI

The primary UI for Tokaido will be a logging and alerting UI for your Rails application. I expect that you will spend the most time in the “Requests” UI, which will show you a list of the previous requests, and the log of the current request. My goal is to use the logging UI to improve on the typical “tail the log” experience. I’ve been working with Austin Bales, a designer at do.com to develop some mockups, and hope to have something for people to look at in the next few weeks.

In addition to an improved logging experience, Tokaido will alert you when something has gone wrong in your request. This may include exceptions (5xx) or information provided by plugins (bullet can let you know when your code is triggering an N+1 query; see the README for more information). The list of prior requests will highlight requests with issues, and problems in the notifications tray will directly link you to the request where the error occurred.

If everything goes well, the Tokaido UI will replace your logging workflow, without impacting the rest of the tasks you perform from the command-line.

Remote Notifications for Rails 3+

Rails 3 shipped with a new instrumentation API, which provides detailed information about many events that happen inside of Rails. This system is used by Rails’ own logging system, and I would like to use it in Tokaido as well.

The instrumentation API provides all of the information we need, but no built-in way to communicate those notifications across processes. Because Tokaido.app will not run in the same process as the Rails app, I have been working on a gem (remote_notifications) that provides a standard way to send these notifications to another process.

This gem also backports Aaron’s Rails instrumentation work (see his talk at Railsberry for more information) to Rails 3.0 and 3.1, which makes it possible to build a reliable tree from notifications, even on systems with low clock resolution.

It also includes a mechanism for subscribing to notifications sent from another process using the regular notifications API, and pluggable serializers and deserializers. It isn’t quite done yet, but I should have an initial release soon.

Integration with Puma Express

Puma is a new threaded web server built by Evan Phoenix of Rubinius fame. Puma Express manages Puma servers, automatically setting up the DNS (appname.dev) for you. This is similar to the approach used by Pow, but without a Node dependency.

Tokaido will integrate with Puma Express, so you will not need to manually boot and shut down your server. Because Tokaido also takes care of logging, there shouldn’t be any need for dedicated tabs with persistent running processes when you use Tokaido.

You will also be able to install an executable into your system (a la GitX and Textmate) to make it easy to boot up a Tokaido in the context of the current application:

$ tokaido .

This will be especially useful for people using the rvm “mounted” Tokaido.

Integration with Code Quality Tools

By default, Tokaido will integrate with flog, flay and rails-best-practices. It will periodically scan your Rails app for problems and notify you via the app’s notification tray. This mechanism will be extensible, so a future version of Tokaido might integrate with Code Climate or support code quality plugins.

Binary Gems and Heroku

Bundler provides a mechanism for deployment (bundle install --deployment) that specifically rejects deploys that require changes to the Gemfile.lock. When deploying a Gemfile.lock that was built using binary gems from a different platform, this mechanism rejects the deploy. At present, only Windows makes heavy use of binary gems, and Heroku’s solution to date has been to simply remove the Gemfile.lock and re-resolve dependencies.

Unfortunately, this solution eliminates the most important guarantee made by bundler, and is untenable in the long-term. If OS X users started to use binary gems more broadly, this would bring this problem into much wider circulation. Also, because platform-specific gems can contain alternate dependencies (see, for example Nokogiri 1.4.2), it is important that bundle install support alternate platforms more extensively than with a naïve solution.

There are a few possible solutions:

  • Do not use --deployment on Heroku. This would allow bundler to update the Gemfile.lock for the new platform. It would also mean that if a developer didn’t update their Gemfile.lock before deploying, they might run unexpected code. This is a short-term fix, but would probably alleviate most of the symptoms without introducing a lot of new caveats.
  • Fix --deployment to allow changes to the Gemfile.lock, but only in response to a genuine platform change. Unfortunately, because of cases like Nokogiri 1.4.2, this could theoretically result in totally different code running in production.
  • Provide a mechanism for developers to explicitly add a deployment environment: bundle platform add x86_64-linux. This would pre-resolve gems across all available platforms and ensure that the same gem versions could be used everywhere.
  • Improve the bundler resolution mechanism to allow gems with the same name, version and dependencies to be treated the same. Because dependencies can also exhibit this problem (a dependency of some JRuby gem could be another JRuby gem with its own non-standard dependencies), this would need to ensure that the entire subtree was the same.

The last solution is most promising, because the platform-specific gems Tokaido is concerned with are simply precompiled variants of the same gem. In the vast majority of cases, this simply means that the gem author is saving the end-developer the step of using a C compiler, but doesn’t actually change a lot else.

Unfortunately, Rubygems itself doesn’t distinguish between precompiled variants of gems and gems with different dependencies, so we will have to add smarts into bundler to detect the difference. My guess is that we will use a short-term fix like the first option while working on a longer-term fix like the last option.

Work Time

As I said in the original Kickstarter, I planned to take time off from work to work on the project. I have structured that time by working mornings on Tokaido, and shifting client work to the afternoons and evenings.

I don’t have a specific ship-date in mind for Tokaido yet, but you should start seeing more work-product as the project progresses, starting over the next few weeks.

Thanks for your patience. It shall be rewarded!