Ruby

http://s.erious.ly

Snax – scaling Rails

up and running with cassandra

Cassandra is a hybrid non-relational database in the same class as Google's BigTable. It is more featureful than a key/value store like Riak, but supports fewer query types than a document store like MongoDB. Cassandra was started by Face...

distributed systems primer, updated

Well, it's been a long time. But! I have four papers to add to my original distributed systems primer: coordination CRDTs: Consistency Without Concurrency Control, Mihai Letia, Nuno PreguiƧa, and Marc Shapiro, 2009. Guaranteeing eventual ...

object allocations on the web

How many objects does a Rails request allocate? Here are Twitter's numbers:

  • API: 22,700 objects per request
  • Website: 67,500 objects per request
  • Daemons: 27,900 objects per action

I want them to be lower. Overall, we burn 20% of our front-end CPU on garbage collection, which seems high. Each process handles ~29,000 requests before getting killed by the memory...

scribe client

I've released Scribe 0.1, a Ruby client for the Scribe remote log server.

sudo gem install scribe

Usage is simple:

client = Scribe.new
client.log("I'm lonely in a crowded room.", "Rails")

Documentation is here.

about scribe

The primary benefit of Scribe over something like syslog-ng is increased scalability, because of Scribe's fundamentally distributed architecture. Scribe also does away with the...

ree

We recently migrated Twitter from a custom Ruby 1.8.6 build to a Ruby Enterprise Edition release candidate, courtesy of Phusion. Our primary motivation was the integration of Brent's MBARI patches, which increase memory stability.

Some features of REE have no effect on our codebase, but we definitely benefit from the MBARI patchset, the...

memcached gem release

One of the hardest gems to install is no more. It's now easy to install!

Memcached 0.15 features:

  • Update to libmemcached 0.31.1
  • Bundle libmemcached itself with the gem (antifuchs)
  • UDP connection support
  • Unix domain socket support (hellvinz)
  • AUTO_EJECT_HOSTS bugfixes (mattknox)

Install with gem install memcached. Since libmemcached is bundled in, there are no longer any dependencies.

on coordination

distributed systems primer

I've been reading a bunch of papers about distributed systems recently, in order to help systematize for myself the thing that we built over the last year. Many of them were originally passed to me by Toby DiPasquale. Here is an annotated list so everyone can benefit.

It helps if you have some algorithms...

peeping into memcached

Memcached is generally treated as a black box. But what if you really need to know what's in there? Not for runtime purposes, but for optimization and capacity planning? demo $ sudo peep --pretty 2479 time | exptime | nbytes | clsid | ......

ruby gc tuning

I'd like to call out something important from my QCon slides: the Railsbench GC settings. quick study In my experience, a typical production Rails app on Ruby 1.8 can recover 20% to 40% of user CPU by applying Stefan Kaes's Railsbench GC patch t...

qcon presentation

My QCon presentation is available. Improving Running Components at Twitter Some choice Tweets: philwills: Evan Weaver on scaling twitter at #qcon was full of interesting stuff and good questions from audience. markhneedham: fascinating readin...

secret codes

Here are some secret codes I am involved with. They are some of the best codes recently coded. kestrel A replacement for Starling, the distributed message queue. Written on the JVM (Scala) because of the mature garbage collector. Has a consta...