Twitter Gem Reborn with Fewer Features and 100% More OAuth

In which I show off the rewritten Twitter gem and an example app that takes advantage of it.

The Twitter gem is kind of my baby. It was pretty much the first gem I created and as such I care for it a little more. When I originally created it, I knew nothing. I sucked. Now over time, I suck less and Twitter’s addition of OAuth seemed like the perfect time to completely rewrite it. Yay for rewrites!

The gem is now leaner, meaner and works swell with OAuth. I would imagine as I use OAuth a bit more, I’ll figure out some ways to make it even easier in the twitter gem but it works for now.

The Cuts

One of the biggest headaches and coolest things about the Twitter gem was the command line interface. It allowed for multiple accounts and was SQLite database backed. That was cool and all but it is ridiculous to have main and active record as dependencies for someone who just wanted the API wrapping aspects of the gem.

Knowing this, I have finally killed the CLI. Fear not, CLI fans, it will make a triumphant return, but that return will be in a separate gem that merely relies on the Twitter gem. The Twitter gem itself will be only a wrapper for the REST and Search APIs that Twitter provides. No more extra cruft.

OAuth

As I said, the main reason for the rewrite was that Twitter now supports OAuth. I thought about it long and hard and decided to not continue support for HTTP Authentication. Twitter has not discontinued support for HTTP Authentication, but they will in the future and I didn’t want to add a bunch of hours to releasing the updated gem simply to support something that is going to be deprecated.

So how does one use the OAuth stuff in the Twitter gem? First off, familiarize yourself with OAuth. Then, you can check out the example below pulled straight from the examples directory included in the gem (or over here on github).

require File.join(File.dirname(__FILE__), '..', 'lib', 'twitter')
require File.join(File.dirname(__FILE__), 'helpers', 'config_store')
require 'pp'

config = ConfigStore.new("#{ENV['HOME']}/.twitter")
oauth = Twitter::OAuth.new(config['token'], config['secret'])

if config['atoken'] && config['asecret']
  oauth.authorize_from_access(config['atoken'], config['asecret'])
  twitter = Twitter::Base.new(oauth)
  pp twitter.friends_timeline

elsif config['rtoken'] && config['rsecret']  
  oauth.authorize_from_request(config['rtoken'], config['rsecret'])
  twitter = Twitter::Base.new(oauth)
  pp twitter.friends_timeline

  config.update({
    'atoken'  => oauth.access_token.token,
    'asecret' => oauth.access_token.secret,
  }).delete('rtoken', 'rsecret')
else
  config.update({
    'rtoken'  => oauth.request_token.token,
    'rsecret' => oauth.request_token.secret,
  })

  # authorize in browser
  %x(open #{oauth.request_token.authorize_url})
end

The ConfigStore is a simple class that allows me to save the request and access token and secrets as I’m completing the handshakes between the script and Twitter.

As you can see, it is pretty simple. Get a request token, then use that to create an access token. From there, you can reuse the access token to authorize future requests.

An Example App

Now one might think, based on that previous example, “Thanks so much John! I really appreciate you getting the ball rolling for me with a simple example.” Friends do I have news for you. I didn’t stop there. I whipped together a little sample app, that I have been using for a few days now. Eventually, it will become a full Twitter client, but for now it is pretty basic and dirty.

It uses Clearance for authentication for authentication and automatically sends you to Twitter to authorize after firing it up (if you haven’t previously). I used Rails rescue_from to automatically send you to Twitter if any of the authorization stuff gets messed up at a later point to.

rescue_from Twitter::Unauthorized, :with => :twitter_unauthorized  
private
  def twitter_unauthorized(exception)
    redirect_to new_authorization_url
  end

Anytime the Twitter gem throws the Twitter::Unauthorized exception, you are redirected to new_authorization_url, which is powered by the AuthorizationsController, which looks like this.

class AuthorizationsController < ApplicationController
  def new
    oauth = current_user.oauth
    session['rtoken'] = oauth.request_token.token
    session['rsecret'] = oauth.request_token.secret
    redirect_to oauth.request_token.authorize_url
  end

  def show
    oauth = current_user.oauth
    oauth.authorize_from_request(session['rtoken'], session['rsecret'])

    session['rtoken'] = nil
    session['rsecret'] = nil

    current_user.update_attributes({
      :atoken => oauth.access_token.token, 
      :asecret => oauth.access_token.secret,
    })

    redirect_to root_path
  end
end

The new action stores the request token and secret and then redirects you to Twitter to authorize the app. Over at Twitter, I have registered an application and set the redirect url to go to /authorization in the app, which as you can see above, obtains the access token and secret and stores them with the current user.

You’ll probably notice that I am calling current_user.oauth and current_user.client throughout the app. These are helper methods I added to the user model to shorten up what was happening in the controller.

class User < ActiveRecord::Base
  include Clearance::App::Models::User

  attr_accessible :atoken, :asecret

  def authorized?
    !atoken.blank? && !asecret.blank?
  end

  def oauth
    @oauth ||= Twitter::OAuth.new(ConsumerConfig['token'], ConsumerConfig['secret'])
  end

  def client
    @client ||= begin
      oauth.authorize_from_access(atoken, asecret)
      Twitter::Base.new(oauth)
    end
  end
end

I could shorten up the controllers even more (and will at some point) by delegating several methods on user to user.client. I’ll post more on that later when I get around to it. Also, at some point, I’m going to remove the atoken and asecret from the user and allow support for multiple Twitter accounts for one user in the app. This should be pretty easy, but I wanted to get something working first before I complicated things.

Conclusion

From a user standpoint, OAuth is really nice. I love not having to put in my username and password. From a developer standpoint, OAuth is also pretty nice. Basically, you register your application with Twitter, provide the consumer token and key in a config file and this example app is ready to go.

As with any rewrite there are bound to be bugs. I have fixed one already that someone caught. If you catch any weird bugs using the rewritten Twitter gem, there is group and ticket information on the Twitter gem website.

Crack, The Easiest Way to Parse XML and JSON

In which I release a new gem that allows parsing XML or JSON with a simple, consistent API.

An astute reader will remember that a while back, HTTParty divorced both ActiveSupport and the JSON gem in order to make it easier to use all around. With the JSON gem went the last gem dependency, which was kind of cool.

A few days back, it occurred to me that the parsing of XML and JSON that HTTParty used might be handy outside of HTTParty. In the spirit of sharing, I whipped together a new gem, named crack, that contains the XML and JSON parsers that formerly were bundled in HTTParty.

Why Crack?

I figured the name was easy and memorable, which is a requirement for anything I’m going to release. When I thought about parsing XML and JSON, for some reason, cracking the code came to mind and thus crack had a name.

Credits

First, I’d like to make it abundantly obvious that I did not author any of this code. I tweaked it a bit and made sure it had tests, but the XML parsing was extracted from Merb (extlib) and the JSON parsing from Rails (ActiveSupport). I merely packaged them together for all to enjoy. Ok, now that we have that out of the way, let’s move onward.

So I ripped the two parsers out of HTTParty and put them in their own gem and then just set that as a dependency for HTTParty. HTTParty will still work the exact same, but if all you need is a really simple way to parse JSON or XML, crack is his name and parsing is his game.

Details

As always these days, I used shoulda and matchy for testing and jeweler to make the gem maintenance easy. That is pretty much it for details on this project. It is focused and simple so there isn’t much behind the scenes.

Installation

I registered a rubyforge project, but I’m waiting for approval. For now, you can get the gem from Github.

sudo gem install jnunemaker-crack -s http://gems.github.com

Usage

It has always slightly annoyed me that all the different XML and JSON (JSON.parse, ActiveSupport::JSON.decode) parsing mechanisms available in Ruby have different APIs. I think parse is the easiest to remember and it is consistent with HappyMapper, another project of mine, so whether you are working with XML or JSON, all you have to remember is parse.

xml = '<posts><post><title>Foobar</title></post><post><title>Another</title></post></posts>'
Crack::XML.parse(xml)
# => {"posts"=>{"post"=>[{"title"=>"Foobar"}, {"title"=>"Another"}]}}

json = '{"posts":[{"title":"Foobar"}, {"title":"Another"}]}'
Crack::JSON.parse(json)
# => {"posts"=>[{"title"=>"Foobar"}, {"title"=>"Another"}]}

That is pretty much all there is to it. Given XML or JSON, you get back a hash. The repositoryhas been up for a couple days, but I thought I would mention it here as well. The keys here are simple and consistent. If you just want to get dirty and you aren’t worried about performance, crack is a perfect fit.

OAuth Explained and What It Is Good For

In which I attempt to explain OAuth in simple terms and what it is good for.

OAuthTwitter recently announced OAuth support and that eventually they will be deprecating HTTP Basic Authentication in favor of OAuth. Knowing this, I figured it was about time to get familiar with OAuth and update the Twitter gem to use it.

Let me start by explaining my history with OAuth. I have none. There that was fast. I didn’t read the specifications or any articles on OAuth. I simply dove in code first and tried to figure out what was going on and how to make things work. I promise that what I’m about to explain will be simple and that I will not spend any time on hmac-sha1 signatures.

For the code examples below, I’ll be using the OAuth gem. Install is typical and I’ll provide it below for copy and paste.

sudo gem install oauth

Tokens and Secrets

At first, the hardest thing to figure out was all the tokens and secrets. Basically, there are three sets of made up of a token and a secret. Each set builds upon the last. The three sets are consumer, request and access.

The consumer token and secret are provided for you by the OAuth provider, when you register an application with them. These basically define what application is attempting to do the deed. You can create a new consumer like this:

consumer = OAuth::Consumer.new(
  'consumer token', 
  'consumer secret', 
  {:site => 'http://twitter.com'}
)

Before I go on, let me explain the end goal. The end goal with OAuth is to get an access token and secret. Once you have these, requesting a users information is much like it would be with HTTP Basic Authentication from a Ruby API point of view.

Request Token

In order to get the access token, you have to create a request token, keep track of it, and then redirect the user to the provider to authorize your application. You can create a new request token, using the consumer you just created, like so:

request_token = consumer.get_request_token
puts request_token.token, request_token.secret

The other thing that the request token provides is an authorization URL. The authorization URL is where your application redirects to, to allow the user to grant or deny you access to the their data. You can get the authorization URL in quite a predictable manner:

request_token.authorize_url

If you were using Rails, you would simple redirect to this URL. Before you redirect, be sure to store the request token and secret as you’ll need those to create the access token when the user returns to your application.

Whether your app is on the web or something else, the aforementioned steps are the same. The difference is if you have a web app, you provide a redirect URL that the provider sends the user back to upon granting access, whereas if you have a desktop app, or the like, the page will just inform the user to head back to the app.

Access Token

Either way, once the user is back at your app, you use the request token and secret to generate the access token. In the code below, you would use the request token and secret that you stored in the session before redirecting to the authorization URL.

request_token = OAuth::RequestToken.new(consumer, 'request token', 'request secret')
access_token = request_token.get_access_token

Once you have the access token, you can make authenticated requests like so:

puts access_token.get('/statuses/friends_timeline.json')

I haven’t had to go this far yet, but I’m assuming you would store the access token and secret in the database or something for later use. Someone correct me if I’m wrong about this. Next time you need to access the user’s data, you can simply create a new access token from the token and secret, and once again make authenticated requests.

access_token = OAuth::AccessToken.new(consumer, 'access token', 'access secret')
puts access_token.get('/statuses/friends_timeline.json')

The Twitter Gem

So how does all this affect the Twitter gem? Currently, setting up the authentication for the Twitter gem is as easy as this:

Twitter::Base.new('email', 'password')

In the OAuth version of the Twitter gem, setting up the authentication will be something like this:

Twitter::Base.new('access token', 'access secret')

So, pretty much the same. The only difference is that you will have to register your application with Twitter and then go through the process of getting the access token and secret. The good news is I already have a Twitter::OAuth class that will make this process as easy as possible.

What It Is Good For

I see three common methods for authenticating web services—HTTP Authentication, API keys and OAuth. HTTP Auth is simple, but then you end up with usernames and passwords all over the interwebs. OAuth is better in that users don’t give out passwords and can cancel at anytime with any app they have granted permission to.

The next question might be, “Wouldn’t an API key system be more simple than OAuth?” The answer is yes—for developers. For developers, it is easier to just tack an API key on every request than to go through the process of all the aforementioned tokens and secrets. So if an API key system is easier for developers, why use OAuth?

I see OAuth as the easiest solution for users. When determining how something will work for users, I often user the mom test. I guarantee you if my mom wanted to print a picture from flickr, she would not be able to figure out how to create an API key on Flickr and then use it with the printing site. Think about the opposite though. I definitely think that when redirected to flickr from the printing site, she could click yes to grant the printing site access.

OAuth is easier for users and API keys are easier for developers. When you go to build your API, take your primary audience into consideration and pick the solution that suites them best.

Hopefully this primer helps those (like me a day ago) who haven’t taken the time to get to know OAuth, and also those who have enjoyed the Twitter gem and are curious about what implications OAuth brings to the table. Formerly, I thought OAuth was over complicated, but now that I’ve spent a few hours with it, I’m pretty comfortable. Oh, and the other cool thing about this is now I have a base to work from for adding OAuth support to HTTParty.

Any OAuth junkies out there, feel free to correct me or elaborate in the comments.

Update: For more on OAuth, you can read how I updated the Twitter gem to use it.

Some Further Reading

In 1492, Columbus Discovered…A Feed

In which I release a new gem named Columbus that auto-discovers feed urls.

Sorry for the rush of posts the past few days. I’ve been feeling a bit inspired of late, and with the discovery of Jeweler, it is now easier to make a gem than it is to just let the code sit in ~/dev/ruby/ on my computer.

A few weeks ago, I showed how to follow redirects using net/http, but I provided no context really as to why I wanted to follow redirects. Basically, that code was one piece of some code I wrote to auto-discover feed urls for a given url.

I was a little more creative with the name this time than last time, calling it Columbus. Get it? Auto-discovery and Columbus was a discoverer. Yeah, you get it. I’m sure I don’t need to explain it. Right? Yeah.

Usage

There isn’t much code in the gem and using it is even easier.

# get the primary feed
primary = Columbus.new('http://railstips.org').primary
puts primary.url, primary.title, primary.body

# get all the feeds
Columbus.new('http://railstips.org').all

The first returns a single feed if one is found else nil. The second returns all the feeds found in an array. That probably doesn’t feel like much, but there is a lot more going on behind the scenes.

Behind the Scenes

  1. Gets the response for the passed in url.
  2. If the URL is a redirect, it follows the redirect up to 5 times to find the endpoint.
  3. Once it has the endpoint, it uses Hpricot to get all the link tags in the response body that appear to be RSS or Atom feeds.
  4. For each link tag found, it gets the response for the URL and once again follows redirects up to 5 times until it finds an endpoint.
  5. Once the endpoint for each feed is found, it returns the URL, the title and the response body for you to fart around with.

Some Details

Once again, I used shoulda, matchy and fakeweb to do the testing. I didn’t need HTTParty, but I did break out an old friend Hpricot, which I haven’t used since XML parsing in the Twitter gem. Kind of funny that this is the first time I used Hpricot for its original intent, parsing HTML.

Installation

For now the gem is just up on Github so the usual routine will get your going.

sudo gem install jnunemaker-columbus --source http://gems.github.com

Hopefully someone finds it useful someday. 🙂 I’ve already got my mileage out of it.

Building API Wrapping Gems Could Not Get Much Easier

In which I show how easy it is now to create ruby gems that wrap APIs, using Google Weather as an example.

Google has a weather api that is dead simple to use. Just discovered that tonight so I whipped together a wrapper using HTTParty. I decided to try out Jeweler, a project by Josh Nichols, that makes creating gems a snap and it delivered. I used shoulda and fakeweb for the tests. Holy crap has making a gem that wraps a web service become really easy.

The New Way

  1. jeweler google-weather —shoulda —create-repo
  2. %w(matchy fakeweb).each { |x| require x } (in your test_helper)
  3. require ‘httparty’
  4. Add some code and tests
  5. rake version:bump:minor
  6. rake gemspec
  7. git push origin master
  8. blog

I did all of these in about an hour or two tonight.

The Old Way

  1. Create a bunch of files and directories and make a bunch of decisions
  2. mock and stub all net/http stuff
  3. net/http and rexml (or hpricot once that came along)
  4. Add some code and maybe some tests
  5. Add a version
  6. Figure out how to build a gemspec
  7. svn commit your files
  8. Request project to be created on rubyforge
  9. Wait a few days
  10. Project approved, release files, blog

And it would take a few days from first code scratched to gem released. My how times are a changing.

Stuff You Can Learn From This Gem

At any rate, the GoogleWeather gem I just created is a really simple example of how to use:

  • jeweler to create and manage a gem
  • httparty to pwn an API
  • shoulda to test the gem
  • fakeweb to make sure your tests aren’t making real web requests
  • matchy for some syntactical sugar

If you want to learn any of those things, poke around in the code a bit and you should be good to go. Also, if you want a really easy way to get weather information, this gem makes that possible.

Sorry I didn’t give it some fancy name like HTTParty or HappyMapper. Maybe I need to make another gem that spits out fancy names. After all, naming the project is the only thing left that is hard. 😉