ELC in Edinburgh Sponsoring Scottish Ruby Conference

Find ELC at the Scottish Ruby Conference in Edinburgh. The weather may be poor but the people and town are fantastic. We encourage you to find us for hack tomorrow afternoon or a pint in the evening.

How to Handle Credential Storage

This website requires Flash player 9.0.0 or higher.

Amazon’s MySQL in the Cloud (RDS)

Amazon just released the “public beta of Amazon Relational Database Service (Amazon RDS), a new web service that makes it easy to set up, operate, and scale relational databases in the cloud.”

Great! So what is it?

RDS today is an EC2 instance running MySQL 5.1. Instead of starting an EC2 instance with RightScale or EC2 commandline tools, you start an RDS instance with the RDS commandline tools (and I’m sure RightScale will support RDS shortly).

The RDS instance accepts parameters such as root database username and password, db space to allocate, initial instance size (note there are new instances), db backup periods, among others. Amazon’s RDS layer configures MySQL in a proven configuration based on the parameters you specify.

RDS instances cost 10% more than a vanilla EC2 instance for Small, Large and XL instances. For Double and Quadruple DB instances the pricing is higher:
Screen shot 2009-10-27 at 09
Note that EC2 pricing is about to change and you will want to recalculate that 10% if RDS does not shift as well.

And why do we want it?

If you run MySQL in EC2 for reasonably large properties, this service effectively provides a substantial reduction in undifferentiated system administration overhead within your organization. That is, if your deployment is small–you probably run your DB on the same machine as your application–you could use RDS to reduce your SysAdmin costs, but you probably value your time less than AWS costs at this stage of deployment. If your deployment is huge–you probably run your DB on EBS (RAID striped) with hot replication to other instances (for performance and failover) and automated offsite backups. You have a large AWS bill and a large SysAdmin cost to implement, monitor and maintain the configuration. RDS could potentially greatly reduce your SysAdmin bill, but today you cannot hot replicate nor run an RDS DB across multiple underlying instances–and therefore RDS is not for you.

So who is it for?

RDS as it stands today is ideal for deployments that have scale to warrant a separate DB. These are midsize deployments–or smaller deployments that follow best-practices in deployment and scalability over instance cost savings. These deployments may have mid-to-large EC2 instances powering their DB today, and don’t need hot replication for performance nor uptime. Note that with the new double and quad instance sizes, the performance ceiling on a single instance has been greatly raised.

These deployments will benefit from a proven DB configuration, out of the box automated backups, one command time-and-date restores, auto instance failover (some downtime would be associated), monitoring (accessible via CloudWatch), and autoscaling of their DB instance (yes–as load increases, your instance is seamlessly migrated to a larger instance).

Our team routinely deploys the midsize and huge DB configurations described here. We have the highest SysAdmin cost on our huge deployments, but also the highest budgets :-). We have been using RDS as the staging DB for several of our midsized deployments. So far we have seen no real usage difference in using an RDS DB vs our own EC2-hosted DB. We do notice that we don’t worry about whether our DB backup script is working. That alone is worth 10% of our instance cost. The reduction in DB instance deployment time and SysAdmin cost is also a win–but we won’t see those savings until our next deployment–as all our deploys are RightScripted and already include a configured DB. Finally–the autoscale, fine-grained restore and other features sound great, but we won’t have more to say until we use them in practice.

Other Observations

  • Logs are accessible within a MySQL table exposed within the RDS insance.
  • In the future it appears that other DB types (non MySQL 5.1) will be supported.
  • DB allocated storage must be declared in advance. Think of this as allocating the EBS volume connected to your instance. Going to need 50GB for your DB? Allocate 50GB. If it grows? Monitor DB size on CloudWatch and make sure you bump the allocated space. Will the RDS team email you to let you know this is an issue? Try it and find out. 😛
  • New instances were released coincident with the launch. 64GB and 32GB high-memory instances.
  • A new Small 64 bit instance is available, but only for RDS. This would be a nice-to-have for EC2 as well to minimize rebuilding templates when going from Small (32bit) to other sizes (64bit).

[Solved] RightScale CentOS 5.0 Rails 2.3.4 upgrade

Here’s the issue:

[prompt] sudo gem install rails --no-ri --no-rdoc --version=2.3.4
ERROR:  could not find gem rails locally or in a repository


[prompt] gem -v
[prompt] cat /etc/issue
CentOS release 5.0 (Final)
Kernel \r on an \m

OK–no problems. Let’s look at the gem sources…

[prompt] sudo gem sources


RightScale’s gem mirror is stale!

So let’s add rubyforge too…

[prompt] sudo gem sources -a http://gems.rubyforge.org
http://gems.rubyforge.org added to sources

And then update our gems en masse:

[prompt] sudo gem update
Updating installed gems
Updating RedCloth
Building native extensions.  This could take a while...
Successfully installed RedCloth-4.2.2
Updating RubyInline
Successfully installed RubyInline-3.8.3
Updating ZenTest
Successfully installed ZenTest-4.1.4
Updating erubis
Successfully installed erubis-2.6.5
Updating eventmachine
Building native extensions.  This could take a while...
Successfully installed eventmachine-0.12.8
Updating highline
Successfully installed highline-1.5.1
Updating hoe
ERROR:  While executing gem ... (Gem::InstallError)
    hoe requires RubyGems version >= 1.3.1

Our RubyGems is too old, but upgrading gives us:

[prompt] sudo gem update --system
Updating RubyGems
Updating rubygems-update
Successfully installed rubygems-update-1.3.5
ERROR:  While executing gem ... (NameError)
    undefined local variable or method `remote_gemspecs' for #<gem::commands::updatecommand:0xb776f4f0>

Ouch! Let’s try the rubygems update helper:

[prompt] sudo gem install rubygems-update
Successfully installed rubygems-update-1.3.5
1 gem installed
Installing ri documentation for rubygems-update-1.3.5...
Installing RDoc documentation for rubygems-update-1.3.5...

[prompt] sudo update_rubygems
/usr/lib/ruby/site_ruby/1.8/rubygems.rb:578:in `report_activate_error': Could not find RubyGem builder (>= 0) (Gem::LoadError)
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:134:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:158:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `each'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:49:in `gem'
	from /usr/bin/update_rubygems:18
[prompt] sudo gem install builder --no-ri --no-rdoc
Successfully installed builder-2.1.2
1 gem installed
[prompt] sudo update_rubygems
/usr/lib/ruby/site_ruby/1.8/rubygems.rb:578:in `report_activate_error': Could not find RubyGem session (>= 0) (Gem::LoadError)
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:134:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:158:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `each'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:49:in `gem'
	from /usr/bin/update_rubygems:18
[prompt] sudo gem install session --no-ri --no-rdoc
Successfully installed session-2.4.0
1 gem installed
[prompt] sudo update_rubygems
/usr/lib/ruby/site_ruby/1.8/rubygems.rb:578:in `report_activate_error': Could not find RubyGem hoe-seattlerb (>= 0) (Gem::LoadError)
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:134:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:158:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `each'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:49:in `gem'
	from /usr/bin/update_rubygems:18
[prompt] sudo gem install hoe-seattlerb --no-ri --no-rdoc
Successfully installed hoe-seattlerb-1.2.1
1 gem installed
[prompt] sudo update_rubygems
/usr/lib/ruby/site_ruby/1.8/rubygems.rb:578:in `report_activate_error': RubyGem version error: hoe(1.8.2 not >= 2.3.3) (Gem::LoadError)
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:134:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:158:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `each'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:158:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `each'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:157:in `activate'
	from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:49:in `gem'
	from /usr/bin/update_rubygems:18
[prompt] sudo gem install hoe --no-ri --no-rdoc
ERROR:  Error installing hoe:
	hoe requires RubyGems version >= 1.3.1

[prompt] echo "lost in a maze of tiny dark passages"
lost in a maze of dark tiny passages

So it appears that we can neither satisfy the requirements for installing hoe (need RubyGems >= 1.3.1) nor update rubygems (need hoe >= 2.3.3)… but let’s take a deeper look and find the solution.

[prompt] sudo gem list
rubygems-update (1.3.5, 1.3.1)

The Solution:

[prompt] sudo gem uninstall rubygems-update --version=1.3.5
Successfully uninstalled rubygems-update-1.3.5

And now we’re unblocked and back on track:

[prompt] sudo update_rubygems
Installing RubyGems 1.3.1
mkdir -p /usr/lib/ruby/site_ruby/1.8
mkdir -p /usr/bin
install -c -m 0644 ubygems.rb /usr/lib/ruby/site_ruby/1.8/ubygems.rb
install -c -m 0644 rubygems.rb /usr/lib/ruby/site_ruby/1.8/rubygems.rb
install -c -m 0755 /tmp/gem /usr/bin/gem
rm /tmp/gem
rm -f /root/.gem/source_cache
rm -f /usr/lib/ruby/gems/1.8/source_cache
Removing old RubyGems RDoc and ri
rm -rf /usr/lib/ruby/gems/1.8/doc/rubygems-1.2.0
rm -rf /usr/lib/ruby/gems/1.8/doc/rubygems-1.3.1
Installing rubygems-1.3.1 ri into /usr/lib/ruby/gems/1.8/doc/rubygems-1.3.1/ri
Installing rubygems-1.3.1 rdoc into /usr/lib/ruby/gems/1.8/doc/rubygems-1.3.1/rdoc


= Announce: RubyGems Release 1.3.0

NOTE:  RubyGems 1.1 and 1.2 have problems upgrading when there is no
rubygems-update installed.  You will need to follow the second set of update
instructions if you see "Nothing to update".

Release 1.3.0 fixes some bugs.

Bugs fixed:

* Disregard ownership of ~ under Windows while creating ~/.gem.  Fixes
  issues related to no uid support under Windows.
* Fix requires for Gem::inflate, Gem::deflate, etc.
* Make Gem.dir respect :gemhome value from config.  (Note: this feature may be
  removed since it is hard to implement on 1.9.)
* Kernel methods are now private.  Patch #20801 by Stefan Rusterholz.
* Gem::location_of_caller now behaves on Windows.  Patch by Daniel Berger.
* Silence PATH warning.

Deprecation Notices:

* Gem::manage_gems will be removed on or after March 2009.

For a full list of changes to RubyGems and the contributor for each change, see
the ChangeLog file.

Special thanks to Chad Wooley for backwards compatibility testing and Luis
Lavena for continuing windows support.

== How can I get RubyGems?

NOTE:  If you have installed RubyGems using a package system you may want to
install a new RubyGems through the same packaging system.

If you have a recent version of RubyGems (0.8.5 or later), then all
you need to do is:

  $ gem update --system   (you might need to be admin/root)

NOTE:  RubyGems 1.1 and 1.2 have problems upgrading when there is no
rubygems-update installed.  You will need to follow the second set of update
instructions if you see "Nothing to update".

NOTE: You may have to run the command twice if you have any previosly
installed rubygems-update gems.

If you have an older version of RubyGems installed, then you can still
do it in two steps:

  $ gem install rubygems-update  (again, might need to be admin/root)
  $ update_rubygems              (... here too)

If you don't have any gems install, there is still the pre-gem
approach to getting software ... doing it manually:

1. DOWNLOAD FROM: http://rubyforge.org/frs/?group_id=126
3. INSTALL WITH:  ruby setup.rb  (you may need admin/root privilege)

== To File Bugs

The RubyGems bug tracker can be found on RubyForge at:

When filing a bug, `gem env` output will be helpful in diagnosing the issue.

If you find a bug where RubyGems crashes, please provide debug output. You can
do that with `gem --debug the_command`.

== Thanks

Keep those gems coming!

-- Jim & Chad & Eric (for the RubyGems team)


RubyGems installed the following executables:

If `gem` was installed by a previous RubyGems installation, you may need
to remove it by hand.

Now what we were after again? Right–Rails 2.3.4. Let’s do some general cleanup:

[prompt] sudo gem update --system
Updating RubyGems
Updating rubygems-update
Successfully installed rubygems-update-1.3.5
:0:Warning: Gem::SourceIndex#search support for String patterns is deprecated
Updating RubyGems to 1.3.5
Installing RubyGems 1.3.5
RubyGems 1.3.5 installed

=== 1.3.5 / 2009-07-21

Bug fixes:

* Fix use of prerelease gems.
* Gem.bin_path no longer escapes path with spaces. Bug #25935 and #26458.

Deprecation Notices:

* Bulk index update is no longer supported (the code currently remains, but not
  the tests)
* Gem::manage_gems was removed in 1.3.3.
* Time::today was removed in 1.3.3.


RubyGems installed the following executables:

And a full update as well:

sudo gem update

This completes while reporting one error with Hoe resolvable via this post.

And finally install Rails:

sudo gem install rails --no-rdoc --no-ri 
Successfully installed rails-2.3.4
1 gem installed

ERROR: gitosis.serve.main:Repository read access denied

Use Unfuddle? If you do and you see this error trying to clone a repository, it may be because:

This error looks to be happening because you are not explicitly involved in the project with which this repository is associated. You should note that even account administrators will need to be “involved” in a project in order to receive permissions to repositories associated with that project. In other words, if you add yourself to the project you should be able to connect to the repository.

Thanks to Unfuddle founder David Croswell for the solution. Worked like a charm.

ELC at Rails Underground London 2009

ELC is a sponsor of this year’s Rails Underground in London this Friday and Saturday. The overall lineup looks fantastic and we’re looking forward to the event. If you are in town and want to connect please find us at Saturday’s 14:30 panel or call our office to arrange a time to meet.

[Workaround] PGError: ERROR: column “model.ref” must appear in the GROUP BY clause or be used in an aggregate function

Seeing this error?

ActiveRecord::StatementInvalid: PGError: ERROR:  column "model.id" must appear in the GROUP BY clause or be used in an aggregate function
: SELECT models.*, count(*) AS count FROM ... some join stuff ... GROUP BY models.id ORDER BY hits

What’s Going On?

You are probably trying to get a model entry and the number of times it appears in the join table hence the count(*) and the group by back to the model. Unfortunately MySQL is really bendy and does this happily. Postgres is a stickler and complains that it doesn’t know what values you want for the columns in your model even tho the way you are structuring your query is probably giving a consistent result set. So what to do?

You can dig thru your joins and work some voodoo magic. You can make subselects or multiple queries. Or you can expand your model columns into the select and group by components of your query. It may well be the easiest path forward. Here’s the rub:

# Assuming your model is named Model and its table is models:
cols = Model.column_names.collect {|c| "models.#{c}"}.join(",")
# And this is something like your query
Model.find_by_sql("SELECT #{cols}, count(*) AS count ... some joins ... GROUP BY join_related_column, #{cols}")

Trust me–you’ll read this and forget about it. Then one day, you’ll hit this on Postgres, you’ll start searching and find a load of dead ends and then stumble here. All I can say is you’re welcome. (Fan mail via comments is accepted.)

Ruby FreshBooks Integration

Having trouble integrating FreshBooks? So were we.

Some gotchas:

  • The README is often incorrect.
  • Update doesn’t work. Use our gem.
  • Status for estimates is an integer, not string: 1 draft, 2 sent, 3 viewed, 4 replied
    5 accepted, 6 invoiced
  • Check the result for invoice.update. If it’s false, there’s an error. Edit the gem to turn on debug statements around lib/freshbooks/connection.rb:call_api
  • To download the pdf, you must set the status to ‘sent’ for a draft (remember to set it back!).
  • Install it: gem install elc-freshbooks.rb –source http://gems.github.com

Full code to download an invoice or estimate PDF:

  def get_pdf(inv_or_est)
    is_draft = inv_or_est.status == 'draft'
    if is_draft
      update_success = inv_or_est.update # Fail if this is false
      raise "FreshBooks API has changed. Document could not be readied." unless update_success
    logged_in_session_to_view_doc_url = inv_or_est.links.client_view
    cookiejar = Tempfile.new("cookies")
    # Note this is particularly ugly because we require session cookies + redirect handling + bad SSL chain (thanks godaddy!). Bailed on Net::HTTP approach.
    html_doc = `wget "#{logged_in_session_to_view_doc_url}"  --no-check-certificate --cookies=on --keep-session-cookies --save-cookies='#{cookiejar.path}'  -O -`
    # https://elc.freshbooks.com/getPDF.php
    if inv_or_est.class == FreshBooks::Estimate
      pd = "estimate"
      pd = "invoice"
    pdf_file = Tempfile.new("pdf")
    cmd = "wget 'https://#{self.freshbooks_api_host}/getPDF.php' --post-data='#{pd}id[]=#{inv_or_est.number}' --cookies=on --keep-session-cookies --load-cookies='#{cookiejar.path}'  --no-check-certificate -O '#{pdf_file.path}'"
    pdf = `#{cmd}`
    logger.debug "Get the actual PDF: #{cmd}"
    if is_draft
      update_success = inv_or_est.update;
      raise "FreshBooks API has changed. Document could not be readied." unless update_success

See more at: http://github.com/elc/freshbooks.rb/tree/master

Upgrade GIT on OS/X

There’s a new GIT version out with at least 10 reasons to upgrade. OS/X users of MacPorts can get the GIT upgrade with:

$ sudo port selfupdate
$ sudo port upgrade git-core # not git!

If you try this on your MacBook Air like I did be prepared for massive sluggishness for up to 2 hours.

Don’t Forget

To remove that annoying message you now get with git push:

$ git push
warning: You did not specify any refspecs to push, and the current remote
warning: has not configured any push refspecs. The default action in this
warning: case is to push all matching refspecs, that is, all branches
warning: that exist both locally and remotely will be updated.  This may
warning: not necessarily be what you want to happen.
warning: You can specify what action you want to take in this case, and
warning: avoid seeing this message again, by configuring 'push.default' to:
warning:   'nothing'  : Do not push anything
warning:   'matching' : Push all matching branches (default)
warning:   'tracking' : Push the current branch to whatever it is tracking
warning:   'current'  : Push the current branch

To keep the default setting and remove the warnings, just type:

# Sets ~/.gitconfig which applies as default across all working dirs:
$ git config --global push.default matching
$ cat ~/.gitconfig 
	default = matching

Or if you’d like to set this on a project-by-project basis, make sure you are in a GIT working directory and:

# Per project setting:
$ git config push.default matching