High Earth Orbit » Ruby

Bug in open-uri when dealing with multiple headers

Andrew — Tue, 14 Aug 2007 15:45:56 +0000

I just discovered a bug in Ruby’s open-uri library, caused by the Net::HTTP library, or at best really annoying behavior that will break when you try and use it. The problem lies in dealing with an HTTP request that returns multiple values for a field. This forum topic discusses that having multiple instances of a field is valid in HTTP1.1 (RFC2616).

So when you use Net::HTTP to fetch a document like this, it will create an array for every field, holding each value found for the field. That’s great. However, in open-uri#last_modified then gets this array and joins the values together to get a single string.

The example I ran into is the Last-Modified field. Given HTTP headers like this:

HTTP/1.1 200 OK
Server: Undisclosed-Webserver/0.1
Date: Tue, 14 Aug 2007 15:03:29 GMT
Last-modified: Tue, 14 Aug 2007 15:03:29 GMT
Content-type: application/vnd.google-earth.kml+xml
Last-modified: Wed, 08 Aug 2007 18:34:55 GMT
Connection: close

And then calling response.each_header {|f,v| puts "#{f}: #{v}"} will result in Last-Modified: "Tue, 14 Aug 2007 15:21:22 GMT, Wed, 08 Aug 2007 18:34:55 GMT". Go ahead and try throwing Time.httpdate at that. You’ll get ArgumentError: not RFC 2616 compliant date:.

To simply recreate this bug, try the following ruby code:

require 'open-uri'
res = open("http://popsci.com/popsci/kml/popsci_future_environment.kml")
res.last_modified

A simple solution, or workaround, is to just get the hash and deal with it yourself: response.to_hash.each {|f,v| puts "#{f}: #{v}"} # => last-modified: "Tue, 14 Aug 2007 15:21:22 GMT

I’m not going to mention any names on this particular offending party that was silly to include 2 Last-Modified dates, but you know who you are.

open-uri and can’t convert Hash into String

Andrew — Mon, 19 Mar 2007 04:03:06 +0000

I’m posting this in hopes that search engines grab it and put it at the top of their list when other poor soul’s run into this problem.

If you’re trying to grab a web resource using Open-URI, and you are using basic authentication (username/password) then you’ll need to make sure to require 'open-uri' or you’ll get:

open("http://example.com/site", :http_basic_authentication => ["username", "password"])
TypeError: can't convert Hash into String

Of course, then you toss some yummy Hpricot into the mix for parsing/scraping the good bits of the HTML.

Sunlight Datakit – Congress in your App

Andrew — Wed, 28 Feb 2007 16:09:24 +0000

Sunlight Labs has released a public API, their Sunlight Datakit. It’s a straight-forward, simple API for getting access to their Civic data, like Congressional Representatives, zipcodes, timezones, and some geographic information.

There is some basic information about elected representatives that makes politico mashups easier: the ability to tie a name to a state, the district and zip codes that they represent, their office telephone number, and so on. We have put together a simple labs “datakit” that does this for us, drawing from several publicaly available data sources. We are making this fully available and have provided a fully documented API for the methods we have developed for those sources. Find out about the datakit here.

Of course, any API needs a nice little client to tie it into your applications. Here is my Ruby client. It’s very simple, because is uses the fallback method_missing to handle any function passed to the class. This also allows the class to be extended by implementing specific methods if more processing of the response is needed.

require 'open-uri'
require 'rexml/document'
require 'cgi'
SUNLIGHT_HOST = 'http://sunlightlabs.com/datakit/'

class Sunlight
  def self.method_missing(service_method, *args)
    params = args[0].collect {|k,v| CGI.escape(k.to_s) + '=' + CGI.escape(v.to_s)}.join('&')
    url = SUNLIGHT_HOST + service_method.to_s + "?" + params
    open(url).read.split("|")
  end
end

resp = Sunlight.getDistrictFromZip5({:zip => 20740})
puts resp.inspect
  # MD:5
  # MD:4

resp = Sunlight.getRepresentativeNameFromCityState({:city => 'Detroit', :state => "MI"})
puts resp.inspect
  # Kilpatrick, Carolyn C.
  # Conyers, John  Jr.
  # Levin, Sander M.
  # McCotter, Thaddeus G.
  # Dingell, John D.

The Sunlight Datakit currently offers the following functions. Check out the documentation for information on the parameters and returned values.

getDistrictFromZip5
getStateFromZip5
getDistrictFromZip9
getStateFromZip9.php
getRepresentativeNameFromDistrict
getRepresentativePhoneNumberFromDistrict
getRepresentativeRoomNumberFromDistrict
getCityFromZip5
getCityStateFromZip5
getLatitudeFromCityState
getLongitudeFromCityState
getZipCodesFromCityState
getTimezoneFromCityState
getRepresentativeNameFromCityState
getRepresentativeNameFromState
getStateAbbreviationFromStateName
getStateNameFromStateAbbreviation

Ruby & Python Bindings officially part of Cocoa

Andrew — Wed, 13 Dec 2006 19:47:56 +0000

Leopard will officially support Ruby & Python bindings in Cocoa which is terrific news. I’m a big fan of interpreted languages when appropriate.

Of course, there are about a dozen other very cool technologies for developers in Leopard that should really make application development, faster, better, and more fun. I’ve gotten a chance to play some with Leopard previews, and the actual User-facing improvements are minimal. But what users will eventually get will be lots of better third-party applications. (via Theocacoa and Michael McCracken)

It’s great to see how much effort is going into not just creating slicker UI’s, but better supporting the people that really make or break an operating environment – developers. Microsoft has had a lot of developer support for quite awhile, and I constantly hear about various .NET meetings/presentations/technologies etc. Apple really kicked it off by releasing XCode for free (no ‘professional versions’ need apply) and then having terrific Developer Documentation and examples.

The Leopard developer page makes some odd claims:

Mac users love to exchange quick messages, have video conferences, and collaborate on each other’s desktops across the network.

um… right, ok. I just love collaborating on your desktop. Anyways, I can’t wait to dive in and start developing on Leopard.

Tags: macosx, apple, leopard, ruby, python, cocoa, bindings, objective-c, programming, microsoft, .net, theocacoa, michaelmccracken

Ruby Hardware

Andrew — Fri, 08 Dec 2006 16:19:17 +0000

On Wednesday at the SouthEast Michigan Ruby Brigade I gave a ‘lightning talk’ on Ruby & Hardware. It is a quick succession of slides (written using S5) on some of the cool hardware devices you can control with Ruby, or hardware you can control your computer with.

It has includes examples and links to using Ruby with:

NabazTag
Symbian Mobile Phones
Nokia 770
Roomba vacuum cleaners
Apple Remotes (the kind you get with your MacBook/MacMini)
SlimDevices SqueezeBox and SlimServer
Lego Mindstorms NXT

There are definitely more devices out there that can or should work with Ruby. Given ruby-serialport it’s possible to control just about anything.

Tags: ruby, programming, roomba, nabaztag, symbian, nokia770, apple, squeezebox, slimserver, slimdevices, brigade, rubymi, rubybrigade

Tesly is good Testing

Andrew — Wed, 11 Oct 2006 04:13:56 +0000

Ben Curtis recently released Tesly Jr., is a great web service that runs through your Rails tests, formats the output, displays them, and allows you to share them with other people. It installs as a plugin in your Rails application. Whenever you run rake to test your app, it uploads the results automatically to the Tesly Jr. server.

Using Tesly Jr., its very easy to keep your development team up to date with the status of the project. Combine this with runing tests on deploy with Capistrano (via Jamis Buck) and you have a that much better integration of testing into your development and release cycle.

Tesly Jr. is free when using the hosted service, and available for just $24 if you want your own copy to run on your own systems.

Tags: rails, ruby, programming, rubyonrails, teslyjr, bencurtis, testing, capistrano, jamisbuck

Ruby Hardware

Andrew — Wed, 04 Oct 2006 03:08:48 +0000

I put together a presentation that didn’t get time at the first AnnArbor.rb meeting. The topic is “Ruby & Hardware” and is a quick rundown of some cool devices that can run (or be interfaced by) Ruby, such as NabazTag, SqueezeBox, Nokia 770. They’re all at various levels of support and functionality, but you can run Ruby on all of them.

Check out the presentation here: Ruby & Hardware.

I’ll be talking about them at the November AnnArbor.rb/RubyMI meeting if you want to come by.

Tags: ruby, gadgets, hardware, presentation, nokia, slimserver, squeezebox, nabaztag, roomba, michigan

Writing Microformats

Andrew — Thu, 28 Sep 2006 21:08:51 +0000

Besides consuming Microformats – it’s useful to know how to produce them. Assaf Arkin has extracted a Microformat Helper from his scrAPI plugin. It supports hAtom, basic hCard, and the datetime design pattern currently – and should be easy to add some more microformats. Check out his Microformat Helper Cheatsheet if you’re addicted to that sort of thing.

Out of the Ruby realm, BlogHelper has a good set of tools and howtos for Using Microformats in WordPress.

Tags: ruby, microformatsstandards, , hatom, assafarkin, cheatsheet, wordpress, bloghelper

Yahoo! does Ruby!

Andrew — Sat, 23 Sep 2006 23:57:26 +0000

Yahoo!, has an addition to their quickly growing Developer Network, the Yahoo Developer Network: Ruby Center. (via RubyBlog.de)

Yahoo has really impressed me with their change from “annoying portal with too many links” to “doing lots of cool stuff and telling people how to use it too”. Their developer network is a great resource for both just learning about a language/tool/design pattern, as well as how to leverage really deep and useful Yahoo resources within your own application.

Providing these resources is definitely in their own interests, such as Cache Yahoo! Web Service Calls using Ruby, but is also just a great service to the community. Their Gallery is also a good place for interesting sites and project ideas.

Tags: ruby, yahoo, programming

Converting table-based Calendars to hCalendar

Andrew — Thu, 07 Sep 2006 04:06:22 +0000

I am looking over the FOSS4G Schedule of sessions. It’s all table based, and it’s somewhat difficult to find specific tracks, rooms, etc. So I took what was the table-based, non-semantic, calendar and converted it into a much more useful hCalendar output, which can be easily translated to iCal for your subscription fun using Brian Suda’s X2V.

You can get the hCalendar here and the iCal link here.

The Problem

Here is the current HTML of the schedule. As you can see, this is an absolute mess of DOM. This table is in fact already the 4th embedded table (tables-within-tables-within-tables oh my!)





In the middle there was some actual interesting bits, such as presentation title, author, times, etc. So what we need to do is walk through all this and build up a conference.
The Solution
Employing some slick Ruby scripting – and using the very useful  scrAPI from Assaf we can define scrapers to walk over the multiple days, and then within those days grab each of the sessions. These are then output into proper hCalendar format like: 

 
  Enabling Users to Produce personalized Geodata
  Mr. Andrew TURNERHighEarthOrbit
  Friday,  15 September 2006 from 10:30-
  11:00,
 at the Amphimax MAX 350
 

The code below makes parsing the nightmare above fairly simple, but due to the lack of any proper classes or id’s (each presentation is id="entry" – eep!), we have to find the bits we want by their current markup attributes. Not suggested, but at least this is nicer than trying to figure out the 10-levels of DOM starting at the root.
You can see the parser here.

RubyConf*MI – the word is “Testing”
Andrew — Mon, 28 Aug 2006 03:19:13 +0000
I went to my fourth conference this summer. Conferences, especially small ones, are a great chance to get quick insights into new ideas or technologies, but most importantly to share ideas with a lot of other people.
The presentations were good, many revolved around testing, with some talking about performance and deployment. There were very few presentations of new novel ideas or techniques. That’s where discussions come in.
I’m a new Ruby programmer who came to the party by the way of the popular gateway drug, Rails. So I’m still behind on proper and good Ruby tools and techniques. Pat Eyler gave an overview of various Ruby libraries for testing and performance such as test/unit, autotest, unit_diff, rspec, rcov, ruby-prof, and benchmark. 
He sequed very well into SouthEast Michigan’s own Patrick Hurley’s talk on C optimization. Coming from a C/C++ background, knowing how easy it is to add small chunks of fast C code to dramatically speed up code (Patrick’s example showed a 20-times increase). 
Back in Rails (esque) land, Zach Dennis talked about his ActiveRecord:Extensions (AR:E). ActiveRecord isn’t Rails specific, but makes a big part of the functionality behind Rails. AR:E adds a lot of very convenient functionality as well as speed improvements to database operations. 
And of course, we made another pilgrimage to the Grand Rapids Brewing Comany.
Tags: ruby, conference, rubyconfmi, optimization, programming, pateyler, patrickhurley, activerecord, activerecordextensions, rcov, testunit, autotest, rspec, benchmark

Ruby Conference Michigan – Saturday August 26
Andrew — Tue, 22 Aug 2006 13:34:37 +0000
I’ll be at the RubyConf*MI which is the premiere Ruby event in Michigan. 
hCalendar info
RubyConf*MI: Saturday, August 26, 9AM-5PM, at Science Building at Calvin College, Grand Rapids, Michigan
It’s just a 1-day conference, but should be a great time. Get to see David Black, author of  Ruby for Rails, speak, as well as SouthEast Michigan’s own Patrick Hurley. 
If you’re in the Grand Rapids, Michigan area, stop by.
Tags: ruby, michigan, grandrapids, conference, rubymi, rubyconfmi

How does a Framework Scale and not splinter?
Andrew — Sun, 20 Aug 2006 04:34:00 +0000
One of the discussions that came up at BarCamp Grand Rapids was how can a framework (and language) scale and grow, without splintering too much. There were a disproportionate number of Java developers present, and one of the few complaints they had about Java was the large number of frameworks that were available. None had a market dominance, or clear set of features. Every week new frameworks pop up.
Compare to the few of us that were Rails proponents. Currently, Rails is the only Ruby framework with any market share (are there even any others?). The question was, in the future will more Ruby frameworks show up, steal market/mind-share, and splinter the community. Diversity is good, new ideas help spur innovation. However, large fracturing confuses new developers and makes support and interoperability difficult.
One definite way to address this possible problem is by keeping the core framework simple and effective and having good support for extensions, plugins, and additions by other developers. Rails currently does this very well, and it looks like this will become even more solid in the future.
A Ruby on Rails Plugin Repository is being realized and should be fully supported soon. Luke Redpath has a good discussion on the current progress and future path. To date, the plugin repository has been wiki based, or required knowing the appropriate subversion repository for a plugin. This new effort will centralize plugins, promote proper documentation, annotation, and testing.
Perl has CPAN, an absolutely incredible repository of modules that has probably kept Perl alive and made it a very powerful language. Ruby Gems provide a very similar infrastructure for distributing great enhancements to the language. Hopefully, the Ruby on Rails Plugin Repository will keep the community united and working to support and build on a single framework while still allowing them to bring in their application specific features. 
Tags: ruby, rails, rubyonrails, perl, java, barcamp, barcampgrandrapids, frameworks

Microformat Ruby Parser
Andrew — Sun, 06 Aug 2006 18:42:51 +0000
With help from Assaf, I’ve generalized the Microformat parsers even more. Now, there is a base-class Microformat that provides a structure for any sub-class to be created that specifies a Microformat that you want to parse.
It’s not robust, in that it assumes that the important information is actually in the text of the tag, and nothing is in any of the tag’s attributes. This should probably be added, where the properties would be an array that specify if the value comes from text or a named attribute.

class Microformat < Scraper::Base  def self.properties(*symbols)
    symbols.each do |symbol|
      html_class = symbol.to_s.gsub(/_/, "-")
      process ".#{html_class}", symbol=>["abbr@title", "a@href", :text]
    end
  end
end

class Geo < Microformat
  properties :latitude, :longitude
end
class Adr < Microformat
  properties :post_office_box,
    :extended_address,
    :street_address,
    :locality,
    :region,
    :postal_code,
    :country_name
end


To use you just have a generalized scraper:

class Location < Scraper::Base   array :geos
  array :adrs
  process ".adr", :adrs => Adr
  process ".geo", :geos => Geo
  result :geos, :adrs
end

which will now return an object with a .geos and .adrs for geo and adr attributes, respectively. 
Updated: Assaf pointed out that he added the ability to pull out the correct data if the microformat was stored in an abbr vs. a span.

process ".#{html_class}", symbol=>["abbr@title", "a@href", :text]
Tags: ruby, microformats, geo, adr, scrapi, parsing

scrAPI – Microformat Parsing in Ruby
Andrew — Fri, 04 Aug 2006 23:49:26 +0000
I was looking for some nice Ruby utility to help in parsing out Microformats from webpages. There are 3 projects currently on RubyForge:

uformatparser – looks very flexible and mature. developed by LabNotes
uformats – pretty good, only supports a couple of formats
mfTools – nothing released yet, unknown status

Talking on #microformats I was pointed to LabNotes newer incarnation of a parser: scrAPI. It’s a much more generic HTML parser/scraper, that can handle getting data from HTML by structure, class, or id. Here is Assaf’s presentation at Mashup Camp II where he gives some good tutorials and discussion about the API. 
Down and dirty with the code
To illustrate scrAPI, I’ll show you the code needed to parse geo location data from a webpage. 
First we just do bring in the necessary libraries, and get an example HTML page:

require 'scrapi/lib/scrapi'
require "net/http"

h = Net::HTTP.new("code.highearthorbit.com", 80)
resp, data = h.get("/greaseroute/index.php")

Then we define our scrapers. The geo microformat looks like:



  35.126,

  -80.764


The process method of the Geo class can take a HTML structure path, CSS class or id, and then the attribute to store. Also, our general Location scraper will look for all geo class tags in the HTML, and fill out the geos array using the Geo class Scraper.

class Geo < Scraper::Base
  process ".latitude", :latitude => :text
  process ".longitude", :longitude => :text
end

class Location < Scraper::Base
  array :geos
  process ".geo", :geos => Geo
  result :geos
end

Finally, now that we’ve built up our “tools”, we can scrape the data, and output all the found locations.

locations = Location.scrape(data)

locations.each {|loc| puts "[#{loc.latitude} x #{loc.longitude}]" }

That was really easy, and effective. Additionally, due to the Microformats standards, we can feel pretty confident on changes to the original site’s markup to not mess up our parsing. 


RubyCocoa
Andrew — Fri, 04 Aug 2006 15:17:53 +0000
RubyCocoa is a bridge between the Mac OS X Cocoa Framework API that makes it callable from Ruby code. You can create applications, objects, call Mac OS X Services. You can even call Applescript:

require 'osx/cocoa'
include OSX
def speak (str)
  str.gsub! (/"/, '\"')
  src = %(say "#{str}")
  NSAppleScript.alloc.initWithSource(src).executeAndReturnError(nil)
end
speak "Hello World!"
speak "Kon nich Wah. Ogan key desu ka?"

Ben Bleything submitted a talk to RubyConf, titled “Harmonize: Exploiting RubyCocoa and Sync Services for Fun and Profit”  where he discusses making a wrapper around the SyncAPI in Ruby. (blog post introducing Harmonize) 
RubyCocoa looks like it may have stagnated a little as the last release was November 2005, but perhaps with such a renewed interest in Ruby, and Mac OS X gaining popularity, perhaps new life with be breathed into it. Projects such as SyncBridge and Harmonize also obviously help. 
Update: Check out the very good examples and information at RubyCocoa.com, including how to use your Apple Remote from RubyCocoa .
Tags: ruby, cocoa, rubycocoa, programming, macosx

Making your Rails project setup easier
Andrew — Thu, 27 Jul 2006 14:39:23 +0000
Luke Redpath has come up with a brilliant, and simple solution for having to install a large set of standard Rails plugins with each new Rails project.  Rails Plugin Packs™ is a YAML file defined set of plugins that his Plugin will then go through and install.
Therefore, you can build up your set of Location based plugins, such as:

YM4R Mapstraction 
Georuby
Yahoo Geocoder and Search
Metacarta Geoparser

Plugin Packs

about:
  name: Mapping Pack
  description: A set of mapping and geocoding plugins
  author: Andrew Turner
  email: rails@highearthorbit.com
  website: http://www.highearthorbit.com
plugins:
  ym4r_mapstraction:
    source: svn://rubyforge.org/var/svn/ym4r/Plugins/Mapstraction/trunk/ym4r_mapstraction
  GeoRuby:
    source:
  metacarta-geoparser:
    source: http://opensource.agileevolved.com/svn/root/rails_plugins/unobtrusive_javascript/tags/rel-0.1
  yahoo-geocode:
    source: http://opensource.agileevolved.com/svn/root/rails_plugins/navigation_mappings/trunk


about:
  name: User site
  description: Plugins for a standard site with users for authentication and basic cms
  author: Andrew Turner
  email: rails@highearthorbit.com
  website: http://www.highearthorbit.com
plugins:
  acts_as_authenticated:
    source: http://svn.techno-weenie.net/projects/plugins/acts_as_authenticated
  comatose:
    source: http://mattmccray.com/svn/rails/plugins



Ruby and Mapstraction
Andrew — Wed, 26 Jul 2006 18:21:00 +0000
Wow, Guilhem has been busy. He recently announced YM4R for Mapstraction. Now you can embed Mapstraction maps into your Ruby on Rails projects. 
In case you didn’t know, Mapstraction is a Javascript library that encapsulates the API’s for GoogleMaps, YahooMaps, and Microsoft Virtual Earth maps. You can write once, but switch between any of the map types.


YM4R – Rails mapping with GeoRSS support
Andrew — Tue, 11 Jul 2006 22:30:49 +0000
The newer, more full-featured kid on the embedding Maps-in-your-Rails-App-block has gotten an update. YM4R v0.5.1 was released a couple of days ago. It’s now mostly a plugin, instead of a gem, since it includes a lot of Rails specific stuff. There are Ruby libraries for creating GoogleMaps tiles, and geocoding that are still in the gem, which is great too.
And I’ve even done my part to contribute by submitting my GeoRSS Simple Parser. 

Technorati Tags: ym4r, plugin, rubyonrails, rails, ruby



Geographic support in Ruby gems
Andrew — Sat, 17 Jun 2006 21:26:25 +0000
I ran across GeoRuby, a Ruby gem for handling spatial data types in a database, awhile ago. However, I never sunk my teeth into how it worked and how to get it going.
In the meantime, I settled myself to using the adequate Cartographer Plugin to easily create Google Maps. I even went so far as to extend Cartographer to switch to Yahoo! maps and MapQuest Maps. 
I just came across another mapping gem, YM4R (yellow maps for Ruby) which handles GoogleMaps v2 (there is a patch for Cartographer to do the same), and Yahoo! maps, local, and traffic. Wow!
Not only that, it’s by the same developer of GeoRuby, The Pochi Superstar Mega Show!. 
Check out the very good, and complete Ym4r + GeoRuby + Spatial Adapter tutorial



Tuesday, 12 September 2006









07:00

    Registration(Amphipôle (niv. 3): 07:00 - 09:00)
  













08:00













09:00




   [20] Getting Started with MapServer
   by Mr. Jeff MCKENNA (DM Solutions Group)
 

  paper