Bug in open-uri when dealing with multiple headers

I just discovered a bug in Ruby's open-uri library, caused by the Net::HTTP library, or at best really annoying behavior that will break when you try and use it. The problem lies in dealing with an HTTP request that returns multiple values for a field. This forum topic discusses that having multiple instances of a field is valid in HTTP1.1 (RFC2616).

So when you use Net::HTTP to fetch a document like this, it will create an array for every field, holding each value found for the field. That's great. However, in open-uri#last_modified then gets this array and joins the values together to get a single string.

The example I ran into is the Last-Modified field. Given HTTP headers like this:

HTTP/1.1 200 OK
Server: Undisclosed-Webserver/0.1
Date: Tue, 14 Aug 2007 15:03:29 GMT
Last-modified: Tue, 14 Aug 2007 15:03:29 GMT
Content-type: application/vnd.google-earth.kml+xml
Last-modified: Wed, 08 Aug 2007 18:34:55 GMT
Connection: close

And then calling response.each_header {|f,v| puts "#{f}: #{v}"} will result in Last-Modified: "Tue, 14 Aug 2007 15:21:22 GMT, Wed, 08 Aug 2007 18:34:55 GMT". Go ahead and try throwing Time.httpdate at that. You'll get ArgumentError: not RFC 2616 compliant date:.

To simply recreate this bug, try the following ruby code:

require 'open-uri'
res = open("http://popsci.com/popsci/kml/popsci_future_environment.kml")

A simple solution, or workaround, is to just get the hash and deal with it yourself: response.to_hash.each {|f,v| puts "#{f}: #{v}"} # => last-modified: "Tue, 14 Aug 2007 15:21:22 GMT

I'm not going to mention any names on this particular offending party that was silly to include 2 Last-Modified dates, but you know who you are.

About this article

written on
posted in ProgrammingRuby Back to Top

About the Author

Andrew Turner is an advocate of open standards and open data. He is actively involved in many organizations developing and supporting open standards, including OpenStreetMap, Open Geospatial Consortium, Open Web Foundation, OSGeo, and the World Wide Web Consortium. He co-founded CrisisCommons, a community of volunteers that, in coordination with government agencies and disaster response groups, build technology tools to help people in need during and after a crisis such as an earthquake, tsunami, tornado, hurricane, flood, or wildfire.