status

location
Washington, DC
Subscribe to GeoRSS Subscribe to KML


VoteReport mapping and data feeds

Published in Community, GeoRSS, KML, OpenSearch, Project  |  5 Comments


twitter-report.pngOver the past two weeks I’ve been working with a great team of people helping to build VoteReport – an open public reporting system to be used during the 2008 US Election to track the situation as citizens cast their ballots. The simple goal is to make it easy for anyone to send in a report describing the wait time, overall rating and any complications that are impairing their ability to participate in the election. For more information check out http://twittervotereport.com.

Dave Troy has put together a solid backend that is aggregating together Twitter, SMS, voice, iPhone and Android native applications, and even YouTube. Others have built the iPhone specific applications. I’ve been working on the mapping and data sharing side of the project. The first goal was to provide a number of mechanisms to share the data that we’re gathering with everyone. Additional mashups and visualizations are free to use the data streams to pull all the data that VoteReport itself has – so definitely go wild with your ideas. A quick breakdown of what’s available:

OpenSearchhttp://votereport.us/opensearch.xml
This is the OpenSearch description document that outlines all of the feeds and various filters that you can use when getting to the data. Always check this as we’ll update it with new parameters or data streams. In addition, the various responses discussed below include OpenSearch styling pagination so you can walk through the entire database of reports without having to drink right from the firehose. This also includes the OpenSearch-Time extension.
KMLhttp://votereport.us/reports.kml
Getting the reports.kml will give a Network Link – this is useful for GoogleEarth and other KML clients to automatically update every 60 seconds with new reports. You can append live=1 to get the full KML document. I have included all the useful attributes in the ExtendedData element of all the Placemarks. Each Placemark also has an id for easy reference.
GeoRSS-Atom – http://votereport.us/reports.atom
Just want to subscribe to the feed in your RSS reader, this feed is useful for getting updates.
GeoJSONhttp://votereport.us/reports.json
JSON is super nice for doing client-side mashups and visualization. This is what the VoteReport Map itself is using. It includes a lot of information for each report, including reporter, icon, location.

All of these feeds even can take a dtstart= with an ISO-8601 date for getting reports after a certain time (and optionally dtend= for getting time-bounds of reports). A useful geographic filter is to use state= with the capitalized two-letter state code to just get reports within a state. So for example http://votereport.us/reports.atom?state=VA is a GeoRSS feed of reports in Virginia. As I mentioned, I did build a quick map that you can view at http://votereport.us/reports/map.

We’re continuing to build it out with new features as more data comes in. You can easily embed the map in your site using (and optionally remove the state=):

<iframe src="http://votereport.us/reports/map?state=VA" frameborder="0" class="stream" width="535" height="500" scrolling="no" ></iframe>

The difficulty with creating more visualizations is the lack of pre-election data. This system has been built to primarily capture a huge amount of valuable information for one day. We’re not sure before hand what this data will look like, coverage or attributes. Typically visualizations are made by exploring and playing with the data to see what emerges. In this case, we’re making estimates (and guiding via the tutorials) on what data we’d like. Therefore, the map itself has simple mechanisms for styling markers based on the user-supplied report. But the data is far to dispersed so far for something like a heatmap.

Fortunately, the team consists of a large number of public advocates that are spreading the word which should encourage more citizens to use the system and contribute both good and bad reports. Andy Carvin of NPR put together this NPR coverage, and we’ve also received coverage from Time, Huffington Post, New York Times, TechCrunch and even Craig Newmark. Check out the TVR press page for more coverage links.

And if you would like to help contribute to the project, check out the VoteReport Wiki. I imagine there will also be a number of post-election visualizations and analysis to come out of the reports.

Similar Posts


Responses

  1. Sean Wohltman says:

    November 1st, 2008 at 6:48 pm (#)

    Andrew, fantastic job – putting the network link on a tour is something that should be done everywhere possible on election day, and I’ll make sure its done at the Reston Google Office’s lobby.

    May want to tie in some sort of color-coded feed as the networks start to call states, and if a large volume of tweets are coming in, filter the top tweets not just by time, but by the geocoding result to the states of particular interest.

    i.e , If Ohio is in a tight race for contention, start filtering the feed to show time-relevant Ohio tweets until the state is called.

    Also, is it too late to try to tie the tweets to the actual polling locations?

    Kind of odd to see tweets popping up in the middle of interstates because that’s where the geocoding returns.

    ~Sean

  2. James Baker says:

    November 4th, 2008 at 2:19 pm (#)

    I’m trying to suck down the JSON data, and I’m not getting much. Sometimes I’ll get ~15K (5-10 tweets), sometimes an empty file. Is this supposed to be streaming data? I can’t seem to get anything using dtstart. I’m trying to download *everything* for analysis; is this intended, and the site just overwhelmed?

  3. James Baker says:

    November 4th, 2008 at 7:04 pm (#)

    Well, so much for day-of analysis. I’d love to do a day-after analysis, if a download of the entire tweet stream could be made available.

  4. James Baker says:

    November 5th, 2008 at 6:33 pm (#)

    I’m starting to feel a bit awkward about being in an echo chamber here… I guess I’ll finish out my little saga.

    Found the count/page params from the OpenSearch xml, but there are still problems with the json response code path. The largest count= # I’ve gotten to work is 8400 (there are 11,000 tweets total) – anything above that seems to push the server beyond a timeout and it returns nothing.

    Breaking up the request into several pages is also failing: For example, http://votereport.us/reports?count=1000&page=11 works while adding “.json” to that reports request returns nothing.

    I’m posting here because this post is newer than anything on the wiki – are none of the project developers paying attention? I’m trying to do some visual analysis and then bring more attention to this project. I’d also like to redistribute the (flattened and abridged) CSV version of the data I’ll be analyzing – which seems to be in line with the spirit of this project, but it’d be nice to confirm that with someone.

  5. Andrew says:

    November 7th, 2008 at 3:40 pm (#)

    Sorry – for some reason my blog wasn’t emailing me with your comments. I apologize! Also, things were quite hectic around the day.

    First off – the HTML page is using hAtom, so well structured if you wanted to parse that. Also, you could grab the KML files in count of 4000 over 3 pages: reports.kml?page=1&count=4000&live=1

    Looks like the JSON is throwing an error. I’ll look at it.