Last Week I participated in a panel with spatial archival experts at the at the Society of American Archivists. Led by Butch Lazorchak of the Library of Congress, and also joined by Steve Morris from GeoMAPP, and John Faundeen from USGS, the panel was a full spectrum discussion of “Geospatial Data Preservation” ranging from the Library of Congress’ $10 million acquisition and access to the infamous Waldseemüller 1507 map Universalis Cosmographia of ‘America’ USGS’s environmental conditions for storing historic satellite imagery to GeoMAPP’s work in gathering time-stamped state geospatial data. Butch in particular provided an inspiring overview on what’s special about Spatial – density of data, representation vs data, and the difficulty in capturing interactivity of more modern digital maps.

The Archivists were a new community to me – people that are passionate about the capturing and storing of data – often until the end of time! But they also vary in their core missions – often diverging on the utility of the captured data and information. Very few seem to be really thinking about archives as a useful resource today and only focusing on the long-time storage and eventual access of the data by some unknown entity. As one member of GeoMAPP said: “All of the Archives are storing this superseded GIS data in dark archives and aren’t really providing access to the datasets and don’t have web mapping interfaces”

Clearly, we think a bit differently about archiving – choosing to focus foremost on access to data which will result in improved archiving of data, distribution, and analysis on utility and benefit. My presentation Maps as Narratives: Making Spatial Archives Accessible

focused on the concept that maps have been, and are increasingly a vital resource for people in their daily lives and work. By providing users tools to access and use historic and realtime data, we can then capture this data and provide it to other users and data repositories.

Particular to internet feeds, and social media we can’t easily predict what data will be useful. Neogeographers create visualizations of twitter streams, photos, foursquare checkin’s, friend locations. How do we know which of these are the modern correspondances of tomorrow’s US President or Global business leader? Through easy mechanisms for sharing data and maintaining links we can begin tracking this information in it’s varied forms, providing better insight and archiving of data for later reuse, whether it is tomorrow or in 100 years.

One of the issues we encountered were the vast amount of pre-rendered tile images that needed to be moved to the device. The overall size was not that large – in the hundreds of megabytes. It was the number of files that caused issues in copying and replicated these USB sticks in order to aid in the proliferation of data.

I’ve long been an ardent supporter of SQLite and Spatialite as Open Data containers for geospatial data. It’s a portable, offline, open standard, relational data store that provides great access and compression. About a year ago we even added Spatialite support to GeoCommons – so anyone can convert data to a SQLite database.

Almost exactly three years ago, Mikel put OSM on the iPhone after realizing that Apple was using SQLite to store the tile cache for maps. It makes simple sense to put blobs of images inside a table schema for fast storage and retrieval.

Earlier this week Development Seed released a command-line toolset called MBTiles to bundle tiles into SQLite. You can get the source code here. It’s great to finally have the beginnings of a set of tools to better utilize SQLite for storing and sharing tilesets.

Chris Schmidt has shared his ideas and added broadening support to TileCache in support of storing tiles in SQLite so that anyone using TileCache can now easily load tiles offline.

I’m excited to see more adoption of easy mechanisms for interchanging data – raster and vector. We have a couple of ideas and things brewing in how to combine these tiles with other vector data as well as rendering that could really provide some good mechanisms for open spatial data stores.

Announced today, the World Bank is openly releasing all of their indicator data. Previously, the World Bank had provided an API, but the full data downloads is a welcome move in the realization that access to raw data can enable many possible projects and analyses that a simple interface cannot.

The World Bank’s Open Data initiative is intended to provide all users with access to World Bank data. The data catalog is a listing of available World Bank data sources.

It is clear that an organization as wide reaching and impactful as the Bank has a vast amount of data across many organizations and groups. Pulling these data together, normalizing, and sharing them is a noble, and well done, effort.

Besides just the data catalog, the World Bank has provided an excellent inspection by country and indicator for actually moving through the data without having to be a developer. For example, the Central African Republic demonstrates the depth of information on economics, social welfare, health, business development, and the environment.

I also believe I see the indelible fingerprint of the excellent work of Development Seed on the design, and layout of a complex catalog of data, indicators, and communication. Having also worked with the World Bank on several projects, it’s interesting to see a large, multinational organization embracing innovative tools, open data, and information sharing in the pursuit of global development. There are also some more great announcements coming in the future.

And don’t forget to get your World Bank Data iPhone Application.

Schuyler Erle and Tom Buckley will be heading down on Tuesday to provide on the ground support between the government agencies and the community.

A tremendous thank you to the numerous individuals and groups that helped and provided tools or data: World Bank, San Diego State University / Calit2, Internet2, Georgetown University, DigitalGlobe, Delta State University, Sahaha, Crisis Mappers, OpenStreetMap, NOAA, Ushahidi, DevelopmentSeed, TelaScience, STAR-TIDES, CrisisCommons, USAID, GeoCommons, OpenSGI, GeoEye.

There have been several tremendous groups that have actively been contributing data and tools both with remote developers and responders on the ground. CrisisMappers, CrisisCommons, Ushahidi, OpenStreetMap, just to name a few.

Many data providers have been making their data freely available. This is most notable when looking at Mikel’s screenshots of OpenStreetMap before the quake and after volunteers began tracing over historic maps and newer satellite imagery from Digital Globe and GeoEye.

Other efforts:

• Ushahidi Haiti is crowd-sourcing reports. You can send a text message to 447624802524, send an email to haiti@ushahidi.com, or send a tweet with the hashtag/s #haiti or #haitiquake.
• The CrisisCommons Wiki has a list of available data and organizations
• Sahana has a form to list offices and organizations that are working on the ground
• GeoCommons search for Haiti has all the datasets and maps that people have contributed for download as Spreadsheet, Shapefile, KML, and more
• OpenStreetMap’s Project Haiti has a list of datasets and people tracing data

However, unlike the open source world, the legal and cultural frameworks in which to share data have not yet emerged. In code, there are a gamut of well known and widely used licenses: GPL, BSD, MIT, Apache, and more. While each has unique characteristics, their overall meaning and implications are easily understood by developers and comply with business operations that wish to use open-source software. The licenses are each unique, and can sometimes be confusing, yet there is a small vocabulary of regular licenses that developers can easily picked and choose.

In the media world, Creative Commons developed an ingenious mechanism of licenses with clear verbage and branding that makes it readily accessible by nearly anyone. With miscible license options such as “Share-Alike”, “No-Derivatives”, and “Non-Commercial”, media producers and consumers can clearly mark appropriate uses of their works. The impact is most clearly seen on content sharing sites such as Flickr and Slideshare where users can choose from a very small list of licenses to publish information, or search for information under certain terms.

Because of these attractive features of understandability, small set, and branding, Creative Commons is increasingly utilized to openly share data. However, the Creative Commons apply only to creative works: stories, songs, photographs, and other media and as such are not truly valid when applied to databases.

Instead, the current landscape of data licenses are all completely unique, incompatible, and difficult to understand. This situation is further complicated by the existing data business ecosystem that thrives on charging large amounts of money to write, verify, and mix unique data licenses and prescribe legal uses of multiple combined data sources.

The implications are that the situation persists and groups that would like to share, or use, open data are relegated to complex, and expensive, legal counsel, or must accept risk and hope they remain within compliance, or at least outside of notice.

There are currently only two potential solutions currently in development. Creative Commons has developed CC0, (”C, C, Zero”) – which essentially removes all copyright from a work or data. Flickr utilizes the CC0 license for their derived boundary datasets.

The other upcoming option is the Open Database License (ODbL), which is being put forth by the OpenStreetMap community in order to create the equivalent Creative Commons, By-Attribution, Share-Alike (CC-BY-SA) for databases.

However, even these two licenses have problems. CC0 is drastic in that it removes all copyright from the data, and so may not work with anything less than full, global release of data. Alternatively the ODbL is criticized for being “too left”, where there is an unclear potential that any utilization of data such as from OpenStreetMap would subsequently have to be released. This is similar to the GPL licenses, or “viral” licenses.

What is missing is a clear set of data applicable open licenses that would allow anyone to easily demarcate the terms of the data they are releasing, and provide confidence to data consumers that they are in compliance with the data rights. The effect will be to allow data to more easily and justifiably be made available as well as tools to interact with this data. It will also address the many questions around collective, or combined, databases and derivative works, such as when deriving vector data from satellite imagery.

A few weeks ago I spoke on a panel at the OGC Summit on Spatial Law and Policy, which is one effort to build a community of developers, companies, data providers, and legal experts to address just such a need. In particular times of disaster response illuminate the immediacy of clear data sharing, which was the focus of the panel, but also more long term use of such data, and derivative works such as in rebuilding and recovery after an event.

I’ve discussed the pitfalls of licensing and Creative-Commons style modules before, which raised some initial questions. But in the year since then nothing has really changed towards this broader vocabulary. Will Creative Commons merely become the de-facto, if non-applicable, licensing? For example, the Ordnance Survey just released data under Creative Commons-Attribution 2.0 UK. I would be interested to hear more about other efforts that are seeking to create a simple, clear set of data licenses. In addition, how else have you dealt with confusing, and complicated data licensing issues in building new datasets, applications, or use cases?

The map itself at first is not very impressive by modern digital mapping expectations. It has simple smooth panning or zoom, with an interface reminiscent of MapGuide style portals.

Where it becomes impressive is after turning on some of the layers of public safety, services, and infrastructure that simple markers that open on hover make it very easy to move around and discover information and links to other municipal databases such as census, architecture, neighborhoods, polling information, lot information, and much more – all without overloading the user. The impressive connection of so much data, especially in a city of the density of New York, is impressive.

The map is an example of simplicity, familiar interfaces, and rich data presentation that As Mr. Klinkenborg states,

There is a pleasing logic to this kind of organization, to layer after layer of data embedded within a scalable map. In a sense, it approximates how we tend to know the world… Think of returning to your neighborhood after a trip or driving to your parents’ house. You can almost feel the increasing depth of your knowledge as the terrain becomes more familiar. What you know isn’t just the superficial arrangement of streets and highways. You have a rich array of geographically organized information, some of it practical — how far to the good grocery store — and some of it emotional.

Obviously I would hope the underlying data is also made available. Imagine if the map existed as a feed of data sources that linked to one another – any queried point returned a GeoJSON item that linked to the Sanitation Collection Schedule in GeoRSS, Elected Officials as hCards, Building outline as a KML, and lot information as GML. The map portal is just a single, simple entry point into this information that the NYC.gov can guide and control. However, the data can and will be available via any number of interfaces that go beyond the device itself, but provide for a seamless integration of this information at our fingertips to query and drape over the very urban landscape as we navigate and interact with it.

Mr. Klinkenborg summarizes,

It’s easy to assume that the real revolution in mapping is the global positioning satellite and Google revolution… But the real revolution lies in the layering of data onto these already kinetic methods of viewing the world. In a very real sense, the virtual planet becomes our index to what we know about the actual planet.

I’m looking forward to future incarnations that include boundaries estimating daily resident happiness, suggesting cultural relevance, and heatmaps of amount of sunlight and sky-view.