One of the issues we encountered were the vast amount of pre-rendered tile images that needed to be moved to the device. The overall size was not that large – in the hundreds of megabytes. It was the number of files that caused issues in copying and replicated these USB sticks in order to aid in the proliferation of data.

I’ve long been an ardent supporter of SQLite and Spatialite as Open Data containers for geospatial data. It’s a portable, offline, open standard, relational data store that provides great access and compression. About a year ago we even added Spatialite support to GeoCommons – so anyone can convert data to a SQLite database.

Almost exactly three years ago, Mikel put OSM on the iPhone after realizing that Apple was using SQLite to store the tile cache for maps. It makes simple sense to put blobs of images inside a table schema for fast storage and retrieval.

Earlier this week Development Seed released a command-line toolset called MBTiles to bundle tiles into SQLite. You can get the source code here. It’s great to finally have the beginnings of a set of tools to better utilize SQLite for storing and sharing tilesets.

Chris Schmidt has shared his ideas and added broadening support to TileCache in support of storing tiles in SQLite so that anyone using TileCache can now easily load tiles offline.

I’m excited to see more adoption of easy mechanisms for interchanging data – raster and vector. We have a couple of ideas and things brewing in how to combine these tiles with other vector data as well as rendering that could really provide some good mechanisms for open spatial data stores.

OpenSearch-Geo

Pedro Goncalves presented his work taking the OpenSearch-Geo specification and forming into an OGC acceptable document. Pedro has been doing great work extending OpenSearch-Geo for accessing earth observation data. I also talked about how OGC services could be described within OpenSearch templating over a year ago in Atlanta.

Unfortunately the presentation and document is currently locked away behind OGC’s portal. Hopefully Pedro will release it publicly. In addition, it’s not clear within the OGC how to adopt such as suggested standard. It’s not part of the OGC and must go through various OGC architecture boards and discussions to be accepted potentaill as a whitepaper.

We wrote OpenSearch-Geo at WhereCamp 2007 and since then it has stayed as a draft standard with various uptake across projects. The adoption as a more formalized standard should have a very positive effect on its adoption.

GeoCommons supports 3 OpenSearch description documents, one each for Finder, Maker, and all of GeoCommons.

In the end, Pedro’s paper was accepted as a “discussion paper”. Hopefully we can push this forward at the next Technical Committee meeting in Mountain View in December – where DeWitt Clinton (OpenSearch original author) will hopefully pop in to push it forward.

The rest of the working group meeting discussed a potential GeoSMS format that ITRI from Taiwan is working on.

Unfortunately, we didn’t get to talking about the potential <geomap> HTML element ideas. There will be more discussion about that on the OGC Mass Market email list.

There are many possible solutions, and a few that are within easy grasp that we can build our tools to encompass, and develop practices that encourage utilization of these solutions while still moving forward onto better ones as the GeoWeb matures. So we’ll take a few articles to look at specific solutions.

Discoverability

Perhaps the most prevalent issue, and the one that is most easily addressable, is the findability and discovery of geodata on the web. Mano Marks reflected this same sentiment in his blog post on standards.

In thinking about discoverability, there are several primary use cases to consider: Machine crawling, Human discovery, and Tool discovery. Providing data via just a single mechanism means that it doesn’t get utilized and consumed to it’s potential and so somewhere along the chain of utilization it will be a burden to actually incorporate into a workflow.

Think of the machines

Machine crawling is the ability for any spider to walk links, find data, metadata, and formats automatically. It’s what Google, or GeoNetwork does to find and register data sources.

There was recently a discussion on auto-discovery in the GeoWeb suggesting the use of robots.txt, sitemaps, or embedded META tags in HTML pages.

Consider how a spider would get to a site: it follows a link to a geospatial portal from some blog, resource, or directly entered as a good place to get data. It does a GET request on the root homepage, “/” which most likely returns the index.html equivalent. The program then parses through that for links or additional information.

If the spider knows about them, then it may ask for a sitemap.xml or robots.txt. But nothing in the original page request noted that this potentially very complete listing of data was there. This problem is the equivalent of an application having to know that it needs to ask for a GetCapabilities or other method to even discover what is available. Too much implicit knowledge of the specification is required for a program to easily discover new data and services.

What the program does see are these links that can contain information such as a link to a list of available resources. The simplest is a link to the Atom or RSS feed that can simply be a paginated list of all the resources available to the application. Within Atom, there is then the ability to link to various representations of that data in different formats. So applications are able to take the most appropriate format based on what they can consume.

Several years ago I first proposed how KML and GeoRSS could easily support one another via cross-links and with HTML documents. Atom has very nice rel and type attributes that allow for linking to all sorts of different representations. You can even link to OGC services like WMS and WFS using atom links.

Of particular interest here are looking at the currently approved list of Atom link relation types that provide basic semantics for telling you how what this link means. Is it another page? just related? It’s a limited set, but one that covers an approachable majority case for developers to begin using.

For example, mechanisms like OpenSearch, specified in a rel="search", simply notify the application that here is a service that it can query to get at additional resources. And with OpenSearch-Geo, a geoweb crawler can query information within a specific location or bounding area.

Humans need data too

Crawlers are great, they provide a say to pull together information into various other sites and tools to provide customized interfaces to users. However, within any site or tool, how should we expose geodata in a way that humans can easily use for whatever purposes the may have.

Again, links have become a very well understood concept on the Web. That underlined blue line states “beyond me lies an unspecified amount of information about this topic“. However, these links typically imply that they will open another human readable HTML page in the browser. A problem caused by links to media such as geospatial data is that the content behind a link may not be just text, it could be an image, audio, movie, KML, database, or a service. Clicking on that link relies on the browser interpreting the MIME-type (remember the point about how vital mime-types are?) and opening the application the user has specified, or left as default.

So consider what this means for generic media. Clicking open a link to an image probably just opens the image in your browser, or opening a movie loads an embedded video player. Geodata browsers, however, probably doesn’t have the same install base as say, Quicktime. Except perhaps GoogleEarth. The Web has become much more comfortable with clicking on a KML link and seeing Google Earth open up and show the data on a globe.

But something very vital often exists with a link to KML data – a recognizable icon that notifies the user (as they learn) that it is a file that will open in Google Earth, or another KML viewer. This is the same as the very widely used RSS icon.

I discussed this idea before about the geotag icon showing various other formats – and now sites like Data.gov actually show the various data format options.

So what we need for GeoWeb standards are some visual representation to people that they are can clink on this link and open a spatial relational database, or an OGC service, and perhaps have some confidence that there is an application that will provide them a useful way to access the data. (and I’m still waiting for Sean Gillies’ ISO and Dublin Core icons)

Of course, we should also employ emergent interfaces that show users the type of data links that are appropriate for them based on their profile or registered MIME-type handlers.

Man-Machine hybrids

So we have discovery links for machine crawlers to register and harvest geodata, and links for humans to click on to follow to data and within data. However, this can easily become overwhelming to need to click through to every link. Imagine if browsing Flickr through lynx.

Browsers already do a lot to assist users in finding relevant extra pieces of data in a page. RSS autodisovery links show up in URL bars notifying our feed readers that we can subscribe to this page. OpenSearch allows someone to embed this search into their browsers (most of them at least) to easily search the repository later.

The decreasing cost of links

These various approaches for different needs and use cases are all very well aligned. They don’t rely on additional external files that we need to make sure stay up to date or that tools are built to just know that the file can be found at a pre-defined location. Links cost next to nothing, mostly measured in bandwidth sizes, but provide a wealth of accessibility and discovery of geospatial data. Especially data in formats that make sense depending on the tools and use cases for different problems.

Of course, links alone don’t address all the needs of the evolving GeoWeb, they merely provide for the integration of geospatial data with the rest of the web. An important, necessary, but not entirely sufficient first step. We need to consider the actual uses and interfaces of these standards, archival, synchronization, conflation and more.

1. Introduction
2. Where We Are
3. Problems
4. Where We Need to Go
5. Solutions: Discoverability

Too often standards and tools are being by domain experts and technologists that lead to overly complex, and irrelevant formats that become a burden and introduce as many problems as they are trying to solve.

What they’re often not considering are the end user experiences. Who are the users, what are they trying to achieve, and how can these formats make for better, and easier utilization of these tools.

Granted, there are expert users. People who really want to make intricately related, projected, spatially and spectrally bounded queries into data and utilize them in advanced analytics engines. But these are not the majority and they’re not what is driving the long-term demand on the GeoWeb (you can use ‘long-tail’ here if you would like). Who are the users that want to engage with this information on a daily basis in their personal lives, businesses, family, safety, governance, and goals.

Grassroots is an option

I’m a very big fan of grassroots organization and emergent structures. The needs tend to grow from real demand, and solutions are built through actual demonstrated benefit and impact. They are agile, evolutionary, and garner broad support amongst users and developers. These are all aspects that are beneficial to achieving standards that meet the needs of end users and provide good experiences.

However, it is not the only solution. Grassroots tends to look at the immediate needs and may not incorporate more distant issues and expected needs. They seek for broad appeal, and “good enough” rather than totally encompassing all potential aspects of all interested domains. Top-down, industry derived, committee driven standards provide more directed needs and objectives that can serve different types of users.

So the solution is a hybrid – where grassroots solutions are encouraged as demonstrators and emergent needs – that are then accepted and supported by more formal organizations.

Conversation is required

But we also need to open up the conversation beyond just technologists and experts. We need to be engaging and understanding users – and not merely from the “how do I sell them more of my coffee”, but “what can I do to make their lives better”? And actually asking and engaging with them in dialogues.

This technique of user stories, and engagement is not new or unused. However it appears to be missing from the GeoWeb standards developments. We’ve been designing standards for ourselves first, and then foisting these upon others. Instead, we need to understand their needs and issues, and then apply our expert knowledge in how to approach solutions properly.

Other articles in the GeoWeb Standards series:

1. Introduction
2. Where We Are
3. Problems
4. Where We Need to Go
5. Solutions: Discoverability

Part 3 in looking at the current state of GeoWeb Standards. See the introduction here.

It’s time to take a hard look across the board at where we’re coming up short and issues that need to be addressed. One way to summarize:

GeoRSS, KML, and GeoJSON are the itching powder, squirting ink pen, and dribble cup of geodata formats.
– Sean Gillies

Sean is definitely known for his candor, and his viewpoint definitely has merit. Overall the various formats and standards fulfill various needs, but still don’t provide for all use cases, align well with best practices, or make sense to users and developers.

The simplest overall problem with many of these formats, and how they fit into the Web, is that they lack proper web-type descriptions. One primary mechanism that Web clients know how to present data is through the use of MIME-Types. MIME types provide a way for the server to notify clients that the data is in a format such as XML, Text, a PNG Image, and so on. These must be formally registered, but also ad-hoc, or vendor specific, types are commons.

In addition, MIME types allow crawlers and registries to easily record the type of the file in the metadata.

Looking over our list of various GeoWeb standards, it’s very easy to identify which formats abide by this and which don’t.

Atom, JSON, HTML, and SQLite all provide format specific MIME-Types, allowing clients to easily employ the proper applications. However, none provide a special mechanism for notifying that the data includes geospatial markup. Not necessarily a problem, geo shouldn’t be that special.

KML is perhaps the only format that has a geospatial specific MIME-type. However, despite it now being an OGC standard, the MIME Type is still the vendor specific: application/vnd.google-earth.kml+xml. However, KML was particulary ingenious in also providing for the compressed, or zipped, format as a unique MIME-type: vnd.google-earth.kmz.

GML is just XML, so that is entirely not useful in notifying a client that it should try and pass this onto a geo-enabled application. And Shapefiles are agglomeration of multiple files, and even zipped up are only marked as compressed files.

More broadly in services, the OGC has a mime type for service descriptions and responses: application/vnd.ogc.wms_xml, though errors have their own MIME-types: application/vnd.ogc.se_xml.

OpenSearch has a special MIME Type, and obviously Tiles and Image files have MIME-types.

Doesn’t matter if you can’t download it

Another major issues facing many of the GeoWeb formats is their file size. Generally, the web bounces back and forth between disregarding sizes due to assumed, ubiquitous high-speed and reliable connectivity, and trying to speed up pages. But even more important is the fact that many potential users don’t have access to high-speed internet and so their is a huge difference between 10k and 100k or 1MB of data.

To compare the sizes, I took a relatively large dataset from GeoCommons, Statistics Canada, Land and freshwater area, Canada, 2005 and exported it in a variety of formats, both uncompressed, and compressed via standard zip algorithms.

CSV just includes latitude and longitude columns of the centroid – so obviously not fully representative. An option would be to include the EWKB in a column for the full geometry – but that is far from any kind of ’standard’ that other tools would know how to intepret.

Perhaps most surprising from these results are that JSON is so large. Unfortunately, the syntax for complex geometries requires a lot of syntax that adds up in representing polygonal data.

Linkability, Durability, and Discoverability

Moving past purely file format and data type specifications brings up the issue of discoverability and linkability in GeoWeb standards. The Web is more than a list of documents that mention resources, but that they can actually link to durable endpoints that can be resolved, queried, accessed, and parsed.

Non-web native formats have no concept of linking. CSV, Shapefiles, and SQLite contain data, but no links. By contrast, Atom, GML, and KML are chock-full of links, although not always used to great effect. JSON can contain links, but without a schema, who knows what the link means.

Obviously the best model to follow here is HTML, which provides automatic links to feeds, OpenSearch description documents, pages, media, styles, and scripts.

However, what happens when a resource disappears and is no longer resolvable? How do you know where else to get another version of the same data, and is it the same data? This is becoming a big problem in the larger web, made more problematic by the use of URL shorteners, but also especially disconcerting when it affects the provenance and accuracy of geospatial data.

But without Complexity

While linkability, durability, and discoverability are vital to GeoWeb standards, the cost of complexity inhibits adoption and probability of support.

This is a long argument in many circles – often made more difficult by practitioners that have been working in a field for years or decades and consider the most opaque formats or concepts commonplace. Look to the OWL/RDF/SemanticWeb space for an example of how there is a mismatch between proponents and the general public.

A standard needs to have clear value to developers and users for it to even begin to be considered. No one is going to dive into a dense specification of a format without even knowing why they would want to use it or how it fits into workflows and architecture.

And complexity can also surface in small ways – inconsistant capitalization of element names (you know who you are KML), or by supporting a plethora of similar, but different flavors making it unclear which to use (GeoRSS).

Tools

In this last section of the overall problems we’re facing with GeoWeb standards, the most prevalent, and easy to address, is the lack of tools that interact and convert between these formats. Really, formats don’t matter to users – they have data from one source such as their camera, PND, blog posts, Government agency, etc. and they want to do something with it like understand what’s going on around them, find their favorite restaurant, save the rainforest, provide services, get their car fixed, or just share stories with their family.

Easy to use, engaging, and data agnostic tools are vital for adoption of any formats. Again you only have to look as far as KML’s meteoric rise from application specific format to perhaps the most ubiquitous, and growing, GeoWeb standard due to the compelling reason of “I want to see my house and things going on around the world”.

Why do none of the major RSS news readers really support GeoRSS? Every site should offer KML and Atom output of their data. Mobile devices should allow me to open in whatever mapping interface or app any of my data from any of my services.

Missing Middle Ground

Amongst the plethora of formats, we’re really missing some middle ground. Each of these formats are quite independent and unique of one another, with little cross pollination and linking occuring.

• Why can’t my KML file link to Atom updates and also to other formats?
• Can OpenSearch describe my tile pyramid?
• How do I describe my path through life, media, events, places I’ve lived, worked, and people I’ve known?

We too easily get caught either in this “this format must solve all possible problems”, or “it’s good enough so why change it”. In between we need to converge to understand use cases, and how these formats and specifications can cross various barriers – connecting the experts with the amateurs, the citizens and the authorities, one with another.

GeoWeb Standards Series

1. Introduction
2. Where We Are
3. Problems

I started in the introduction by talking about the general Web and some considerations of how geospatial data standards face unique challenges in resolving to broader data interoperability.

In evaluating the current status of standards, it’s useful to give an overview of the current standards, and brief thoughts on where they are working and need specific addressing.

I will also note that this discussion is focused on web-oriented geospatial data standards. There are many other geospatial data formats that exist – but are either too esoteric, proprietary, or not Web-aligned, to be useful in considering their application to utilization in the broad Web.

Shapefiles

Of course, with the first example I will slightly bend the statement above. Shapefiles are the bastard geodata denizens of the web. They are annoying in multiple ways. Foremost being that they are a proprietary data standard that is found entirely too common across geodata portals – especially government portals. However, there is too much information shared, and open tools that can use them, to ignore as serving a place in the GeoWeb.

Shapefiles are difficult to work with in the web. They are like portable databases, but actually consist of several files: datastore (dbf), geometries (shp). data to geometry join (shx), and optionally a projection definition (prj). Deriving from the usage as a binary data definition for desktop software storage developed by ESRI, they have historic shortcomings such as 12-character limit on attribute names, and restriction to a single geometry type (i.e. can’t mix lines, points, and polygons).

In addition with respect to web standards, they obviously have to deal with the multiple files, lack of a Mime-Type, and no web characteristics such as linking.

Microformat-Geo & Adr

Microformats are basic attempt at embedding data within generic HTML markup. The geospatial formats include simple 2-D coordinates, or an address; geo and adr respectively.

Microformats are nice because they align well within a prevalent data format and allow non-geographic expert users to easily embed information, either directly or via simple tools. Google and Yahoo both openly provide support for Microformats through improved search reliability and even some basic data manipulation tools via APIs. Other tools such as libraries and extensions also provide compelling use of Microformats with geospatial documents.

However, basic limits are that geo only allows for latitude and longitude, without any support for a height. Adr at least can provide more complete information, but neight geo nor adr allow for linking to external geometries – a common shortcoming of most the formats discussed.

Another problem with Microformats are that they don’t allow linking to context within a document. So while you can include location information in a paragraph, it is not possible to express how this location relates to the rest of an article or narrative.

So, for example, while it’s possible to markup the location of the White House, one can’t easily denote if this was the location of a press conference, or just that the U.S. President was there, or whatever else may have occured.

GeoRSS

GeoRSS arose out of the simple desire to include location in the increasingly prevalent RSS and Atom feeds from blogs and news sources. It’s another community driven and owned standard, like Microformats, that met existing needs from a bottom-up approach.

GeoRSS over the past few years has become increasingly common amongst web sites using maps and geospatial data. Google Maps, Yahoo Maps, Microsoft Bing all support exporting and importing data via GeoRSS, and major news outlets such as Reuters, and Al-Jazeera output GeoRSS.

Despite it’s widespread adoption, GeoRSS has some complexities that arose out of it’s development. There are 9 potential “flavors” of GeoRSS, although this is largely due to the 3 different formats of feeds: RDF, RSS, and Atom. There are still 3 formats of GeoRSS itself that can be utilized in any of the 3 feed formats: W3C, Simple, and GML. This causes confusion for developers, especially since W3C format is deprecated but still widely used. Perhaps this is one reason that despite GeoRSS being a simple extension to existing feed formats, there still is not GeoRSS support in any of the major news feed readers, except perhaps limited support in FriendFeed.

In addition, GeoRSS hasn’t really advanced in quite awhile despite multiple requests and discussions of extensions for multiple locations, time spans, external geometries, and feature identification.

KML

KML, or Keyhole Markup Language, became a defacto standard out of the popularization of Google Earth, formerly Keyhole Earth, and the wide creation and sharing of geographic data to use inside of this compelling 3D geobrowser.

KML offers a rich markup supporting feature locations, attributes, visual styling, 3D models, addresses, and even Atom links. In addition, it is now an OGC standard, and recently Google announced there were more than 500,000 KML files and 2 billion KML placemarks, or features (making an average of 4 placemarks per KML file).

However, KML is very clearly a direct object representation of the Google Earth application. Attribute names follow a rough camel-case convention based on parent or child classes, but sometimes this simple rule is broken in unclear ways, making it difficult for tool developers to create compliant tools. In addition, the styling capability is rudimentary with little true cascading support and no attribute or class styling capabilities.

Google continues to push forward the KML specification with vendor specific, gx:, extensions. The rest of the geospatial community has yet to attempt to influence the spec in any way despite these apparent problems.

GeoJSON

A much more recent community standard, GeoJSON merely adds geographic markup to the JSON format. This is primarily targeted to client to server communication and takes advantage of the compact size, and quick evaluation of JSON data.

GeoJSON nominally followed the GeoRSS definitions, making it easy to understand and leverage existing tools and knowledge. However, JSON itself does not provide for any actual format specification or schema definition, leaving clients to determine the layout of the JSON to agreed upon documentation rather than actual standards. This is becoming especially problematic as more services expose GeoJSON via APIs to third-party developers. It is really little more than arbitrary, unique XML without the extensive syntax.

GML

In response to the history of arbitrary, unique XML, Geography Markup Language, GML, was developed. GML follows a very strict and feature-rich mechanism for creating geographic schemas and domain specific semantics. It is used for very precise data interchange, typically over OGC services like WFS. GML is targeted to bridge the span from 1-D to 4-D geometries, multiple domains, and entirely customizable profiles, or versions, depending on a user or developer’s needs.

With GML’s power comes much complexity. Developers are typically required to devise and include their own unique schema definitions when using GML. The scope of writing a generic GML client is akin to writing a Ruby script interpreter and is daunting to general web developers that only want to include simple geographic capabilities to their general services. This complexity hampers it’s widespread adoption.

Other formats

There are a variety of other formats that are beginning to emerge on the broader web through a variety of fronts.

Spatialite is the set of spatial extensions to the open, portable SQLite format. SQLite is a file database that provides for full relational capabilities in a single file. Spatialite therefore adds geographic columns and rudimentary geospatial query support.

SQLite is already used by a variety of tools such as Google Gears for offline support and the Google Maps on the iPhone for storing tiles. We chose Spatialite for our Geocoder due to it’s compact nature and deployability. Spatialite makes for a very compelling option when you need to have access to an entire geographic database and perform operations on the data.

GeoPDF is working to become an open format. There is a pending OGC adoption of the georegistration embedding, and Adobe is pushing the ISO 32000 spec that includes how to embed vector and geographic drawing. There is still however a very fragmented ecosystem of tools and interoperability that threatens the format as a mechanism for disseminating geographic data.

CAP, Common Alerting Protocol, is a realtime focused format for sharing out alerts such as emergency news, earthquakes, or municipal signals. It is still an XML format with no real mechanism for ensuring delivery or timliness, and it is not clear the advantages over more broadly used and extended formats such as Atom.

There are still even more formats that are used in the GeoWeb such as CSV, GPX, RDF, and even OpenStreetMap (OSM). However, it is not really worth discussing these here as they are either too generic (CSV), or still too nascent (OSM) to really consider as an existing GeoWeb standard. They will, however, be discussed later in looking to the future of geodata formats.

Also, Semantic data such as Linked Data, RDF, or OWL are continuing to bubble beneath the surface. I will go in depth later on the potentials of semantic geospatial data standards.

Services

Beyond just data formats, there are a number of GeoWeb service, or interface, standards. Open Geospatial Consortium (OGC) dominates this landscape and provides various querying specifications such as Web Feature Service (WFS) and Web Map Service (WMS) in addition to other cataloguing and location-based service interfaces.

WFS and WMS both provide very full-featured capabilities, but also follow older paradigms of interfaces. Fortunately neither of them are SOAP-based, but instead rely on simple Query parameters for specifying bounding boxes, layers, formats, projections. Perhaps the biggest difficulty is that the service description is at the same endpoint as the service itself and often servers use the wrong MIME-types for the documents and errors.

More recently, general web standards organizations such as the World Wide Web Consortium (W3C) have been adopting geospatial additions for browser DOM geolocation, HTTP location information an privacy. ISO and OASIS are looking at OpenSearch-Geo for possible integration into their harvesting and cataloging standards.

OpenSearch-Geo follows the same concepts as GeoJSON and GeoRSS, providing a simple extension to a broadly adopted interface and merely adding geospatial components to it. In addition, by being only a templating specification, it can easily apply to describing general API’s such as Flickr, KML network links, or even WFS when applying the appropriate template markup.

However, while OpenSearch-Geo has garnered a lot of interest, it’s actually prevalent use isn’t clear. There are limited services that offer an explicit, compliant description of their geospatial search interfaces.

The View is Mixed

So the current state of GeoWeb data standards is quite mixed. There is no denying that they have becoming mainstream. We’re seeing some emergence, and divergence of more popular formats in Mapufacture and GeoCommons, both on upload as well as downloads or links. Google has released their figures for KML they’ve crawled on the web. By contrast, GeoCommons and Mapufacture rely on users to vet and register data sources, providing a different viewpoint into the utility of geospatial data formats on the web.

The above charts show the composition of data uploads, links, and entire composition of geodata uploaded to GeoCommons and Mapufacture. As an interesting comparison, downloads are definitely trending towards lighter weight standards: KML downloads account for 67.8% of all downloads, with CSV’s at 25.7% and Shapefiles for merely 6.3%. It is worth noting that this is merely a narrow viewpoint in the larger web – not accounting for OGC standards, raster data, and services. However it is still an enlightening consideration in looking at how people are actively engaging with the GeoWeb.

The different formats have all been used extensively, but when which format is most appropriate isn’t clear. This leads many applications to include multiple formats, an easy and appropriate solution but also one that can confuse users and provide for duplication. We’ll dive into more general problems in the next article.

1. Introduction
2. Where We Are
3. Problems
4. Where We Need to Go
5. Solutions: Discoverability

Later this week I’m speaking at GeoWeb about the current progress of GeoWeb standards, how far we have to go, and how to get there. We have KML and GeoRSS leading the way in searchable, linkable formats, but still a plethora of Shapefiles strewn about. There are questions of findability, semantic ontologies, durability, and expressiveness. What are the adoption rate of these formats and their utility in the future real-time, mobile, linked, open web?

What else do you think is the good and bad of GeoWeb standards?

There were several issues he quickly ran into. Currently imagery is not easy to find for the non-geo expert. The primary interface the broad public is using GoogleEarth which has started doing some nice things with imagery such as time and history.

The second requirement was the ability to run “offline”. Obviously he isn’t going to have any kind of bandwidth for downloading new imagery in the field. GoogleEarth allows you to increase your cache size, and you can “pre-fly” an area to grab data. You could then store this cache (on my machine it is located at /Users/ajturner/Library/Caches/Google Earth) and burn to a DVD or USB stick if you had to move it to another machine.

Let’s do the ‘right’ thing

However, obviously the above solution is bending the terms of service. Probably not a large concern when you’re deploying to the middle of Nigeria – but all the same, DigitalGlobe probably wouldn’t be too happy if you started sharing this imagery.

So my next suggestion was to look for some other source of data – the first of which came to mind is OpenAerialMap. The imagery isn’t nearly as good here, is still under questionable terms of service from iCubed, and is served to the user in tiles. Granted OpenAerialMap is a very new, and as of yet under-served tool. But there are some very valuable lessons from this use case to apply in building out OAM and similar imagery sharing services.

To get the OAM imagery offline there are a few simple approaches. One would be to setup a script to walk the tile pyramid and save the tiles locally. Then use TileCache to serve these off of a disk to a local OpenLayers instance. Not bad – but many new gears and prone to be difficult to maintain.

Peek under the covers

So I began investigating the underlying datasource. Zooming way in to Jos reveals that the imagery is being served by a WMS – a Web Mapping Service.

Clicking on the i-Cubed Landsat copyright link you get the metadata page. The imagery is from 2007 – so not very old, and a very helpful “License: Unknown”, but with a note that the license issue is being worked on. Under “More information” there is now a valuable link to the WMS “Capabilities Document”.

http://hypercube.telascience.org/cgi-bin/landsat7?
request=GetCapabilities&Service=WMS&Version=1.1.1

If you are not familiar with OGC services, the common recipe is that any service provides a GetCapabilities method. This method will return a service definition that tells you what types of data, format, versions, and more that this particular service provides. Unfortunately, in most browsers clicking on the link downloads a “landsat7″ file which does nothing to tell the client (your computer and applications) what kind of file it is. Simple point in why RESTful url’s are much nicer for users. I would have liked to see a landsat7.xml at the very least.

Looking into the file, you can see a number of links proscribed the various methods. However, another problem with OGC Services is that they never actually link to the method or description. In fact, there are 13 pointers to http://hypercube.telascience.org/cgi-bin/landsat7? – the client (developer, application, etc.) is expected to have read the OGC specification document to know that they need to append the request=, and version= and potentially a number of other parameters to actual call that method.

Anyways, RESTful OGC services are a bigger discussion and here we were just focusing on getting imagery before deploying.

The next step is to pull out QGIS, and add the WMS URL. You have the option of several layers. Global is good for finding the region and zooming in – and then you can use the original layer for higher-resolution imagery.

Of course, we could have just used a the WMS URL to GET the image directly – that is, if you know your WMS spec by heart:

curl "http://hypercube.telascience.org/cgi-bin/landsat7?
request=GetMap&Service=WMS&Version=1.1.1
&LAYERS=original&STYLES=&SRS=EPSG:4326
&WIDTH=512&HEIGHT=512
&FORMAT=image/jpeg
&BBOX=9.9,8.8,10,8.9">landsat.jpg

And for more, non-imagery, maps – there is Harvard’s new AfricaMap. It points to 46 topographic and historic basemaps that would be useful to download to a drive an use in the field. For example, to get a topomap:

curl "http://cga-3.hmdc.harvard.edu/cgi-bin/mapserv?
map=/opt/CGA/data/img/uscomp.map
&request=GetMap&Service=WMS&Version=1.1.1
&LAYERS=ONC&STYLES=&SRS=EPSG:4326
&WIDTH=512&HEIGHT=512
&FORMAT=image/png
&BBOX=8,8,10,10">topomap.png

So the answer was – there is some data out there, definitely not as high quality and resolution as the proprietary datasets. And even when there is data, it’s still very difficult to find and utilize. Vector data is finally becoming common (to find, search, and create) but there are still large steps to make raster data as easy to use for non-experts.

By comparison, this summit was meant as a brainstorming around geo and search interfaces and responses. Pulling from the announcement:

We would as much as possible like to bound the discussion to: 1) common ground for geospatial search for web resources and 2) integrating spatial search into search protocols. As part of the discussion we would also like to get advice from the other communities about which catalog/registry search protocol is the ‘mainstream’ one (or more?) that we (OGC) should align with and in turn, be sure that spatial search is supported in a thoughtful but not cumbersome way by the broader IT standards community.

You can see a partial list of attendees here.

There was a good overview of existing, albeit often quite complex, search interfaces. As is potential in meetings like this where attendees have their own history, investments, and beliefs in standards, the discussion can become difficult to easily resolve.

A couple of interesting agreements came out of the meeting. Foremost was the understanding for guidance of using simple, common formats as they already exist when appropriate. This means using OpenSearch as a base URI templating mechanism and follow GeoRSS-Simple specification for geographic data. Of course, a format can expand upon this and offer more complex formats that conform to more complex specs. But by at least providing a common baseline means that almost any service can easily interconnect with another service.

One difficult mechanism that is missing is a way for geographic search to specify the type of spatial operation. Typically most services assume a “within” or intersects”. For example, what restaurants are within a 5-mile radius of my position. However, it’s apparent that this can be confused based on assumptions and also does not provide for any other type of operations. Again, for example, find me all the hospitals that are not within the hurricane path.

A long-standing model for this is called the DE-9IM spatial operation set. It was presented by Eliseo Clementini, and also frequently attributed to Egenhofer. You can read more about it. Granted, a majority of geospatially-capable search interfaces may not require this, but it’s nice that there is a relatively straight-forward model that everyone can agree on.

I hope more attendees share their thoughts and outcomes. There are definitely many who point out the problems of designing standards in a smoke filled room, and I much rather bringing the discussion out into the open where more people can chime in and contribute.

However, once you pull the feed out of the original service, you should being to wonder about the privacy. Many of these services required authorization before allowing you to pull down feeds. This way they can make some assurance that only allowed people can grab your location feed. However, once the feed is pulled out, it is out of the hands of an authorization system and has a very easy potential to be made unwittingly public.

The onus of security is on the application or aggregator that pulled the feed on behalf of the authorized user. But at the same time once the feed has been retrieved, there is no storage of the authorization credentials with the feed itself. It has essentially been stripped of it’s shell of potential privacy and looking at the feed itself you would have no idea if it was supposed to be kept private, and visible only to certain, unknown persons.

What would be nice would be a mechanism to store at least references to permissions and authorization credentials within the feed itself. That way if an application still has the feed, or wishes to store it and re-aggregate it, they can apply the same authorization as the feed originally had.

Existing Mechanisms?

Brian Suda pointed me to the, currently suspended, Platform for Privacy Preferences. But this appears to be a rather heavy-handed approach. The W3C GeoPriv Working Group is also looking at location privacy but not in terms of feeds, and the idea of permissions and privacy aren’t specific to location (though that is typically where it gets a large amount of attention).

Simple Soutions

I’m wondering if there exists, or could easily be formulated, an additional markup in Atom to specify permissions. It would still be the responsibility of the application to abide by these permissions – but at least they would have the information necessary to do so.

Here is a possible solution. Provide a default access (private), but then refer to authorization endpoints for who would be allowed to view this feed. In this example, if the user can provide OpenID authorization to this URL, then they can view the feed:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" >
   <title>Andrew's location</title>
   <permissions>
     <default access="none"/>
     <permission access="view" href="http://myopenid.com/bobdingle" type="openid"/>
   </permissions>
   ...

Tags: atom, georss, rss, feeds, syndication, privacy, permissions, openid, oauth, plazes, dopplr, location, fireeagle

This is a really great idea. The doc contains some nice layout of use-cases for mobile users sharing and loading landmarks. I should be able to easily send someone a location, especially my location, our meeting spot, a great hike, or whatever. However, it’s not clear why Nokia defined their own specification. Or at least, why would they continue to push their own specification.

One of the purposes is interoperability – as stated in the specification:

Landmark messaging should work across different manufacturers’ devices. This specification is open and license-free and allows incorporating the functionality to different software platforms and interoperable Java MIDP landmark messaging applications for the devices of other manufacturers.

While the specification is “open and license-free” it is still controlled by Nokia. Look at the MIME-type: application/vnd.nokia.landmarkcollection+xml. What that means is that while you are free to use the spec, you probably won’t have any say in the format if you need some functionality or find a problem.

Now compare to a location-documentation format like KML. Now, to be fair, KML wasn’t as ubiquitous in 2005 (or the year Nokia was probably working on their spec), and the MIME-type of KML is still application/vnd.google-earth.kml+xml. However, KML is now undergoing external standards adoption, and will be outside the specific control of a single entity, Google.

More than just ownership

While allowing community input into a format is nice, there are more reasons to use a more widely used format like KML – namely that it’s more widely used and therefore there are a lot of clients and services that can make use of the format. More players means more fun.

Nokia’s Sports Tracker app does export KML, as well as GPX. This makes it very nice to email, share, and upload your hikes or workouts. Perhaps this is a trial for them. However you can’t load KML into Sports Tracker to follow someone else’s track.

Here’s to hoping that Nokia supports KML and even GeoRSS for subscribing to a source of location information (such as a stream of sensor data or friends’ locations).

I give this introduction because if you’re a neogeographer, or somehow ancillary to traditional GIS, then you may not really know what the OGC does or how it operates. From my experience, and the crowds I hang out in, the OGC is a mysterious entity that rains down schemas and complex formats. One major problem, in my opinion, of the OGC is that it requires membership to join or participate, and sometimes to even view working documents, and membership requires sums of money (amount depending on the size of your organization). So if you use geospatial standards, but not heavily invested in formulating new ones, you’re kind of left out in the cold to wait and see what emerges from these ’smoke filled rooms’.

Therefore, it was interesting to see that Google decided to ‘open’ the KML format by submitting it to OGC for standardization and further development. By moving the format outside of itself, Google assumedly hopes to gain broader acceptance of the format so that other companies and software feel more comfortable implementing KML and also possibly guiding its future development.

Organizing Geospatial Committees

OGC itself is really just an administrative organization. By itself the OGC doesn’t actually develop and implement standards. Instead, it coordinates the activities of companies and developers to do this. Technical Committees (TC) are formed from OGC Member organizations that meet regularly to discuss the standards, and then develop reference implementations of these standards. For fairly stable formats, such as GML, this is done incrementally.

However, when a sponsoring organization (OGC member that pays a lot of money) has a particular task they would like completed, a Testbed is formed that will more quickly, and focused, develop aspects of a standard. This is the case with Google and KML. They would like the OGC to quickly discuss and adopt KML and move forward on future directions that KML will take. And then they would like several organizations to develop example, or reference, implementations of KML for publishing and consuming to exercise the standard (and find any little bugs along the way).

The OGC lumps several of these tasks together into a larger OGC Web Services Testbed, or OWS. The newest one is the OWS-5 testbed, that involves several ‘threads’, one of which is “Agile Geography”. It is this thread that includes the task to adopt and advance KML.

Raj Singh, Director of Interoperability Programs at the OGC gives a great introduction to KML as part of the OWS-5 testbed.

Enter the fray

So… Why am I going into such depth on the OGC? Our newest venture, Mapufacture, was selected to participate in the Agile Geography thread of the OWS-5 testbed. In particular, we will be implementing KML 2.2 and also moving forward on many aspects of advancing KML to better operate with other standards such as GML and WFS (heavy-weight), as well as express the interoperability with lighter-weight standards such as GeoRSS and OpenSearch-Geo.

But I also have another goal. Having been an outsider of the OGC and the standardization process, it has seemed like an opaque operation that is disconnected from the emergent and dynamic geospatial web. If you’ve dealt with any of the standards before you too may be frustrated by the silent communications, the thick (hundreds of pages) PDF documents behind click-through licenses, and obscure explanations and esoteric formats.

So my goal is to make my participation in the process as transparent as possible. As part of the thread, we are going to share our conversations and drafts along the way to elicit feedback and comments from the broader community. The entire purpose of the testbed is to develop a format that is useable and understandable by people that are not GIS experts – but just want to use the format to express locations and geography as it relates to their non-geo-centric activities. (i.e. someone building a photo-sharing site does not want to read a 400 page document on WFS and then have to deal with 10-level deep XML).

However, I’ve also that there are justifiable concerns with how open the process can be. Intellectual Property rights concerns, and more importantly the concern that unscrupulous types would try to claim-jump formats and patent them for their own nefarious uses exists. This isn’t as much as a concern for KML, which already exists as a format – and we’re making use of other existing formats – so this shouldn’t be as much a concern. But it was enlightening to learn why in the past there has been a somewhat lack of outside communication during a development process. Of course, there was also the somewhat greedy desire to have “first dibs” on formats by being a member.

The kick-off meeting was yesterday and today in Reston, VA where we are meeting with the ’sponsoring organization’ (Google) and the other members of the thread to do general discussion of goals, issues to discuss, and tasks to perform for the next couple of months and over the entire 6-month length of the testbed efforts. I already have several pages of notes on what we’ve discussed, so watch this space for specific requests for feedback on the updates to KML. I’m looking forward to sharing the experience of developing a standard and how the inner cogs of the OGC operate.