status

location
Washington, DC
Subscribe to GeoRSS Subscribe to KML


Technology

Desire Paths to Open Data

Published in Government, Technology


Last week I made a quick statement sharing my concern for civic organizations promoting ETL – Extract Transform and Load – of open data instead of developing APIs. I felt it warranted a more thorough response than the terseness of microbursts.

Desire Lines and Road Surfaces

Walk through most parks and any college campus you will quickly notice that dirt worn pathways that connect between the sidewalks indicating pedestrian shortcuts. These desire lines indicate an initial and repeated optimization that lay outside the paved paths. Often these ad-hoc networks are a type of ‘footstream’ that are adopted and paved – or they are left to individual use, muddy in rain and undocumented or supported through groundskeeping. At scale this often explains entire city and county or even national road networks that started as ‘cowpaths’ and through continued and growing usage became official infrastructure – roads and highways – which are relied upon as a matter of business.

Desire Paths - Virginia Tech - DigitalGlobe.png

This road network is the infrastructure that government develops and promises to support as a necessary mechanism for citizens to build communities and businesses to operate commerce. Information infrastructure is the next generation that government is developing which increasingly becomes the relied upon and required tools for community and commerce. Tim O’Reilly has referred to “Government as a Platform” which means that we must be able to rely on these services as a durable backbone where which we build our numerous and diverse applications.

Opening Data: Prototype or Infrastructure

Open data started as simple file sharing. In my own city the data catalog was a large and easy to read list of datasets with metadata, links to common formats, and updated dates. Through a series of public contests, developers used these file downloads to build some compelling applications to highlight the future of Government IT. In 2008 Apps for Democracy was iLive.at (link dead) and ParkItDC.com (link dead) and again in 2009, Apps for America winner was 311.socialdc.org (n.b. link is dead).

It should be apparent that these contests and applications were interesting desire lines that did not provide or sustain a platform of information which citizens could rely on. Albeit just simple examples, they are indicative of the tendency to build simple one-time applications which unfortunately miss that next step of becoming part of the platform they seek to improve. I’ve heard similar examples from other cities where civic hackers created well-meaning and well-built applications but that sit so far outside the existing government operations that they require continued manual maintenance by these unpaid volunteers with the common outcome that the service stops updating (maybe while even still operating, arguably a worse condition than simply shutting down).

Unfortunately, even the original data catalog has slowly atrophied. Based on my own experience when looking for more recent crime data (the data catalog stops about September 2013) I learned that the internal system was being migrated and the transformation process had faltered and it just wasn’t a priority to get the system back online. It was too removed from their actual job of analyzing and responding to crime to make a separate feed available with any defined timeline.

Federal Highway Intersection.jpeg

Which is a stark reminder that despite the amazing capabilities technology can deliver, Government is foremost responsible for serving people, not serving technology. Everything it does in the end is to serve the communities that elect, fund, and generally are employed by, these governments. When most civil engineers are designing roads they don’t apply grandiose design aesthetics and creativity. They pull open their codes & standards, determine the appropriate concrete mixture, depth and rebars based on specs, and get to work developing the road that fits the expected and reliable operations that citizens need.

Operational Open Data is Sustainable Open Data

There are numerous studies, reports, case studies and general community practice have made the case that open data has a great potential benefits to civilian and business communities. Not least of which is the ability for government agencies to more easily share data between one another in addition to improving business efficiency and consumer decision making.

Government has a difficult and extremely important job. As an entity, it is not enamored with new techniques or formats. Attempting to create unidirectional bifurcations of the data create strain which will eventually give way when there is any pressure: time, fiscal, or personnel. New technologies need to understand these processes and costs in order to align themselves if they are to ever become part of the government platform.

For open data to move from a shortcut to part of the stable infrastructure, we need to design it from the beginning to be practical, sustainable and ultimately operational. Open data needs to be the way government operates, and it needs to be part of the living systems that manage and process the data as part of day-to-day business.

To the original question, generalized techniques such as ETL – extract, transform and load – have tremendous flexibility to explore new paths and opportunities. By enabling freedom to explore applications, new formats, and communities, government can observe and understand these desire lines. It can then make the decision if these paths become part of the supported network or if they indicate a necessary redesign of the large system to accommodate these concepts. Personally I’ve seen, and built, many ETL tools and community applications that worked on the outside of government. While fast moving an extremely agile, they ultimately are untenable to provide ongoing and durable platforms for information access.

By comparison we should be encouraging and working directly with government technical staff to specify and prototype API – application programming interface as they provide an excellent mechanism for this prototype to adoption. By developing an external interface to a service the provider is making a contract with end-users that is independent of the implementation details. This enables a developer to use their tools of choice with the intent that it could be rebuilt within government infrastructure while maintaining the promised interface that applications already rely upon. And finally like the observing the increasing depth and width of a dirt path the measured analytics behind and API help prioritize incorporation and operationalization.

Exemplar of this has most recently been the DCAT distributed catalog specification. Neither new nor novel as far as federated data catalogs are concerned, it was an API that was created in conjunction with technologists and government agencies and adopted independent of any technology implementation that is now poised to easily share links to data between numerous national and local government agencies, all in the public. Instead of building more data harvesters, an API means that anyone can both participate in production as well as usage of the open data however best fits their needs.

Desire to Collaboratively Craft

Perhaps the most exciting thing I have observed in my six years living in DC and watching the Open Government movement surge has been the positive growth and excitement of people within government to actively and publicly collaborate. More than merely publishing a catalog and running a competition, government representatives are eager to talk about ideas, share code and data, and hear where they can open their infrastructure for these types of creative developments.

While much of the commercial web is becoming ‘appified’ (and often eschewing access via common or open APIs), perhaps this is one case where it’s superb the government moves more slowly and is just now entering the time of the programmable web. For many of us who volunteer our time and expertise hoping to improve the civil societies in which we live, the best thing we can do is work closely to advise and create the best platform possible.


The Fourth Screen

Published in Technology


People viscerally engage with their personal technology devices. Recent studies indicate that we spend 6.5 minutes of every hour awake with our phone, and even more time than we do with our partner. Anecdotally I have heard that we have mobile in our hand more often than we are wearing pants.

Fortunately we adapt. At least in my experience etiquette now precludes phones being left on tables during meals or friends reading during a conversation. We modify our behavior to selectively utilize and then put away our technology. Send that message and the device is then relegated back into the dark, invisible recess of our pockets or bag.

iPhoneBook.pngWhat made mobiles initially so pervasive as information access devices was that they were prevalent, but more importantly non-invasive. Unlike a laptop which creates a physical perceived wall between the user and public – a phone masqueraded like a hollowed book as a normal device in which a person could hide the internet. This cloak faded as people acculturated the new interfaces.

Displays continue to shrink and conform. The Kindle broadly introduced the concept of a screen that performs as a book, and little else – affording an acceptance much as someone reading the paper. We now have the emergence of a new paradigm of social device interaction.

Fourth Screen

In the early 20th century, at the birth of aviation pilots found it precarious to operate their aircraft while using pocket watches to plan flight paths. As a result, wristwatches were created to enable pilots to quickly and constantly monitor the time without removing their hands from the flight stick. This new heads-up type display became extremely popular over the following century until it was in fact replaced by the aforementioned mobile phone.

9580161610_1eb56845cf_m.jpgBut the wrist-watch again has the opportunity to replace our pocketed phones and provide us with a heads-up interface to our connected devices. The recent Pebble Watch displays incoming messages, calls, and can even be extended to call to various web services for weather, delivery information, and location.

This is not entirely new. I had a DataLink watch in 1995 that wirelessly synchronized to my computer via a mesmerizing flickering display. These smartwatches, dating back to 1972, have iterated through popular and usable formats but, like the early smartphones or a web desktop required several cycles to truly establish.More recently Samsung released their Galaxy Gear Watch and you have been able to wear your iPod Nano as a watch for a few years. There are rampant rumors that Apple is working on a Watch.

Google Glass is the harbinger of ocular augmentation that directly overlay on our visual field. There already exists a backlash over the obtrusive glowing screens and the concern if someone is paying attention to you, referencing something, or even photographing you; much to the chagrin of nearby people .

Ambient Awareness

Connected watches and other wearable devices offer a larger opportunity for ambient awareness. Our sensor-laden mobile phones, replete with microphone, light sensor, accelerometers, and GPS, are enclosed in dark caves of our pockets; missing so many contextual clues.

By contrast our wrists remain exposed – constantly dappled by the light, reverberating from the punctuated sounds, sensitive to air temperature changes, visible to wireless signals, and even experiencing a wide range of motions from walking to waving, shaking hands, and opening doors.

d119772edbe3756052ca37ad125f03dd.jpegAs a highly-ambient information display, the watch offers an unparalleled platform. Consider also that Apple bought Color, a team experienced with multi-sensor fusion and device content distribution and a future of integrated contextual passive alerts with casual interaction appears imminent.

Surreptitious Engagement

In the end, computing is becoming ubiquitous, pervasive, and non-invasive. We accept new technology with its foibles and obligations but we ultimately desire it to blend into our periphery where we can always engage, but never interfere.

Fortunately it is still about the human interaction, even if we get a little machine help in the process.


Codename Farewell or Clone at your Own Peril

Published in Design, Technology


Buran space shuttle/></a>Last week <a href=Aaron and I were discussing how the web facilitates inspiration and sometimes even copying of other sites or applications.

This is a positive outcome of open access that can create evolutionary improvements. However in my experience I have also seen people clone an interface where they missed the larger context of the interaction, or even worse they cloned something that was internally known to be a quick hack or incomplete solution that was delivered to meet a deadline or as the first phase of a multi-phase story that was never completed.

Buran and Enterprise

This reminded me of the fascinating history of the U.S. and Russian space programs. During the space race of the 1970′s and ’80′s the Russians were known to be accessing the unclassified engineering plans for the upcoming Space Shuttle. While this was a necessary, and arguably a greater good to science and industry, the US program did not want the Russians to beat them to the delivery of a reusable launch vehicle.

I proposed using the Farewell material to feed or play back the products sought by [the Soviets], only these would come from our own sources and would have been ‘improved,’ that is designed so that on arrival in the Soviet Union they would appear genuine but would later fail. U.S. intelligence would match Soviet requirements supplied through Vetrov with our version of those items, ones that would not — to say the least — meet the expectations of that vast Soviet apparatus.

From “How the Soviets stole a space shuttle”. And apparently this ploy was successful: “Soviets have ablative material in their elevon gaps, just like we did. We fooled them and now use tiles in the gaps.”

Receiving inspiration, and even copying the aspects, of other applications is clearly an effective means to jumpstart features. However this should not preclude your own diligence in engineering and design to ensure you have appropriately incorporated these concepts for an effective, and operational, means.


Quantified Baby

Published in Technology



Quantified Baby Chart

As any parent I am constantly concerned about my child’s health. With data it is easier to identify emerging problems and also diagnose the underlying problem as a precaution rather than merely reactive. For these reasons we spent the past year on our Quantified Baby project.

Throughout the year we have been maintaining various quantitative and qualitative data points of our son’s habits and growth. From the day he was born we gathered every time he ate, slept, pooped, peed, and took a bath. This included the quantity of food, time of sleep, and even (optionally) the color, consistency and leakiness of his output.

Overall our son has been extremely healthy, happy, and effusive. To date his he only been sick with a common cold once for about 5 days, had no allergies, and slept through the night after the first three weeks when he was effectively gaining weight. While I know that our data capture and analysis doesn’t account for our fortune, we do believe that it was beneficial to our own parenting, regularity, and ability to be aware and informed of his health for any doctor visits. We also didn’t sweat all of the details. There are a few gaps due to lack of sleep, general distraction, or sometimes simply bucking the machine (i.e. “I don’t wanna”).

We learned a lot of insights along the way, particularly in the methods, benefits, and difficulties of quantifying your baby. For our measuring, we used the capable, full featured, and easy to use Total Baby. Designed by an engineer for his own family, it has the practicality of design by someone that is also forced to use it in a bleary eyed state with a crying baby in another hand, and a bottle/diaper/blanket in the other. This is not meant as a review of this particular application but more as a highlight to what the baseline and required features should be of any quantified self applications.

Make it Useful

Many ‘quantified self’ applications take a conceited view of their data capture and require you to enter data with little to no information value returned to you. These tools are fun at first but quickly become forgotten since they required action outside of the normal activity.

By contrast, quantified self tools must provide at least immediate value. By example our measurement of feeding provided a timer of the current activity, and also set a (configurable) alarm for the next feeding that would typically occur in 2 hours at the beginning. When you’ve been cycling through several days of polyphasic short sleeps, this type of simple arithmetic becomes harder than you may imagine.

So by providing an immediate benefit (automatic recurrence timer) we were clearly incentivized to keep using the application. A missed feeding would quickly get entered so we knew how he was doing throughout the days and a simple count of times and amount he fed to ensure positive weight gain. You can imagine the potential for existing social networks to measure and alert on emerging trends that may impact your health through continuous input for all of your data.

Make it Easy

Quantified Baby 3

While we are both engineers, when first learning to deal with another human life your focus tends to be on their needs and not learning complex applications. Actions and questions need to be extremely clear and operable with a single hand and thumb.

It should also be easy to get quick statistics from a glance so that you can take action if necessary. With our tracking we could immediate see the time since last feeding, changes, sleep, and even bath or other customizable timers. This bio-dashboard meant we were reassured through parallel mechanisms that we were tracking data and everything was on track with him.

Automated tracking is even better as they don’t require remembering or only capturing observed actions. Deb Roy’s work in ‘The Birth of a Word’ demonstrates passive monitoring of language development that would not be as clear in limited measurements. I had originally planned a few hardware sensors that would measure crib pressure for motion in sleep and rolling over – but as they say “finish all your projects before the kid is born; you won’t get any done afterwards”.

Make it Shareable

The data are useful to one person, but arguably are just a substitute for memory. However with two or more people, a quantified self application becomes the collaboration center-point to have multiple inputs and actions on the information. When my wife would change diapers or finish a feeding my own phone would update with these events and I would be continuously informed without having ask, or even worse, wake her up.

The data also worked on any of our iOS-based devices. So instead of having to chase down a single unique device we could instead grab anyone iPhone or iPad that we had nearest to us – which usually wasn’t far as they are also great devices for reading while rocking, or playing some soothing music – proving the mobile phone in and of itself the greatest invention for parents.

Make it Open

A quantified self application should be expandable, and open to customization. While by default Total Baby has a good configuration of feedings, colors (if you need to be that specific), food types, and activity types – you are also able to add your own activities, timers, and as I mentioned before, automatic reminders. That means if you have special needs, medications, therapy, or just want to ensure validation that grandparents are getting equal time, then it’s important that the application allow you to add these as necessary.

A surprising and welcome feature of Total Baby was that all of the data were exportable as a Spreadsheet CSV and SQLite database file. While not the cleanest data model (times are just strings in the event descriptions which are not enumerated), it allowed us to play with charting and general metrics as our son grew. The growth charts and other timelines are also all exportable and email-able to family, doctors or caretakers. I can also make backups in case any of our devices are lost.

Opportunities for Health

“It’s your body, make sure you know what is going on with it.”

In personal health, I’m continually astonished by lack of personal data that is measured and retained. How many times do I need to fill out basic details such as address or blood type that I am dubious that real information such as blood levels, heart states, or other extremely important information is gathered, available, or used for any kind of passive analysis and alerting.

While I do trust my health practitioner, no one is as invested in the health of my body as I am. Doctors are busy and laden by antiquated technology that prevents them from providing the care they would really like to. There have been numerous attempts are revolutionizing the healthcare industry that fail through lack of focus, complex policy, privacy issues, or just lack of general initiative.

I want to see the same technology that reroutes traffic, recommends new shoe ads, and sorts my email to understand when I need to alter medications, change my behavior or visit a specialist who upon my arrival would have a full, and appropriate, analysis and access to my data history.

By focusing on the quantified self, we are empowered to monitor and maintain our own well being – and obviously with children to assist in theirs as they grow. I can invest the time, effort, and even technological development that creates a solution now that will aid my own family.


smile!


Faith in Services, Trust in Data

Published in Technology


Moleskine Concept Diagram 1

Earlier this week Google announced Keep, their new service for quickly storing notes from your phone and online. As others have pointed out, it’s similar to the popular Evernote. Feature comparisons aside, the launch of new services providing common, even essential, access to my information highlights a disconcerting trend.

Faith in Services

Underlying these product decisions is the manifestation of the products that we use. Historically we purchased physical media that we would hold and use as much as we wanted. Despite a company going out of business, ending a product or even upgrading the version we could choose to keep using the product as we had originally acquired and intended. Whether that was Word Perfect or Windows 95, we were in control.

With the advent and ubiquity of connectivity and the Internet, we have become more reliant on services – popularly referred to as The Cloud. There are numerous benefits to this architecture such as scalability, reliability, accessibility, and maintainability. Users can access information from almost any device whenever they need it without worrying about location, versions, or upgrades.

Consequently we put our faith in the service and the trust that there is connection to the internet, the service will be working, that our information is not leaked or shared, and even that the company or product continues to exist. We are now reliant on decisions for which we have little or no control.

Reality of Providers

Evernote has a vision to create a 100-year company, a nearly unique perspective for a new information technology company. It is the kind of pledge that instills a confidence that considers building a long-term relationship with users and exhibiting through products and services.

By contrast, invested and publicly traded companies have a fiduciary responsibility to maximize profits and increase shareholder value. They are legally required to change their products and business to contiuosly get the most money possible. Users are not shareholders and as such are beholden to the decisions to improve revenue. This is not bad per se, but can have consequences in products that live or are shutdown that may seem arbitrary to users.

Google is well known for this behavior, as recently in shutting down Google Reader, though even more relatedly the Keep-similar closing of Google Notebook that was part of a regular “spring cleaning”.

Beyond mere existence of these services we are still reliant on the stability of the service provider. The recent launch of SimCity was plagued with service failures due to scaling and security. And there are many more intentional service related restrictions, particular if the service provider is a hardware manufacturer with clear incentives to lock you into their physical platform.

Trust in Data

The point is that we must realize the vitalness of protecting and accessing our data. Whether my personal notes, email, photos, business plans, or any other information that I have, it is imperative that we retain ownership and rights to the underlying data. Users should be able to hold their data with permission to access, use and reuse regardless of future business decisions.

Services are a value-add. They make my data more useful and perform amazing capacity such as character recognition, entity extraction, geocoding, analysis, and recommendations. But these cannot come at the detriment of control and access to the information that exists independent of a particular product.

Consumers are relatively new to technology, and there is a constant flow of marketing, features, and new products to try. These are easily appealing and exciting while overlooking to the potential implications of investing into a particular service. I hope the culture evolves to where access to the data is not an esoteric or unrelated conversation but is a forthright requirement of any new service.