Now what? Given the large and growing pile of data we will optimistically uncover, we will run into new frustrations. People will claim that the published data formats are not the ones that their analysis tool requires. People will be overwhelmed by dataset size, not knowing where to start. People will unknowingly recreate someone else’s data-munging workflows on the way to repeating analyses of the same data. People will become the next bottleneck if data ever ceases to be.

There’s no one answer to the concerns listed above because everyone has a different goal for the data. To handle these issues, we will need more than a place to find up-to-date datasets—-we will also need a place where it is easy for people to share ideas and strategies for tackling data. We will need a social data commons.

Whereas blogs and wikis help report findings, steps, and missteps, a social data commons can be the place to go to “talk shop” about the available data. Even if people post their solutions using decentralized means, there will be benefit to pooling all of these resources in one place on the web. Here are some tools that will help the data-tinkerers get things done:

• Data-munging war stories. The first stage in data analysis is often long and frustrating. One must digest the dataset in the form they received it, and transform, clean, and filter out the subset that they wish to analyze, visualize, or otherwise present. The workflow differs for each dataset and application, but to the extent that people can share tools and instructions for processing each dataset, these should be written up in the form of recipes for baking the data.
• Crowdsourced analysis. Datasets can be overwhelming. While many exploration tasks are easily automated, it is often easiest to leave certain tasks (e.g., “Find the interesting pictures”) to humans. Mechanical Turk gives us a hint at what this might look like, and the Guardian provides a wonderful example of crowdsourced public data analysis in action.
• Current uses showcases. To spark competition, avoid duplicating work, and inspire follow-on projects, visitors should see a showcase of the current uses of each dataset. Aside from links to sites built around a dataset, the list can include embedded visualizations of finished work.
• Analysis wishlists. Given that data released by a government reaches more than just programmers, there will be more people with ideas than people who can implement the ideas. People with ideas should be given an outlet, and passers-by should be asked to vote on these ideas to help data geeks with some free cycles discover the most insteresting unimplemented project.
• Data wishlists. If an agency were to dedicate resources to releasing another dataset, which one is in highest demand? As Ted mentioned, governments should let demand drive delivery.
• Forums. No set of tools will encompass all use cases for social data analysis. A discussion forum can lead to the formation of interest groups while serving as a catch-all for needs not served by the list above.

The US government might hit a few bumps trying to implement some of these social features. For example, a conflict of interest might arise if the showcase of uses of a dataset includes a site critical of the current administration. Having the executive branch ban spam or abusive comments on a forum draws concern over limitations of free speech. These details are not roadblocks, but they do signal that we can’t expect a social overlay to spring out of data.gov per se—-if we want these features, we may have to build and manage them on a third party.

I’m sure there’s more to the social data commons than I listed here. What did I miss, and where can we seek further inspiration?

Thanks to Ted for reading the first version of this entry.

1) Strongly typed worksheets with “advisory” error checking. For instance, the user can designate a particular column to hold numbers only, and maybe proceed to enter a date, but would then see an Excel-style warning dot in the cell in question.

2) Transparent many-to-many or one-to-many relationships between worksheets in a workbook (think foreign key relationships in database-speak). The user can designate a particular column to hold references to rows in another worksheet, or lists of such. The other worksheet will then automatically have a corresponding column added containing references going in the other direction.  (E.g. if each row in the “Departments” worksheet has a column referencing “Courses”, then “Courses” has a column referencing the corresponding rows in the “Departments” worksheet.

3) Hierarchical presentation of relationships between worksheets in the workbook. Columns that reference other worksheets may be configured to show any subset of columns from the referenced worksheet, and so on.

User testing with multiple prototypes showed that the user interface needed to be very similar to that of a traditional spreadsheet (e.g. Excel) to be usable by most users in the target population. Significant features hypothesized to make the interface more efficient (e.g. automatic report layout management) proved only to confuse the users and make it harder to design consistent editing affordances. Nevertheless, we did manage to integrate the key high-level features of the application (relationships between worksheets and the presentation of resulting hierarchical data on screen) into a prototype bearing very much of a resemblance to Excel.

(This project was done by Paul Grogan, Yod Watanaprakornkul, and me.)

I argued in favor of letting individuals make their own RDF collections (using, for example, our Exhibit framework) and worry about merging them with other people’s data later.  Stefano’s response accused me of using “RDF” and “structured data” interchangeably, asserting Exhibit is really just a nice UI over spreadsheet (tabular) data—that although it can export RDF, it is “not properly using RDF” because it has “lost the notion of globally unique identifiers (and in that regard, is much more similar to Excel than to Tabulator)”.  Tim Berners Lee has made similar complaints to me about Exhibit not using RDF.

This argument highlights for me yet an important ambiguity about what RDF is.   I occasionally have to help people understand that RDF is a model, not a syntax.  That some data can be RDF even if it isn’t serialized to RDF/XML.  That the key is to have items named by URIs, connected by relations named by URIs.  Stefano’s argument suggests a different blurring: between the model and its intended use.  Stefano’s “not properly using” phrase implies that if you don’t intend to merge your data into the global namespace, then even if you implement the model  and wrote it down as RDF/XML to boot, you won’t be “properly using RDF”.

I want to address both these claims: that Exhibit is just a UI over spreadsheets, and that using RDF this way isn’t proper.

RDF and spreadsheets

Regarding the spreadsheet claim, I’ll begin by admitting that Stefano is absolutely right:  Exhibit is a visualization tool for tabular (spreadsheet) data.  But notice that all RDF is spreadsheet data—I can take all the RDF in the world and throw it into one spreadsheet.  In fact, I only need three columns to contain the subject (tail), object (head), and predicate (link) for each RDF statement.  Admittedly none of today’s spreadsheets would have enough rows, but that’s an engineering detail.  So, the spreadsheet model isn’t the problem.   And we also agree that Exhibit’s interface is nothing like spreadsheets’, and far better for the collection visualization tasks it is designed for.

I think instead that what Stefano is objecting to is a usage characteristic of spreadsheets versus RDF.  When I open a spreadsheet, the data it shows me is right there, in a file on my own system.  Global identifiers don’t matter because the data is all there (and presumably self-consistent) in the one spreadsheet.   In contrast, in Stefano’s image of RDF (and in Tim’s, as one can see from the Tabulator project) the data about a particular entity is spread all over the web, and it is the globally unique identifier that lets you go out, gather all that data together, and know that it is all about the same entity.

This is certainly an appealing vision.  But I want to argue that a focus on globally unique identifiers neglects two benefits of RDF that I consider equally important: data portability and schema flexibility.

Spreadsheets suffice

To illustrate this argument, I’ll hark back to a previous post where I discussed a data integration problem that should have been easy but wasn’t.   I keep an  Exhibit of folk dance videos on the web.   Recently, Nissim Ben Ami posted a collection of 511 new dance videos on Youtube.  I wanted to incorporate it into my site.  But it quickly became apparent that said incorporation would basically require my entering all 511 video descriptions manually into my system, and I still haven’t gotten around to it.

The major barriers were twofold.  The first was syntactic:, the structured descriptions of the videos were delivered as XML.   That meant that in order to get at the data, I was going to have to learn XSLT—something I’ve been putting off for years.   The second hurdle is semantic: Youtube has the wrong schema for my folkdance videos.  I care about choreographer, dance type, and year choreographed; YouTube only offers slots for submitter and submission date of the video.  So, as you can see from this example, the contributor takes the usual approach: he takes his nice structure data and shoves it into the generic comment (info) field as free text.  All that structure is instantly lost.

Suppose instead that spreadsheets (or, in a pinch, RDF) were the accepted framework for publishing information on the web.  The YouTube “spreadsheet” would contain submitter and submission date information, but Nissim could just add “artist” and “composition-date” columns to hold the data he wanted to enter.   I would then be in a great position to download his data and incorporate it into my own catalog (spreadsheet).  What would I have to do?  After opening his spreadsheet and mine, I’d have to match columns—perhaps he called his “artist” and “composition date” while mine are “choreographer” and “year”.  But a simple copy and paste fixes that discrepancy.  Merging entities is not much harder than merging properties: a simple global replace will convert his choreographer “Israel Ya’akovi” to my “Israel Yakovee”.  The local consistency of his data and mine means that I only have to work once per choreographer (and in most cases I won’t have to: there’s a standard spelling for almost every choreographer’s name, which serves as a unique identifier in this context even if it isn’t a URL).

Overall, my work has reduced by order of magnitude.  Instead of laboriously entering 511 new records, I just download a spreadsheet and match up a handful of properties (columns) and a few tens of choreographer names.

Stepping back, observe that I’ve relied on two things.   First, on data portability—my being able to download the data in a convenient form: not XML, which is a programmer’s friend but an end-user’s enemy; rather, something I can just look at and understand.  Second, on schema flexibility—on Nissim’s being able to add whatever columns/properties he decides are important, instead of being limited to those used on the hosting web application.

I’m also relying on some features of this particular scenario, but I believe they often hold.   I am relying on Nissim’s data having only a small number of properties so that I can map them manually to mine.   I also rely on there being a small number of choreographers, and hope to take advantage of most of them having matching names in his data and mine—these names certainly aren’t globally unique identifers, but they are “unique enough” when considering just my data and his.  Critically, I am not thinking of pulling all data about a given dance from a multitude of different web sites—this would demand global unique identifiers to link data since I would never have the patience.  Rather, I am considering a pairwise data acquistion: taking data I want from one internally consistent site.

Such pairwise acquisition is commonplace: any time a scientists wants to pull a data set from some other scientist’s lab, or a consumer wants to download product information about several cameras from a review site, or a student wants to include a Wikipedia data set in a report they are writing, there is an obvious single source and target for a data merger.   And there’s a human being who has the incentive, and with the right tools the capability, to do the limited amount of work needed to accomplish that merger.

This is a simple low-hanging fruit argument.  It would be wonderful to be able to automatically merge data from thousands of different sources into a coherent whole.  And this is a problem Freebase will need to solve, if they want to become the hub for aggregation of structured data.  But right now we can’t even manually merge data from two sites without doing a ridiculous amount of grunt work—so perhaps we should give some attention to that easier problem on our way to solving the hard one.

Don’t skip the wild west

I’d like to so these efforts proceed in parallel, but I’m worried about enthusiasm for the more ambitious goal blocking movement toward the low-hanging fruit.  I recently submitted a proposal to NIH on the topic of data integration that reflected my perspective above.  I argued that the current efforts in the Biology community to force everyone to adopt a common ontology (and sometimes repository) for their experimental data are being resisted by biologists who think they know best how to present their data.  I suggested as an alternative that we give biologists tools, such as Exhibit, that would encourage them to publish their data in a common structured syntax, and worry about integrating all that data after it has become available in structured form.  The proposal rejection was accompanied by a review that said, on the one hand, “The benefit of the proposed approach is that it is very different from some multi-institutional data sharing projects (like caBIG), which have used a very rigid, top-down approach to creating semantics. Even if this project is unsuccessful it could bring to light new ideas and strategies that might make those large-scale projects more responsive to investigators and more successful.”  At the same time, it argued for rejection because “The absence of any control over the information models and ontologies – truly a semantic wild west – is daring and may ultimately be the downfall of this project.”

I’m fascinated to see, in the same review, a recognition of the problems that the current centralized approach is bringing (lack of buy-in to common ontologies by individual scientists who think they know better and probably do), and an unwillingness to tolerate the contrary (anarchic) solution.  I also love the metaphor of the “semantic wild west” because I think it supports my argument.  Would anyone have suggested establishing a city of several million people just after the west was opened for settlement?  The west’s early wildness was an unavoidable phase of its evolution towards the thickly settled and uniformly governed area it is now.    In the same vein, I think that our semantic web is best grown by encouraging individual semantic-web settlers to create their own data homesteads and begin looking for the trails that connect them to neighboring collections.  We need to get the data into plain view first.   Later we can send in the data sheriffs and place all those data sets under uniform governance.

There are now hundreds of ontologies with tens of thousand of terms.  However, it has always been a “cottage industry”—various groups develop their own ontologies, then publish them for use by others.  Is there a way to open the development of the ontology up to the community?  Community might be just a few or thousands.

As an example, the gene ontology (28K terms) has 3 full time curators.  People from the community submit to an issue tracker to get new terms etc.  A ne version is released daily.  In contrast, the NCI thesaurus (for cancer) has 20 full time editors with 1 lead editor who runs everything, and a slow cycle of “releases” with less community input.  Others work like typical open source projects with 20-30 team members involved in active discussions.

Natasha’s group builds on Protege, a very old open source ontology editor that is now one of the most popular, with 120,000 registered users.  It has a very open plugin architecture with dozens of plugins for visualization, import, export, nlp, and lots of unknowns.  They’ve been working to augment protege with support for collaboration.  It works in a distributed fashion (desktop and web clients).  It support simultaneous editing, but also annotation, discussion, proposals and voting in the context of the ontology.  There are many types of annotations—questions, comments, proposals—on any elements of the ontology—classes, properties, instances.   While the tool handles most types of structured data, it is focused on taxonomic hierarchies were stuff gets inherited down the hierarchy.

They investigated use of their tool for several tasks.  One is ontology evaluation—finding existing ontologies that might be useful for you.  This source of information for this is author-contributed metadata about the ontologies—domain, key classes and concepts, who the developer is, etc. Another is automatic tools that compute quality metrics, and another is annotations by other users of the ontologies.

This last is important because some ontology metrics are subjective—a feature that is “good” in one setting can be awful in another.  An example might be a high level of axiomatization.  This is important for inference, but creates clutter if you just want description.  There’s also the problem of crosscutting taxonomies—you might have two different ways of describing the same domain that form a “matrix” of non-overlapping hierarchies.  To address this sort of subjectivity, they allow users to record evaluations of ontologies.

These tools can be explored at their bioportal web site where they have a large library of biomedical ontologies.  On that site, users can describe their ontology based projects, and list/review the ontologies they are using.  Reviewers give general reviews, usage information problems encountered, coverage of the key terms, major gaps, and issues with specific elements of the ontology.  This site aims to make ontology evaluation/creation a truly democratic process.  This is controversial—some argue that ontologies need a more rigorous editorial process (mirroring a current debate about open vs. traditional journal publication).

Another big task is mapping: connecting two ontologies by asserting that terms in two different ontologies “match”.  They aren’t trying to find mappings, but want to enable others to upload the mappings they have found.  Mappings can be created manually or uploaded in bulk (if computed by someone’s tool).  Mappings are themselves metadata, which can be annotated and discussed just like other data in the ontology.

Of course a big question is whether people will use these tools.  Right now, many users are asking for these features and reporting lots of bugs—good signs of demand.

A lot of questions have now arisen that need some serious user studies—what are the dynamics of the social networks that form around collaborative ontologies?  What are the different types of users/editors?  What produces the most discussion/controversy?  Do these tools help or hinder collaboration?

Is time just another attribute of data? Or is it something more? Time brings meaning to data, especially data about the real world. Time is also essential for understanding human activity and an essential element of design processes. Sometimes we address time explicitly, sometimes implicitly. It structures how people interact with computers, but is also a measurable effect of that interaction. The goal of this workshop is to explore human-computer interaction from a temporal perspective.

We were pleased that our workshop drew 35 participants with a variety of interests and backgrounds — from architects, interaction designers to data mining analysts, doctors, ubicomp researchers, and of course HCI researchers.  As can be seen in the workshop proceedings, our participants were interested in a number of different types of temporal data:

• Media: audio + video capture, manipulation, editing, sharing
• Personal health
• Personal information management
• Life logging (Personal activity data recording + Reflection)
• Air traffic control
• Financial data analysis
• Sensor networks
• Environmental impact monitoring
• Product research
• Software engineering

Despite the diversity, several common themes emerged.

The first was empowerment: the idea that accurate, low-cost-to-capture rich records of people’s everyday activities could thoroughly change the way we live.  Participants highlighted several creative examples of how such records of our lives could help us — in personal, social and work contexts.  For example, getting accurate records of one’s daily routines (such as exercise and diet) could let people identify ways to live healthier [a la Thomas Goertz's Decision Tree].  Or, to enable the hacking of social dynamics: for example, to analyze in situ or post-hoc repeated patterns of conflict in interactions with particular individuals so as to be able to better understand sources of stress related to collaboration.  Or, simply helping the user more easily retrieve and manage their personal information in an activity-centric manner than complements human episodic memory.

The essential challenge was the question of how to give individuals (just-plain-folks, end-users) access to this rich data about themselves in a way that they could easily analyze, understand, manage and use.  One participant commented that such information was “turning citizens into intelligence analysts — about their own lives”.  Intelligence analysts, of course, have extensive training in how to look at data; end-users don’t.

Another was the question of accountability, access, protection and privacy: we have never previously had access to accurate records about any aspects of our lives.  Once we have these records, what sort of implications will this have on our interactions with others? (e.g., ineffable records of where people were, how long they were there, what they did)  The process of scientific discovery, process/protocol and how this will impact how scientists work with one another? How will we control or grant others access to these records in a way that provides individuals privacy?  If individuals are employees/members of organizations, who “owns” the data about an individual’s activities at work, and what rights does the individual have towards accessing/ it and what rights does an individual have to their own activity records? Finally, after an individual departs (passes away or leaves an organization), how should such data be handled or retired? Who has rights to a deceased individual’s life log?

Other themes and topics of discussion included : the need for interfaces to help reconcile subjective/emotional memories of the past with “cold, hard lifelogs”, implicit versus explicit representations of time; e.g., different ways of portraying dynamic processes, and explanation facilities for time-dynamic pattern recognition.

Based on the strong interest from our workshop participants, we have decided to start a discussion group / online watering hole for us to further discuss some of the issues surrounding interaction with temporal data.  We welcome anyone interested (not only workshop participants) to join and post their thoughts, questions, projects and ideas:

• Temporal-Data @ groups.google.com – Temporal Data Google Group

With this google group we wish to continue our consolidated, cross-application domain discussion of interaction issues with the hopes of taming the complexities of our data rich environments.

The magic behind raw data

The best thing about raw data is that almost everyone knows how it works. This means that as far as the data (re)user is concerned, the datasets are text files (or perhaps a close variant) that they can download, open in some default application, and get some immediate use out of it.

If the US Federal budget dataset is released as a comma-separated file, a middle-schooler can download the file, open it in a spreadsheet application, and sum the columns to see how much we’re spending on the Department of Education this year. A more skilled high-schooler can upload the file to Many Eyes, make a pie chart out of it, and post it to their blog. A first-year college student can write a php script to allow people to comment on various parts of that pie chart, allowing you to drill in to various slices to get a finer granularity.

With raw data, you’ve opened more people to more visualization, exploration, and discussion than was available through the original web application that acted as a firewall to your database.

Hugging the data to death

During his talk, Tim spoke about “Database Huggers,” or people who, for various reasons, hide their data away in databases. Once the data sits in a database, the publisher might provide a specific and constrained view of the data by way of a website, or they might hide it even more, simply calculating some aggregate statistic over the data and claiming, without verification, that the data has certain properties.

There are several legitimate reasons for database hugging. Some data was meant to be private—academic, medical, and financial information are all datapoints we’d prefer to keep private. We’d hope our service providers will keep it out of the hands of others. Similarly, a company might have competitive reasons for keeping information private, especially when it would be equally valuable to their competitors and not too valuable to the public—lists of customers and transaction histories come to mind. Keeping this information far from the publicly accessible web is responsible and wise.

There are other cases, however, where the data should legitimately stay open and publicly accessible. Open government initiatives will result in many datasets published by organizations that will or should exist in the public domain. Many Long Tail websites, maintained by small groups of hobbyists, probably would not mind if the datasets they generate are published in their full glory. For these types of applications, raw data is ideal.

Even in the case of datasets that should be open to the public, database huggers will sometimes disable direct access to the data, instead opting to place it in a database that sits behind an html-generating web application. Thinking that you’ve hidden your data behind HTML, thus making it safe from reuse, is an unwise assumption. In about an hour, a decent programmer can write a perl script to crawl your site and tease the data apart from the obfuscated HTML that surrounds it, reverse-engineering your database without asking for permission. In fact, there are tools that make this process easier than writing a one-off perl script. And if you think you can block the person from accessing every page on your site in a short period of time, then they will just collaborate with everyone else who wants the data, write a Greasemonkey script to collect parts of the site that they browse, and eventually collect your entire presented dataset.

Databases are not inherently evil. They provide an excellent way to store, index, and query data, but they also have a way of separating the average user from that data. Most websites, for example, do not publish a read-only username and password to their database, for fear of arbitrary queries that could easily take down their machines, or at least keep the machines busy for a long time. We should design tools to maintain the excellent services that databases have been built to provide over the last four decades, without limiting the access to the raw data when such access would be most valuable.

Are APIs the future of raw data?

There is a middle ground between the highly private datasets and the obviously open ones. Most forward-thinking organizations have realized this. They have also realized that if they have something to sell, be it in meatspace or screenspace, it’s better to release the data about their offerings to anyone that wants to use it, so that people eventually end up at their site. They do this by providing a web API to make their dataset queriable, essentially telling other software developers which questions they can answer about the dataset (query for books by author, query for restaurants by cuisine). Amazon has some APIs, as does Yelp, and you’d have to be a pretty self-loathing web 2.0 company to not provide an API over some portion of your data. So are APIs the solution? Not always.

APIs are a step in the right direction—open data is better than obfuscated data. APIs help both third-party developers and dataset publishers get more out of a dataset. They have a few drawbacks as well:

• The API is an HTTP interface to your database. This means that if someone else makes a third-party application that is immensely popular, it’s your database that pays for the brunt of its popularity. You weren’t expecting a huge ramp-up in server load? Too bad.
• As kind as the dataset publisher is, they can’t predict every use of the data—if they could, they already would have implemented the best use cases. If they can’t predict how the consumer/developer will use the data, they might not publish a good hook into the dataset. This would either prevent or make awkward the interaction between the third-party application and the publisher.
• Building an API for a dataset makes the people who are nice enough to share their data do more work on top of designing their application. Following common REST or CRUD conventions makes this easier, but still puts the onus on the developer. As a corollary, APIs don’t change with the data. APIs are frequently revised, meaning that a change in your data requires constant upkeep of your API.

One might argue that some of the criticisms of APIs are unfair:

• Saying that raw data will reduce the load on your database implies that the third party has some cache of the data, which is thus slightly out-of-date. You could imagine some sort of Comet-updated raw dataset system, but it’s unlikely for now that dataset publishers will be willing to stream live updates to third parties.
• Perhaps the limited API functionality is for good reason. Amazon might never want you to be able to download their entire dataset—they don’t want to waste the bandwidth and they don’t want competitors to know exactly how many items they have on hand.
• Publishing any sort of raw data will require extra work on behalf of the dataset publisher. Perhaps API-writing is the least invasive of their time?

An ideal data management tool would allow raw data publishing when possible, and make it easier to build APIs when some limited access is desirable. We should not pretend to know the point at which raw data is superior to APIs, but the point exists somewhere. It’s important to understand the benefits that raw data provides on top of web APIs, so that you can think about when it would be valuable to use.

After all this time, the answer was text files?

You’ve probably become skeptical of these suggestions. Are we really supposed to throw away decades of database research in how to properly store, index, and query reasonably sized datasets so that a middle-schooler can look at the data in a different way? Of course not. The interesting research question becomes whether we can give the user the illusion of raw data while still benefiting from database technology where possible.

That’s one research direction we’re taking within the Haystack group. With the constraint that the raw data, in human-readable text files, should always be available, we’d like to blur the boundaries between databases and data-aware webservers.

Specifically, what we plan on designing is an apache web server module that recognizes when it is serving a dataset, perhaps by taking note that it is serving a .csv, .rdf, or .json file. In such cases, the server would cook the data into a database behind the scenes. Data-aware clients (in javascript for the time being, but in the browser one day) can then query the web server about the data directly. Updates become difficult, but we can make consistency guarantees about the original raw data text files to ensure that someone can download them and see up-to-date information.

If you prefer programmatic access to the files, the module turns into a REST(, SQL, SPARQL, you favorite path language)-capable endpoint. If you prefer to get down and dirty with the data, you’ve got the text files.

We certainly don’t want to stand in the way of a world with Linked Data, so if you’d like, the tool will eventually return data with URIs. We can’t guarantee the URIs will resolve to anything useful, but that just might require a human’s touch. We’re not sure how that fits into the picture for the average data publisher, since the marginal benefit to the individual of universally addressing your own data is small, whereas the benefit to everyone else of adding another linked dataset grows with the number of datasets it is linked to.

And now, for some questions

We’re early in the development of our tools, so we’re open to your ideas and suggestions. Keeping text files up-to-date with the database that’s proxying them is nontrivial. Thinking of the ideal client/server mode of operation will also take time. We probably haven’t thought of the most important must-have feature yet, so any suggestions are welcome.

Thanks to Ted Benson, Sam Madden, and David Karger for their thoughts on this post.

(Cross-posted on my blog)

Suppose I’m developing some sort of academic course management system. Chances are I’ll want to display to the user a list of course offerings and their associated course codes, readings from the syllabus, meeting times etc. Maybe something like this:

Now according to Good Rules of Normalization and Decency, I probably stored this data across several database tables, related by foreign keys. I might have tables named “offerings”, “course_codes”, “readings”, “sections”, “meetings” and so forth. So how do I retrieve all this related data from the database?

The good news is that relational databases are made for just this kind of task: joining tables efficiently is what they do for a living. Unsuspectingly, I run my query [1]:

SELECT o.title, cc.code, r.author, r.title, s.name,
       m.start_time, m.end_time, m.day, m.place
FROM   offerings o, course_codes cc, readings r, sections s,
       meetings m
WHERE cc.oid = o.id
AND   r.oid = o.id
AND   s.oid = o.id
AND   m.sid = s.id;

The bad news is: That didn’t work too well. The mistake may seem obvious to seasoned database application developers: I can’t just do several unrelated joins in parallel like that, or I’ll get a gazillion rows [2] back. Not only does this lead to exponentially bad performance, but the result is also in a rather annoying form as far as the client application is concerned. There is even another problem: if any of the courses in the database do not happen to have any sections or readings listed, they will be omitted from the result. SQL “fixes” this through a hack known as outer joins. It introduces NULL values into the result and, rather undeclaratively, requires each join to have its particular join condition specified explicitly rather than as part of the more general WHERE clause.

So how do we retrieve data like this from a relational database? We pull the joins out of the database and evaluate them ourselves, in our own application-specific data structures. Just about every non-trivial database web app out there does this in some way or another. The data is stored across multiple related tables in some MySQL or Postgres database. When the Javascript in the end user’s browser needs to present data to the user in some hierarchical fashion like the example above, it issues a request to a server-side middle layer, written in PHP, Ruby on Rails, Python, Java, awk or whatnot. The middle layer, possibly with the help of a persistence library, then issues a bunch of separate SQL queries to the database to retrieve all the data involved, assembles (read: joins) this into some hierarchical data structure, and returns it to the Javascript app in JSON or XML form. True, the database does help limit the data enough that this assembly process is not too much of a performance concern. But joining tables is the job of the database, and we shouldn’t have to write middle layers to do it ourselves.

There should be a general and declarative way to make big joiny queries like the above work efficiently, returning the data in exactly the hierarchical form we want it — strictly relational result sets are not expressive enough. I am currently working on a simple SQL-like query language that does just this: send my generalized middleware a single big, declarative (no for loops or outer joins here!) query, and you’ll get back the JSON equivalent of the relational result set with the data nested into arrays and objects any way you want it.

[1] “No one does this!” some may object. Actually, Ruby on Rails’ own ActiveRecord did for a while.
[2] I believe the technical term is “The Cartesian Product.” Darn you, Descartes.

The basic approach is for the owner to manipulate the visible parts of the system—the forms that people use to enter data, and the pages that show the data in the system—and for the server to automatically create the schemas and databases needed to support those interfaces.  For example, if the owner adds a field in the form, the backend will add a field in the back-end database, without the owner knowing anything about that database.  This class of tools are known as “forms driven applications”.

The main contribution here is that an important part is to manage “workflows”—they way content is entered and then flows through various stages of the system, evolving and changing who can and should see it as it goes.   There needs to be a notion of roles and access permissions, and for pages to behave differently depending on both the state of the data and who is accessing the page.   It’s hard to do this if you only work with one page/form at a time.  Their tool tries to provide “guided debugging” of the entire workflow, suggesting the next steps that should happen to a particular piece of data, data it should be combined with, and roles it should be assigned to.

These ideas have been pushed into a startup called app2you.  I quite like the approach and hope it is successful.

There’s a problem of hunting through all the queries to find the one you want.  They want effective search and browsing, and also assistance in composing new queries.    There are challenges:

• queries are not just strings, but complex objects with inputs, outputs, and semantics.  2 similar queries can have very different outputs, and 2 different queries can return the same
• typical search problem: need to avoid giving too many matches
• efficient algorithms (this is a database conference after all)

An application they have in mind is scientific data management.  There’s tons of data and lots of (shared) data analysis with complex queries that are freqently evolving.

Consider the scenario of a novice user trying to create a query, given a large repository of past queries by others. He’ll try to find a perfect match but will probably need to take something close and then modify it.  There must be a metaquery language for describing the kind of query you want.  Since that query was probably built over time, there may be many versions that evolved, and it can be useful to see all the different versions and find the best ones for his use.   It willbe useful to explain to the user how these versions are related, e.g. this refines that.  One needs to watch out for the metaquery being more complicated to construct than the query they want to find.  One approach is “partial query”—for the user to build as much of the query as they can, then look for other queries that are similar.

• When you are trying to decide whether you should merge data from two different sources.  e.g. to find out what portion of keys and concepts they have in common.
• To decide between different approaches to integration.
• To become aware of what information you even have.  E.g., the department of homeland security was formed by mashing a bunch of different agencies together, and it isn’t even clear if they know what they know.
• To help form communities, by discovering subcommunities with overlapping knowledge who could benefit from talking to each other
• The government often uses a “one to rule them all” massive schema as a means of data exchange, a hub and spoke model where everyone migrates their data into and out of the huge schema.  To use it, you have to find out where your little schema fits into the huge one.

He described a case study of such schema matching and outlines limitations in existing tools and needs for the next generation.  Schema A had 1374 elements, was a relational schema envisioned as being a hub schema for the whole military.  Schema B was relatively small (800 element) legacy schema.  They hoped to subsume B away into A and forget about it going forward.  But they asked, do these schemas overlap (and what is the nature of overlap)?  If not, maybe B should be left as an island.  What is distinctive about each?  Can you produce a comprehensive vocabulary of terms participating in one or both?  Nobody wanted any mappings (yet).  They did want summaries, statistics, high level concepts of what the schemas address.  What were the commonalities and distinctions?

They used a schema mapping tool called Harmony.  It was hard to identify high level concepts from lists of matches: what is “date_begin+156″ property.  The started by manually identifying “high level concepts” on both.  For each A concept, they looked for strongest matches in B.  Reported numbers of overlaps and distinct concepts.  They concluded there weren’t many overlaps.  Customers said “great, can you incorporate these 7 other schemas?”.  They really needed a way to automatically summarize a huge schema into coherent parts.   They found schema centric views (showing whoe schema as one object) were insufficient.  They discovered belatedly that spreadsheets were actually a good way to show the pairwise matches of schema terms.  But this doesn’t work beyond 2 schemas.  Multi-way matching is hard and vital.