Now what? Given the large and growing pile of data we will optimistically uncover, we will run into new frustrations. People will claim that the published data formats are not the ones that their analysis tool requires. People will be overwhelmed by dataset size, not knowing where to start. People will unknowingly recreate someone else’s data-munging workflows on the way to repeating analyses of the same data. People will become the next bottleneck if data ever ceases to be.

There’s no one answer to the concerns listed above because everyone has a different goal for the data. To handle these issues, we will need more than a place to find up-to-date datasets—-we will also need a place where it is easy for people to share ideas and strategies for tackling data. We will need a social data commons.

Whereas blogs and wikis help report findings, steps, and missteps, a social data commons can be the place to go to “talk shop” about the available data. Even if people post their solutions using decentralized means, there will be benefit to pooling all of these resources in one place on the web. Here are some tools that will help the data-tinkerers get things done:

• Data-munging war stories. The first stage in data analysis is often long and frustrating. One must digest the dataset in the form they received it, and transform, clean, and filter out the subset that they wish to analyze, visualize, or otherwise present. The workflow differs for each dataset and application, but to the extent that people can share tools and instructions for processing each dataset, these should be written up in the form of recipes for baking the data.
• Crowdsourced analysis. Datasets can be overwhelming. While many exploration tasks are easily automated, it is often easiest to leave certain tasks (e.g., “Find the interesting pictures”) to humans. Mechanical Turk gives us a hint at what this might look like, and the Guardian provides a wonderful example of crowdsourced public data analysis in action.
• Current uses showcases. To spark competition, avoid duplicating work, and inspire follow-on projects, visitors should see a showcase of the current uses of each dataset. Aside from links to sites built around a dataset, the list can include embedded visualizations of finished work.
• Analysis wishlists. Given that data released by a government reaches more than just programmers, there will be more people with ideas than people who can implement the ideas. People with ideas should be given an outlet, and passers-by should be asked to vote on these ideas to help data geeks with some free cycles discover the most insteresting unimplemented project.
• Data wishlists. If an agency were to dedicate resources to releasing another dataset, which one is in highest demand? As Ted mentioned, governments should let demand drive delivery.
• Forums. No set of tools will encompass all use cases for social data analysis. A discussion forum can lead to the formation of interest groups while serving as a catch-all for needs not served by the list above.

The US government might hit a few bumps trying to implement some of these social features. For example, a conflict of interest might arise if the showcase of uses of a dataset includes a site critical of the current administration. Having the executive branch ban spam or abusive comments on a forum draws concern over limitations of free speech. These details are not roadblocks, but they do signal that we can’t expect a social overlay to spring out of data.gov per se—-if we want these features, we may have to build and manage them on a third party.

I’m sure there’s more to the social data commons than I listed here. What did I miss, and where can we seek further inspiration?

Thanks to Ted for reading the first version of this entry.

Clay’s criticisms are well founded, but what I’d like to see more of is some brainstorming about what our ideal data.gov would look like. A recent post about the coming data.gov.uk site provides a nice foil for us, for one, as the UK seems to be taking a very different approach. But more importantly, what would you want to see in a government data site, and how would you use it?

I heard once that it is a good exercise to try to compress an idea into three sentences or less — it forces you to understand what you really want to say. So here is my three sentence suggestion:

• Bring it all under one roof. The current data.gov site is like Yahoo! from the mid-90s: it is just a directory of links to other sites. This is a noble start, but we really need to get a single point of access if we want to revolutionize eGovernment. The government is an immense, heterogeneous organization, so this is as much an organizational challenge as a technical one. But there are plenty of precedents of systems which allow individual data publishers (the government agencies) to retain control over the publishing and updating of their own data, while allowing data consumers (the public) to access it all from a single location.
• .. But don’t forget to give credit. When offering a single access point for all the data, it is essential to keep metadata that tracks which data came from where. This is as important for book-keeping and data integration reasons as it is for simply giving credit where credit is due. Agencies that publish data sets of great use should be recognized for their work.
• Build it as you go. We don’t need the perfect system overnight. No single ontology, schema, or data format will be able to encompass all the government’s data. That’s OK — it doesn’t have to. Don’t let fear of not getting it perfect slow down incremental progress toward our goal. Just bringing the data under one roof is a fantastic start; you can always try to begin standardizing formats and “linking” it later. My next blog post will specifically address this topic.
• …And version data sets. The benefits of offering “versions” of datasets are threefold. First, it allows you to maintain a system in which data providers feel comfortable updating their data at will. Second, it allows the implementors of the system to feel comfortable experimenting with data integration techniques and knowing that, if it doesn’t work out, users still have access to the same system they did last week. Third, it is the ultimate expression of openness: like a subversion repository for the government, everyone will be able to see the evolution of data over time.
• Help users discover data. With the sheer volume of data available, publishing it isn’t enough — you have to help people find what they want. The current data.gov site already does a decent job of offering search functionality. We can go further, providing data “footnotes” for bloggers to link back into the data.gov site (see the DataPress project for an idea of how this might work), suggestions of “hot” data sets for particular areas of interest, or a government data blog that highlights new and important data that has been recently published.
• .. And let them tell you what they want. Your users — the citizens — are your best assets. Let them prioritize your tasks for you by allowing them to suggest and vote on features and data sets they would like to see added. This type of decentralized management strategy is making waves among the business community, and the same mindset can be applied to government.

So there are my three sentences: Bring it all under one roof, but don’t forget to give credit. Build it as you go, and version data sets along the way. Help users discover data, and let them tell you what they want.

What are your three?

One set of changes we made was aimed at squeezing the gap between submission and conference.  It seems obvious to me that conferences should inform us of new work.   This doesn’t happen if the conference presents work that was submitted 7 months earlier (cf CHI)!  We made a few changes to tighten things up

1. We compressed the initial reviewing period to two weeks.  While many conferences give reviewers several months to examine their papers, I’m certain that few people look at them more than a week ahead of the deadline.  Certainly most reviews don’t arrive until day-of!  We got exactly one reviewer  rushecomplaint abodut this schedule.   Perhaps we’ll lose some unusually disciplined and organized reviewers who like to plan ahead; I’m still convinced this is a big improvement for the community.  For the same reason, we offered the program committee only a week to digest and summarize these reviews before the committee meeting.
2. Similarly, we gave authors only a week to revise their papers for publication after acceptance.   Again, we assumed most revision happens at the last minute, so we jumped straight to the last minute.  I’d say that a paper requiring more than a week of revision probably wasn’t ready to submit anyway.
3. We advocated moving to a model, like NIPS, of producing the printed proceedings after the conference.   Given the slow printing process, distributing printed proceedings at the conference requires that final versions be submitted long before the conference.  I haven’t opened a paper proceedings for years, but enough people are still attached to them that I don’t think the time is ripe to do away with them entirely.  But distributing them post-conference lets us keep them without keeping the delay they introduce.    In the end, we weren’t able to convince the steering committee to make this change; however, our push led the publisher to offer an unusually swift timeline for publication.

At the end, we squeezed out enough delay that the critical path became something outside our control: travel to the US.  For cheap fares, authors need to buy their airlines tickets a month before the conference.  On top of this we had to add a month for non-US authors to get visas for travel to the US (sadly, one of next year’s co-chairs could not get a visa in time, and missed the conference).  Combining these requirements with our one month review process produced our final, 3-month lag between submission and conference.

Looking ahead, I’d advocate for ISWC to eliminate the paper proceedings and to select countries with more sensible visa frameworks—Canada anyone?  This ought to let us cut the submission lag by another factor of 2.

We also made changes to the reviewing process.

1. We eliminated topic tracks, instead allowing authors to choose their preferred program committee member for review of their paper (as well as an alternate).  My experience has been that most track descriptions tend to be ambiguous gobbledygook that produce tremendous ambiguity about the “right” track to submit—especially if, as often seems to be the case for me, your work seems to span multiple tracks.  Given that decisions are made by the people on the program committee rather than abstract tracks, an “end-to-end” argument says picking the person makes more sense.  Authors understand their papers well and can study committee members to identify the one most likely to favorably receive their paper.   This is analogous to the submission process for journals where one chooses an editor.   There was some worry about load balance—what if everyone picks the “nice” reviewer?   But in practice we were able to assign every paper to one of the two chosen committee members.
2. We eliminated the paper “bidding” process.  In past ISWCs, all potential reviewers were able to survey all submissions and bid for those they wanted to review.   We felt that it made more sense for our knowledgeable program committee members to select reviewers who they thought were best suited to each paper—aiming for a “best for the paper”, rather than “fun for the reviewer” assignment of papers.  We feel this worked well, guaranteeing good reviews of each paper.  Interestingly, this was the one change on which we got negative pushback at the ISWC town hall meeting—even though it was populated by the beneficiaries of our reviewing process, the majority advocated a return to reviewer bidding.  I still think our approach is superior.
3. By a second end-to-end argument, we eliminated generic 1-5 scoring of papers,which ultimately must be translated into an accept/reject, and replaced it with actionable descriptions: we asked each reviewer to take a position for or against acceptance of each paper, or to “give up” and indicate they didn’t care.
4. Perhaps most important, we sought controversial papers.  The obvious and common measure of  paper quality is the average of the reviewer scores.  However, this buries one of the most important potential roles of a conference—to encourage debate about research.  Rather than a paper that everyone considers OK, a paper that half the reviewers love and half hate seems much more important to present at a conference, since it indicates a fundamental disagreement about research that is well worth airing in public, exploring, and trying to resolve.   To seek out such papers we applied two policies.  The first was that each paper with at least one vote for acceptance was discussed at the PC meeting.  The second was that any paper that, after discussion, had at least one program committee member advocate was accepted to the conference, regardless of the amount of opposition.  Because “average scoring” is so deeply embedded in reviewing practice, this change required changes to our conference management software—which were cheerfully carried out by the folks at precision conference, along with all the other great customer support they offered.  We never had to wait more than a few hours for an answer to a support request.  I highly recommend them for other conferences.
5. We introduced conditional acceptances.  We got some pushback that this is a big burden for authors, but I don’t buy that—obviously, the authors could choose to treat this as a rejection and skip the hassle, but in fact many of our conditional accepts were worked on and turned into accepts—including one of the four papers we ultimately highlighted as a best paper.  It is certainly a big burden for the program committee, and I’d like to thank our member for being willing to tackle this extra responsibility.

Finally, we made some changes to the conference itself.

1. We scheduled our parallel tracks based on surveys of attendees about which talks they wished to attend. It was trivial to gather this information using a google spreadsheet.   Using it, we created a schedule with almost no conflicts among the papers people stated plans to attend.
2. We introduced a category of “general interest” papers.  These were specifically not intended to be “best” papers; rather, they were papers we felt ought to be attended to by anyone interested in achieving a broad sense of the work going on around the Semantic Web.   The Semantic Web community is made up of subcommunities that risk becoming quite isolated from one another—the description logic community, the Semantic Web user interface community, the scalability community, the standards community, and so on.   To combat isolation we need to share knowledge, so we identified papers that we felt were representative of each subcommunity but also broad enough that members of the other subcommunities could understand them.
3. We introduced a town-hall meeting so the community could give us feedback about all the innovations I’ve just described.  The town hall was reasonably well attended even without any food or beer incentive.  With the exception of reviewer bidding, all of our changes were favored by a majority of the town hall attendees.

There were three changes we completely failed to pull off:

1. ISWC offers “in use” and “industry” tracks with their own submissions, committees, and schedules independent of the research track.  This creates ambiuity for authors about where they should submit and ambiguity about which track should accept, and also leads to tracks that are not coordinated at the conference.  I would like to see a single committee considering all this work as a whole.
2. We had hoped to install Semantc-Web tools for managing information at the conference—letting people annotate the papers/talks they saw, provide information about restaurants they’ve attended, coordinate meetings with others at the conference.   We failed—creating a system for this still requires too much special purpose engineering.    I see that as a (to-date) failure of the Semantic Web community to provide tools that are easy to install and use.
3. Both Avi and I were eager to nurture the study of user interaction with the Semantic Web.  We both consider this a topic that is not being sufficiently explored—without good user interfaces, we don’t think the promise of the Semantic Web can be fulfilled. Unfortunately, our plan was frustrated by a lack of submissions in this area—it seems the work simply isn’t being done.   I wonder how we can convince the community of the importance of pursuing it.

In all, chairing the conference was surprisingly fun and easy.  For years, I’ve declined chairing requests, on the theory that with my organizational capabilities any conference I chaired would not actually take place.   For ISWC, my worries were assuaged by a fantastic and well organized (Swiss) co-chair who made sure everything actually happened on time.  So, a big thank you to Avi.

Best paper went to Ugur Kuter and Jennifer Golbeck for “Semantic Web Service Composition in Social Environments”.   This paper reflected a nice linking of two disparate fields.  The first is semantic web service composition.   This area looks ahead to when users will create workflows by composing—chaining together—a series of “web services” (processes on the web with exposed APIs that can be invoked from elsewhere on the web), passing data from one to the next in order to perform some computation.  Generally, web service composition is looked at as a logic problem—figuring out which web services can be composed to meet a particular specification.  Kutur and Goldbeck instead look at a trust problem.   Many of these services on the web might not be completely trustworthy—perhaps they are fault-prone or even malicious.   The paper framed the problem of choosing, among many compositions, the most trustworthy one.  The algorithms in the paper are heuristic and surely not the last word, but real value of the paper was in framing the problem.

The best student paper, by Vicky Papavassiliou, Giorgos Flouris, Irini Fundulaki, Dimitris Kotzinos, and Vassilis Christophides, “On Detecting High-Level Changes in RDF/S KBs”,  was also a problem-framing paper.   Ontologies are obviously an important part of the Semantic Web (my last post notwithstanding) and are already heavily used to characterize data in a variety of domains (medical being a significant one).    Over time, these ontologies will evolve.   People will need to make sense of the changes to these ontologies.  It’s easy to describe “atomic” changes (this element was added, this removed) but these atomic changes are likely to occur in groups to achieve some higher level change.  It is these higher level changes that will serve as the most meaningful description to an end user.   This paper poses the two related questions of “how should these higher level changes be described?” and “how can they be detected from the raw description of atomic changes to the ontology?” Again, there is likely to be followon work, but this paper did a nice job initiating study of the problem.

We also awarded an honorable mention in each category.  For best paper, honorable mention went to Daniela Petrelli, Suvodeep Mazumdar, Aba-Sah Dadzie, Fabio Ciravegna for “Multi Visualization and Dynamic Query for Effective Exploration of Semantic Data.” Working at Rolls-Royce, they mapped a large amount of information (about jet engine design) into a Semantic Web framework and deployed a new user interface for visualizing and navigating that information.  They then studied its use and usefulness in the company and reported conclusions.   The paper was a real user study; something of whichwe see far too little in the semantic web community.  It was particularly nice to give this award after hearing a researcher declare the previous day that “there is no scientific method for evaluating semantic web user interfaces.”  The community needs more papers of this sort.

An honorable mention went to Jacopo Urbani, Spyros Kotoulas, Eyal Oren, and Frank van Harmelen for ” Scalable Distributed Reasoning using MapReduce.”  This was a performance paper, showing how to do typical semantic-web logical inference tasks on a large parallel cluster.  As is usual, the main problem is data communication bottlenecks between the processors, and this paper showed how a very limited amount of replication could dramatically reduce those communication bottlenecks.

When I attended the panel I was sorry to discover that the panelist did in fact all agree.  Far worse, they all said “yes” and wanted to debate what kind of ontologies were needed. Those who’ve followed my slow conversation with Stefano Mazzocchi won’t be surprised at my reaction—ajump to the audience microphone to voice a strong “no!”   I asserted that a bunch of data presented in spreadsheets was already a big step forward over our current unstructured web.  This led to some interesting discussion that helped me clarify some points in my mind that I’ll try to lay out here.

The panelists’ general reaction was amazement that I could be opposed to ontologies.  Without ontologies, how could any tool actually use the data?  What good would that data be without an explanation of what it meant?

Tim Berners Lee tried to mediate by suggesting that I did support ontologies.  After all, a spreadsheet has an ontology: the ontology specifies rows, columns, cells, and the relationship between them. But by this definition, any structured data necessarily has an (implicit) ontology, and saying “ontology” is just another way of saying “structured data”.   And I think this diverges from the standard meaning of “ontology” in the Semantic Web community, which I would read as “an explicitly recorded, machine readable description of the ontology of the given data.”   While I am a big proponent of structured data I’m going to bet that the panelists would not consider their implicit ontologies to be ontologies in the Semantic Web sense.   So we do in fact disagree.

Why then do I think we don’t need (explicit) ontologies?   Because I’m focused on the ways that human beings, rather than machine agents, will consume the data being shared.  And for humans, a machine-readable explanation of the data’s meaning is often unnecessary because the human who is consuming that data can figure it out in other ways.  For example, the meaning of the data elements might be explained in English, a “caption” of the data I am inspecting.     Even without captions, if I get a data table with column headings, I can use my comprehension of English to understand the meaning of those headings and from it infer the roles of the columns.  Even if there aren’t column headings, the “shape” of the data can tell me a lot—I’ll recognize standard person names, phone numbers, addresses, prices, book titles, and such from the textual patterns or from matches to my large wetware database of known entities.  And if I see enough examples I can draw conclusions about the values in the column (indeed, Google Squared suggests that you might not even need a human in the loop to make these inferences).

So humans can understand data without (explicit) ontologies, but is it any use?  Sure!  Just to plug some of my own group’s tools, they can use Exhibit to throw it into a rich visualization—a map, timeline, or list with faceted browsing and sorting.   Or they can combine it with another data set using Potluck, and throw the combined data into an Exhibit visualization.  I can make a post on ManyEyes or throw the data into DabbleDB for further processing.  These activities typically require me to match certain properties (columns) of the data set into roles in the UI (Exhibit, ManyEyes) or to properties in the other data set (Potluck, DabbleDB)—a straightforward task.  They don’t require the machine to understand the data, because I’m the one taking these actions.  They do require that the data be structured, since otherwise there’s no way for me to say “which column” to the tools I’m trying to use.

That’s the argument I wanted to make at the panel, but it’s a bit hard to squeeze into 20 seconds at the audience-feedback microphone.  So I’m afraid the panelists instead thought that I was arguing against ontologies, asserting that they should not be deployed at all.

On the  contrary, I like ontologies.  But I’m convinced that ontologies are a luxury, not a necessity. They’re certainly nice to have, and there are some things you can only do if you have them–for example, theycan help me understand column headings written in Russian or Spanish by connecting them to explanations in English.  But I remain captivated all the opportunities that arise just by making data easily accessible in raw form.   Too often, what people want to do with information is perfectly easy to explain, but impossible to do without serious programming, for silly reasons.

And it’s that enthusiasm for open data that keeps me energetically arguing that we don’t need ontologies.  If we need ontologies, then work on freeing data needs to stop until we get them.  I think that’s a very dangerous perspective.  It’s the one that says “there’s no point to building tools for scientists to publish their data, until we’ve figured out the right huge ontology that we’ll force them all to publish in.”

Instead, I think we should go right ahead with our research on ontologies and tools for them, but in the meantime, let the data fly!

P.S. When someone rose to support me, arguing that we should forget ontologies and concentrate on Linked Open Data, I mudied things further by asserting that we don’t really need the “Linked” part, and Open Data is useful in its own right.  While it comes from the same place as my perspective on ontologies above, that’s the substance of my discussion with Stefano, and I won’t repeat it here.

So here, from the perspective of a human-computer interaction guy, is what I’d like to see changed about the semantic web:

Stop Calling Everything ‘Semantic’

At worst, the term ’semantic’ in a title can mean “we re-did existing research using {RDF, RDFa, N3, OWL, SPARQL, DBpedia, Semantic MediaWiki},” without a clear notion of why this would be a good thing. The strength of the semantic web is its ability to interoperate heterogeneous data, yet the inclination is to ignore this and work on the problem, any problem, in a semantic web framework. Semantic web research papers can feel like a bunch of hammers running around in search of a nail. And there are plenty of oft-hammered nails: semantic query visualizations, semantic desktops, semantic wikis, semantic ontology alignment, and semantic web service composition, to name a few. Why do these benefit from being semantic, any moreso than taking another approach?

At best, the term is still very unclear about what it implies. ‘Semantic’ should mean more than a language or a framework. It is an idea, and the idea should drive the research. Saying that something is semantic should imply something as clearly as saying that it is a proof by reduction, or a tangible user interface, or a static code analysis technique.

Who’s the User? (And Why Would They Ever Use This?)

That’s not a jab. It composes two specific critiques:

To solve important problems, you need to know who your users are. What are their problems? What biases and constraints do they bring to your system? This is true whether you’re composing web services or creating a Linux desktop. Semantic web technology’s greatest strength and greatest weakness is that it is very general. Too many projects focus on trying to help everybody; but too often, “everybody” is too vague to give you a good foothold, and it trends toward “semantic web-interested people”. This leads to many issues with the research, not the least of them is the Big Fat Graph solution to every semantic web problem and the requirement that I manually author RDF triples. When you’re defining the problem, define the set of users! “Everybody” is too vague; start with some personas or scenarios, or build systems that aim at some subset of the world. This will give you the insights necessary to generalize back out to “everybody”.

Second, there are some serious questions about user motivation. The semantic web suffers from a real cold start problem — how to get all that data into linked format.  Again, no single motivator will work for everybody, so the resulting motivators are so general, or so tied to implicit semantic web assumptions, that few get off the ground. Nobody wants to sit and re-encode their data into semantic web format.  But given a real problem, and the promise of a solution that just so happens to involve RDF, it will happen.

This is why I think Semantic Web UIs is something of a misnomer. It’s like “Java Swing UIs” or “UIs based on a relational database backend and a PHP frontend”. The critical irony of a good semantic web UI is that there should be no indication that it’s semantic. You could do this using a standard database and data model, but it’s easier because it uses semantic web technologies. Again, the interface should flow from the problems, not from the data model (flexible as it is).

I’d love to hear a semantic web researcher’s critique of human-computer interaction. Or your thoughts on my thoughts…

1. Data Attributes, are key-value pairs that may be added to any DOM node
2. Microdata provides a way to interweave objects and object-properties amidst the DOM
3. RDFa provides a way to interweave RDF amidst the DOM
4. Client-side Database Support provides a full relational data access from JavaScript (the spec says this will be SQL compliant, but in reality it will likely just be the SQLite subset of SQL).

These are all great developments, and will no doubt bring about a lot of creativity about how data can be used on the client-side, but what interests me the most is why the HTML5 working group felt the need to include Microdata alongside RDFa.

The capabilities of HTML5 Microdata and RDFa are nearly identical, albeit with slightly different terminology. Both provide a way to embed data within HTML attributes and tag contents. Both allow for both named entities and blank nodes. And both allow for a variety of more complex constructions, such as lists and HREF property values. One of the only real differences, as I can tell from glancing over the specs, is that RDFa requires URIs whereas Microdata simply uses ordinary strings to reference entities and properties. And that is what worries me: one of the biggest benefits of RDF is its use of URIs, yet URIs seem to be exactly what is preventing the adoption of RDF.

One problem is probably that URIs look funny as data model elements, even to a programmer. “A person has name” is much more natural sounding than “A http://csail.mit.edu/Contact#Person has a http://csail.mit.edu/Contact#name”. We think of our code in natural language terms, and URIs obfuscate our real world metaphors.

Far more serious a problem is the namespace traffic jam that currently exists. If I want to publish an RDF document that describes this blog, for example, best practice would have me draw class types and property types from no less than six ontologies!

• The RDF ontology to describe object properties
• The RDFS ontology to describe object classes and labels
• The Dublin Core (DC) ontology to describe the titles, authors, and the like
• The Friend of a Friend (FOAF) ontology to describe my contact information
• The XSD ontology to describe literal dates, strings, and numbers
• And yet another, custom, ontology to describe everything else particular to the blog

That is already 6 ontologies, and we haven’t even raised the possibility of using OWL Time, Snap, Span, and GeoOWL for things like time and space description! Even for a semantic web developer, the complexity of managing all of these ontologies, and the namespaces that go with them, becomes pretty burdensome pretty quickly.

And that is why I worry about the future of RDFa in HTML5. It appears that the Microdata specification in HTML5 is essentially the RDF graph data model with the URIs neutered out. Given essentially the same data model, no doubt most developers will pick the easier of two formats to implement.

In order to get more people on the RDF bandwagon, we need to make the RDF path just as easy to follow as the Microdata one. How can this be done? If you ask me, the best way is to get rid of this namespace traffic jam and cultivate a set of community-oriented ontologies.

Rather than trying to create base ontologies that address abstract universal concepts, why not try to have each community standardize a single ontology for their particular domain. Have WordPress and Blogger sponsor the Blog Ontology. Have Amazon.com and eBay sponsor the Marketplace Ontology. Have Facebook and MySpace sponsor the Social Ontology. Then, instead of reusing bits from other ontologies, such as dc:creator or foaf:name, have each of these community-focused ontologies be self-sufficient, covering all the concepts necessary for their domain. We can always apply mapping rules to distinguish between social:name and store:book-author-name later. With only a single ontology per domain area to worry about, the namespace traffic jam will disappear and it will be easier for people to get on board with RDF and RDFa.

All in all, it seems the good news coming out of the HTML5 spec is that we can expect rich data annotation to soon be arriving to HTML content everywhere. But what we need to work on as a community is a way to make URIs, and the Ontologies that give them meaning, easier for programmers to use so that the web won’t just be full of data with Microdata, but full of linked data with RDFa.

I argued in favor of letting individuals make their own RDF collections (using, for example, our Exhibit framework) and worry about merging them with other people’s data later.  Stefano’s response accused me of using “RDF” and “structured data” interchangeably, asserting Exhibit is really just a nice UI over spreadsheet (tabular) data—that although it can export RDF, it is “not properly using RDF” because it has “lost the notion of globally unique identifiers (and in that regard, is much more similar to Excel than to Tabulator)”.  Tim Berners Lee has made similar complaints to me about Exhibit not using RDF.

This argument highlights for me yet an important ambiguity about what RDF is.   I occasionally have to help people understand that RDF is a model, not a syntax.  That some data can be RDF even if it isn’t serialized to RDF/XML.  That the key is to have items named by URIs, connected by relations named by URIs.  Stefano’s argument suggests a different blurring: between the model and its intended use.  Stefano’s “not properly using” phrase implies that if you don’t intend to merge your data into the global namespace, then even if you implement the model  and wrote it down as RDF/XML to boot, you won’t be “properly using RDF”.

I want to address both these claims: that Exhibit is just a UI over spreadsheets, and that using RDF this way isn’t proper.

RDF and spreadsheets

Regarding the spreadsheet claim, I’ll begin by admitting that Stefano is absolutely right:  Exhibit is a visualization tool for tabular (spreadsheet) data.  But notice that all RDF is spreadsheet data—I can take all the RDF in the world and throw it into one spreadsheet.  In fact, I only need three columns to contain the subject (tail), object (head), and predicate (link) for each RDF statement.  Admittedly none of today’s spreadsheets would have enough rows, but that’s an engineering detail.  So, the spreadsheet model isn’t the problem.   And we also agree that Exhibit’s interface is nothing like spreadsheets’, and far better for the collection visualization tasks it is designed for.

I think instead that what Stefano is objecting to is a usage characteristic of spreadsheets versus RDF.  When I open a spreadsheet, the data it shows me is right there, in a file on my own system.  Global identifiers don’t matter because the data is all there (and presumably self-consistent) in the one spreadsheet.   In contrast, in Stefano’s image of RDF (and in Tim’s, as one can see from the Tabulator project) the data about a particular entity is spread all over the web, and it is the globally unique identifier that lets you go out, gather all that data together, and know that it is all about the same entity.

This is certainly an appealing vision.  But I want to argue that a focus on globally unique identifiers neglects two benefits of RDF that I consider equally important: data portability and schema flexibility.

Spreadsheets suffice

To illustrate this argument, I’ll hark back to a previous post where I discussed a data integration problem that should have been easy but wasn’t.   I keep an  Exhibit of folk dance videos on the web.   Recently, Nissim Ben Ami posted a collection of 511 new dance videos on Youtube.  I wanted to incorporate it into my site.  But it quickly became apparent that said incorporation would basically require my entering all 511 video descriptions manually into my system, and I still haven’t gotten around to it.

The major barriers were twofold.  The first was syntactic:, the structured descriptions of the videos were delivered as XML.   That meant that in order to get at the data, I was going to have to learn XSLT—something I’ve been putting off for years.   The second hurdle is semantic: Youtube has the wrong schema for my folkdance videos.  I care about choreographer, dance type, and year choreographed; YouTube only offers slots for submitter and submission date of the video.  So, as you can see from this example, the contributor takes the usual approach: he takes his nice structure data and shoves it into the generic comment (info) field as free text.  All that structure is instantly lost.

Suppose instead that spreadsheets (or, in a pinch, RDF) were the accepted framework for publishing information on the web.  The YouTube “spreadsheet” would contain submitter and submission date information, but Nissim could just add “artist” and “composition-date” columns to hold the data he wanted to enter.   I would then be in a great position to download his data and incorporate it into my own catalog (spreadsheet).  What would I have to do?  After opening his spreadsheet and mine, I’d have to match columns—perhaps he called his “artist” and “composition date” while mine are “choreographer” and “year”.  But a simple copy and paste fixes that discrepancy.  Merging entities is not much harder than merging properties: a simple global replace will convert his choreographer “Israel Ya’akovi” to my “Israel Yakovee”.  The local consistency of his data and mine means that I only have to work once per choreographer (and in most cases I won’t have to: there’s a standard spelling for almost every choreographer’s name, which serves as a unique identifier in this context even if it isn’t a URL).

Overall, my work has reduced by order of magnitude.  Instead of laboriously entering 511 new records, I just download a spreadsheet and match up a handful of properties (columns) and a few tens of choreographer names.

Stepping back, observe that I’ve relied on two things.   First, on data portability—my being able to download the data in a convenient form: not XML, which is a programmer’s friend but an end-user’s enemy; rather, something I can just look at and understand.  Second, on schema flexibility—on Nissim’s being able to add whatever columns/properties he decides are important, instead of being limited to those used on the hosting web application.

I’m also relying on some features of this particular scenario, but I believe they often hold.   I am relying on Nissim’s data having only a small number of properties so that I can map them manually to mine.   I also rely on there being a small number of choreographers, and hope to take advantage of most of them having matching names in his data and mine—these names certainly aren’t globally unique identifers, but they are “unique enough” when considering just my data and his.  Critically, I am not thinking of pulling all data about a given dance from a multitude of different web sites—this would demand global unique identifiers to link data since I would never have the patience.  Rather, I am considering a pairwise data acquistion: taking data I want from one internally consistent site.

Such pairwise acquisition is commonplace: any time a scientists wants to pull a data set from some other scientist’s lab, or a consumer wants to download product information about several cameras from a review site, or a student wants to include a Wikipedia data set in a report they are writing, there is an obvious single source and target for a data merger.   And there’s a human being who has the incentive, and with the right tools the capability, to do the limited amount of work needed to accomplish that merger.

This is a simple low-hanging fruit argument.  It would be wonderful to be able to automatically merge data from thousands of different sources into a coherent whole.  And this is a problem Freebase will need to solve, if they want to become the hub for aggregation of structured data.  But right now we can’t even manually merge data from two sites without doing a ridiculous amount of grunt work—so perhaps we should give some attention to that easier problem on our way to solving the hard one.

Don’t skip the wild west

I’d like to so these efforts proceed in parallel, but I’m worried about enthusiasm for the more ambitious goal blocking movement toward the low-hanging fruit.  I recently submitted a proposal to NIH on the topic of data integration that reflected my perspective above.  I argued that the current efforts in the Biology community to force everyone to adopt a common ontology (and sometimes repository) for their experimental data are being resisted by biologists who think they know best how to present their data.  I suggested as an alternative that we give biologists tools, such as Exhibit, that would encourage them to publish their data in a common structured syntax, and worry about integrating all that data after it has become available in structured form.  The proposal rejection was accompanied by a review that said, on the one hand, “The benefit of the proposed approach is that it is very different from some multi-institutional data sharing projects (like caBIG), which have used a very rigid, top-down approach to creating semantics. Even if this project is unsuccessful it could bring to light new ideas and strategies that might make those large-scale projects more responsive to investigators and more successful.”  At the same time, it argued for rejection because “The absence of any control over the information models and ontologies – truly a semantic wild west – is daring and may ultimately be the downfall of this project.”

I’m fascinated to see, in the same review, a recognition of the problems that the current centralized approach is bringing (lack of buy-in to common ontologies by individual scientists who think they know better and probably do), and an unwillingness to tolerate the contrary (anarchic) solution.  I also love the metaphor of the “semantic wild west” because I think it supports my argument.  Would anyone have suggested establishing a city of several million people just after the west was opened for settlement?  The west’s early wildness was an unavoidable phase of its evolution towards the thickly settled and uniformly governed area it is now.    In the same vein, I think that our semantic web is best grown by encouraging individual semantic-web settlers to create their own data homesteads and begin looking for the trails that connect them to neighboring collections.  We need to get the data into plain view first.   Later we can send in the data sheriffs and place all those data sets under uniform governance.

There are now hundreds of ontologies with tens of thousand of terms.  However, it has always been a “cottage industry”—various groups develop their own ontologies, then publish them for use by others.  Is there a way to open the development of the ontology up to the community?  Community might be just a few or thousands.

As an example, the gene ontology (28K terms) has 3 full time curators.  People from the community submit to an issue tracker to get new terms etc.  A ne version is released daily.  In contrast, the NCI thesaurus (for cancer) has 20 full time editors with 1 lead editor who runs everything, and a slow cycle of “releases” with less community input.  Others work like typical open source projects with 20-30 team members involved in active discussions.

Natasha’s group builds on Protege, a very old open source ontology editor that is now one of the most popular, with 120,000 registered users.  It has a very open plugin architecture with dozens of plugins for visualization, import, export, nlp, and lots of unknowns.  They’ve been working to augment protege with support for collaboration.  It works in a distributed fashion (desktop and web clients).  It support simultaneous editing, but also annotation, discussion, proposals and voting in the context of the ontology.  There are many types of annotations—questions, comments, proposals—on any elements of the ontology—classes, properties, instances.   While the tool handles most types of structured data, it is focused on taxonomic hierarchies were stuff gets inherited down the hierarchy.

They investigated use of their tool for several tasks.  One is ontology evaluation—finding existing ontologies that might be useful for you.  This source of information for this is author-contributed metadata about the ontologies—domain, key classes and concepts, who the developer is, etc. Another is automatic tools that compute quality metrics, and another is annotations by other users of the ontologies.

This last is important because some ontology metrics are subjective—a feature that is “good” in one setting can be awful in another.  An example might be a high level of axiomatization.  This is important for inference, but creates clutter if you just want description.  There’s also the problem of crosscutting taxonomies—you might have two different ways of describing the same domain that form a “matrix” of non-overlapping hierarchies.  To address this sort of subjectivity, they allow users to record evaluations of ontologies.

These tools can be explored at their bioportal web site where they have a large library of biomedical ontologies.  On that site, users can describe their ontology based projects, and list/review the ontologies they are using.  Reviewers give general reviews, usage information problems encountered, coverage of the key terms, major gaps, and issues with specific elements of the ontology.  This site aims to make ontology evaluation/creation a truly democratic process.  This is controversial—some argue that ontologies need a more rigorous editorial process (mirroring a current debate about open vs. traditional journal publication).

Another big task is mapping: connecting two ontologies by asserting that terms in two different ontologies “match”.  They aren’t trying to find mappings, but want to enable others to upload the mappings they have found.  Mappings can be created manually or uploaded in bulk (if computed by someone’s tool).  Mappings are themselves metadata, which can be annotated and discussed just like other data in the ontology.

Of course a big question is whether people will use these tools.  Right now, many users are asking for these features and reporting lots of bugs—good signs of demand.

A lot of questions have now arisen that need some serious user studies—what are the dynamics of the social networks that form around collaborative ontologies?  What are the different types of users/editors?  What produces the most discussion/controversy?  Do these tools help or hinder collaboration?

Being argumentative by nature, I wanted to highlight some of the benefits of the looser, sloppier approach to data sharing that we took for SIMILE.   Obviously, being able to link data from multiple sources, and feed it into a search engine as Stefano describes, is a great thing.  But there are some tremendous advantages that accrue when even a single individual decides to create a blob of structured data with no reference to anyone else’s.

The first is interaction.  As shown with our Exhibit framework (created by David Huynh, now also at Metaweb), structured data enables rich visualization.  If my data objects have coordinates, I can plot them on a map.  If they have dates, I can put them on a timeline.  If they have colors, I can filter or sort by color.  It doesn’t matter if I call those properties latitude, longitude, date and color, or northSouth, eastWest, sinceTheCreation and elementOfTheRainbow, and whether I decide that my city is Chicago or the Windy City—as long as I have my own internally consistent names for them, I can use them to hook my data into interesting visualizations and interactions.

The second benefit is portability.  If I publish some interesting data as part of an HTML document, then anyone who wants to use that data for something else—to rebut my argument, to mash it up with some other data, to put it some use I never thought of—has the unpleasant job of scraping said data out of the HTML into a usable form.  This generally requires a programmer, and even for them it’s a tedious task that distracts them, and may deter them, from what they really want to do with the data.  But if that data is published as data—even in something old fashioned as a spreadsheet—it becomes way easier to grab it and reuse it.  Look at how much of the blogosphere is made up of cross-references, trackbacks, and responses to other blog postings.  If you’re going to argue about something involving data—for example, whether a single payer system is going to end up saving or costing money, or whether today’s perfect game is all that unusual—you probably want to publish that data to support your argument.  At which point, someone who wants to refute your argument is going to want to use that same data.  That’s going to be a lot easier if they can get that data from your posting.  That’s the theory behind our Datapress project, which aims to let you post data sets (and visualizations of them) in your Wordpress blog, and lets other people refer to and reuse that data.  In that sort of one-on-one debate over data, it really doesn’t matter whether I use the same identifiers as Freebase—you can take my identifiers and use them to build your rebuttal.

Uniformity does start to matter when someone wants to mash up data from multiple sources.  If those sources haven’t agreed on identifiers beforehand, then the masher has some work ahead—this is a case where a centralized vocabulary is really helpful.  But again, getting the data at all is such a big jump over the current state of affairs—I imagine how grateful mashup makers would be if all they had to do was merge some identifiers instead of retyping a whole spreadsheet from scratch.  The point here is that unlike Stefano’s hypothetical search engine, that wants to issue a query against all the world’s data at once, your typical mashup author just needs to deal with a couple of (probably small) data sets.  His or her particular data integration problem is quite manageable a posteriori.

I’ll also dust off an argument David Huynh once made to me, even if it might get him in trouble with his current employer.   Unification is not an absolute, but contextual.  Whether two things are the same may change depending on what you are doing with them.   Continuing my never-before attempted forays into sports analogies, are the Brooklyn Dodgers the same as the L.A. Dodgers?  If you want to talk about the team that moved from Brooklyn to LA, the answer must be yes!  But in a different context you might be interested in comparing the lifetime records of these two distinct teams.  (In fact, Freebase tries to have it both ways: it asserts that the Brooklyn Dodgers were “later known as” the Los Angeles Dodgers (implying they are the same team with a name change) but asserts that Los Angeles Dodgers were founded in 1958, which clearly isn’t true of the Brooklyn Dodgers that folded in 57.)

This is obviously one of those half-empty half-full debates:  We both recognize the value of both approaches, but are compelled by different aspects.  Stefano looks at the amazing things that could be done with a single consistent data universe, and worries about how to create it.  I look at the amazing things that can already be done with a host of disjoint but internally-consistent data microverses, and find that compelling enough to allay any worry about whether we’ll ever need http://www.freebase.com/view/en/lion to unify with http://en.wikipedia.org/wiki/Lamb .