One set of changes we made was aimed at squeezing the gap between submission and conference.  It seems obvious to me that conferences should inform us of new work.   This doesn’t happen if the conference presents work that was submitted 7 months earlier (cf CHI)!  We made a few changes to tighten things up

1. We compressed the initial reviewing period to two weeks.  While many conferences give reviewers several months to examine their papers, I’m certain that few people look at them more than a week ahead of the deadline.  Certainly most reviews don’t arrive until day-of!  We got exactly one reviewer  rushecomplaint abodut this schedule.   Perhaps we’ll lose some unusually disciplined and organized reviewers who like to plan ahead; I’m still convinced this is a big improvement for the community.  For the same reason, we offered the program committee only a week to digest and summarize these reviews before the committee meeting.
2. Similarly, we gave authors only a week to revise their papers for publication after acceptance.   Again, we assumed most revision happens at the last minute, so we jumped straight to the last minute.  I’d say that a paper requiring more than a week of revision probably wasn’t ready to submit anyway.
3. We advocated moving to a model, like NIPS, of producing the printed proceedings after the conference.   Given the slow printing process, distributing printed proceedings at the conference requires that final versions be submitted long before the conference.  I haven’t opened a paper proceedings for years, but enough people are still attached to them that I don’t think the time is ripe to do away with them entirely.  But distributing them post-conference lets us keep them without keeping the delay they introduce.    In the end, we weren’t able to convince the steering committee to make this change; however, our push led the publisher to offer an unusually swift timeline for publication.

At the end, we squeezed out enough delay that the critical path became something outside our control: travel to the US.  For cheap fares, authors need to buy their airlines tickets a month before the conference.  On top of this we had to add a month for non-US authors to get visas for travel to the US (sadly, one of next year’s co-chairs could not get a visa in time, and missed the conference).  Combining these requirements with our one month review process produced our final, 3-month lag between submission and conference.

Looking ahead, I’d advocate for ISWC to eliminate the paper proceedings and to select countries with more sensible visa frameworks—Canada anyone?  This ought to let us cut the submission lag by another factor of 2.

We also made changes to the reviewing process.

1. We eliminated topic tracks, instead allowing authors to choose their preferred program committee member for review of their paper (as well as an alternate).  My experience has been that most track descriptions tend to be ambiguous gobbledygook that produce tremendous ambiguity about the “right” track to submit—especially if, as often seems to be the case for me, your work seems to span multiple tracks.  Given that decisions are made by the people on the program committee rather than abstract tracks, an “end-to-end” argument says picking the person makes more sense.  Authors understand their papers well and can study committee members to identify the one most likely to favorably receive their paper.   This is analogous to the submission process for journals where one chooses an editor.   There was some worry about load balance—what if everyone picks the “nice” reviewer?   But in practice we were able to assign every paper to one of the two chosen committee members.
2. We eliminated the paper “bidding” process.  In past ISWCs, all potential reviewers were able to survey all submissions and bid for those they wanted to review.   We felt that it made more sense for our knowledgeable program committee members to select reviewers who they thought were best suited to each paper—aiming for a “best for the paper”, rather than “fun for the reviewer” assignment of papers.  We feel this worked well, guaranteeing good reviews of each paper.  Interestingly, this was the one change on which we got negative pushback at the ISWC town hall meeting—even though it was populated by the beneficiaries of our reviewing process, the majority advocated a return to reviewer bidding.  I still think our approach is superior.
3. By a second end-to-end argument, we eliminated generic 1-5 scoring of papers,which ultimately must be translated into an accept/reject, and replaced it with actionable descriptions: we asked each reviewer to take a position for or against acceptance of each paper, or to “give up” and indicate they didn’t care.
4. Perhaps most important, we sought controversial papers.  The obvious and common measure of  paper quality is the average of the reviewer scores.  However, this buries one of the most important potential roles of a conference—to encourage debate about research.  Rather than a paper that everyone considers OK, a paper that half the reviewers love and half hate seems much more important to present at a conference, since it indicates a fundamental disagreement about research that is well worth airing in public, exploring, and trying to resolve.   To seek out such papers we applied two policies.  The first was that each paper with at least one vote for acceptance was discussed at the PC meeting.  The second was that any paper that, after discussion, had at least one program committee member advocate was accepted to the conference, regardless of the amount of opposition.  Because “average scoring” is so deeply embedded in reviewing practice, this change required changes to our conference management software—which were cheerfully carried out by the folks at precision conference, along with all the other great customer support they offered.  We never had to wait more than a few hours for an answer to a support request.  I highly recommend them for other conferences.
5. We introduced conditional acceptances.  We got some pushback that this is a big burden for authors, but I don’t buy that—obviously, the authors could choose to treat this as a rejection and skip the hassle, but in fact many of our conditional accepts were worked on and turned into accepts—including one of the four papers we ultimately highlighted as a best paper.  It is certainly a big burden for the program committee, and I’d like to thank our member for being willing to tackle this extra responsibility.

Finally, we made some changes to the conference itself.

1. We scheduled our parallel tracks based on surveys of attendees about which talks they wished to attend. It was trivial to gather this information using a google spreadsheet.   Using it, we created a schedule with almost no conflicts among the papers people stated plans to attend.
2. We introduced a category of “general interest” papers.  These were specifically not intended to be “best” papers; rather, they were papers we felt ought to be attended to by anyone interested in achieving a broad sense of the work going on around the Semantic Web.   The Semantic Web community is made up of subcommunities that risk becoming quite isolated from one another—the description logic community, the Semantic Web user interface community, the scalability community, the standards community, and so on.   To combat isolation we need to share knowledge, so we identified papers that we felt were representative of each subcommunity but also broad enough that members of the other subcommunities could understand them.
3. We introduced a town-hall meeting so the community could give us feedback about all the innovations I’ve just described.  The town hall was reasonably well attended even without any food or beer incentive.  With the exception of reviewer bidding, all of our changes were favored by a majority of the town hall attendees.

There were three changes we completely failed to pull off:

1. ISWC offers “in use” and “industry” tracks with their own submissions, committees, and schedules independent of the research track.  This creates ambiuity for authors about where they should submit and ambiguity about which track should accept, and also leads to tracks that are not coordinated at the conference.  I would like to see a single committee considering all this work as a whole.
2. We had hoped to install Semantc-Web tools for managing information at the conference—letting people annotate the papers/talks they saw, provide information about restaurants they’ve attended, coordinate meetings with others at the conference.   We failed—creating a system for this still requires too much special purpose engineering.    I see that as a (to-date) failure of the Semantic Web community to provide tools that are easy to install and use.
3. Both Avi and I were eager to nurture the study of user interaction with the Semantic Web.  We both consider this a topic that is not being sufficiently explored—without good user interfaces, we don’t think the promise of the Semantic Web can be fulfilled. Unfortunately, our plan was frustrated by a lack of submissions in this area—it seems the work simply isn’t being done.   I wonder how we can convince the community of the importance of pursuing it.

In all, chairing the conference was surprisingly fun and easy.  For years, I’ve declined chairing requests, on the theory that with my organizational capabilities any conference I chaired would not actually take place.   For ISWC, my worries were assuaged by a fantastic and well organized (Swiss) co-chair who made sure everything actually happened on time.  So, a big thank you to Avi.

Best paper went to Ugur Kuter and Jennifer Golbeck for “Semantic Web Service Composition in Social Environments”.   This paper reflected a nice linking of two disparate fields.  The first is semantic web service composition.   This area looks ahead to when users will create workflows by composing—chaining together—a series of “web services” (processes on the web with exposed APIs that can be invoked from elsewhere on the web), passing data from one to the next in order to perform some computation.  Generally, web service composition is looked at as a logic problem—figuring out which web services can be composed to meet a particular specification.  Kutur and Goldbeck instead look at a trust problem.   Many of these services on the web might not be completely trustworthy—perhaps they are fault-prone or even malicious.   The paper framed the problem of choosing, among many compositions, the most trustworthy one.  The algorithms in the paper are heuristic and surely not the last word, but real value of the paper was in framing the problem.

The best student paper, by Vicky Papavassiliou, Giorgos Flouris, Irini Fundulaki, Dimitris Kotzinos, and Vassilis Christophides, “On Detecting High-Level Changes in RDF/S KBs”,  was also a problem-framing paper.   Ontologies are obviously an important part of the Semantic Web (my last post notwithstanding) and are already heavily used to characterize data in a variety of domains (medical being a significant one).    Over time, these ontologies will evolve.   People will need to make sense of the changes to these ontologies.  It’s easy to describe “atomic” changes (this element was added, this removed) but these atomic changes are likely to occur in groups to achieve some higher level change.  It is these higher level changes that will serve as the most meaningful description to an end user.   This paper poses the two related questions of “how should these higher level changes be described?” and “how can they be detected from the raw description of atomic changes to the ontology?” Again, there is likely to be followon work, but this paper did a nice job initiating study of the problem.

We also awarded an honorable mention in each category.  For best paper, honorable mention went to Daniela Petrelli, Suvodeep Mazumdar, Aba-Sah Dadzie, Fabio Ciravegna for “Multi Visualization and Dynamic Query for Effective Exploration of Semantic Data.” Working at Rolls-Royce, they mapped a large amount of information (about jet engine design) into a Semantic Web framework and deployed a new user interface for visualizing and navigating that information.  They then studied its use and usefulness in the company and reported conclusions.   The paper was a real user study; something of whichwe see far too little in the semantic web community.  It was particularly nice to give this award after hearing a researcher declare the previous day that “there is no scientific method for evaluating semantic web user interfaces.”  The community needs more papers of this sort.

An honorable mention went to Jacopo Urbani, Spyros Kotoulas, Eyal Oren, and Frank van Harmelen for ” Scalable Distributed Reasoning using MapReduce.”  This was a performance paper, showing how to do typical semantic-web logical inference tasks on a large parallel cluster.  As is usual, the main problem is data communication bottlenecks between the processors, and this paper showed how a very limited amount of replication could dramatically reduce those communication bottlenecks.

When I attended the panel I was sorry to discover that the panelist did in fact all agree.  Far worse, they all said “yes” and wanted to debate what kind of ontologies were needed. Those who’ve followed my slow conversation with Stefano Mazzocchi won’t be surprised at my reaction—ajump to the audience microphone to voice a strong “no!”   I asserted that a bunch of data presented in spreadsheets was already a big step forward over our current unstructured web.  This led to some interesting discussion that helped me clarify some points in my mind that I’ll try to lay out here.

The panelists’ general reaction was amazement that I could be opposed to ontologies.  Without ontologies, how could any tool actually use the data?  What good would that data be without an explanation of what it meant?

Tim Berners Lee tried to mediate by suggesting that I did support ontologies.  After all, a spreadsheet has an ontology: the ontology specifies rows, columns, cells, and the relationship between them. But by this definition, any structured data necessarily has an (implicit) ontology, and saying “ontology” is just another way of saying “structured data”.   And I think this diverges from the standard meaning of “ontology” in the Semantic Web community, which I would read as “an explicitly recorded, machine readable description of the ontology of the given data.”   While I am a big proponent of structured data I’m going to bet that the panelists would not consider their implicit ontologies to be ontologies in the Semantic Web sense.   So we do in fact disagree.

Why then do I think we don’t need (explicit) ontologies?   Because I’m focused on the ways that human beings, rather than machine agents, will consume the data being shared.  And for humans, a machine-readable explanation of the data’s meaning is often unnecessary because the human who is consuming that data can figure it out in other ways.  For example, the meaning of the data elements might be explained in English, a “caption” of the data I am inspecting.     Even without captions, if I get a data table with column headings, I can use my comprehension of English to understand the meaning of those headings and from it infer the roles of the columns.  Even if there aren’t column headings, the “shape” of the data can tell me a lot—I’ll recognize standard person names, phone numbers, addresses, prices, book titles, and such from the textual patterns or from matches to my large wetware database of known entities.  And if I see enough examples I can draw conclusions about the values in the column (indeed, Google Squared suggests that you might not even need a human in the loop to make these inferences).

So humans can understand data without (explicit) ontologies, but is it any use?  Sure!  Just to plug some of my own group’s tools, they can use Exhibit to throw it into a rich visualization—a map, timeline, or list with faceted browsing and sorting.   Or they can combine it with another data set using Potluck, and throw the combined data into an Exhibit visualization.  I can make a post on ManyEyes or throw the data into DabbleDB for further processing.  These activities typically require me to match certain properties (columns) of the data set into roles in the UI (Exhibit, ManyEyes) or to properties in the other data set (Potluck, DabbleDB)—a straightforward task.  They don’t require the machine to understand the data, because I’m the one taking these actions.  They do require that the data be structured, since otherwise there’s no way for me to say “which column” to the tools I’m trying to use.

That’s the argument I wanted to make at the panel, but it’s a bit hard to squeeze into 20 seconds at the audience-feedback microphone.  So I’m afraid the panelists instead thought that I was arguing against ontologies, asserting that they should not be deployed at all.

On the  contrary, I like ontologies.  But I’m convinced that ontologies are a luxury, not a necessity. They’re certainly nice to have, and there are some things you can only do if you have them–for example, theycan help me understand column headings written in Russian or Spanish by connecting them to explanations in English.  But I remain captivated all the opportunities that arise just by making data easily accessible in raw form.   Too often, what people want to do with information is perfectly easy to explain, but impossible to do without serious programming, for silly reasons.

And it’s that enthusiasm for open data that keeps me energetically arguing that we don’t need ontologies.  If we need ontologies, then work on freeing data needs to stop until we get them.  I think that’s a very dangerous perspective.  It’s the one that says “there’s no point to building tools for scientists to publish their data, until we’ve figured out the right huge ontology that we’ll force them all to publish in.”

Instead, I think we should go right ahead with our research on ontologies and tools for them, but in the meantime, let the data fly!

P.S. When someone rose to support me, arguing that we should forget ontologies and concentrate on Linked Open Data, I mudied things further by asserting that we don’t really need the “Linked” part, and Open Data is useful in its own right.  While it comes from the same place as my perspective on ontologies above, that’s the substance of my discussion with Stefano, and I won’t repeat it here.

The sets of other pages consisted of the following conditions:

• “none” – Only the current page the user was viewing
• “interaction” – The 5 last pages visited prior to the page
• “task” – the set of pages resulting from a random walk from a page to the queries that lead to that page, and other pages returned by those same queries
• “collection” – pages linking to the current page
• “historic” – long term visit history of the user
• “social” – combination of long term visit histories of others who visit the same url

Their results revealed that the “best” sources of context depended on whether one wanted to predict the next immediate pages, or next set of pages over the long term.  Not surprisingly, the last five pages the user visited (”interaction” above) was found to most often predict the category of the next immediate page(s) visited.  “Task” contexts performed second best at these immediate predictions. And, the long-term history best predicted overall the pages the user visited in the future over a long term.

These results back up our intuitive notion that people tend to have a mixture of short term (task-oriented) interests, and long-term (low-frequency) interests reflecting the user’s perhaps, personal interests.

What I thought was somewhat misleading about the study was its title — I was initially drawn to the paper because I expected to see the paper address anticipating user information needs via activity/situational context based implicit information retrieval.  The study did not examine user interests at all — but merely looked at the categories assigned to pages subsequent to a particular visit.  Moreover, instead of examining the predictive power of particular document collections, these merely assessed the overlap in these categories.  I had expected instead the paper to examine the performance of various types of classifiers trained on subsets of page visits to predict subsequent pages — such an analysis would lead to greater insight surrounding the predictive power of various information sources towards future browsing than measuring simple category agreement.

Nonetheless, this is interesting, simple and important work, as it provides evidence surrounding sources of information that might be used in the future to anticipate users’ information needs.  This study also highlights the current situation in web-scale research: that only companies like Microsoft, Google or Yahoo! have access to the sheer volume of data needed to do such an analaysis.  Out of the three companies, Microsoft/MSR has been the most forthcoming with papers and involvement in conferences like SIGIR, WWW and SIGCHI, particularly over the last two years, with fascinating papers like the Large Scale Revisitation Patterns analysis by Adar, Teevan and Dumais et al. that won best paper at CHI ‘07.

One of our In the future, if academic research institutions like universities or the Web Science Research Initiative (WSRI) had access to such detailed usage data, there would likely be a significantly greater wealth of interesting insights about human behavior surrounding information access and retrieval.  The UMass Lemur Project and our own projects list.it and eyebrowse (forthcoming) hope to contribute to greater access to real-life search, web-browsing and note-taking behaviors for academic institutions through open-source tools that facilitate the capture of user behavior, and real user-donated corpora.

Of course, you can only draw limited confidence from this kind of simulation.  Their simulated users fit their model of the world (spammers labeled late) so of course a tool designed to their model will do well on their simulated users.  I wonder, would it have been that hard to just do manual labeling of expertise on some real delicious users?  This would obviously give more trustable results than simulations.   Indeed, they found that by manual examination, the top 50 users of the tag “mortgage” were spammers.  However, they say that the problem was finding a good ground truth for experts.   But that suggests it would still be possible to evaluate differentiation of spammers from non-spammers, even if you can’t evaluate differentiation of experts.

I asked about some past work on e.g. semisupervised learning suggests various approaches to combining small amounts of high-quality data (experts) with large amounts of messier data (mass user ratings).  It suggests, for example, some sort of weighted combination of expert and mass user opinion.  They know this could help a lot, but don’t have a general approach to separating the export from everyone else in a large mass of recommendations (Rotten Tomatoes did it for them).

All in all, a relatively inconclusive outcome. But this is an area that needs more attention—support for a complex process involving many related searches, management of the result sets, etc. Merry Morris has done some nice work here, with a focus on the collaborative aspects.

Followup: more coverage from Fxpal.

The problem with term suggestion is that it’s based on common terms at the top, but these may be the unwanted/distracting documents. How do you suggest ways deeper into the corpus? Also, there are problems with terms being presented out of context. There are also basic low level UI annoyances. With query suggestion, there’s the problem of finding a good corpus of queries and figuring which ones are related/similar to the user’s query. Kelly proposes using the automatic term selection techniques as a way to generate whole queries. First extract some terms, then suggest ways of combining them to make a good query. To generate terms, they cluster the documents, took the 5 largest, then selected “good” terms from each of the clusters. The considered offering these terms individually to users, but also just offering a “new query” consisting of old query with top terms appended to the query. They also considered user-generated suggestions.

Diane can always be trusted to carefully work out a good user study protocol so I won’t describe details of the corpora (TREC Robust track, with queries of different levels of difficulty) or user conditions or metrics (”Session-Based Normalized Discounted Cumulative Gain”). That’s all well done but details are in the paper.

As queries became more difficult, users made more queries, and also used more query suggestions. users saw term suggestions as a way to modify their query, but query suggestions as a way to make a whole new query. A but funny, as the query suggestions were in fact modifications of their original query. Qualitative feedback was that people liked the flexibility of term suggestion, and its use to refine the query. They didn’t like the jumbling together of terms, and said it was too much effort to use. People said it was hard to see how terms related to their search. People liked the query suggestions for its “all in one” approach. They liked the specificity and focus of the query—the query made a more meaningful semantic unit than the individual term suggestions. The liked that the queries suggested ways of manually changing their query. In cons, they wished they could click on individual query terms and felt many queries were redundant. In a followup study they found that people used lots of query suggestions. Query suggestions were generally rated higher than terms. Especially, those who got user-generated suggestions preferred the whole queries to having them chopped up into term suggestions; perhaps the user suggestions did not make a lot of sense in individual terms. They’d like to go back an develop a hybrid that lets people get the whole queries but then manipulate pieces of them.

I quite liked this talk but it made me think back to our old work on Scatter Gather. Just like Diane’s, our system clustered the document collection, then picked important terms from each cluster. But instead of presenting “queries”, we just presented each cluster—through its descriptive terms, and also through titles of some “representative documents”. Scatter/Gather offered less flexibility to the user to mix and match terms for a new query; on the other hand, I think there is some interesting difference between the “cluster” metaphor versus the “query” metaphor. I bet that presenting terms in clusters would fix some of the complaints about terms not making sense in isolation.

Followup: this talk was much tweeted and blogged about elsewhere:

• Fxpal

Update: I’ve fixed the download problem outlined in the comment.  Now clicking on the thumbnail should give you the full image. ]]> http://groups.csail.mit.edu/haystack/blog/2009/07/21/sigir09-search-engine-predeliction-towards-news-media-providers/feed/ 1 http://groups.csail.mit.edu/haystack/blog/2009/07/20/sigir09-enhancing-cluster-labeling-using-wikipedia/ http://groups.csail.mit.edu/haystack/blog/2009/07/20/sigir09-enhancing-cluster-labeling-using-wikipedia/#comments Mon, 20 Jul 2009 19:33:52 +0000 David Karger http://groups.csail.mit.edu/haystack/blog/?p=363 David Carmel from IBM Haifa spoke about the problem of labelling document clusters.  The goal is to find short labels for the clusters that describe them well to end users.  The typical approach seeks important terms in the clusters.  But sometimes important terms aren’t helpful/meaningful, and sometimes the best labels don’t show up in the cluster at all.  For example, at the Open Directory Project, its category labels appeared in the text of documents with that label clusters only 85% of the time, and were rarely among the statistically important terms.

In this work, they try to match the cluster contents to articles in wikipedia, then look at the wikipedia articles’ metadata (titles, categories) to find good descriptive labels for the clusters.  It seems to work pretty well.

to test this they took a bunch of text documents with labeled categories, and tested whether the manual label got selected by their wikipedia algorithm.  They tested on some standard corpora: the 20 newsgroups, and the open directory project for which they manually labeled 100 categories.  They carefully explored effects of cluster coherence, noise, etc.