Try out Habitbug!

Filed under: Social Computing by Katrina Panovich on Tuesday, 7 May 2013 | No Comments

One of the things that’s been interesting to us for a while now is how we can use our friends to help us get things done – friendsourcing. (See some example previous posts here, here, here, and here.)

We’re excited to launch habitbug, which is a Twitter app that helps you form and maintain habits by holding you accountable to all of your friends.

Since this is an experimental study, there are some slight differences for different users, but here’s the basic system:

  1. Pick a habit. This should be something you want to do (or avoid) every day. My current goal is to meditate, even for just a few minutes.
  2. Set a time that you want us to remind you. We’ll send you an @reply when it’s time.
  3. Check in every day that you do your habit. To make it easy, you can just reply to your reminder from @habitbug with any text you want.

With this system, we’re hoping to learn more about how we can use your friends as an accountability mechanism. I’d love to hear more about what you think! Try it out here! If you find any bugs or issues, please send us an email.

Converging Online Education and Online Journalism

Filed under: Uncategorized by David Karger on Monday, 17 December 2012 | 1 Comment

The Neiman Journalism Lab recently collected a number of opinions on interesting trends in online journalism.  You can read the whole set here, but for those too lazy to click, here’s my own contribution.

Massive open online courses (MOOCs) are widely believed to be revolutionizing education. But I think they also suggest some really interesting futures for journalism. In particular, I’m excited about the online discussion forums that accompany the MOOCs. These forums transform students from passive consumers of information into a community of inquiry who are actively engaged in asking questions and collaboratively working out answers. We need the same in journalism.

Too often, the forums hanging off news sites are troll-filled wastelands, where the best content one can hope for is a particularly well crafted putdown. In contrast, the MOOC forums exhibit high quality discussion where questions are asked, answers proposed and critiqued, and conclusions drawn in a style that supports and encourages other students. We’ve even seen the emergence of student leaders who are particularly adept at guiding others to find or construct needed information.

For most people who’ve finished school, journalism is probably the primary source of new information. What can we do to improve the news consumer’s “education”? Can the news “anchor” become the course “teacher”? With current events as the source material, what kind of MOOC in foreign affairs or government policy could be taught by a big-name journalist? Driven purely by interest in learning, thousands of MOOC students are doing “homework” to improve their knowledge, exercises that are graded by the computer and essays graded by peers in the class. What assignments could the journalist create to enhance a student’s understanding of a foreign country or a difficult budget or policy question? What would it be like if readers could submit peer-graded essay responses instead of grouchy complaints about biased media? Could this student-authored content actually start contributing to the news?

Journalism and education are siblings: if you’re informed but not educated, you have no context to interpret the information you’re getting; if you’re educated but not informed, you’re living in an ivory tower. In MOOCs I see the beginnings of a trend that might draw these two information-delivery mechanisms together in a powerful way.

Two Funny Things at the 2012 International Semantic Web Conference

Filed under: Uncategorized by David Karger on Monday, 26 November 2012 | 1 Comment

I spent last week at the 2012 International Semantic Web Conference.  This conference addresses the important topic of structured data on the web.  I had two “funny” experiences; one humorous and one peculiar.

At the beginning of the conference, I was amused to see that ISWC, whose central theme is linking the web’s data together into a coherent whole, had more trouble than any other conference I’d been to in picking a twitter hashtag for the event.  Most conferences just announce one at the beginning, but at ISWC it was left to emerge “organically”, which meant tweets were inconsistently tagged as #iswc, #iswc12, #iswc2012, or #iswcboston.  I tweeted a joke to this effect.  The responses that I got back were classic.  Reflecting the philosophy of the Semantic Web, various individuals argued that this was a good thing; that expecting everyone to agree on a single vocabulary was contrary to the Semantic Web vision of linking disparate ontologies.  Another pointed out that if only twitter were “doing things right”, treating their hash tags as ontological entities, and letting different people label each entity differently, then we wouldn’t have this problem.  These responses are completely logical but ignore reality.  We may know better than twitter how things ought to be, but in the meantime there’s an easy solution (that most other conferences have adopted) that works fine with the way twitter is now.

The more seriously funny experience was at the ISWC demo session.  The two demos that most impressed me were systems for  (i)  browsing upcoming events (concerts etc.) and (ii)  browsing academics and their publications.  Both of these systems were characterized by rich data models and nicely designed user interfaces that delivered valuable information and insights from their chosen domains.

The funny part is that neither of these applications should really be called a “Semantic Web application.”  Someone unaware of the Semantic Web, tasked with building these applications, would see a traditional data management and visualization problem that they would solve using traditional database tools (SQL) and web APIs.  The fact that these tools are storing their data in a triple store instead of a SQL database is irrelevant to the user experience.  And the fact that at least one of them is exposing a SPARQL endpoint for querying the data they are managing is good citizenship, helpful to the next project, but not important for this one.

This story fits what I argued in a talk at an ISWC workshop on programming the semantic web.  The original description of the semantic web envisioned applications that could wander through a linked world of hundreds of different ontologies, discovering/learning those new ontologies as it went and combining information from all of them to produce valuable answers.  It seems to me that the vast majority of applications don’t need this power.  Instead, these applications have fixed ontologies imposed by their creators at creation time.  They can therefore be created using traditional techniques.

This begs an obvious broader question: what kinds of work is Semantic Web research that should appear at ISWC.?  I ask this not in the interest of jealously guarding the “purity” of a discipline—I like breadth—but in the interest of directing research to the venue where it can be best disseminated and evaluated.  The semantic web technology stack is pretty mature at this point.  But that means that using a semantic web back-end doesn’t automatically turn your project into semantic-web research.   If I build a traditional interactive application on top of a triple store, my contribution is the application, and it should probably go to a conference like CHI that specializes in assessing human computer interaction.  A system that uses natural language processing or machine learning to recognize entities in text doesn’t suddenly become a semantic web contribution by outputting its results in RDF; instead it should be submitted to a venue like NAACL or ICML where it can be assessed by the best researchers in NLP and ML.

One might worry that the Semantic Web is going to suffer the same image problem as AI: that as soon as it works, it isn’t Semantic Web.  But I don’t think that’s the case.  There are certain research questions that are, and will continue to be, core to the Semantic Web.

With regard to the Semantic Web’s role in traditional applications, I would love to see at ISWC some studies that compared the relative developer effort required to build applications using the traditional and Semantic Web tool stacks.   Nobody’s going to argue against making data easier to reuse.  But the Semantic Web community still has to prove, I think, that their approach to reusability is better than others.  If I’m going to build a traditional application that consumes and manipulates data from one or two fixed sources, does using a triple store instead of a SQL (or noSQL) database make it easier for me to build that application or maintain it later?  Most applications hide their databases behind object-relational mappers, so will it even be noticeable which underlying database technology I am using?  When I want to pull data from my target source, does it help me to have that data available via a SPARQL endpoint, or would it be just as effective to present it me via a SQL endpoint, or an API that returns JSON objects?

If we are able to make a case that the Semantic Web really does help with reuse of data, then there’s a host of ISWC-relevant questions around transitioning the legacy of traditional data repositories to the Semantic Web.  For example, this paper shows how to “scrape” a traditional web API so it can be used with other Semantic Web tools.

Then there are the true Semantic Web applications, pan-schematic systems with no built-in assumptions about the schemas to which they are applied.  Almost by definition, these systems aren’t designed for domain specific tasks; however, they can be really useful for general-purpose information seeking, browsing, or organization.  Tools like tabulator try to support generic data browsing; semantic desktops like our old Haystack system provide personal information management over arbitrary schemas.   There’s also the Semantic Web search problem, of being able to search data that is structured but has no particular schema, more effectively than we can via text search.  Progress on these problems has been far slower than I expected or hoped; it seems like we’re mostly still stuck in the world of “big fat graph” visualizations.   This is a place where I’d really like to see ISWC focus its attention.  Perhaps this could serve to define a Semantic Web Challenge for next year: build an application that would let you win a scavenger hunt over the linked open data web.

 

 

“Living with Big Data: Challenges and Opportunities”, Jeffrey Dean and Sanjay Ghemawat, Google Inc.

Filed under: Bigdata by Anant Bhardwaj on Saturday, 29 September 2012 | 1 Comment
Tags: ,

As part of the Big Data Lecture Series — Fall 2012, Google’s Jeff Dean gave a talk on how Google manages to deliver services which involve managing huge amounts of data. In order to make things work over the distributed infrastructure of Google’s several data-centers, they use services and sub-services. Each service uses a protocol to communicate with other services. These protocols are language independent. Dean gave an example of a simple spell correction service which takes a request, such as correction{query:”……”}. The advantage of this model is that it is independent of client and it’s easy to make changes with no ripple effect. For an instance, to add a language feature in their spell correction service, they just need to add an extra optional request parameter: correction{query:”…….”, lang:”en_US”}. Also, it allows them to build each service independently.

Since Google has a lot of clusters in different datacenters, the list of potential things that can go wrong is long – rack failure, router failure, hard drive failure, machine failure, link failure (especially long distance links which are susceptible to external failures like attacks from wild dogs, drunken hunters etc.) are just a few! So, the software itself must provide reliability and availability. Replication allows them to solve hardware failures and issues such as data loss, slow machines, excessive load, bad latency etc. In order to tolerate latency, they primarily use two techniques – cross request adaptation and within request adaptation. The cross request adaptation technique examines recent behaviour and accordingly makes a decision for the future requests. On the other hand, the within request adaptation technique copes with the slow subsystems in context; it uses “tied requests” i.e. each request is sent to two servers (with a delay of 2ms). As soon as one of the two starts processing the request, it notifies the other to stop. Google ran experiments and they deduced that the latency hugely improves with a small overhead of a few extra disk reads.

In order to manage huge amount of data over distributed infrastructure, Google has several cluster level services, such as GFS/Colossus, Map Reduce, Cluster Scheduling System, Big Table. Although these services solve many problems, they also introduce several cross-cluster issues. To solve these cross-cluster issues, Google has built the ‘spanner’, a large-scale storage system that can manage data across all of Google’s data-centers. The ‘spanner’ has a single global namespace for data. It supports constant replication across data-centers and auto-migration to meet various constraints, such as a resource constraint (“file system is getting full”). A migration could be an app-level hint — “place this data to Europe”. The key idea is to build high level systems which provide a high level of abstraction. This black box is incredibly valuable since applications don’t need to deal with low level issues.

Monitoring and debugging are crucial in distributed environment. Every server in Google supports request tracing (call graph), online profiling, debugging variables and monitoring. Google has a tool called dapper which allows them to monitor and debug their infrastructure.

Much of Google’s work is approximating AI. Recently, they have been working on infrastructure for deep learning. Deep learning is an algorithmic approach to automatically learn high level representation from raw data. It can learn from both labelled and unlabelled data(unsupervised learning). The model could be as complex as number of parameters in billions, and requires several CPUs. In order to deal with this huge amount, Google partitions the model, adding another dimension of parallelism with multiple model instances communicating with each other. Google has built a deep network for machine learning (learning image representation, natural language processing (speech and text both)) that has significant reduction in the training time. They in fact trained an acoustic model for speech recognition in approximately 5 days with 800 machines in parallel.

Faculty Summits and Industy-Faculty Collaborations

Filed under: Thought Piece by David Karger on Wednesday, 15 August 2012 | 1 Comment

By some statistical fluke this summer I got invitations to and attended faculty summits at Google, Microsoft, and Facebook within a period of two weeks.  All were well run and a lot of fun, but left me wondering whether there are better ways to foster collaborations between faculty and these great companies.

Each company put on an admirable event.  The scales were different—400 attendees at Microsoft, 100 at Google, and 30 at facebook.  But the overall structure was pretty similar for all three.  The bulk of the time was devoted to conference-type presentations by company engineers and researchers, highlighting a bunch of the interesting work they were doing.  A few faculty also presented on the results of their collaborations with the company.  There was a presentation on funding opportunities.  And, in the evenings, conversational dinners.

The summits were very well done and entertaining to attend.  But I’m not convinced they took best advantage of the opportunity presented by the gathering of faculty.  This was really brought home to me at a Google summit presentation on online education: It explained how the traditional model of education, with one faculty member presenting to a room full of passive students, has become outmoded now that such presentations can be recorded for online consumption by anyone.  Of course, this presentation was given to a room full of passive summit attendees.   And it felt a lot like a classroom, with many “listeners” directing their attention to their email.

Given these summits’ strong similarity to conferences, centered on a sequence of talks, it’s worth remembering that the real value of conferences is in the hallways where attendees can have one-on-one conversations.  I saw many of these conversations happening at the summits, but the majority seemed to be among faculty who already knew each other, and have plenty of opportunity to talk to each other at real conferences.

What was disappointingly scarce was the one thing that these summits seem distinctively suited to generate: faculty-industry dialog.  Emphasizing this point, each summit offered one event that did support such dialog; in each case I found it to be the most valuable part of the summit which highlighted the limits of the rest.  Microsoft held a demo/poster session, where faculty circulated among various Microsoft groups who presented the projects they were working on.   With the faculty spread out among numerous projects, there was lots of opportunity for small-group discussions that could really dive into technical issues and possibly identify shared research interests.  At Google, the summit held “breakout sessions” on various topics; small mixed groups of faculty and Googlers spent an hour discussing specific topics of interest posed by Google.   I’ve already held a followup discussion with some Googlers around the topic of one such discussion, and can see some great research questions emerging.  Facebook held a single mixed “round table” (feasible given its small size) that went meta, discussing the question of how to enhance Facebook-faculty collaboration.   Also noteworthy at Facebook was my lucky dinner seating between two Facebookers that gave us time for lengthy discussion of some research questions.

These relatively short interactions gave me a sense of how much potential these summits have to foster interaction.  Working off them, here are some thoughts on how to fulfill that potential.

  1. Can the lectures.  Instead of presenting company research to the few physically-present faculty, record them and post the canned lectures so everyone, not just attendees, can see.  Have summit attendees watch them in advance to prepare for the summit.
  2. Bipartite poster/demo sessions.   Copy Microsoft’s demo/poster session which lets faculty learn about lots of different projects happening at the company and engage in small focused dialogs on projects they’re interested in.  Then invert it: set up a mirror session where the faculty are the ones with the demos and posters and employees circulate to discover interesting connections.
  3. Bring the mountain to Mohammed.  In particular, a faculty poster/demo session is a much cheaper way for potentially thousands of employees to get a sense of faculty research, and a chance to influence it, than sending those thousands of employees to conferences.  Faculty seemed to significantly outnumber company attendees at these summits, suggesting a missed opportunity for interaction.  I know these companies are full of PhDs who enjoy talking about research.  Where were they?
  4. Mix things up.  Break up the knots of old-buddy faculty and get them talking to employees.  Enforce mixed seating at meals.  Use some if the company’s cool technology to decide which faculty should be meeting which employees, and make sure it happens.
  5. Questions not answers.  Most of the talks presented finished work.  If the work is done then there’s no collaboration opportunity.  It would be great instead to see presentation of research objectives, that might help flush out others with related objectives, or with tools or data that could help meet those objectives.
  6. Unconference planning.  I’ve attended a few “unconferences” such as Foo camp where attendees set the agenda communally after they arrive.  This ensures that the topics are what the attendees actually want to talk about (the planning process also helps identify shared interests).  And the inability to prepare in advance means less presentation and more productive discussion.

Admittedly, a these suggestions are predicated on the assumption that faculty research might have something useful to offer to these companies.  Given the tremendous creative talent that these companies employ, that isn’t clear: perhaps our research is unimportant and the goal is simply to impress us into recommending these companies as good employment destinations for our students.  I’ll try to tackle that question in a separate post.

 

 

Congress, the NSF, and Social Science Research

Filed under: Social Computing, Thought Piece by David Karger on Tuesday, 5 June 2012 | 1 Comment
Tags:

For the past few weeks I’ve been following the Monkey Cage blog as it has followed the vote by the House of Representatives to prohibit the National Science Foundation (NSF) from funding Political Science research.   These days I tend to roll my eyes and feel helpless when Congress takes silly positions for political reasons (though I enjoy the irony of the House deciding that its own activity isn’t worthy of study).  But today an op-ed appeared in the Washington Post, which I expect to be more sensible/less political than Congress, arguing that the NSF should defund research not only politics but all social science, on the grounds that it isn’t possible to design rigorous controlled experiments in the social sciences or draw objective conclusions from the results.  While I suppose I might benefit from the redirection of that money towards “hard” science like CS, I don’t want to let such a blatantly false claim stand unchallenged (Monkey Care also has a nice rebuttal).    Especially since, if they do, an renewal of the congressional assault on research in human computer interaction is surely around the corner.

So, to falsify the claim, I’ll just mention two fine examples of careful controlled studies in social science of computer supported cooperative work.  Matthew Sagalnik and Duncan Watts did some a great controlled experiment showing how the popularity of music is substantially “self fulfilling”—music that is initially highly ranked is ultimately highly ranked by the whole community, even if the initial rankings are random.  At the conference on Computer Supported Cooperative Work, a source of plenty of such studies, Farzan, Pal, Kraut and Konstan used a controlled experiment to assess the impact of different kinds of socialization strategies on the socializiation of new entrants to a technical support forum.

These are just two examples of many; even one would serve to falsify the claims made in the Post’s op-ed.  I wonder if the author will be scientific/objective enough to retract his piece given the new data?

Oh well.  I suppose we should all write our congressmen before this goes any further.

To improve the CHI conference, would you share which talks you attended?

Filed under: CHI, Collective Intelligence, Publication, Social Computing, Thought Piece by David Karger on Thursday, 10 May 2012 | No Comments
Tags:

I’m having a great time at CHI (including my first time two-stepping today) but I strongly believe, as Jonathan Grudin asserted today, that we can make use of data to improve the conference.  I’ve already analyzed historical data that demonstrates that we can substantially reduce reviewer workload.  We’ve also created a way you can use current data to meet new people at CHI.  And if you’d be willing to share some of your own data, I’d like to try to use it to improve future conferences.

For the current conference, there’s a nice tool, created by Alireza Sahami, for meeting interesting new people at CHI this year.  If you visit the mobile version of the CHI program (which also works fine in a regular laptop browser) you can mark some papers you’ve attended or noted to read (just as you can with the CHI program apps—unfortunately, the data is not shared between them, so you’ll need to repeat).  Then, clicking the recommendations link and enabling recommendation will give you a list of users who are most similar to you in terms of the papers they’ve chosen to attend.  If you don’t know them, send an email and meet them to discuss your shared interests!

For future conferences, I’d really like to investigate the talk attendance data to see if we can do a better job of scheduling the conference to avoid conflicts.  In a past life I did a lot of work in combinatorial optimization, and I have my eye on some algorithms that I believe would do an excellent job of automated low-conflict scheduling.  But to test this hypothesis (and convince the CHI chairs to give it a try for real) I need actual talk preference data from the attendees.  Right now, if you’re marking those interesting talks using the android or iphone apps , that data is locked inside the app.  Would you be willing to share it with me so I can experiment with it?

I still need to get IRB approval to use this (personally identifying) data.  So I don’t want it yet.  Instead, please fill out this form that will let me contact you once I have set up a repository and a procedure for collecting the data?  In the meantime,  don’t delete your app!  It’s the only place where your talk selections are stored.

For CHI 2012: Discussion Forums in the Document Margins

Filed under: CHI, Collective Intelligence, News, Publication, Social Computing by David Karger on Monday, 7 May 2012 | No Comments
Tags:

Would you like some feedback on your CHI paper?  We’ve set up a site to let people read and comment on it.

On Wednesday at CHI, we’ll be presenting our paper on nb, a discussion forum situated in the margins of documents being discussed.  Its original intended usage was for discussion of classroom lecture notes, but we have discovered through our reading group that it is also quite useful for reading, commenting on and discussing research papers.   With this in mind, we’re making nb available to the CHI 21012 conference community.

We’ve created a folder on nb for CHI papers; there, you can read and annotate/discuss papers that have been uploaded by their authors.  There are currently 4 CHI papers there.  The first, of course, is our own paper on nb.  We’d love your comments, especially if they arrive before we present on Wednesday morning!  If you want to read and discuss, you just register on the site.  If you want to make your paper available for comments, please email your paper to sacha@mit.edu and he’ll upload it (yes, this is clunky—the system is currently designed for a faculty member who is uploading their course content, without means for “mere participants” to do the same).  I’d like to suggest, since you are seeking comments on your paper, that you “pay it forward”—read and comment on one of the papers already there to balance the hope that someone will then do the same for you.

The paper itself is focused on classroom deployment.  Nb has been used in about 60 classes at 5 universites, with several faculty choosing to use it repeatedly.  We study a particular successful use of the tool, in a class at MIT where it acquired over 14,000 comments from 100 students.   Using both quantitative and interview data, we explain why situating discussions in the margins can work better than the typical separate-forum approach.

If you’re teaching a class, we’d love to help you use nb for it.  Contact us!

Allocating CHI reviewers, a sequel

Filed under: CHI, Publication, Thought Piece by David Karger on Saturday, 5 May 2012 | 4 Comments
Tags:

Last year I used an analysis of CHI review data to argue that we could save a lot of reviewers’ time on low quality papers by modiyfing our review process.  With all the current talk of the value of replication, I figured it was worth testing the same procedure with this year’s review data, which Dan Olsen was kind enough to provide.

CHI currently collects three external reviews on every paper before engaging the program committee to make an accept/reject decision.   Last year I did an analysis that suggested rejecting papers with two bad reviews (scores below 3) without requiring a third, and showed that this would have saved 469 reviews (of papers that should have been rejected)  while accidentally rejecting just 6 of about 300 papers.  I argued that 6 papers was probably dwarfed by other mistakes the PC makes, so this would be a worthwhile tradeoff.

Because I’m replicating, I don’t need to detail the analysis again; you can find that in last year’s post.  I explored the following procedure: send each paper to two reviewers and, if the scores are too low, reject it without further consideration.  Otherwise, send it to a third reviewer and then on the the program committee for decision.  Varying the definition of “too low” provides a tradeoff between false positives (extra reviews for papers that ultimately get rejected) and false negatives (accidental early rejection of papers that would have been accepted).  Looking at last year’s data, a pretty appealing threshold was to reject any paper whose two scores were both lower than 3.   This is a natural rule because 3 is the “neutral” score in CHI reviews; both reviews below signifies that both reviewers recommend rejection.  It was also a good threshold because it saved 469 reviews while creating only 6 accidental rejections.

So, did the results replicate this year?  Applying exactly the same procedure, we get a remarkable level of agreement.  Of the roughly 1567 papers submitted, there were 369 acceptances and 1198 rejections.  Using the “two below 3″ rule, we skip 491 reviews of low-rated papers, while accidentally rejecting 4.7 papers (see last year’s post, or the discussion below, to understand why we get a fraction).  An almost identical outcome, just slightly better.

Some caveats: the data I got was slightly messy, with a few papers showing no reviews or less than three.  I presume these were withdrawn submissions, but haven’t had time to find out.  There were less than ten of these, not enough to influence the results significantly.

For those who want to explore the data themselves, I’ve prepared a table of the relevant numbers.  Score 1 and Score 2 are the reviewer scores; Accepts counts the number of pairs (remember, three pairs to a paper) from accepted papers that got those two scores, Rejects the number of pairs from rejected papers that got those two scores.  All other columns are functions of these two.  Ratio measures the ratio of accepts to rejects for a given pair of scores.  I’ve order the rows by this ratio, which is useful for visualizing the optimum false positive/false negative tradeoff.  Cum acc and Cum rej total the accept and reject pairs in the lines above, then divides by three so we can count papers (averaged over outcomes) instead of counting pairs.  These cumulative totals count the number of accepts and rejects above the current line, thus telling you the number of papers from each category that would be given a third review if you used the given line as the threshold for deciding on that third review.  False rejects then shows how many papers that were ultimately accepted would have been rejected in the first round using the given threshold, while saved reviews counts the number of ultimately rejected papers for which the threshold would have led to skipping the third review.

Note that the accepts and rejects columns are counting “review pairs”.  Each paper in the data set got three reviews.  If we imagine the chair selecting three reviewers but then requesting a review from only 2 of them (keeping the third in reserve for when the first reviews are good), then there are three possible outcomes.  Counting over all three outcomes of all three papers yields the average outcome—the expectation should the chair select their two reviewers at random from the pool of three.  It is this averaging that produces fractions in the other columns.

Score 1 Score 2 Accepts Rejects Ratio Cum acc Cum rej false rejects saved reviews
5 5 14 0 NaN 14 0 364.3 1197.3
4.5 4.5 37 3 12.33 51 3 352.0 1196.3
5 4 63 8 7.87 114 11 331.0 1193.7
5 4.5 41 6 6.83 155 17 317.3 1191.7
4.5 4 113 21 5.38 268 38 279.7 1184.7
4 4 125 31 4.03 393 69 238.0 1174.3
4.5 3.5 65 20 3.25 458 89 216.3 1167.7
5 3 22 8 2.75 480 97 209.0 1165.0
5 3.5 19 8 2.37 499 105 202.7 1162.3
4 3.5 129 68 1.90 628 173 159.7 1139.7
4.5 3 32 23 1.39 660 196 149.0 1132.0
4 3 70 81 0.86 730 277 125.7 1105.0
3.5 3.5 52 62 0.84 782 339 108.3 1084.3
4.5 2.5 20 29 0.69 802 368 101.7 1074.7
5 1 2 3 0.67 804 371 101.0 1073.7
5 2.5 9 17 0.53 813 388 98.0 1068.0
4 2.5 43 95 0.45 856 483 83.7 1036.3
4.5 1.5 8 18 0.44 864 501 81.0 1030.3
4.5 2 11 25 0.44 875 526 77.3 1022.0
4 2 39 95 0.41 914 621 64.3 990.3
3.5 3 48 117 0.41 962 738 48.3 951.3
5 2 6 20 0.30 968 758 46.3 944.7
5 1.5 2 8 0.25 970 766 45.7 942.0
3 3 14 66 0.21 984 832 41.0 920.0
4 1 5 28 0.18 989 860 39.3 910.7
3.5 2.5 30 172 0.17 1019 1032 29.3 853.3
4 1.5 6 36 0.17 1025 1068 27.3 841.3
4.5 1 2 13 0.15 1027 1081 26.7 837.0
3.5 2 24 199 0.12 1051 1280 18.7 770.7
3.5 1.5 6 68 0.09 1057 1348 16.7 748.0
3 2.5 15 188 0.08 1072 1536 11.7 685.3
3 1 3 52 0.06 1075 1588 10.7 668.0
3.5 1 3 55 0.05 1078 1643 9.7 649.7
3 2 10 214 0.05 1088 1857 6.3 578.3
3 1.5 2 107 0.02 1090 1964 5.7 542.7
2.5 2.5 3 155 0.02 1093 2119 4.7 491.0
2 2 4 223 0.02 1097 2342 3.3 416.7
2 1.5 3 227 0.01 1100 2569 2.3 341.0
2.5 2 4 331 0.01 1104 2900 1.0 230.7
2.5 1 1 83 0.01 1105 2983 0.7 203.0
1.5 1 1 144 0.01 1106 3127 0.3 155.0
2 1 1 157 0.01 1107 3284 0.0 102.7
1 1 0 94 0.00 1107 3378 0.0 71.3
1.5 1.5 0 76 0.00 1107 3454 0.0 46.0
2.5 1.5 0 138 0.00 1107 3592 0.0 0.0

 

Forums in the Document Margins for Classes and Reading Groups

Filed under: CHI, Collective Intelligence, Publication by David Karger on Friday, 4 May 2012 | No Comments
Tags:

This year at CHI we’ll be presenting a paper on nb, a tool that lets students have forum-style threaded discussions in the margins of pdf documents.  We’ve posted it in advance at the link above in hopes of getting some comments on it that can help us prepare our presentation.  We’re also making nb available to you, if you have a paper at CHI and want some feedback on it.

We originally designed nb for class materials—lecture notes and such—under the hypothesis that attaching the forum to the content provides useful context that helps students see and participate in relevant discussions at the right moment in their reading.  Our paper shows support for this hypothesis.

But we’ve discovered that nb is also quite useful for our research reading group.  If we post the paper in advance and everyone annotates it, we get great material for reading group discussion, nicely organized for us as we traverse the paper.   Recently, when we asked Parmit Chilana for a pdf of their CHI Lemonaid paper so we could mark it up this way, she mentioned that she’d like to see those comments to help her prepare her CHI talk.  We thought this was a great idea, so we’re making it more generally available.

We’ve created a CHI channel on the nb site.  You can sign on by clicking this link.  There, you’ll be able to read and annotate/discuss any paper whose authors have provided it.   If you’re an author and want to get some reader feedback, either before or after your talk, send the paper to sacha@mit.edu and we’ll add it to the site (a bit old-fashioned, but the system isn’t currently designed to let “student readers” upload papers to their classes).

Of course, to get some annotations you need some readers, and our system can’t help you with that!  You’ll need to use some old fashioned techniques, like mentioning it on your blog and encouraging people to read and annotate it.  For example, consider this blog post, whose first line encourages you to read and annotate our (paper on nb) on nb.  I look forward to your comments!

Next Page »