Blogs and the Dissemination of Scientific Research
HCI research needs to get better at spreading the word, sooner, in the Web 2.0 era. Typically, by the time that CHI rolls around, the research being presented is at least 7 months old. When (or if) a group decides to post PDFs early, the papers are so distributed that interested readers can’t find them. What’s more, the research that is posted isn’t presented in a web-friendly way: how many web pages do you read in PDF form? But, when HCI research is made available in an interesting and accessible form, it often gets great press.
What I’m thinking might remedy the situation is a CHI early results blog. It would work like this: when a paper is accepted to CHI, the SIGCHI blog administrators e-mail the authors and invite them to write a short blog post describing the results that will be appearing at the conference. It should be written for a general web audience and other HCI researchers and practitioners. This would not just be the abstract and intro; the blog would highly encourage pictures, videos, and any other media. The drafts would be vetted for readability, and then posted as soon as they are ready (with some flow control to make sure we don’t post too many items at once). This means results could be posted as early as December.
If successful, this could be a great way to accelerate the dissemination of important research results, generate lots of positive buzz for the conference and the papers in it, keep research conversations going year-round, and increase the number of HCI posts on Slashdot. And I mean, Slashdot is our currency, really.
What do you think? Would you post in a centralized SIGCHI blog when your papers get accepted?
This is a great idea; I’d definitely post work there. It might be nice if the blog supported crossposting from specific research groups’ blogs, which is where they might actually want to announce their results.
Such a blog would exacerbate one big problem with CHI: as you say, presentations there are for 7-month-old work. With the blog, it would become 7-month-old and _broadly known_ work—a recipe for boredom. The right fix there, of course, is to shorten the submission-to-publication timeline. I’ve just finished chairing ISWC 2009 where we had quite a bit of success with a tighter schedule: from submission, 2 weeks for reviews (nobody does them before the last 2 weeks anyway), 1 week for metareviews, 1 week for rebuttals and final decision, 1 week for camera-ready copy. If you abandon paper proceedings, that camera ready can arrive the day before the conference, giving you a 5-week timeline instead of 7 months. We ended up stretching it a bit farther to give attendees time to get visas and book flights, but still had a June submission deadline for our October conference.
When I read about psychology studies, I often read that they tend to get re-run by several independent labs in the world in order to see whether the results are actually reproducible. From my experience with HCI research, each user study tends to get run only once, maybe 2 weeks before a deadline. The results are then taken as bible and quoted by subsequent papers without any attempt to re-run the study to reproduce its result. Is it possible that we might be building on a shaky foundation of irreproducible results?
One reason why studies are not re-run is because research tools are not reusable. They fall apart as soon as the last usability subject leaves the lab. And no researcher would be willing to re-implement a competitive tool to do a proper comparison, let alone re-run a study that earns him/her no novelty point.
So the publication process is not just problematic for the time delay, but also for the reward scheme. So, fixing the time delay might just get us bogus papers faster.
A friend of mine claims that most if not all things you use a lot on your computer and on the Web today have not come out of HCI academic research proper in the last 2 decades. I am having trouble arguing with him
@David K–
It’s a legitimate worry about the work becoming broadly known. I’d personally worry that people who read the post wouldn’t come to my talk at the conference. But, if you do a good job as an advertisement, it could also potentially increase turnout too.
@David H–
Unlike in psychology, there is very little reward in HCI for reproducing previous results. I think this is partially because HCI sometimes has tenuous claims to being “a science”. But maybe levying the criticism at systems papers is unwarranted; their core contribution is rarely about the evaluation anyway. Interfaces are replicable by observing the design and the description.
But I don’t think a results blog would affect this issue either way. The paper still had to get through peer review, which required its evaluation to be strong enough to be believable. But I wouldn’t be surprised if blog posts suppressed the evaluation parts of systems papers — if you built Air Guitar Hero, few people on the web care about the results of your ANOVA.
It’s a great idea, and something I consider problematic in HCI. I read a lot of economics, psychology, neuroscience etc. blogs that all discuss recent and interesting research, but there’s very little from HCI.
I wonder why HCI doesn’t do it. To throw a provocative statement out, perhaps researchers would do more interesting research if they thought they had to blog about it and get others interested
.
DK’s point about the work being well known by the time of the conference may also a) conference organisers to rethink the format (which is already being discussed), and b) present the work in a new or interesting manner. As you say, it may even serve as a good indicator of what to go see.
“Interfaces are replicable by observing the design and the description.” – I have not seen this done by HCI researchers before.
My point is that the reward system may be the more important thing to fix than the publication time lag.
I think this is a great idea! Two important but simple questions are:
1) I think that the audience will likely need to be either for other HCI researchers or for a general audience. Focused effort on explaining results in an accessible way to a general audience might go a long way towards getting people interested in HCI topics, the way that NASA seems to do for space exploration, which is a great goal.
2) Who would do the vetting? I am relatively new to the whole CHI process, but I do know that it seems like reviewers are often already stretched just with reviewing papers and may not be eager to take on blog post reviews as well. I think that just forcing authors to vet their own posts would encourage them to think critically about how to explain results and would probably make for better CHI presentations anyway – I wouldn’t worry about posting the results all at once, but you could invite people to post in waves throughout December and January.
In general, I look forward to more blogging and more idea-sharing from HCI writers – it’s nice reading papers for depth, but blogs would be a great way to understand someone’s research interests and capture some interesting findings.