Distributed Web Crawling
Idea: use volunteer computing to increase not only computation power, but also communication power
- Allow users with slow links to do communication-bound tasks; e.g., web crawling/searching
Experimental Results:
- web crawling through slow link is slow even with fast PC (w/ JIT)
- web crawling using volunteers with slower Java (4x slower) but faster links resulted in speedup
- using “multi-work” Engine and Manager gave more speedup transparently (without recoding)
- ongoing work on even more speedup