The Chunk Project explores a way of approaching distributed and cloud computing that considers the “space” computation and data structures take up.
Modern computing substrates like cloud computing clusters, massively multi-core processors, and general-purpose GPUs offer a wealth of computing power with the caveat that programmers must carefully structure their programs to fit within various hardware and communication limits in order to get high performance. Unfortunately, modern abstractions expressly hide machine structure from the program and program structure from the machine, hindering automatic optimization. We introduce an alternative computing abstraction—a linked graph of finite-sized memory chunks—that explicitly exposes the size and structure of programs to the operating system. The chunk graph enables the operating system to use size and structure to optimize how programs are mapped onto complex machine structure.
Publications:
ChunkStream is a system for efficient streaming and interactive editing of online video. Rather than using a specialized protocol and stream format, ChunkStream exposes video clips as large, but fine-grained, data structures built from a generic foundation, allowing a client to retrieve and work with only the portions of the clips that it needs. ChunkStream supports resource-adaptive playback and “live” streaming of real-time video as well as fast, frame-accurate seeking; bandwidth-efficient high-speed playback; and simple compilation of editing decisions from a set of clips. Our benchmarks indicate that ChunkStream uses less bandwidth than HTTP Live Streaming while providing better support for editing primitives.
Publications:
Source code snapshots: chunkstream.tar.gz and chunkstream-benchmarks.tar.gz. These snapshots are unsupported and are only available to aid replicating the benchmarks from our 2010 papers. If you plan to use chunks in your project, please use our current source distribution.
Cloud systems differ fundamentally in how they offer and charge for resources. While some systems provide a generic programming abstraction at coarse granularity, e.g., a virtual machine rented by the hour, others offer specialized abstractions with fine-grained accounting on a per-request basis. In this paper, we explore Tasklets, an abstraction for instances of short-duration, generic computations that migrate from a host requiring computation to hosts that are willing to provide computation. Tasklets enable fine-grained accounting of resource usage, enabling us to build infrastructure that supports trading computing resources according to various economic models. This computation model is especially attractive in settings where mobile devices can utilize resources in the cloud to mitigate local resource constraints.
Publications:
The latest Chunk source code distribution is from 2011 June 23 with hash 9ba6ffd.
The Chunk code is distributed under the MIT License:
Copyright (c) 2009-2011 Justin Mazzola Paluska
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@mit.edu.