Database War Stories

There’s an interesting series of posts over at O’Reilly Radar. Tim O’Reilly asked people how they were using databases in their “Web 2.0” applications (although I think the Web 2.0 part of it is for the most part irrelevant). The responses so far have made for interesting reading. So far we’ve heard from Second Life, Bloglines and Memeorandum, Flickr, NASA World Wind, and craigslist. One of the lessons learned is that with a high-traffic site, at some point you have to break your database up so that the “hot” data is spread across a number of boxes. This got me thinking. It should be possible to build a tool that analyzes your database usage and, given a number of slave boxes to configure as it sees fit, automatically configures masters and slaves and distributes your data across those boxes as necessary. This would not be a one-time only process either; it would continue to monitor usage and performance and adjust accordingly. Certainly not an easy task, but should be doable.