I was blown away when I read about how IMVU runs a million tests a day in the course of about 50(!) production deployments.
The way it happens is downright shuttle-like :
The code is rsync’d out to the hundreds of machines in our cluster. Load average, cpu usage, php errors and dies and more are sampled by the push script, as a basis line. A symlink is switched on a small subset of the machines throwing the code live to its first few customers. A minute later the push script again samples data across the cluster and if there has been a statistically significant regression then the revision is automatically rolled back. If not, then it gets pushed to 100% of the cluster and monitored in the same way for another five minutes.
I freaking LOVE that. The machine dipping its toes in the water before moving code, then again after to make sure things are OK. The Pragmatic Programmer talks about ubiquitous automation, but this is something approaching a whole new level, man.
