Wednesday, December 10, 2008
Major Version Upgrade with No User Downtime Using Virtuals Machines
This last week we upgraded our web-based software to a new major version without any downtime for our users. It was a great success. This only works if the databases backing the site are upwardly compatible. Here's the steps:
1. It all started last year when we hosted our application in a virtual machine (who doesn't these days anyway) even though our application was the only one running on the server.
2. On each of the production boxes we recently installed a parallel shadow virtual machine and loaded the new version of the software. We created test copies of the databases on our database server farm.
3. We created a shadow farm address (www.siteB.OurFarm.com instead of www.siteA.OurFarm.com) in our local router/load balancer and pointed the siteB domain to our shadow virtuals. Since all the nunit and functional tests were all done in our internal test farm, this round of testing was for configuration issues (can all the machines see all the databases? Can the farm machines see each other? Can outside users hit all the shadow boxes?).
4. We upgraded the databases on the live system by adding our new tables. The old system ran on the upgraded databases, but did not use all the tables. We incrementally pointed our shadow virtuals to the three live databases and tested them.
5. On the big transition day, we simply switched our domain address "SiteA" to point to our new shadow virtuals and all was golden.
PostScript: If any errors had occurred with the new version, we could back out by pointing "SiteA" back to the original servers (assuming no damage to the databases). We had made copies of the databases before had so we could switch back to them if something had corrupted the databases.