Thursday, September 06, 2012

"Don't Just Do Something, Sit There!"

During a recent upgrade I panicked. We were upgrading our system to be on a Universal Time so all the countries could schedule and see events in their local time (don't get me started on daylight savings time).
Our team in India ran my upgrade script to convert all the historical records to Universal Time from the local server time, but my web application was still showing the old time, just as if the script had not run. I spoke with the person doing the SQL script execution and he assumed me he already ran it.
But our system wasn't showing the new times. This is when I panicked. Instead of carefully thinking the situation through and talking with some other knowledgeable people, I had the India team rerun the script. Then it hit me. The servers where caching the times. The database had really been updated, but the app was clever enough to cache the time to save load on the database.
I restart the web app and sure enough the historic times were now changed by twice the intended amount. After rerunning the upgrade script with a negative time to set the clock back everything was fine. The upgrade proceeded and I learned a valuable lesson,
"Don't just do something, sit there! (and think)."

No comments: