Friday, December 28, 2007

Holiday Upgrade

Not upgrading the Holidays - upgrading our Sakai deployment. (The holidays have been very nice, thank you. Starting off with a trip to Vancouver, in my mind, and ending with some skiing in Tahoe. Friends, family and feasting in between...)

Stanford has told us we must take vacation during the holidays as a cost-saving measure, and yet we must release software. So here we are! We get to gossip about the holidays.

Our Sakai / CourseWork upgrades are, on the whole, pretty easy. More time is taken giving the new deployment a live shakedown. Perhaps Mary Mak can manage to get automated testing in place - it would be great to have robots crawl the site after deployment.

The procedure was that QA did the final build for release, and Julian Morley dropped it onto a private preprod box. There he reviewed any property changes. If a table had changed Sam would of cut a clone of the production schema before hand. Julian would apply the Operations lens to the deployment and look for issues.

On release day QA would run any of the tested SQL necessary to prepare / update the databases, and then Operations would manage the servers.

Ops takes the Big5 Load Balancer thingy and puts up the standard "out of service" link. This goes to some static HTML in the Stanford www server pool. Then one of the tomcat pool is chosen as a candidate. The tomcat from the ops-private preprod box is dumped onto the candidate machine, and started up. The non-load-balanced machine name is used for smoke testing.

Cost cutting measures have lead to a loss of that level of software release process. The only real difference is that developers now do the final build & SQL tweaking. This provides the double edged sword of having people around who can tweak the deployment target as it is slammed into production. (When QA ran the show they would rightly punt the deploy of something had a bad smell. Now it can be hacked. You decide how you want to run your institution. :) )

In this case we're rollout a build containing a Sakai-provisional tool of our own making. It is an alternate "home page" tool intended to be used in course sites. Lydia is handling the DB conversion. We're also doing some CourseWork Classic conversion, and Julian has wrangled a blob dump. (we switched to extracted blobs long ago - summer? - but kept them in the DB. the dump now allows us to move the tables and recover the space from disk.)

When we're not doing database table changes lots of the ritual is disposed of. The process boils down to a tomcat drop, a restart and a smoke test.

I'm expecting this to go out without any operational difficulties.

No comments: