More migrations than waterfowl

Migrations. We all have em. If you’ve ever rebuilt your personal computer, you’ve had a migration. The act of upgrading a “foundation” system (like an OS or app platform) can be pretty daunting, but needn’t be too much of a pain in the neck. I mean, if you’re upgrading, then you should end up better off on the other side, right? Otherwise, what’s the point?

Having participated in dozens of migrations of varying kinds, I’ve discovered that release management can help significantly reduce the overall risk and impact of any widespread upgrades or migrations.

OS upgrades – upgrading your Windows or Unix server to the latest operating system.

Hardware upgrades – upgrading server hardware resources (CPU, disk, networking).

App Server upgrades – upgrading your application server and/or app platform. Examples include simply upgrading to the latest distribution of Apache, Jrun, Tomcat, etc., or a widely deployed enterprise systems/application (PeopleSoft, Exchange, or a Content Management System).

Miscellaneous migrations – migrating source control systems, CM systems, App Platforms (say, moving a Tomcat app to Weblogic).

Release Management can help with all of these types of migrations in several ways. Firstly, the release manager is well positioned to provide essential environment information such as inter-application, inter-server and other various dependencies.

The release manager can also provide insight into usage patterns, and can help plan for and communicate about coming migrations.

Lastly, the release manager can assist with coordinating all of the various players involved in a wide-scale migration and/or upgrade. From working with engineers to make any required app updates, to following up with Sys Admins to validate application and/or server availability post-upgrade, the release manager can help make sure that everyone has what they need to do their part.

Advertisement

Backing out of a release

Nobody likes to delay a scheduled release. However, risk management is a big part of release management, and sometimes it just makes sense to abort and reschedule. When deploying applications to a production environment, introducing unknowns can potentially have widespread impact. In some cases, the negative impact can far outweigh the benefits of troubleshooting and tweaking to push an application update out.

Following are some tips for helping to mitigate the possible negative consequences of a failed production deployment. This posting primarily applies to deploying web based applications to a production server environment hosting multiple applications.

Use a maintenance window – A key step in mitigating risk and minimizing user impact is establishing a “maintenance window”. Identify when server traffic is typically low, and block out a time frame for deploying applications or application updates. Regardless of whether the build/deployment/release is automated or not, it’s useful to have a set time when changes to the system occur. A release window adds a certain level of objectivity and predictability to system changes. It offers a boundary to risk exposure in that risk is minimized by literally limiting the scope of the time you’re systems are being changed. An additional benefit is the ability to notify your users that this time is when system updates occur, and they needn’t worry if they experience issues while attempting to use online resources. Depending on your user base, this can be helpful in reducing unnecessary trouble tickets or bug reports.

Ideally, most everything would be automated, happen almost instantly, and you’d have elegant, intelligent maintenance pages/handlers to gracefully redirect users to a “this application is being updated” page, but that’s a topic for another posting. Then again, ideally, app deployments would just auto-magically happen all by themselves, all the dependencies would always be in place, and config files would never mysteriously have the QA values still in them (“I could have sworn I ran the ANT task to update that file..”). Wouldn’t that be cool?

OK, so you’ve got a maintenance window – now what? How to decide when to investigate, troubleshoot, and make the darn application work right whether it wants to or not? When to say “Uncle”, back out of the release, and hang your head in shame?

Estimate time to complete – Accurately estimating how long a particular task or automated job takes to complete is essential to managing smooth application deployments. Deciding when to back out all comes down to accurate estimates of how long tasks take to complete. Consider the following when coming up with estimates: how long does the actual deployment take if nothing goes awry? how long would it take to double check some basic dependencies or configuration files? How long would it take to back out and undo all changes and restore the system to it’s pre-release state (worst case scenario)?

The rest is simple – if you’re deep the in release window and things are bumpy, ask yourself, “Can I still backout of this before the window closes?”. This is your “point of no return”. If the answer is yes, keep troubleshooting and get things sorted out. If it’s a close call, take your medicine and back out.

If you have somehow managed to get past the point of no return, well, you better make sure you get everything running ok (without introducing new problems!) before the window closes, or else you might as well just start updating your resume.

Subversion everywhere…

OK, so maybe we won’t use Perforce.  A mischevious consultant casually mentioned that a lot of his clients use Subversion, and essentially recommended that we use it too.  So now the management types have decied we need to evaluate and consider Subversion.  Fun.  We’ve been evaulating source control tools for about a year now, and we were sooo close to finally actually making a decision.  But alas, the saga continues.

It looks like Subversion is basically CVS with some improvements.  It’s all open source, which I suppose is kind of neat.  But along with open source comes all sorts of fun little troubleshooting.  At least with a commercial vendor, they’ll help us get things set up, and support us if we need it.  Of course, that kind of thing comes with a hefty price tag, and, well, you know, it’s not like our source code integrity is really a big deal or anything.  Certainly not worth spending a few grand to make things better.  Nah…

Also, the other fun thing is trying to figure out how to get our stuff from VSS into Subversion, while keeping all our labels and history and all that jazz intact.

Well, the good side of all this is:  we’ve thought this thing through to death, and whatever we do decide on, we’ll never be able to go back and say “oh, why didn’t we consider that?”.  It never hurts to be thorough when choosing a toolset you’re planning to have for years and years to come.

Goodbye VSS, Hello Perforce!

It looks like we’re finally gonna get off of VSS entirely and just move on over to Perforce.  Yay!  I’ve never actually used Perforce, but from what I’ve heard and read on their site, it looks like it’s gonna be sweet.  Plus, just about everyone who’s anyone is using Perforce or CVS, so it can’t be that bad.

Gonna hafta start mapping out the migration path from VSS to Perforce.  Looks pretty straightforward, especially since they’ve been kind enough to provide some VSS to Perforce migration scripts.

More later, gotta go watch the Shield.