Release Management – QA or engineering?

There seems to be some debate around where in an organization Release Management really belongs.  Is release management a function of engineering, or is it a function of quality assurance or quality control?  A case can be made for both, or neither.

On the one hand, release management is certainly a function of engineering or software development.  What good is a slick piece of code if it can’t be integrated, tracked, and released into the world?

On the other hand, the release manager is the last line of defense for what actually gets deployed.  So in that sense, the release manager must be acutely aware of quality issues.  Obviously, the release engineer can’t be completely in on all the requirements or business needs – that’s outside of his scope.  However, he must be aware of things like system resources, monitoring, logging, etc.  All the things going on behind the scenes to ensure quality of response and uptime.

Release Management as an engineering function

Often times, relatively small software development teams don’t recognize or appreciate the need for a central resource to handle build, deployments, and environment maintenance.  Typically, you’ll find one engineer in the group who’s familiar enough with SCM systems, web services, UNIX/Windows admin, etc to step in and fill the role.  Depending on the size of the team and the complexity of their development processes, this is often sufficient. 

In any case, someone needs to take ownership of these issues, and that someone should have at least some knowledge of complex systems and how applications are built and deployed.

Release Management as a Quality function

The release manager definitely needs to work closely with the QA team, in order to properly plan and allocate resources to test apps before releasing them into production.  In some cases, the release manager can help a QA engineer determine the nature of a bug, and whether it’s an environment/server config issue, or simply a bug in the app.

My humble opinion is that release management sort of “straddles” both engineering and QA, and as such really belongs in either it’s own organizational unit, or could fall in with operations/IT management.  I guess it really comes down to how your team is organized and what the real needs are.

More migrations than waterfowl

Migrations. We all have em. If you’ve ever rebuilt your personal computer, you’ve had a migration. The act of upgrading a “foundation” system (like an OS or app platform) can be pretty daunting, but needn’t be too much of a pain in the neck. I mean, if you’re upgrading, then you should end up better off on the other side, right? Otherwise, what’s the point?

Having participated in dozens of migrations of varying kinds, I’ve discovered that release management can help significantly reduce the overall risk and impact of any widespread upgrades or migrations.

OS upgrades – upgrading your Windows or Unix server to the latest operating system.

Hardware upgrades – upgrading server hardware resources (CPU, disk, networking).

App Server upgrades – upgrading your application server and/or app platform. Examples include simply upgrading to the latest distribution of Apache, Jrun, Tomcat, etc., or a widely deployed enterprise systems/application (PeopleSoft, Exchange, or a Content Management System).

Miscellaneous migrations – migrating source control systems, CM systems, App Platforms (say, moving a Tomcat app to Weblogic).

Release Management can help with all of these types of migrations in several ways. Firstly, the release manager is well positioned to provide essential environment information such as inter-application, inter-server and other various dependencies.

The release manager can also provide insight into usage patterns, and can help plan for and communicate about coming migrations.

Lastly, the release manager can assist with coordinating all of the various players involved in a wide-scale migration and/or upgrade. From working with engineers to make any required app updates, to following up with Sys Admins to validate application and/or server availability post-upgrade, the release manager can help make sure that everyone has what they need to do their part.

Backing out of a release

Nobody likes to delay a scheduled release. However, risk management is a big part of release management, and sometimes it just makes sense to abort and reschedule. When deploying applications to a production environment, introducing unknowns can potentially have widespread impact. In some cases, the negative impact can far outweigh the benefits of troubleshooting and tweaking to push an application update out.

Following are some tips for helping to mitigate the possible negative consequences of a failed production deployment. This posting primarily applies to deploying web based applications to a production server environment hosting multiple applications.

Use a maintenance window – A key step in mitigating risk and minimizing user impact is establishing a “maintenance window”. Identify when server traffic is typically low, and block out a time frame for deploying applications or application updates. Regardless of whether the build/deployment/release is automated or not, it’s useful to have a set time when changes to the system occur. A release window adds a certain level of objectivity and predictability to system changes. It offers a boundary to risk exposure in that risk is minimized by literally limiting the scope of the time you’re systems are being changed. An additional benefit is the ability to notify your users that this time is when system updates occur, and they needn’t worry if they experience issues while attempting to use online resources. Depending on your user base, this can be helpful in reducing unnecessary trouble tickets or bug reports.

Ideally, most everything would be automated, happen almost instantly, and you’d have elegant, intelligent maintenance pages/handlers to gracefully redirect users to a “this application is being updated” page, but that’s a topic for another posting. Then again, ideally, app deployments would just auto-magically happen all by themselves, all the dependencies would always be in place, and config files would never mysteriously have the QA values still in them (“I could have sworn I ran the ANT task to update that file..”). Wouldn’t that be cool?

OK, so you’ve got a maintenance window – now what? How to decide when to investigate, troubleshoot, and make the darn application work right whether it wants to or not? When to say “Uncle”, back out of the release, and hang your head in shame?

Estimate time to complete – Accurately estimating how long a particular task or automated job takes to complete is essential to managing smooth application deployments. Deciding when to back out all comes down to accurate estimates of how long tasks take to complete. Consider the following when coming up with estimates: how long does the actual deployment take if nothing goes awry? how long would it take to double check some basic dependencies or configuration files? How long would it take to back out and undo all changes and restore the system to it’s pre-release state (worst case scenario)?

The rest is simple – if you’re deep the in release window and things are bumpy, ask yourself, “Can I still backout of this before the window closes?”. This is your “point of no return”. If the answer is yes, keep troubleshooting and get things sorted out. If it’s a close call, take your medicine and back out.

If you have somehow managed to get past the point of no return, well, you better make sure you get everything running ok (without introducing new problems!) before the window closes, or else you might as well just start updating your resume.

Subversion everywhere…

OK, so maybe we won’t use Perforce.  A mischevious consultant casually mentioned that a lot of his clients use Subversion, and essentially recommended that we use it too.  So now the management types have decied we need to evaluate and consider Subversion.  Fun.  We’ve been evaulating source control tools for about a year now, and we were sooo close to finally actually making a decision.  But alas, the saga continues.

It looks like Subversion is basically CVS with some improvements.  It’s all open source, which I suppose is kind of neat.  But along with open source comes all sorts of fun little troubleshooting.  At least with a commercial vendor, they’ll help us get things set up, and support us if we need it.  Of course, that kind of thing comes with a hefty price tag, and, well, you know, it’s not like our source code integrity is really a big deal or anything.  Certainly not worth spending a few grand to make things better.  Nah…

Also, the other fun thing is trying to figure out how to get our stuff from VSS into Subversion, while keeping all our labels and history and all that jazz intact.

Well, the good side of all this is:  we’ve thought this thing through to death, and whatever we do decide on, we’ll never be able to go back and say “oh, why didn’t we consider that?”.  It never hurts to be thorough when choosing a toolset you’re planning to have for years and years to come.

Goodbye VSS, Hello Perforce!

It looks like we’re finally gonna get off of VSS entirely and just move on over to Perforce.  Yay!  I’ve never actually used Perforce, but from what I’ve heard and read on their site, it looks like it’s gonna be sweet.  Plus, just about everyone who’s anyone is using Perforce or CVS, so it can’t be that bad.

Gonna hafta start mapping out the migration path from VSS to Perforce.  Looks pretty straightforward, especially since they’ve been kind enough to provide some VSS to Perforce migration scripts.

More later, gotta go watch the Shield.

Scripting with ANT… Groovy

While ANT definitely is a great tool, and it’s benefits over MAKE well proven by now, it does have some limitations when it comes to scripting complex build and deploy tasks.  Well, not so much ‘limitations’, cuz it’s pretty open to extend it any way you could want. 

But anyone who’s ever tried to do something fancier than “grab code, compile, package into .ear file”, has had to get real creative with XML to get it to act like a scripting language.

Groovy looks like it could help make life easier for us build/release managers.

Article in JavaWorld… 

It’s even got an ANT Task.

Migrating apps from Windows to Unix – Intro

App migrations can be a real headache.  They don’t have to be, but they often are.  Especially when moving from one platform to another.  This is the story of such a migration…

The powers that be in my organization have decided to move away from MSFT technology, and get into the whole Java/Linux thing.  While I’m all for learning new stuff (especially Java/*nix stuff), this is turning out to be a pretty major effort.  We’ve currently got about 80 or so apps (ASP or .NET) on a handful of IIS servers.  All of these apps will eventually need to either be rewritten or ported to run in a java environment.  Can you say job security?

We decided on Weblogic as our new Java app platform (with Apache out front).  From what I can tell, it’s pretty sweet.  As a step in the direction we’re headed, we decided to use Apache to front our IIS servers, so we can just proxy our IIS apps though a unified web server.  Apache is cool.  Using Apache as the out-facing web server, we can then proxy app request to either IIS, or Weblogic, or that pesky little Tomcat service we’ve got runnin – all through one unified namespace.  Neat!

This is just the introduction to the app platform migration saga, and I have a feeling it’ll turn into a pretty regular series of articles for a while.  It sort of ties into the VSS / Source control discussions, since part of the migration is to create an entire new development environment for our engineers.

And here’s where it gets real interesting – we don’t have very thorough docs on all of our apps and their depelndencies (I know, shocking!), since some of them were written years ago, when we didn’t have much in the way of processes and proper documentation.  So trying to figure out how to map all the inter-app and server dependencies has been quite a challenge.

One really cool thing about the whole effort is that I’m getting to play with Linux, Apache, and Weblogic a lot more, which is fun!

More to follow…

VSS 2005 Upgrade

So while we’re trying to decide what do do with source control long-term, we figured we’d take VSS 2005 for a test run – just to see if it’d hold us over for a while. 

The main thing we’re missing right now is access over http for our remote crew.  Sure there are some nifty tools for VSS over http, but VSS 2005 has it built-in.

The good: VSS 2005 client sure is purty.  It’s definitely a little slicker, and the branching/merging seems a bit smarter (it couldn’t get worse, right?).  Seems to run faster too, which is nice.

The bad: Turns out, you need to use the latest VStudio (2005) client to access the repository over http.  Pretty lame, you’d think you could at least just use the VSS client.  I suppose since it’s really just exposed as a web service, someone’ll make a better interface to it, but I haven’t seen it yet.

The ugly:  Took me about 3 days to get the darn thing runnin.  For some (as yet unknown reason), VSS and IIS just would not play nicely together.  After a couple days and a few unfruitful hours on the phone with MSFT techs, I decided to just install it on another machine (an IIS box built to the same specs as the first one, just without anything on it yet).  20 minutes later it was up and runnin… go figure.

Automated build examples

Stumbled acrosss an article with some nice examples of how to automate builds. Check it out:

http://www.wrox.com/WileyCDA/Section/id-131031.html

Which source control system?

So we’re thinkin about getting off of this clunker VSS.  Sounds like MSFT is plannin to phase out VSS as a stand-alone solution, and bundle it in with the VS Team Foundation suite.

In doing some research and product comparison, it looks like PerForce is the way to go.  Anyone have any thoughts/tips about migrating source control systems?  Sounds like it’s gonna be a fun one!