Using QuickBuild for building and deploying apps is fun!  I’ve got 3 of my apps into QB now, and their build and deployment is almost completely automated.  I’m getting closer and closer to the Holy Grail of release management – the “magic button”.  You know, that button you push that just automagically a) pulls your code from the repo, b) builds and/or copies all dependent jars to the app local folder, c) builds (compiles and packages into a .war) the app and d) (the grand finale) automatically deploys the application to our Weblogic farm.

One thing we had to do (or rather, get the engineers to do!) was standardize on how we use environment-specific variables.  Obviously, for DB connections your best bet is to always use datasources whenever you can.  Then your apps just need to refer to those, rather than using database servername, db name, user, password, all that junk.  Not only does this streamline the migration from dev to qa to prod, but it also abstracts your database connection info, which is nice for several reasons (not the least of which is security).

But what about other variables?  What if your log location is different in dev vs prod?  Or what if you want to send error e-mails to a different address from qa vs prod?  You need to do some kind of variable substitution at build time.  Enter the .tmpl file.

Require (or ask, or beg, or bribe) your engineers to keep all of these “configurable” values in one file – typically an ApplicationContext.xml or something that looks like it.  Then, have them put the values for these variables in a properties file somewhere (app.properties or whatever).  The last step is to actually use a .tmpl file, and dynamically generate that .xml file using this instead.  If no values are passed through to the file from ANT or QuickBuild, then the app will just use the properties specified in the properties file.  But if you do want to pass them through at build time, all you need to do is create a variable for that config in QB (or ANT, if you’re not using QB yet!), and viola! – you’ve got the right stuff where it needs to be.

I wrote up a simple doc for my team to help smooth the transition from VSS to Subversion. Here’s the text of it, maybe it’ll help you too!

What is SVN?
Subversion (SVN) is our new version control system that is very similar to CVS. It fixes many bugs and limitations of the current version of CVS. We have chosen SVN as their version control system and have installed the SVN server on <your server here> for production use.

How can I connect to Subversion?
There are several open source Subversion clients available on the internet. There is a pretty long list of svn clients and plugins available. The quickest and easiest client to use is TortoiseSVN. For the remainder of this document it is assumed that TortoiseSVN is being used. The following steps outline the process for connecting to the svn server:

1. Download and install the Tortoise SVN client. It can be downloaded at http://tortoisesvn.tigris.org.
2. Install using the default settings.
3. Checkout the project that are interested in checking out.

Note – if you are working on drive over a network, you‘ll need to update a setting in TortoiseSVN to get the icons to show up: Go to Settings -> Look and Feel -> Icon Overlays and check the drive types for which you want to see overlay icons.Be aware that enabling overlays for network drives will slow down TortoiseSVN.

SVN Directory Structure
The following is the current directory structure:

https://your.repo.url – Doc root for production subversion repository

/applications – a listing of all current apps in the repository
-><appname> – application name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of code/db – folders for each database’s SQL scripts
->/<dbname> – database name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of SQL scripts/documentation – space in the repo for storing docs requiring revision control –>/<project name>
->/qa – space in the repo reserved for storing QA-related docs and scripts

It is possible to have other directories under the trunk for other requirements. For example, if a prototype is required for the project, a /prototype directory can be created for this purpose. Similarly, other directories can be created on an as needed basis.You may have noticed we’ve separated out the documentation from the source code. There are several reasons for this:

1. Isolating source from docs is generally a good practice
2. We’d like to reduce user/permissions management overhead
3. We plan to move as much documentation as possible into a document management / collaboration tool in the near future.

We’re trying to maintain a similar tree to what we’ve used in VSS, and make changes only where it makes sense.

Checking out a project
To check out a project from svn using TortoiseSVN, click on “File – SVN Checkout”. Enter the URL of the directory that you would like to check out and the local directory to put it into. For example, to check out app “foo”, the URL would be:

https://your.repo.url/applications/foo/trunk/

Simply enter your password when prompted and the download will begin.

Importing a new project
To import a new project using TortoiseSVN, using the Windows file browser, go to the directory above the one you would like to import and verify it meets the directory requirements specified above under the /trunk directory. Click on the directory to import. Click on “File – TortoiseSVN – import”. Enter the URL of that you want the directory to be imported into. For example, if you are importing a directory called “foo”, the URL to use is:https://your.repo.url/applications/foo/trunk/

Once the file(s) is imported you need to do a checkout to get the project back out of svn so that you can now modify the files.

Transitioning from VSS to SVN
Below are some common tasks we’re used to performing with VSS, and their equivalent in Subversion. SVN does some things fundamentally different from VSS, so it’s not exactly a 1:1 mapping, but this can serve as a guide during the transition:

Where to find more information:
More information on Subversion can be obtained from the subversion book.

There seems to be some debate around where in an organization Release Management really belongs.  Is release management a function of engineering, or is it a function of quality assurance or quality control?  A case can be made for both, or neither.

On the one hand, release management is certainly a function of engineering or software development.  What good is a slick piece of code if it can’t be integrated, tracked, and released into the world?

On the other hand, the release manager is the last line of defense for what actually gets deployed.  So in that sense, the release manager must be acutely aware of quality issues.  Obviously, the release engineer can’t be completely in on all the requirements or business needs – that’s outside of his scope.  However, he must be aware of things like system resources, monitoring, logging, etc.  All the things going on behind the scenes to ensure quality of response and uptime.

Release Management as an engineering function

Often times, relatively small software development teams don’t recognize or appreciate the need for a central resource to handle build, deployments, and environment maintenance.  Typically, you’ll find one engineer in the group who’s familiar enough with SCM systems, web services, UNIX/Windows admin, etc to step in and fill the role.  Depending on the size of the team and the complexity of their development processes, this is often sufficient. 

In any case, someone needs to take ownership of these issues, and that someone should have at least some knowledge of complex systems and how applications are built and deployed.

Release Management as a Quality function

The release manager definitely needs to work closely with the QA team, in order to properly plan and allocate resources to test apps before releasing them into production.  In some cases, the release manager can help a QA engineer determine the nature of a bug, and whether it’s an environment/server config issue, or simply a bug in the app.

My humble opinion is that release management sort of “straddles” both engineering and QA, and as such really belongs in either it’s own organizational unit, or could fall in with operations/IT management.  I guess it really comes down to how your team is organized and what the real needs are.

Migrations. We all have em. If you’ve ever rebuilt your personal computer, you’ve had a migration. The act of upgrading a “foundation” system (like an OS or app platform) can be pretty daunting, but needn’t be too much of a pain in the neck. I mean, if you’re upgrading, then you should end up better off on the other side, right? Otherwise, what’s the point?

Having participated in dozens of migrations of varying kinds, I’ve discovered that release management can help significantly reduce the overall risk and impact of any widespread upgrades or migrations.

OS upgrades – upgrading your Windows or Unix server to the latest operating system.

Hardware upgrades – upgrading server hardware resources (CPU, disk, networking).

App Server upgrades – upgrading your application server and/or app platform. Examples include simply upgrading to the latest distribution of Apache, Jrun, Tomcat, etc., or a widely deployed enterprise systems/application (PeopleSoft, Exchange, or a Content Management System).

Miscellaneous migrations – migrating source control systems, CM systems, App Platforms (say, moving a Tomcat app to Weblogic).

Release Management can help with all of these types of migrations in several ways. Firstly, the release manager is well positioned to provide essential environment information such as inter-application, inter-server and other various dependencies.

The release manager can also provide insight into usage patterns, and can help plan for and communicate about coming migrations.

Lastly, the release manager can assist with coordinating all of the various players involved in a wide-scale migration and/or upgrade. From working with engineers to make any required app updates, to following up with Sys Admins to validate application and/or server availability post-upgrade, the release manager can help make sure that everyone has what they need to do their part.

Nobody likes to delay a scheduled release. However, risk management is a big part of release management, and sometimes it just makes sense to abort and reschedule. When deploying applications to a production environment, introducing unknowns can potentially have widespread impact. In some cases, the negative impact can far outweigh the benefits of troubleshooting and tweaking to push an application update out.

Following are some tips for helping to mitigate the possible negative consequences of a failed production deployment. This posting primarily applies to deploying web based applications to a production server environment hosting multiple applications.

Use a maintenance window – A key step in mitigating risk and minimizing user impact is establishing a “maintenance window”. Identify when server traffic is typically low, and block out a time frame for deploying applications or application updates. Regardless of whether the build/deployment/release is automated or not, it’s useful to have a set time when changes to the system occur. A release window adds a certain level of objectivity and predictability to system changes. It offers a boundary to risk exposure in that risk is minimized by literally limiting the scope of the time you’re systems are being changed. An additional benefit is the ability to notify your users that this time is when system updates occur, and they needn’t worry if they experience issues while attempting to use online resources. Depending on your user base, this can be helpful in reducing unnecessary trouble tickets or bug reports.

Ideally, most everything would be automated, happen almost instantly, and you’d have elegant, intelligent maintenance pages/handlers to gracefully redirect users to a “this application is being updated” page, but that’s a topic for another posting. Then again, ideally, app deployments would just auto-magically happen all by themselves, all the dependencies would always be in place, and config files would never mysteriously have the QA values still in them (“I could have sworn I ran the ANT task to update that file..”). Wouldn’t that be cool?

OK, so you’ve got a maintenance window – now what? How to decide when to investigate, troubleshoot, and make the darn application work right whether it wants to or not? When to say “Uncle”, back out of the release, and hang your head in shame?

Estimate time to complete – Accurately estimating how long a particular task or automated job takes to complete is essential to managing smooth application deployments. Deciding when to back out all comes down to accurate estimates of how long tasks take to complete. Consider the following when coming up with estimates: how long does the actual deployment take if nothing goes awry? how long would it take to double check some basic dependencies or configuration files? How long would it take to back out and undo all changes and restore the system to it’s pre-release state (worst case scenario)?

The rest is simple – if you’re deep the in release window and things are bumpy, ask yourself, “Can I still backout of this before the window closes?”. This is your “point of no return”. If the answer is yes, keep troubleshooting and get things sorted out. If it’s a close call, take your medicine and back out.

If you have somehow managed to get past the point of no return, well, you better make sure you get everything running ok (without introducing new problems!) before the window closes, or else you might as well just start updating your resume.

It looks like we’re finally gonna get off of VSS entirely and just move on over to Perforce.  Yay!  I’ve never actually used Perforce, but from what I’ve heard and read on their site, it looks like it’s gonna be sweet.  Plus, just about everyone who’s anyone is using Perforce or CVS, so it can’t be that bad.

Gonna hafta start mapping out the migration path from VSS to Perforce.  Looks pretty straightforward, especially since they’ve been kind enough to provide some VSS to Perforce migration scripts.

More later, gotta go watch the Shield.

While ANT definitely is a great tool, and it’s benefits over MAKE well proven by now, it does have some limitations when it comes to scripting complex build and deploy tasks.  Well, not so much ‘limitations’, cuz it’s pretty open to extend it any way you could want. 

But anyone who’s ever tried to do something fancier than “grab code, compile, package into .ear file”, has had to get real creative with XML to get it to act like a scripting language.

Groovy looks like it could help make life easier for us build/release managers.

Article in JavaWorld… 

It’s even got an ANT Task.

App migrations can be a real headache.  They don’t have to be, but they often are.  Especially when moving from one platform to another.  This is the story of such a migration…

The powers that be in my organization have decided to move away from MSFT technology, and get into the whole Java/Linux thing.  While I’m all for learning new stuff (especially Java/*nix stuff), this is turning out to be a pretty major effort.  We’ve currently got about 80 or so apps (ASP or .NET) on a handful of IIS servers.  All of these apps will eventually need to either be rewritten or ported to run in a java environment.  Can you say job security?

We decided on Weblogic as our new Java app platform (with Apache out front).  From what I can tell, it’s pretty sweet.  As a step in the direction we’re headed, we decided to use Apache to front our IIS servers, so we can just proxy our IIS apps though a unified web server.  Apache is cool.  Using Apache as the out-facing web server, we can then proxy app request to either IIS, or Weblogic, or that pesky little Tomcat service we’ve got runnin – all through one unified namespace.  Neat!

This is just the introduction to the app platform migration saga, and I have a feeling it’ll turn into a pretty regular series of articles for a while.  It sort of ties into the VSS / Source control discussions, since part of the migration is to create an entire new development environment for our engineers.

And here’s where it gets real interesting – we don’t have very thorough docs on all of our apps and their depelndencies (I know, shocking!), since some of them were written years ago, when we didn’t have much in the way of processes and proper documentation.  So trying to figure out how to map all the inter-app and server dependencies has been quite a challenge.

One really cool thing about the whole effort is that I’m getting to play with Linux, Apache, and Weblogic a lot more, which is fun!

More to follow…

Stumbled acrosss an article with some nice examples of how to automate builds. Check it out:

http://www.wrox.com/WileyCDA/Section/id-131031.html

As systems, processes, and resources become more distributed, they invariably become more specialized.  Furthermore, software products (especially web-based apps) are typically in an ongoing cycle of development, testing, and release.  Add to this an evolution in the platforms on which these systems run, and you’ve got a lot of moving pieces that must fit together to guarantee the success and long term value of a product or project.

The need exists for a resource to oversee the development, testing, deployment, and support of these systems.  This resource must have a general knowledge of every aspect of the SDLC, various operating systems and software application platforms, and an understanding of different business functions and perspectives.  Release Management addresses this need.

Release Management is the discipline (art, science, strategy, headache) of building, packaging, and deploying software for consumption.

A Release Manager is:

• Architect – the release manager helps to identify, create, and implement processes or products to efficiently manage the release of code.
• Gatekeeper – a release manager “holds the keys” to production systems and takes responsibility for their quality and availability.
• Facilitator – a release manager serves as a liaison between varying business units and users to guarantee smooth and timely delivery of software products or updates.

There’s an apparent scarcity of information out there on release management in general (at least there is a tag for it at Technorati).  There’s stuff about change management, configuration management, defect tracking, source control, etc., etc.  But there’s not a lot of general info about the overall process of managing code releases.  I’d like to change that.  I’ll be posting on various release related issues as often as I can.

If you’ve ever had issues with building, deploying or releasing code for consumption, I’d like to hear about it.  This is the place for it!