Intro to Continuous Integration

OK, so it’s been a while since I posted… yes, I am a bad, bad man. But, in my defense, I did start a new job so I’ve been kinda busy tryin to figure out how to do some fancy new tricks. My new gig is pretty far removed from production releases (and the support required for that – yay!!), my primary focus now is on build automation and continuous integration.

I’m no longer working in the academic space, I’ve entered the “corporate world”. Fun stuff. It’s actually a nice change – things move a bit faster, and we get the opportunity to work with some cooler, more sophisticated stuff. For instance, Continuous Integration.

While CI may not necessarily be a function of Release Management, it certainly is a factor when trying to get working software out the door. If you’re not implementing CI, you should be. It’s just that simple. Check out the penultimate article on CI, and then tell me you couldn’t benefit from it.

The basic idea is to integrate code changes as early and as often as humanly possible. In fact, more often than is humanly possible. Set up a build server and have builds run all day long – you’ll be glad you did when it comes time to release your app and you know with a high level of confidence that everything is working as it should be. I had started getting some of this in place in my last gig using QuickBuild, but the place I’m working at now is all about CruiseControl.

CruiseControl is pretty much the “grand-daddy of CI” servers, and it’s really quite nice to work with. I even heard that ThoughtWorks is planning to roll out an even better version in the coming months, so watch for that!

I’ve seen just how to use CI properly, and I can’t imagine an app of any scale being developed without it. Though, note that without solid unit tests (automated unit tests!), CI doesn’t really buy you much (tho it’s still better than not having it – at least you’ll catch compilatin errors!).

More to come on this topic, but I just wanted to get something up – I feel like I’ve been neglecting my readers for far too long.

Anyone have experience with CI? How has it worked on your project? What pitfalls have you run into that we should all watch out for?

Update: Oddly, WordPress won’t let peopl epost URLs in comments, so I’m posting somehting for a buddy.  The gang at ThoughWorks have some exciting stuff goin on wtih Cruise, check it out:

http://studios.thoughtworks.com/cruise

A useful tip for variable substitution

Using QuickBuild for building and deploying apps is fun!  I’ve got 3 of my apps into QB now, and their build and deployment is almost completely automated.  I’m getting closer and closer to the Holy Grail of release management – the “magic button”.  You know, that button you push that just automagically a) pulls your code from the repo, b) builds and/or copies all dependent jars to the app local folder, c) builds (compiles and packages into a .war) the app and d) (the grand finale) automatically deploys the application to our Weblogic farm.

One thing we had to do (or rather, get the engineers to do!) was standardize on how we use environment-specific variables.  Obviously, for DB connections your best bet is to always use datasources whenever you can.  Then your apps just need to refer to those, rather than using database servername, db name, user, password, all that junk.  Not only does this streamline the migration from dev to qa to prod, but it also abstracts your database connection info, which is nice for several reasons (not the least of which is security).

But what about other variables?  What if your log location is different in dev vs prod?  Or what if you want to send error e-mails to a different address from qa vs prod?  You need to do some kind of variable substitution at build time.  Enter the .tmpl file.

Require (or ask, or beg, or bribe) your engineers to keep all of these “configurable” values in one file – typically an ApplicationContext.xml or something that looks like it.  Then, have them put the values for these variables in a properties file somewhere (app.properties or whatever).  The last step is to actually use a .tmpl file, and dynamically generate that .xml file using this instead.  If no values are passed through to the file from ANT or QuickBuild, then the app will just use the properties specified in the properties file.  But if you do want to pass them through at build time, all you need to do is create a variable for that config in QB (or ANT, if you’re not using QB yet!), and viola! – you’ve got the right stuff where it needs to be.

VSS to SVN – How To

I wrote up a simple doc for my team to help smooth the transition from VSS to Subversion. Here’s the text of it, maybe it’ll help you too!

What is SVN?
Subversion (SVN) is our new version control system that is very similar to CVS. It fixes many bugs and limitations of the current version of CVS. We have chosen SVN as their version control system and have installed the SVN server on <your server here> for production use.

How can I connect to Subversion?
There are several open source Subversion clients available on the internet. There is a pretty long list of svn clients and plugins available. The quickest and easiest client to use is TortoiseSVN. For the remainder of this document it is assumed that TortoiseSVN is being used. The following steps outline the process for connecting to the svn server:

1. Download and install the Tortoise SVN client. It can be downloaded at http://tortoisesvn.tigris.org.
2. Install using the default settings.
3. Checkout the project that are interested in checking out.

Note – if you are working on drive over a network, you‘ll need to update a setting in TortoiseSVN to get the icons to show up: Go to Settings -> Look and Feel -> Icon Overlays and check the drive types for which you want to see overlay icons.Be aware that enabling overlays for network drives will slow down TortoiseSVN.

SVN Directory Structure
The following is the current directory structure:

https://your.repo.url – Doc root for production subversion repository

/applications – a listing of all current apps in the repository
-><appname> – application name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of code/db – folders for each database’s SQL scripts
->/<dbname> – database name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of SQL scripts/documentation – space in the repo for storing docs requiring revision control –>/<project name>
->/qa – space in the repo reserved for storing QA-related docs and scripts

It is possible to have other directories under the trunk for other requirements. For example, if a prototype is required for the project, a /prototype directory can be created for this purpose. Similarly, other directories can be created on an as needed basis.You may have noticed we’ve separated out the documentation from the source code. There are several reasons for this:

1. Isolating source from docs is generally a good practice
2. We’d like to reduce user/permissions management overhead
3. We plan to move as much documentation as possible into a document management / collaboration tool in the near future.

We’re trying to maintain a similar tree to what we’ve used in VSS, and make changes only where it makes sense.

Checking out a project
To check out a project from svn using TortoiseSVN, click on “File – SVN Checkout”. Enter the URL of the directory that you would like to check out and the local directory to put it into. For example, to check out app “foo”, the URL would be:

https://your.repo.url/applications/foo/trunk/

Simply enter your password when prompted and the download will begin.

Importing a new project
To import a new project using TortoiseSVN, using the Windows file browser, go to the directory above the one you would like to import and verify it meets the directory requirements specified above under the /trunk directory. Click on the directory to import. Click on “File – TortoiseSVN – import”. Enter the URL of that you want the directory to be imported into. For example, if you are importing a directory called “foo”, the URL to use is:https://your.repo.url/applications/foo/trunk/

Once the file(s) is imported you need to do a checkout to get the project back out of svn so that you can now modify the files.

Transitioning from VSS to SVN
Below are some common tasks we’re used to performing with VSS, and their equivalent in Subversion. SVN does some things fundamentally different from VSS, so it’s not exactly a 1:1 mapping, but this can serve as a guide during the transition:

VSS Command	SVN Command	Difference
GetLatestVersion	Update	The SVN and CVS update will notify if files are pruned from repository so you can remove them from local repository (VSS would not).
Add	Add	Even though they have the same name, there are very big differences. The VSS adds the file directly to the repository whereas in CVS and SVN it is a local note that is made. The file is only added during the commit.
Delete	Delete	Same as add but for delete
Checkin	Commit	These commands pretty much do the same
Checkout	–	There is no edit in SVN, but there is a lock command to lock a file before using it.
–	Checkout	The checkout in SVN is essentially telling SVN “I want to track changes in these files”. You do a SVN checkout when you want to initiate a project.
Undo Checkout	Revert (and unlock if necessary)
ShowDifferences	Compare	In SVN, you need to view the log for the file, and select 2 rpevious revisions. Then you can right-click and view diffs
Label	Copy	A tag in SVN is actually a copy. Therefore it must be copied in the tags section (see previous explanation).
(manual copy)	Copy	In VSS dragging and dropping a branch was the equivalent of a “branch”.

Where to find more information:
More information on Subversion can be obtained from the subversion book.

Quick Build

In my search for an effective tool for managing builds and releases, I’ve decided on QuickBuild.  It’s basically a commercial version of LuntBuild, with some nifty management features added in. 

One of the nicer things about it is the ability to set properties for an app, and have them propagate to different configs for that app (dev, QA, staging, UAT, production), while having the ability to override certain values (say, DB username and password, etc…).

Like CruiseControl, it works well for continuous builds when integrated with your source control system.  Subversion and QuickBuild work together quite nicely.  Though, unlike CruiseControl, QuickBuild allows for “on-demand” deployments, and it’s promotion management features make it a cinch to migrate code from dev -> QA -> staging -> produciton.

I’m currently adding my apps to QuickBuild, and I’ve found it to be really easy to setup, use, and manage.  The end goal is to automate as much of the release process as possible, allowing for more predictable, repeatable, and consistently “problem-free” production releases.

Plus, it certainly would be nice to be able to release apps remotely via a web-based interface.  Now, if I can just figure out how to access QuickBuild from the golf course…

VSS to Subversion migration

Sorry for the dearth of posts lately, I’ve just been having too much fun with Subversion to post. Bad Release Guy! That, and about a hundred releases or so…

We have finally *officially* decided to use SVN, rather than Perforce. I’m actually pretty happy about the decision, I’m finding SVN to be a nice, simple, straightforward source control system. It doesn’t have all the “bells and whistles” associated with Perforce or ClearCase, etc., but it is a solid tool and does what a good version control system should.

The setup and install was pretty trivial, not a big deal at all. The only remaining item is to tie this thing up to my local authentication/authorization system. It’s proving to be more of a pain than I had hoped, and I’m not thrilled about just using SSH or Apache’s basic auth to control access to our repository.

I’ve put together some VSS to Subversion migration scripts (in perl) that have worked out quite well. It’s basically just a simple “get latest and add to SVN”, no labels or file history or anything gets moved over, but I’ve noticed the “VSS way” of doing things is not exactly based on best practices anyway. So we’ll just keep our old, VSS repository around in the meantime in read-only mode, in case we need to do some historical reporting.

If you’re interested in checking out my VSS to SVN migration scripts, have a ball!

Release Management – QA or engineering?

There seems to be some debate around where in an organization Release Management really belongs.  Is release management a function of engineering, or is it a function of quality assurance or quality control?  A case can be made for both, or neither.

On the one hand, release management is certainly a function of engineering or software development.  What good is a slick piece of code if it can’t be integrated, tracked, and released into the world?

On the other hand, the release manager is the last line of defense for what actually gets deployed.  So in that sense, the release manager must be acutely aware of quality issues.  Obviously, the release engineer can’t be completely in on all the requirements or business needs – that’s outside of his scope.  However, he must be aware of things like system resources, monitoring, logging, etc.  All the things going on behind the scenes to ensure quality of response and uptime.

Release Management as an engineering function

Often times, relatively small software development teams don’t recognize or appreciate the need for a central resource to handle build, deployments, and environment maintenance.  Typically, you’ll find one engineer in the group who’s familiar enough with SCM systems, web services, UNIX/Windows admin, etc to step in and fill the role.  Depending on the size of the team and the complexity of their development processes, this is often sufficient. 

In any case, someone needs to take ownership of these issues, and that someone should have at least some knowledge of complex systems and how applications are built and deployed.

Release Management as a Quality function

The release manager definitely needs to work closely with the QA team, in order to properly plan and allocate resources to test apps before releasing them into production.  In some cases, the release manager can help a QA engineer determine the nature of a bug, and whether it’s an environment/server config issue, or simply a bug in the app.

My humble opinion is that release management sort of “straddles” both engineering and QA, and as such really belongs in either it’s own organizational unit, or could fall in with operations/IT management.  I guess it really comes down to how your team is organized and what the real needs are.

More migrations than waterfowl

Migrations. We all have em. If you’ve ever rebuilt your personal computer, you’ve had a migration. The act of upgrading a “foundation” system (like an OS or app platform) can be pretty daunting, but needn’t be too much of a pain in the neck. I mean, if you’re upgrading, then you should end up better off on the other side, right? Otherwise, what’s the point?

Having participated in dozens of migrations of varying kinds, I’ve discovered that release management can help significantly reduce the overall risk and impact of any widespread upgrades or migrations.

OS upgrades – upgrading your Windows or Unix server to the latest operating system.

Hardware upgrades – upgrading server hardware resources (CPU, disk, networking).

App Server upgrades – upgrading your application server and/or app platform. Examples include simply upgrading to the latest distribution of Apache, Jrun, Tomcat, etc., or a widely deployed enterprise systems/application (PeopleSoft, Exchange, or a Content Management System).

Miscellaneous migrations – migrating source control systems, CM systems, App Platforms (say, moving a Tomcat app to Weblogic).

Release Management can help with all of these types of migrations in several ways. Firstly, the release manager is well positioned to provide essential environment information such as inter-application, inter-server and other various dependencies.

The release manager can also provide insight into usage patterns, and can help plan for and communicate about coming migrations.

Lastly, the release manager can assist with coordinating all of the various players involved in a wide-scale migration and/or upgrade. From working with engineers to make any required app updates, to following up with Sys Admins to validate application and/or server availability post-upgrade, the release manager can help make sure that everyone has what they need to do their part.

Backing out of a release

Nobody likes to delay a scheduled release. However, risk management is a big part of release management, and sometimes it just makes sense to abort and reschedule. When deploying applications to a production environment, introducing unknowns can potentially have widespread impact. In some cases, the negative impact can far outweigh the benefits of troubleshooting and tweaking to push an application update out.

Following are some tips for helping to mitigate the possible negative consequences of a failed production deployment. This posting primarily applies to deploying web based applications to a production server environment hosting multiple applications.

Use a maintenance window – A key step in mitigating risk and minimizing user impact is establishing a “maintenance window”. Identify when server traffic is typically low, and block out a time frame for deploying applications or application updates. Regardless of whether the build/deployment/release is automated or not, it’s useful to have a set time when changes to the system occur. A release window adds a certain level of objectivity and predictability to system changes. It offers a boundary to risk exposure in that risk is minimized by literally limiting the scope of the time you’re systems are being changed. An additional benefit is the ability to notify your users that this time is when system updates occur, and they needn’t worry if they experience issues while attempting to use online resources. Depending on your user base, this can be helpful in reducing unnecessary trouble tickets or bug reports.

Ideally, most everything would be automated, happen almost instantly, and you’d have elegant, intelligent maintenance pages/handlers to gracefully redirect users to a “this application is being updated” page, but that’s a topic for another posting. Then again, ideally, app deployments would just auto-magically happen all by themselves, all the dependencies would always be in place, and config files would never mysteriously have the QA values still in them (“I could have sworn I ran the ANT task to update that file..”). Wouldn’t that be cool?

OK, so you’ve got a maintenance window – now what? How to decide when to investigate, troubleshoot, and make the darn application work right whether it wants to or not? When to say “Uncle”, back out of the release, and hang your head in shame?

Estimate time to complete – Accurately estimating how long a particular task or automated job takes to complete is essential to managing smooth application deployments. Deciding when to back out all comes down to accurate estimates of how long tasks take to complete. Consider the following when coming up with estimates: how long does the actual deployment take if nothing goes awry? how long would it take to double check some basic dependencies or configuration files? How long would it take to back out and undo all changes and restore the system to it’s pre-release state (worst case scenario)?

The rest is simple – if you’re deep the in release window and things are bumpy, ask yourself, “Can I still backout of this before the window closes?”. This is your “point of no return”. If the answer is yes, keep troubleshooting and get things sorted out. If it’s a close call, take your medicine and back out.

If you have somehow managed to get past the point of no return, well, you better make sure you get everything running ok (without introducing new problems!) before the window closes, or else you might as well just start updating your resume.

Subversion everywhere…

OK, so maybe we won’t use Perforce.  A mischevious consultant casually mentioned that a lot of his clients use Subversion, and essentially recommended that we use it too.  So now the management types have decied we need to evaluate and consider Subversion.  Fun.  We’ve been evaulating source control tools for about a year now, and we were sooo close to finally actually making a decision.  But alas, the saga continues.

It looks like Subversion is basically CVS with some improvements.  It’s all open source, which I suppose is kind of neat.  But along with open source comes all sorts of fun little troubleshooting.  At least with a commercial vendor, they’ll help us get things set up, and support us if we need it.  Of course, that kind of thing comes with a hefty price tag, and, well, you know, it’s not like our source code integrity is really a big deal or anything.  Certainly not worth spending a few grand to make things better.  Nah…

Also, the other fun thing is trying to figure out how to get our stuff from VSS into Subversion, while keeping all our labels and history and all that jazz intact.

Well, the good side of all this is:  we’ve thought this thing through to death, and whatever we do decide on, we’ll never be able to go back and say “oh, why didn’t we consider that?”.  It never hurts to be thorough when choosing a toolset you’re planning to have for years and years to come.

Goodbye VSS, Hello Perforce!

It looks like we’re finally gonna get off of VSS entirely and just move on over to Perforce.  Yay!  I’ve never actually used Perforce, but from what I’ve heard and read on their site, it looks like it’s gonna be sweet.  Plus, just about everyone who’s anyone is using Perforce or CVS, so it can’t be that bad.

Gonna hafta start mapping out the migration path from VSS to Perforce.  Looks pretty straightforward, especially since they’ve been kind enough to provide some VSS to Perforce migration scripts.

More later, gotta go watch the Shield.