Meetup talk on CI

I presented an overview of CI in the enterprise at last week’s Large Scale Production Engineering meetup.

It’s been posted on slideshare, check out CI In The Enterprise: Lessons Learned!

vss to svn – 7 steps to svn goodness

Using Subversion for source control is all the rage these days. Everyone’s doing the svn dance. In fact, my most popular blog posting is about vss to svn migration, so at least it’s a hot topic here anyway. I assumed there’s lots of people making the switch out there, and figured I’d write up a few things to think about if you’re considering the move.

Why subversion? Mostly because svn just works. It’s a great little tool for doing what you’d expect of a solid source control system. It’s lightweight, so you can use it just about anywhere. Also – it’s free, which is always nice. There are plenty of other options out there, and each have their merits. This particular post is about the journey from vss to subversion.

1. Why? Are you sure?  Switching source control systems can be a real pain, especially if you’ve got a boatload of source. It’s definitely worth the effort in 99% of the cases, but sometimes the payoff just isn’t worth the disruption. If you’re a heavy Microsoft shop for example, you might want to consider jumping in with both feet and just moving on to the whole Team Foundation thing.

That being said, subversion rocks. Use it. 🙂

2. Preparing the new repo – You’ve got to install subversion somewhere, and create your new repository.  You’ll need to decide if you want to use the svn: or http: protocol to access your repo remotely, most people just tunnel it thru apache over http:.

You’ll want to start off with a nice clean, standardized layout for your new repo.  A safe bet is to use the old “trunk, tags, branches” route, with some default dirs in place (src, web, db, qa, lib, whatever).

If you’re not sure about installing/hosting it yourself, there are a handful of places that’ll do it for ya (dreamhost, cvsdude, … )

3. Exporting the stuff from vss –  The trick here is to do it cleanly.  You’ll most likely end up needing to delete all the .vssver files and whatnot.  Also, decide if you really need the entire history of files in your new repo… that could get hairy. Migrating to svn AND keeping all of your checkin history intact in svn is outside the scope of this posting. We decided we didn’t and kept a backup (read only) copy of the old vss db around “just in case”. You know, in case we ever needed to refer to something 3 or 4 years old.

Guess what? We never looked back. 🙂

4. Importing the stuff to svn – This is where it can get fun.  You’ll want to map your existing vss repository’s directory layout to your shiny new one. I ended up using a little perl script to get them all added into svn correctly. (vss2svn.zip)

5. Updating other various things and people –  You may have some existing systems already pointing to your vss repo, make sure to list them all out and get em updated. We had a handful of build scripts that needed to be update to point to the new repo.

Also, be sure to communicate effectively to your team. You might even need to do a little PR for the effort, so make sure you’re clear on #1 above. Provide your users with ample docs and support info to make the transition easier. Maybe host a couple “svn how to” sessions, to get things moving.

6. Testing This needn’t be too complicated, just walk through a handful of typical use cases. For example, do a few svn checkouts, make sure you can connect and everything’s cool, try a build or 2 from svn, etc.

The key here is to find and address (or figure out workarounds to) the most likely issues, and be ready to help your users get around em.

7. Relax and have a beer You’re off! Now just relax, watch your source get modified, and start forgetting about “vss analyze and fix”

🙂

Advertisement

Build and Release – a match made in heaven?

So I’ve taken to digging a bit deeper into the build side of things, so please don’t be upset if this turns into more of a build automation blog… It’s just where I’m spending a lot of my time lately.

I’ve started to see Release Management as a bit closer to the business side of things, and looking at what I do as more “build and deployment automation”. At my previous job (where I was when I started this blog), I did pretty much everything form managing SVN, to automating builds and deployments, to managing a bunch of servers. Since getting out of Academia and into the “real world” of business, I’ve noticed that these roles tend to be a little more separated.

Now that I’m consulting, I even made a little online resume thingie for your enjoyment – check it out!

http://www.justinlittle.com/

I am available for consulting too! Wouldn’t it be neat to be able to say you hired the guy who “wrote the blog” on release management? 🙂

Right now, I’m working on helping a small software company in Silicon Valley automate the heck out of their build. We’ve got a desktop app written in C#, and we didn’t really have any automated builds at all. It’s pretty cool walking into a place that has a clean slate, and being able to completely build out a system from scratch.

I can’t really say I love working with MSFT products anymore, I think I’ve gotten spoiled with the explicit and transparent nature of working with Linux/Java stuff. It seems like MSTF is trying to do everything for you, which is cool until things start breaking and you need to troubleshoot. OK, that’s my mini MSFT-bashing rant, I’ll stop now, I promise.

So Bill G and the Gang have some pretty interesting little tools together, most notably MSBuild in conjunction with Team Foundation Services. I can’t say I love it, but it does (sort of) work. I pretty much eschewed their whole solution, and just wrapped a call to MSBuild inside my ANT scripts, and then wrapped that in CruiseControl.NET for scheduling of builds and for email notifications on builds events. It all seems to work pretty well, the only real hiccup was getting ANT to play nicely with Team Foundation’s source control.

I just use the tf.exe command line client form an exec inside my ANT script, but it’s not pretty. Mostly cuz doing a “get” makes everything read-only (kinda like vss), and that doesn’t really work for me cuz the build actually produces some artifacts/shuffles around some files. Since I don’t really know what MSBuild is up to behind the scenes, I ended up just making them writable after “get”ing them using hte old “attrib” command. Weak sauce.

Other than that, CruiseControl.NET seems to be pretty happy. We’ve now got nightly builds popping, and all is good in the world. Now if I could only get my hands on a decent GUI based testing tool for WinForms, then I could run a handful of smoke/regression tests after the build happens, so we know we didn’t break anything…

VSS to SVN – How To

I wrote up a simple doc for my team to help smooth the transition from VSS to Subversion. Here’s the text of it, maybe it’ll help you too!

What is SVN?
Subversion (SVN) is our new version control system that is very similar to CVS. It fixes many bugs and limitations of the current version of CVS. We have chosen SVN as their version control system and have installed the SVN server on <your server here> for production use.

How can I connect to Subversion?
There are several open source Subversion clients available on the internet. There is a pretty long list of svn clients and plugins available. The quickest and easiest client to use is TortoiseSVN. For the remainder of this document it is assumed that TortoiseSVN is being used. The following steps outline the process for connecting to the svn server:

1. Download and install the Tortoise SVN client. It can be downloaded at http://tortoisesvn.tigris.org.
2. Install using the default settings.
3. Checkout the project that are interested in checking out.

Note – if you are working on drive over a network, you‘ll need to update a setting in TortoiseSVN to get the icons to show up: Go to Settings -> Look and Feel -> Icon Overlays and check the drive types for which you want to see overlay icons.Be aware that enabling overlays for network drives will slow down TortoiseSVN.

SVN Directory Structure
The following is the current directory structure:

https://your.repo.url – Doc root for production subversion repository

/applications – a listing of all current apps in the repository
-><appname> – application name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of code/db – folders for each database’s SQL scripts
->/<dbname> – database name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of SQL scripts/documentation – space in the repo for storing docs requiring revision control –>/<project name>
->/qa – space in the repo reserved for storing QA-related docs and scripts

It is possible to have other directories under the trunk for other requirements. For example, if a prototype is required for the project, a /prototype directory can be created for this purpose. Similarly, other directories can be created on an as needed basis.You may have noticed we’ve separated out the documentation from the source code. There are several reasons for this:

1. Isolating source from docs is generally a good practice
2. We’d like to reduce user/permissions management overhead
3. We plan to move as much documentation as possible into a document management / collaboration tool in the near future.

We’re trying to maintain a similar tree to what we’ve used in VSS, and make changes only where it makes sense.

Checking out a project
To check out a project from svn using TortoiseSVN, click on “File – SVN Checkout”. Enter the URL of the directory that you would like to check out and the local directory to put it into. For example, to check out app “foo”, the URL would be:

https://your.repo.url/applications/foo/trunk/

Simply enter your password when prompted and the download will begin.

Importing a new project
To import a new project using TortoiseSVN, using the Windows file browser, go to the directory above the one you would like to import and verify it meets the directory requirements specified above under the /trunk directory. Click on the directory to import. Click on “File – TortoiseSVN – import”. Enter the URL of that you want the directory to be imported into. For example, if you are importing a directory called “foo”, the URL to use is:https://your.repo.url/applications/foo/trunk/

Once the file(s) is imported you need to do a checkout to get the project back out of svn so that you can now modify the files.

Transitioning from VSS to SVN
Below are some common tasks we’re used to performing with VSS, and their equivalent in Subversion. SVN does some things fundamentally different from VSS, so it’s not exactly a 1:1 mapping, but this can serve as a guide during the transition:

VSS Command	SVN Command	Difference
GetLatestVersion	Update	The SVN and CVS update will notify if files are pruned from repository so you can remove them from local repository (VSS would not).
Add	Add	Even though they have the same name, there are very big differences. The VSS adds the file directly to the repository whereas in CVS and SVN it is a local note that is made. The file is only added during the commit.
Delete	Delete	Same as add but for delete
Checkin	Commit	These commands pretty much do the same
Checkout	—	There is no edit in SVN, but there is a lock command to lock a file before using it.
—	Checkout	The checkout in SVN is essentially telling SVN “I want to track changes in these files”. You do a SVN checkout when you want to initiate a project.
Undo Checkout	Revert (and unlock if necessary)
ShowDifferences	Compare	In SVN, you need to view the log for the file, and select 2 rpevious revisions. Then you can right-click and view diffs
Label	Copy	A tag in SVN is actually a copy. Therefore it must be copied in the tags section (see previous explanation).
(manual copy)	Copy	In VSS dragging and dropping a branch was the equivalent of a “branch”.

Where to find more information:
More information on Subversion can be obtained from the subversion book.

Quick Build

In my search for an effective tool for managing builds and releases, I’ve decided on QuickBuild.  It’s basically a commercial version of LuntBuild, with some nifty management features added in. 

One of the nicer things about it is the ability to set properties for an app, and have them propagate to different configs for that app (dev, QA, staging, UAT, production), while having the ability to override certain values (say, DB username and password, etc…).

Like CruiseControl, it works well for continuous builds when integrated with your source control system.  Subversion and QuickBuild work together quite nicely.  Though, unlike CruiseControl, QuickBuild allows for “on-demand” deployments, and it’s promotion management features make it a cinch to migrate code from dev -> QA -> staging -> produciton.

I’m currently adding my apps to QuickBuild, and I’ve found it to be really easy to setup, use, and manage.  The end goal is to automate as much of the release process as possible, allowing for more predictable, repeatable, and consistently “problem-free” production releases.

Plus, it certainly would be nice to be able to release apps remotely via a web-based interface.  Now, if I can just figure out how to access QuickBuild from the golf course…
🙂

VSS to Subversion migration

Sorry for the dearth of posts lately, I’ve just been having too much fun with Subversion to post. Bad Release Guy! That, and about a hundred releases or so…
🙂

We have finally *officially* decided to use SVN, rather than Perforce. I’m actually pretty happy about the decision, I’m finding SVN to be a nice, simple, straightforward source control system. It doesn’t have all the “bells and whistles” associated with Perforce or ClearCase, etc., but it is a solid tool and does what a good version control system should.

The setup and install was pretty trivial, not a big deal at all. The only remaining item is to tie this thing up to my local authentication/authorization system. It’s proving to be more of a pain than I had hoped, and I’m not thrilled about just using SSH or Apache’s basic auth to control access to our repository.

I’ve put together some VSS to Subversion migration scripts (in perl) that have worked out quite well. It’s basically just a simple “get latest and add to SVN”, no labels or file history or anything gets moved over, but I’ve noticed the “VSS way” of doing things is not exactly based on best practices anyway. So we’ll just keep our old, VSS repository around in the meantime in read-only mode, in case we need to do some historical reporting.

If you’re interested in checking out my VSS to SVN migration scripts, have a ball!