Software as an organism

I hadn’t realized quite how fitting my tagline was when I started this blog.  The “care and feeding” of software is one of the main roles of a release manager, or anyone involved in managing change to software for that matter.  It occurred to me today, that software has a lot in common with living things, more and more every year.

“What?!” you ask?.  How is a web based app like a living organism you wonder?  Well, there’s a lot of similarities, and it makes for a good analogy.  We’re decades into software engineering now, and we’ve discovered a lot of stuff along the way.

One key development I’ve noticed is that more and more, organizations (both software consumers and creators) are beginning to realize that developing apps is less like mechanical engineering and more like giving birth.  I’ve even heard people use giving birth as an analogy, and we’ve all heard some kind of app referred to as “my baby”.  It’s no longer an exercise of “identify requirements, design, code, test, release” in isolation – it’s a nearly never-ending repetition of this cycle.  Agile methodologies are compelling because they acknowledge this at the outset of a project, and cater to this reality.  Software is never “done”.  If it is, so is the company that made it…

Mechanical or even electrical engineering approaches are not really suited to software.  Sure, they provide a framework for getting things done, but there’s a key difference between what’s typically been “engineering” and the engineering of software solutions.

Engineering: the art or science of making practical application of the knowledge of pure sciences, as physics or chemistry, as in the construction of engines, bridges, buildings, mines, ships, and chemical plants.

Rarely do people start off to develop some kind of app, thinking they’ll “nail it” in version 1.0, and never need to spend more time/energy on it.  Certainly, there are phases in the development of an app, as there are in the life of an organism.  So while it is an ongoing cycle (the SDLC), there is a sort of linear path that an app goes through, that’s made up of each of these iterations.

In this 5 part series, I’ll ponder the following topics:

1. Definition of an “application”

2. Comparison of an app with an organism

3. Trends in software design, development

4. Release manager, build engineer as doctor, triage nurse

5. What does it all mean?

Advertisement

Hyperlink Legal goes live!

Hyperlink Legal has just launched their website. They’re a small company, specializing in creating hyperlinks in PDF docs for the legal industry. Check em out!

www.hyperlinklegal.com

Shmoop it!

Shmoop has just been released to the general public, check it out!  Shmoop is a learning aid, providing tons of awesome content about History, Literature, Poetry, and more to come.

It’s still in beta, so there’s bound to be some stuff missing… but it’s an swesome start!  I wish this was around when I was in school!

vss to svn – 7 steps to svn goodness

Using Subversion for source control is all the rage these days. Everyone’s doing the svn dance. In fact, my most popular blog posting is about vss to svn migration, so at least it’s a hot topic here anyway. I assumed there’s lots of people making the switch out there, and figured I’d write up a few things to think about if you’re considering the move.

Why subversion? Mostly because svn just works. It’s a great little tool for doing what you’d expect of a solid source control system. It’s lightweight, so you can use it just about anywhere. Also – it’s free, which is always nice. There are plenty of other options out there, and each have their merits. This particular post is about the journey from vss to subversion.

1. Why? Are you sure?  Switching source control systems can be a real pain, especially if you’ve got a boatload of source. It’s definitely worth the effort in 99% of the cases, but sometimes the payoff just isn’t worth the disruption. If you’re a heavy Microsoft shop for example, you might want to consider jumping in with both feet and just moving on to the whole Team Foundation thing.

That being said, subversion rocks. Use it. 🙂

2. Preparing the new repo – You’ve got to install subversion somewhere, and create your new repository.  You’ll need to decide if you want to use the svn: or http: protocol to access your repo remotely, most people just tunnel it thru apache over http:.

You’ll want to start off with a nice clean, standardized layout for your new repo.  A safe bet is to use the old “trunk, tags, branches” route, with some default dirs in place (src, web, db, qa, lib, whatever).

If you’re not sure about installing/hosting it yourself, there are a handful of places that’ll do it for ya (dreamhost, cvsdude, … )

3. Exporting the stuff from vss –  The trick here is to do it cleanly.  You’ll most likely end up needing to delete all the .vssver files and whatnot.  Also, decide if you really need the entire history of files in your new repo… that could get hairy. Migrating to svn AND keeping all of your checkin history intact in svn is outside the scope of this posting. We decided we didn’t and kept a backup (read only) copy of the old vss db around “just in case”. You know, in case we ever needed to refer to something 3 or 4 years old.

Guess what? We never looked back. 🙂

4. Importing the stuff to svn – This is where it can get fun.  You’ll want to map your existing vss repository’s directory layout to your shiny new one. I ended up using a little perl script to get them all added into svn correctly. (vss2svn.zip)

5. Updating other various things and people –  You may have some existing systems already pointing to your vss repo, make sure to list them all out and get em updated. We had a handful of build scripts that needed to be update to point to the new repo.

Also, be sure to communicate effectively to your team. You might even need to do a little PR for the effort, so make sure you’re clear on #1 above. Provide your users with ample docs and support info to make the transition easier. Maybe host a couple “svn how to” sessions, to get things moving.

6. Testing This needn’t be too complicated, just walk through a handful of typical use cases. For example, do a few svn checkouts, make sure you can connect and everything’s cool, try a build or 2 from svn, etc.

The key here is to find and address (or figure out workarounds to) the most likely issues, and be ready to help your users get around em.

7. Relax and have a beer You’re off! Now just relax, watch your source get modified, and start forgetting about “vss analyze and fix”

🙂

Build and Release – a match made in heaven?

So I’ve taken to digging a bit deeper into the build side of things, so please don’t be upset if this turns into more of a build automation blog… It’s just where I’m spending a lot of my time lately.

I’ve started to see Release Management as a bit closer to the business side of things, and looking at what I do as more “build and deployment automation”. At my previous job (where I was when I started this blog), I did pretty much everything form managing SVN, to automating builds and deployments, to managing a bunch of servers. Since getting out of Academia and into the “real world” of business, I’ve noticed that these roles tend to be a little more separated.

Now that I’m consulting, I even made a little online resume thingie for your enjoyment – check it out!

http://www.justinlittle.com/

I am available for consulting too! Wouldn’t it be neat to be able to say you hired the guy who “wrote the blog” on release management? 🙂

Right now, I’m working on helping a small software company in Silicon Valley automate the heck out of their build. We’ve got a desktop app written in C#, and we didn’t really have any automated builds at all. It’s pretty cool walking into a place that has a clean slate, and being able to completely build out a system from scratch.

I can’t really say I love working with MSFT products anymore, I think I’ve gotten spoiled with the explicit and transparent nature of working with Linux/Java stuff. It seems like MSTF is trying to do everything for you, which is cool until things start breaking and you need to troubleshoot. OK, that’s my mini MSFT-bashing rant, I’ll stop now, I promise.

So Bill G and the Gang have some pretty interesting little tools together, most notably MSBuild in conjunction with Team Foundation Services. I can’t say I love it, but it does (sort of) work. I pretty much eschewed their whole solution, and just wrapped a call to MSBuild inside my ANT scripts, and then wrapped that in CruiseControl.NET for scheduling of builds and for email notifications on builds events. It all seems to work pretty well, the only real hiccup was getting ANT to play nicely with Team Foundation’s source control.

I just use the tf.exe command line client form an exec inside my ANT script, but it’s not pretty. Mostly cuz doing a “get” makes everything read-only (kinda like vss), and that doesn’t really work for me cuz the build actually produces some artifacts/shuffles around some files. Since I don’t really know what MSBuild is up to behind the scenes, I ended up just making them writable after “get”ing them using hte old “attrib” command. Weak sauce.

Other than that, CruiseControl.NET seems to be pretty happy. We’ve now got nightly builds popping, and all is good in the world. Now if I could only get my hands on a decent GUI based testing tool for WinForms, then I could run a handful of smoke/regression tests after the build happens, so we know we didn’t break anything…

Intro to Continuous Integration

OK, so it’s been a while since I posted… yes, I am a bad, bad man. But, in my defense, I did start a new job so I’ve been kinda busy tryin to figure out how to do some fancy new tricks. My new gig is pretty far removed from production releases (and the support required for that – yay!!), my primary focus now is on build automation and continuous integration.

I’m no longer working in the academic space, I’ve entered the “corporate world”. Fun stuff. It’s actually a nice change – things move a bit faster, and we get the opportunity to work with some cooler, more sophisticated stuff. For instance, Continuous Integration.

While CI may not necessarily be a function of Release Management, it certainly is a factor when trying to get working software out the door. If you’re not implementing CI, you should be. It’s just that simple. Check out the penultimate article on CI, and then tell me you couldn’t benefit from it.

The basic idea is to integrate code changes as early and as often as humanly possible. In fact, more often than is humanly possible. Set up a build server and have builds run all day long – you’ll be glad you did when it comes time to release your app and you know with a high level of confidence that everything is working as it should be. I had started getting some of this in place in my last gig using QuickBuild, but the place I’m working at now is all about CruiseControl.

CruiseControl is pretty much the “grand-daddy of CI” servers, and it’s really quite nice to work with. I even heard that ThoughtWorks is planning to roll out an even better version in the coming months, so watch for that!

I’ve seen just how to use CI properly, and I can’t imagine an app of any scale being developed without it. Though, note that without solid unit tests (automated unit tests!), CI doesn’t really buy you much (tho it’s still better than not having it – at least you’ll catch compilatin errors!).

More to come on this topic, but I just wanted to get something up – I feel like I’ve been neglecting my readers for far too long.

Anyone have experience with CI? How has it worked on your project? What pitfalls have you run into that we should all watch out for?

Update: Oddly, WordPress won’t let peopl epost URLs in comments, so I’m posting somehting for a buddy.  The gang at ThoughWorks have some exciting stuff goin on wtih Cruise, check it out:

http://studios.thoughtworks.com/cruise

A useful tip for variable substitution

Using QuickBuild for building and deploying apps is fun!  I’ve got 3 of my apps into QB now, and their build and deployment is almost completely automated.  I’m getting closer and closer to the Holy Grail of release management – the “magic button”.  You know, that button you push that just automagically a) pulls your code from the repo, b) builds and/or copies all dependent jars to the app local folder, c) builds (compiles and packages into a .war) the app and d) (the grand finale) automatically deploys the application to our Weblogic farm.

One thing we had to do (or rather, get the engineers to do!) was standardize on how we use environment-specific variables.  Obviously, for DB connections your best bet is to always use datasources whenever you can.  Then your apps just need to refer to those, rather than using database servername, db name, user, password, all that junk.  Not only does this streamline the migration from dev to qa to prod, but it also abstracts your database connection info, which is nice for several reasons (not the least of which is security).

But what about other variables?  What if your log location is different in dev vs prod?  Or what if you want to send error e-mails to a different address from qa vs prod?  You need to do some kind of variable substitution at build time.  Enter the .tmpl file.

Require (or ask, or beg, or bribe) your engineers to keep all of these “configurable” values in one file – typically an ApplicationContext.xml or something that looks like it.  Then, have them put the values for these variables in a properties file somewhere (app.properties or whatever).  The last step is to actually use a .tmpl file, and dynamically generate that .xml file using this instead.  If no values are passed through to the file from ANT or QuickBuild, then the app will just use the properties specified in the properties file.  But if you do want to pass them through at build time, all you need to do is create a variable for that config in QB (or ANT, if you’re not using QB yet!), and viola! – you’ve got the right stuff where it needs to be.

VSS to SVN – How To

I wrote up a simple doc for my team to help smooth the transition from VSS to Subversion. Here’s the text of it, maybe it’ll help you too!

What is SVN?
Subversion (SVN) is our new version control system that is very similar to CVS. It fixes many bugs and limitations of the current version of CVS. We have chosen SVN as their version control system and have installed the SVN server on <your server here> for production use.

How can I connect to Subversion?
There are several open source Subversion clients available on the internet. There is a pretty long list of svn clients and plugins available. The quickest and easiest client to use is TortoiseSVN. For the remainder of this document it is assumed that TortoiseSVN is being used. The following steps outline the process for connecting to the svn server:

1. Download and install the Tortoise SVN client. It can be downloaded at http://tortoisesvn.tigris.org.
2. Install using the default settings.
3. Checkout the project that are interested in checking out.

Note – if you are working on drive over a network, you‘ll need to update a setting in TortoiseSVN to get the icons to show up: Go to Settings -> Look and Feel -> Icon Overlays and check the drive types for which you want to see overlay icons.Be aware that enabling overlays for network drives will slow down TortoiseSVN.

SVN Directory Structure
The following is the current directory structure:

https://your.repo.url – Doc root for production subversion repository

/applications – a listing of all current apps in the repository
-><appname> – application name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of code/db – folders for each database’s SQL scripts
->/<dbname> – database name
—>/branches – space reserved for development branches
—>/tags – space reserved for tags
—>/trunk – the main development line of SQL scripts/documentation – space in the repo for storing docs requiring revision control –>/<project name>
->/qa – space in the repo reserved for storing QA-related docs and scripts

It is possible to have other directories under the trunk for other requirements. For example, if a prototype is required for the project, a /prototype directory can be created for this purpose. Similarly, other directories can be created on an as needed basis.You may have noticed we’ve separated out the documentation from the source code. There are several reasons for this:

1. Isolating source from docs is generally a good practice
2. We’d like to reduce user/permissions management overhead
3. We plan to move as much documentation as possible into a document management / collaboration tool in the near future.

We’re trying to maintain a similar tree to what we’ve used in VSS, and make changes only where it makes sense.

Checking out a project
To check out a project from svn using TortoiseSVN, click on “File – SVN Checkout”. Enter the URL of the directory that you would like to check out and the local directory to put it into. For example, to check out app “foo”, the URL would be:

https://your.repo.url/applications/foo/trunk/

Simply enter your password when prompted and the download will begin.

Importing a new project
To import a new project using TortoiseSVN, using the Windows file browser, go to the directory above the one you would like to import and verify it meets the directory requirements specified above under the /trunk directory. Click on the directory to import. Click on “File – TortoiseSVN – import”. Enter the URL of that you want the directory to be imported into. For example, if you are importing a directory called “foo”, the URL to use is:https://your.repo.url/applications/foo/trunk/

Once the file(s) is imported you need to do a checkout to get the project back out of svn so that you can now modify the files.

Transitioning from VSS to SVN
Below are some common tasks we’re used to performing with VSS, and their equivalent in Subversion. SVN does some things fundamentally different from VSS, so it’s not exactly a 1:1 mapping, but this can serve as a guide during the transition:

VSS Command	SVN Command	Difference
GetLatestVersion	Update	The SVN and CVS update will notify if files are pruned from repository so you can remove them from local repository (VSS would not).
Add	Add	Even though they have the same name, there are very big differences. The VSS adds the file directly to the repository whereas in CVS and SVN it is a local note that is made. The file is only added during the commit.
Delete	Delete	Same as add but for delete
Checkin	Commit	These commands pretty much do the same
Checkout	—	There is no edit in SVN, but there is a lock command to lock a file before using it.
—	Checkout	The checkout in SVN is essentially telling SVN “I want to track changes in these files”. You do a SVN checkout when you want to initiate a project.
Undo Checkout	Revert (and unlock if necessary)
ShowDifferences	Compare	In SVN, you need to view the log for the file, and select 2 rpevious revisions. Then you can right-click and view diffs
Label	Copy	A tag in SVN is actually a copy. Therefore it must be copied in the tags section (see previous explanation).
(manual copy)	Copy	In VSS dragging and dropping a branch was the equivalent of a “branch”.

Where to find more information:
More information on Subversion can be obtained from the subversion book.

Quick Build

In my search for an effective tool for managing builds and releases, I’ve decided on QuickBuild.  It’s basically a commercial version of LuntBuild, with some nifty management features added in. 

One of the nicer things about it is the ability to set properties for an app, and have them propagate to different configs for that app (dev, QA, staging, UAT, production), while having the ability to override certain values (say, DB username and password, etc…).

Like CruiseControl, it works well for continuous builds when integrated with your source control system.  Subversion and QuickBuild work together quite nicely.  Though, unlike CruiseControl, QuickBuild allows for “on-demand” deployments, and it’s promotion management features make it a cinch to migrate code from dev -> QA -> staging -> produciton.

I’m currently adding my apps to QuickBuild, and I’ve found it to be really easy to setup, use, and manage.  The end goal is to automate as much of the release process as possible, allowing for more predictable, repeatable, and consistently “problem-free” production releases.

Plus, it certainly would be nice to be able to release apps remotely via a web-based interface.  Now, if I can just figure out how to access QuickBuild from the golf course…
🙂

VSS to Subversion migration

Sorry for the dearth of posts lately, I’ve just been having too much fun with Subversion to post. Bad Release Guy! That, and about a hundred releases or so…
🙂

We have finally *officially* decided to use SVN, rather than Perforce. I’m actually pretty happy about the decision, I’m finding SVN to be a nice, simple, straightforward source control system. It doesn’t have all the “bells and whistles” associated with Perforce or ClearCase, etc., but it is a solid tool and does what a good version control system should.

The setup and install was pretty trivial, not a big deal at all. The only remaining item is to tie this thing up to my local authentication/authorization system. It’s proving to be more of a pain than I had hoped, and I’m not thrilled about just using SSH or Apache’s basic auth to control access to our repository.

I’ve put together some VSS to Subversion migration scripts (in perl) that have worked out quite well. It’s basically just a simple “get latest and add to SVN”, no labels or file history or anything gets moved over, but I’ve noticed the “VSS way” of doing things is not exactly based on best practices anyway. So we’ll just keep our old, VSS repository around in the meantime in read-only mode, in case we need to do some historical reporting.

If you’re interested in checking out my VSS to SVN migration scripts, have a ball!