Dr. Dobb's | Improving the Development Process | May 19, 2010

It's just as important to have good development processes as it is a good system architecture

ByEric J. Bruno
May 19, 2010
URL:http://www.ddj.com/tools/224900379

Eric J. Bruno is a contributing editor to Dr. Dobb's. He can be contacted at eric@ericbruno.com.

For an application's long-term success, it's just as important tohave good development processes as it is a good system architecture. Inmy experience, even the best development organizations -- those whomanage their design and coding tasks well -- tend to leave applicationdeployment and management until the end. This includes the proper use ofa source code repository such as CVS, where multiple developmentprojects may be working in parallel. Additionally, the source coderepository should be used, along with a scripted procedure, to pull anddeploy code to production environments.

Too many organizations rely on developers or systemadministrators to build software releases by hand, and then manuallydeploy them to the appropriate servers in production. This manualprocess applies to both new software releases, as well as theprovisioning of new servers. Further, they also rely on developers orother build specialists to manage parallel programming projects in thesame code base, again by hand. These approaches are fraught with dangerbecause they rely on manual processes, as well as specializedknowledge only a few people in the organization may possess. To remedythis, I suggest the following:

• Parallel development: Use source code branching along with a process that I've found to work well (described below).
• Controlled builds: Use an automated build process that pulls code from a repository by label and compiles on a dedicated build machine.
• Software deployments: Use a combination of Ant scripts and source code repository labels to deploy the proper artifacts to different types of servers (i.e. web server, application server, database, and so on).

Before getting into the details, let's cover some basics that,hopefully, you're already following. When developing code for anysystem, it's best to adhere to the following development processprincipals:

• Ensure that all code is kept in a source-code repository (i.e., CVS or Subversion).
• Perform system builds on a regular schedule that may change according to development phase (i.e. once per week early in a development cycle, or once per day towards the end).
• Ensure that all code checked into the repository compiles, is unit tested to some degree, and does not break the build in anyway.
• Any build that is a candidate for release and is either deployed to a test system or a production system is built with the latest code, and is tagged (labeled) in the source-code repository. The build should then be "pulled" from the repository bytag to ensure consistent releases across time and servers.
• Use branching to ensure that changes to a particular release can be made to that release's specific code base, even when development has begun on the next release; i.e. parallel development.

Although I use CVS for examples in this article, most of theconcepts should apply to other source-code repositories as well, be itSubversion, Git, or Mercurial. Let's examine the procedure for handlingparallel application development first.

Improving Release Management

Based on experience on multiple projects in the past, I'vesettled upon a CVS method of working that seems to work best. Theprocess is based on code branching. It's important to note that I'veseen projects go very wrong through the improper use if branching, soI've come up with a recipe that works well.

Even if you don't maintain parallel programming teams, there maybe times when you'll need to work on two versions of the sameapplication at one time. For instance, when a software release isdeployed to production, it's important that you be able to get to theexact source code and all supporting files that were used for thatrelease, even if coding has progressed since then, in case bugs arefound in production.

To achieve this, development is done in the HEAD (sometime calledthe "tip") of the source code repository, and each release cycle mustbe performed in its own branch. This branch, uniquely labeled torepresent the particular release being tested/deployed, is created whendevelopment is complete, and a comprehensive quality-assurance (QA)cycle is about to begin prior to its release to production.

Also note that there is only one active source code branch at anyone time. This rule should be strictly followed. The branch is createdwhen development is "feature complete," and a QA cycle is about tobegin. The following steps discuss this procedure in more detail:

1. Create a new branch with the release cycle/version/name as part of the branch name (i.e., BRANCH_MYAPP_v_2_34)
   1. Important note: At this time, development for the next release can begin in the HEAD again. The branch ensures you can get back to this release's specific code base.

2. Pull the latest code from the branch created in step 1, and do a complete build on a trusted machine. This can be a dedicated build machine, or a developer's machine that is considered the build master.
3. Place the resulting binaries back into the CVS branch.
4. Label the branch with a unique label for that release and build (i.e., MYAPP_v_2_34_build_001)
5. Pull the binaries and all supporting files (i.e. configuration files) from the CVS branch using the label created in step 4, and deploy to your QA servers.
6. If bugs are found, perform code fixes in the branch (not the HEAD), and repeat the QA cycle starting from step 2. Note that this will result in an incremented label in step four due to the new build number.
7. When the release passes QA, pull the binaries and supporting files from the branch using the most recent label created in step 4, and deploy to production.
8. Merge the branch back into the CVS HEAD, paying careful attention to any potential conflicts (remember that development may have started for the next release there). In my experience conflicts are rare, and are easily resolved when they do occur. However, they should never be done automatically; it's crucial to review the conflicts and make careful decisions to resolve them.
9. Development for the next release can continue in the CVS HEAD, and this process is repeated when that cycle is "code complete."

Because the code in a branch is merged into the HEAD when itpasses QA and is released, the HEAD is sure to have all of the codechanges (bug fixes) made during the QA cycle, even if development hasprogressed in the HEAD for the next release. Also, if a problem arisesin an earlier release, because the branches are not deleted or removedin any way, the precise code base is always available for every releaseever made. This guarantees that you can recreate any revision of yourapplication at any point in the future.

This process works well for a one development group working onone release at a time, as well as multiple development groups working onmultiple releases at a time. The most efficient approach to followinvolves two parallel development groups, where you align the releasesso that when one development group is in a QA/bug fix/release cycle, thesecond development group is coding the next release. As a result, ifyou plan each release to be roughly equal in terms size and developmenttime, the second development group should be entering QA as the firstdevelopment group deploys its release and begins coding the nextversion.

This swapping of roles between the development groups (when oneis coding, the other is testing/bug fixing) ensures the most efficientuse of resources, and that the release frequency is increased. Thisprocess has the following advantages:

• Both of your development groups work at a reasonable pace
• You achieve twice as many software releases in a normal time frame
• New features made available to users sooner, without sacrificing quality

Next, let's dive deeper into improvements to the softwaredeployment process. These improvements build from the process we justexamined. The example we'll follow is for enterprise (hosted)applications, but the process should apply to any type of software.

Improving Release Deployment

The CVS development and branching procedures outlined in theprevious section are important to ensure that the exact code andconfiguration for every release can be retrieved when needed. It alsoprovides an efficient process for parallel development groups to follow.However, it's also important that software deployment be doneefficiently, and without error. To make this process painless anderror-free, the solution I use is to combine Ant scripts with CVS, usingthe proper software release label.

Ant scripts that take as parameters the task/software name, thesoftware's CVS label, and the server name to deploy to, make for aquick, easy, and precise deployment process. For example, look at thefollowing command:

> ant deploy.xml -Dcvsuser=ebruno -Dlabel=MYAPP_2_234_build_012 -Denvironment_name=prodserver_01 deploy_app

In this command, the first parameter is the Ant script; next isthe CVS username to use; next is the CVS label to use to pull therelease; next is the server to deploy to; and the final parameter is theant task itself. In this case, the Ant task, deploy_app, clearly indicates that we are deploying software.

Ant Tasks for Application Deployment

Let's take a closer look at the Ant scripts themselves, alongwith the shell script that makes it easier to pass in all of thoseparameters. For instance, the shell script in Example 1 sets the path, classpath,and Ant options (such as heap size) to run Ant. It also collects theparameters (CVS user, CVS label, and the environment name), and callsthe Ant script to execute the deployment.

#! /bin/shANT_HOME=/usr/local/antJAVA_HOME=/usr/javaANT_OPTS="-Xmx512m"PATH=${PATH}:${ANT_HOME}/binCLASSPATH=$CLASSPATH:$ANT_HOME/lib:$ANT_HOME/lib/jakarta-ant-1.4.1-optional.jarexport ANT_HOME JAVA_HOME ANT_OPTS PATH CLASSPATHCVSUSER=$1RELEASE=$2## ex: qa, prodserver_01, prodserver_02, etc.ENVIRONMENT=$3## true or falseFORCE_CHECKOUT=$4ant -buildfile deploy.xml -Dcvsuser=$CVSUSER -Dlabel=$RELEASE -Denvironment_name=$ENVIRONMENT -Dforce_checkout=$FORCE_CHECKOUT deploy_app

The parameters passed into this shell script are set asenvironment variables in the line that executes the ant command at theend. These are read in as part of the Ant script, as shown here (seeListing 1 for the entire script):

......

The properties listed here are set with the parameters entered inthe shell script and specified as -D command-line parameters whencalling Ant. There are other properties, such as the CVS project name,and the name of application archive file being deployed, that arehard-coded in this example. You'll need to replace these with thecorrect names for your project, or you can pass them as additionalenvironment variables if you choose.

The rest of the Ant script contains five targets:

• init sets the CVS user, and prepares to backup the existing deployment.
• check_release determines if we can skip the CVS checkout process. This is done when deploying the same release to multiple servers. The checkout is done once, and the same files are< used to deploy multiple times. This can be overridden by setting the FORCE_CHECKOUT parameter to True.
• backup backs up the existing application files on the server where you're deploying the new release. This is done in case you need to quickly get back to the original files.
• cvs_checkout checks out the release by specified label from CVS, and creates a file to records that release's version number. You can use this as a sanity to check to ensure the deployment actually occured.
• deploy_app is the main target that calls all of the other targets (via the dependancy tree) to backup the existing application files, checkout the new release from CVS, and then copy the files to the right places on the specified server.

Ant has built-in support for source code repositories such as CVS. For instance, the cvs_checkout target uses the following command to do its work:

Remember to replace the CVS server entry with the address of youractual CVS server, and the path to your CVS repository. The packagename, tag name (label), and destination directory are all derived fromthe parameters you specified in the shell script earlier.

This sample Ant script assumes you're deploying a Java EEenterprise application archive (EAR) file. However, you can pull anynumber of files and file types. The deploy Ant target contains the entries to copy the files to the server, as shown here:

The parameters in this Ant task are pretty self explanatory. Theexample script contains additional entries that copy other files, suchas properties files and execution scripts to start start and stop theserver, simply as an example. You can take this further, and createscripts that deploy the actual binaries and other files for your webserver, application server, database, and so on. This will enable you toprovision new servers identically when you need to, and then deployyour software to them from that point forward.

Conclusion

The success of a software project goes beyond good architectureand code; it relies on good processes and procedures throughout thedevelopment, testing, and deployment phases as well. This articleillustrates that maintaining good source-code repository practices whenit comes to release management and parallel development teams ispossible if a proven plan is followed. It also illustrates thatmaintaining Ant scripts toautomate the deployment of labeled releases from your source coderepository is not difficult. You can even use them to provision entireservers into your data center. In the end, having a reliable, anderror-free, deployment process will save you a lot of time andheadaches. Using the processes and scripts provided here in your ownprojects can help you achieve that.