Working in the web app world, I frequently get asked, often by managers who aren't thinking about the product and SOX compliance, if it is OK to plop just a jsp file or a jar instead of rebuilding the entire war each time.
My answer is always "No!". There are a number of reasons why, but most simply is that doing so defeats the basic tenets of release engineering: reproducability, traceability, accountability.
As a simple example that recently occurred in my own job, we were starting to upgrade a pair of server that had not been touched in nearly three years. We started in on the first of the pair however we soon found that the code needed to be reverted. When we redeployed the backup of the war file, the code was exhibiting incorrect behavior, especially compared to its twin. Turns out that soon after the deployment three years earlier, a developer also deployed a single JSP file into the exploded war directory. Three years go by, developer has moved on, managers have forgotten everything, two different release engineers moved on. The developer was playing around with a production system, didn't document what happened and didn't re-release the product to correct the situation. As a result, the product could have been in jeopardy if we had not been more tentative and worked on only one server at a time (another topic of discussion: a lot of developers want me to deploy to all the servers at once). By only working on one, we could compare the old war with the changed war on its twin.
This anecdote illustrates one major issue with partial deployments. There was no accountability for the product: none of the manager, developer or the release engineer at the time of the first release held themselves accountable. There was not tracability: no documentation was kept, no build records, no labeling - just a file plopped down into a directory. No reproducability: the environment could not be reproduced without looking at the twin server. If that twin production server hadn't existed, then there would have been nothing to revert to.
Full deployments from a single, release distribution build lead to all of these tenets. Anything less breaks release engineering practices.
Showing posts with label theory. Show all posts
Showing posts with label theory. Show all posts
Significant builds in continuous integration
With agile methodologies, it is very common to have continuous integration builds. In those methodolog, there are two types of builds: successful and unsuccessful. Unsuccessful builds are builds where either the code could not compile, unit tests fail or regression tests (if part of the build system) fail. Successful builds are ones where the build completes successfully.
Considering how often continuous integration builds are executed, with cascading, downstream builds of other products, even large disks could fill up quickly. Many limit the number of builds preserved or the number of days preserved. Here I suggest a different approach.
There are two types of successful builds. A significant successful build is one where a change occur to initiate this build or an upstream (assumed to be successful) build. An insignificant successful build is one where no change has occurred to initiate this build.
Insignificant builds could be deleted on the next successful build. Significant builds should be kept for some period determined by the SCM.
An example of an insignificant build within the system is builds activated hourly or when changes are detected in version control. It is easily conceivable, especially overnight, that the hourly builds would become insignificant compared to the version-control based builds.
When what would the purpose of having insignificant builds? There may be other tasks outside of the build that are triggered by the build, for example, regression tests on a remote system, load tests run at night. Each of these could be triggered by a specific build, but generating the same build objects (for example, publishing them to the same object repository).
Significant and insignificant successful builds have their place within the build framework. Identification of each and more importantly, getting the build systems to identify each, and managing these would become the next stage.
Considering how often continuous integration builds are executed, with cascading, downstream builds of other products, even large disks could fill up quickly. Many limit the number of builds preserved or the number of days preserved. Here I suggest a different approach.
There are two types of successful builds. A significant successful build is one where a change occur to initiate this build or an upstream (assumed to be successful) build. An insignificant successful build is one where no change has occurred to initiate this build.
Insignificant builds could be deleted on the next successful build. Significant builds should be kept for some period determined by the SCM.
An example of an insignificant build within the system is builds activated hourly or when changes are detected in version control. It is easily conceivable, especially overnight, that the hourly builds would become insignificant compared to the version-control based builds.
When what would the purpose of having insignificant builds? There may be other tasks outside of the build that are triggered by the build, for example, regression tests on a remote system, load tests run at night. Each of these could be triggered by a specific build, but generating the same build objects (for example, publishing them to the same object repository).
Significant and insignificant successful builds have their place within the build framework. Identification of each and more importantly, getting the build systems to identify each, and managing these would become the next stage.
QA tagging
A QA tag is a construct that I invented over a period of time to deal with another concept called slushing. Slushing is to refreeze an intermediate "QA release". Conceptually, you freeze your code, then thaw it slightly to make a change, refreezing the code with the new change (bug fix during QA), repeat each time there is a QA iteration. At the end of QA testing, the QA tag is used to make the release tag.
The QA tag is a label (or tag) that breaks the typical definition of a label - it repeatedly gets changed, much like a branch would.
In Subversion, I create a new peer directory to the standard "branches", "tags", "trunk" directories called "qa". This directory holds the QA tags. The QA tags are copied from trunk as the release tag would have. Changes are merged as needed. When QA approves the product for release, the release engineer will make the release tag from the QA tag.
This has the advantage that development can continue without directly affecting the QA engineer and selective revisions can be merged into the QA tag as needed (not taking HEAD every time).
The QA tag is a label (or tag) that breaks the typical definition of a label - it repeatedly gets changed, much like a branch would.
In Subversion, I create a new peer directory to the standard "branches", "tags", "trunk" directories called "qa". This directory holds the QA tags. The QA tags are copied from trunk as the release tag would have. Changes are merged as needed. When QA approves the product for release, the release engineer will make the release tag from the QA tag.
This has the advantage that development can continue without directly affecting the QA engineer and selective revisions can be merged into the QA tag as needed (not taking HEAD every time).
Types of files to be released
One issue that I have seen come up again and again in badly conceived projects is the mixing of files that should have been properly partitioned when the distributions were created. One very simple example is that log files are created in the application's root directory. This makes it more difficult for various utilities and system maintenance routines to operate as well as for upgrades within the release engineering framework.
I have divided files in the application into four possible types:
Program and configuration files could be set to read-only by the install program; configuration files changed to read-write as needed. The directories for the persistent and transient files should be read-write for the owner of the application and any other application that needs access.
With this simple separation, upgrades and upgrade programs can be made much more easily. Backups can be tailored as needed. And the application's release structure is similar to its memory structure: program code separated from data, separated from heap, separated from kernel, etc.
I have divided files in the application into four possible types:
- program - binary executables, shell scripts and library files. A library file is code that is being used by another program, but is not executed directory.
- configuration - files that define values specific to this installation, e.g. db source, appl dir, peer hosts, etc.
- persistent - files that would need to survive upgrades and/or backups, e.g. databases
- transient - files that are created for temporary use, e.g. log files, caches
Program and configuration files could be set to read-only by the install program; configuration files changed to read-write as needed. The directories for the persistent and transient files should be read-write for the owner of the application and any other application that needs access.
With this simple separation, upgrades and upgrade programs can be made much more easily. Backups can be tailored as needed. And the application's release structure is similar to its memory structure: program code separated from data, separated from heap, separated from kernel, etc.
Project vs. Product
One issue that has been going around my company lately is the difference between "project" and "product". This difference has affected the development cycle, project management and release engineering.
Simply, a project is a work effort and a product is a deliverable. Releases are made of products, not of projects.
A project could consist of multiple products over possibly multiple releases, or just be one release of one product. A project is a statement of work, of change.
Simply, a project is a work effort and a product is a deliverable. Releases are made of products, not of projects.
A project could consist of multiple products over possibly multiple releases, or just be one release of one product. A project is a statement of work, of change.
Generalized Release Process
As many companies have, my current employer has acquired a number of other companies and tried to integrate their technologies and employees into the fold.
Of course, the release processes are completely different. At first, I did not try to merge the different processes, but developers were starting to cross "company boundaries". I decided to create a new theory of process for the developers.
The release process is broken into four components:
Of course, the release processes are completely different. At first, I did not try to merge the different processes, but developers were starting to cross "company boundaries". I decided to create a new theory of process for the developers.
The release process is broken into four components:
- build
- distribution/packaging
- installation
- deployment
Build
The build step is the traditional build of software, but for qa and production, as an isolated build process within a standardized, controlled environment. The developer environment is too fluid and unstable to be used for releases. The production build environment should be separate and only accessible to the release engineers. This helps prevent unknowns from being introduced into the release process. The build is performed on a build host often within a revision controlled work area ("sandbox").Additionally, QA/Prod builds are to be made from tagged code. This allows for reproducibility. This implies that builds do not perform code management/version control. Version control should not be managed by build scripts (exception being CruiseControl-like systems for automated testing based on code changes).
For some code bases, there will be nothing to build, e.g. only perl or php modules. In this case, the build step would be empty, but would still exist.
Distribution/packaging
This step will copy deliverables into a presentable directory structure for the intended target platform or environment (Windows vs. Linux, JDK 1.4 vs. JDK 1.5, and eng vs. qa vs. prod). There may be different environment sets and multiple options. As examples, environments for eng, qa, and production, options for load testing, regression testing or CruiseControl, selecting different database engines.
Build scripts (Makefile, build.xml, maven.xml) may represent the different options and environments by separate targets, variables or options. The build scripts would select the configuration files, libraries or executables to be "shipped".
The distribution step will also package the project for installation at a later time. It will not copy outside of the release engineering environment. Note that this is not packaging of jar/war/ear files – that should be part of the build process.
The purpose of the distribution step is to select proper files, position them, and then package them for later installation.
If there is no need for changes and no need to package deliverables (one file being delivered), then there is no need for this step. Distribution occurs on the build host often within the sandbox (local revision control work area).
In a practical matter, there may be two "targets" for this step: "distribution" and "packaging". Distribution should precede packaging in the process.
Installation
Installation is the traditional step to create a copy of a software distribution and make it available for execution or deployment. There is no distinction between environments; the previous distribution step should have handled that. This step is solely to take a packaged copy of the built project and install it on a machine or machines. Any pre-execution, pre-deployment steps that need to be made are executed here. For example, creation of directories, setting of permissions, changes to databases.
Installation will most often occur on a different machine from the build host and outside of the sandbox. An installation will likely not have targets in the build scripts, unless a build script gets packaged with the application to be used as an install script.
Deployment
Deployment is the activation of the software on the installation machine. This could be deploying a servlet to Tomcat/JBoss, adding a cron job or starting a daemon that has been installed with a startup script or some other means.
After the software has been installed on a machine, the code is deployed from the installation point to the appropriate application server.
Subscribe to:
Posts (Atom)