I recently worked for a client with J2EE applications who had issues with my suggestions on how release engineering should be handled. So I will leave this as a lesson for those of you that might read this little blog.
First, they would copy the software from one environment to the next instead of installing the software from distribution files. This led to a large number of conflicts within the base configurations (usually with the multicast addresses and ports which would be forgotten).
Next, they insisted on having tweaked startup programs that were never committed to a revision control system, replaced the ones that would have been in revision control and that the "installation" system needed to work around (i.e., not overwrite). Instead of the operations team working with development to have a set of startup programs that worked across the board and having those committed to revision control, there are multiple sets of install/start/stop programs that may or not be correctly maintained and most assuredly are not properly tracked and auditable.
Lastly, the operations team insists on configuring their "large number" of J2EE server systems by using GUI Jconsole-like applications using mbeans. This is laboreous, time consuming, error prone and unweildy. Most sites that I have worked at will generate configuration files and have them pushed out as text to the multiple servers, modified for each specific server and environment. These files can be recreated at any time and if properly managed through version control, can even be rolled back to any point in time. Managing a large number of servers through a GUI is an effort in futility.
Distribution program
I've recently created a distribution program for myself and others to use within the company. The program name is, unimaginatively, called "distrib". It reads XML data files from a server and determines the entry from the username of the application account (see previous post).
Names of distribution files, server hosts and pathnames are captured in one of the aforementioned XML file. The distribution files are copied to the distribution directory on the remote server ('server:~/vrel#'); files that have already been copied are checked with md5 checksums. The product's install program is also implicitly copied.
After the distribution files have been copied, the program calls the product's install program on each remote server.
There are options to manage the program flow and to access the XML data files.
The program is simple, quick and efficient.
A lot of the past work has been laid as foundation to be able to write a program such as this. Individual application accounts, a +90% common code install template system for 20+ products, more than 25 products working on the same Generalized Release Process.
Names of distribution files, server hosts and pathnames are captured in one of the aforementioned XML file. The distribution files are copied to the distribution directory on the remote server ('server:~/vrel#'); files that have already been copied are checked with md5 checksums. The product's install program is also implicitly copied.
After the distribution files have been copied, the program calls the product's install program on each remote server.
There are options to manage the program flow and to access the XML data files.
The program is simple, quick and efficient.
A lot of the past work has been laid as foundation to be able to write a program such as this. Individual application accounts, a +90% common code install template system for 20+ products, more than 25 products working on the same Generalized Release Process.
Application accounts
For each of the products that are installed and deployed within my company, there is an application account to go with it. The account will have a shared home directory, /home/username, and a per-host application installation directory, /apps/username.
The application installation directory is managed by the product's install program. The only contents should be from the installation and from the application. Users should not "move" files aside, or copy files to that directory. This is the production installation directory, not a work-area.
The home directory is to house builds, distributions, patch files, other "cruft", depending on the host. Release builds are stored under the "releng" directory by release number, for example "~/releng/1.2". QA builds are stored under the "qa" directory in the same fashion, or as the QA engineer chooses. Distributions are stored in the home directory in sub-directories starting with "v" and the release number, e.g. "~/v1.2". The install program, always found in "sbin" in the source repository, is copied reach release (since there may be changes for that release) to the distribution directory (on the destination server).
Generally builds are performed in the engineering environment and no build tools are available in the production environment. Because of co-location, the engineering environments and QA environments share home directories and the production environment is isolated. Many of the products are installed and deployed on multiple hosts, within each environment. Therefore, it is important to have a shared resource (the home directory) in each environment to be able to install distributions or patches from.
Release numbers are just symbolic names, they could be anything - conventionally they are dotted numeral sets that match tags in the source repository (e.g. "1.4.6.1"). But a release number could be "1.2-test0012" for a QA release. This may correspond to the "1.2" source tag.
The application installation directory is managed by the product's install program. The only contents should be from the installation and from the application. Users should not "move" files aside, or copy files to that directory. This is the production installation directory, not a work-area.
The home directory is to house builds, distributions, patch files, other "cruft", depending on the host. Release builds are stored under the "releng" directory by release number, for example "~/releng/1.2". QA builds are stored under the "qa" directory in the same fashion, or as the QA engineer chooses. Distributions are stored in the home directory in sub-directories starting with "v" and the release number, e.g. "~/v1.2". The install program, always found in "sbin" in the source repository, is copied reach release (since there may be changes for that release) to the distribution directory (on the destination server).
Generally builds are performed in the engineering environment and no build tools are available in the production environment. Because of co-location, the engineering environments and QA environments share home directories and the production environment is isolated. Many of the products are installed and deployed on multiple hosts, within each environment. Therefore, it is important to have a shared resource (the home directory) in each environment to be able to install distributions or patches from.
Release numbers are just symbolic names, they could be anything - conventionally they are dotted numeral sets that match tags in the source repository (e.g. "1.4.6.1"). But a release number could be "1.2-test0012" for a QA release. This may correspond to the "1.2" source tag.
QA tagging
A QA tag is a construct that I invented over a period of time to deal with another concept called slushing. Slushing is to refreeze an intermediate "QA release". Conceptually, you freeze your code, then thaw it slightly to make a change, refreezing the code with the new change (bug fix during QA), repeat each time there is a QA iteration. At the end of QA testing, the QA tag is used to make the release tag.
The QA tag is a label (or tag) that breaks the typical definition of a label - it repeatedly gets changed, much like a branch would.
In Subversion, I create a new peer directory to the standard "branches", "tags", "trunk" directories called "qa". This directory holds the QA tags. The QA tags are copied from trunk as the release tag would have. Changes are merged as needed. When QA approves the product for release, the release engineer will make the release tag from the QA tag.
This has the advantage that development can continue without directly affecting the QA engineer and selective revisions can be merged into the QA tag as needed (not taking HEAD every time).
The QA tag is a label (or tag) that breaks the typical definition of a label - it repeatedly gets changed, much like a branch would.
In Subversion, I create a new peer directory to the standard "branches", "tags", "trunk" directories called "qa". This directory holds the QA tags. The QA tags are copied from trunk as the release tag would have. Changes are merged as needed. When QA approves the product for release, the release engineer will make the release tag from the QA tag.
This has the advantage that development can continue without directly affecting the QA engineer and selective revisions can be merged into the QA tag as needed (not taking HEAD every time).
Types of files to be released
One issue that I have seen come up again and again in badly conceived projects is the mixing of files that should have been properly partitioned when the distributions were created. One very simple example is that log files are created in the application's root directory. This makes it more difficult for various utilities and system maintenance routines to operate as well as for upgrades within the release engineering framework.
I have divided files in the application into four possible types:
Program and configuration files could be set to read-only by the install program; configuration files changed to read-write as needed. The directories for the persistent and transient files should be read-write for the owner of the application and any other application that needs access.
With this simple separation, upgrades and upgrade programs can be made much more easily. Backups can be tailored as needed. And the application's release structure is similar to its memory structure: program code separated from data, separated from heap, separated from kernel, etc.
I have divided files in the application into four possible types:
- program - binary executables, shell scripts and library files. A library file is code that is being used by another program, but is not executed directory.
- configuration - files that define values specific to this installation, e.g. db source, appl dir, peer hosts, etc.
- persistent - files that would need to survive upgrades and/or backups, e.g. databases
- transient - files that are created for temporary use, e.g. log files, caches
Program and configuration files could be set to read-only by the install program; configuration files changed to read-write as needed. The directories for the persistent and transient files should be read-write for the owner of the application and any other application that needs access.
With this simple separation, upgrades and upgrade programs can be made much more easily. Backups can be tailored as needed. And the application's release structure is similar to its memory structure: program code separated from data, separated from heap, separated from kernel, etc.
Release notes... finally
I've finally gotten the company that I work for to standardize the release notes that are being submitted.
Previously, some groups would send text files with some information, other groups would send PDF or MSWord files with other information, still more groups wouldn't send any release notes at all.
The principles of the new release notes format are:
Previously, some groups would send text files with some information, other groups would send PDF or MSWord files with other information, still more groups wouldn't send any release notes at all.
The principles of the new release notes format are:
- easy to write
- target the audience
- document change (only)
- information related to this release of this product
- easy to read (audience targeting again)
- A set of value pairs that will show information like product name, release number, list of developers, etc.
- Descriptions for super-system impact, changes to the product, changes to perform to OS, DB or release procedures.
Project vs. Product
One issue that has been going around my company lately is the difference between "project" and "product". This difference has affected the development cycle, project management and release engineering.
Simply, a project is a work effort and a product is a deliverable. Releases are made of products, not of projects.
A project could consist of multiple products over possibly multiple releases, or just be one release of one product. A project is a statement of work, of change.
Simply, a project is a work effort and a product is a deliverable. Releases are made of products, not of projects.
A project could consist of multiple products over possibly multiple releases, or just be one release of one product. A project is a statement of work, of change.
Subscribe to:
Posts (Atom)