The IT release management best practices explained here are based on the use case of a mid-sized telecom company. The company faced multiple issues that cost them about 25% of their customers, and a decline in their productivity. The main reason was the lack of an efficient release management process and procedures. Find out how this company mitigated release failures and increased their release throughout within three months.
"Release management is the process of managing, planning, scheduling and controlling a software build through different stages and environments" - Wikipedia
"Release management is all about build, test, and deploy while continuous delivery is about build, test, and deploy using automation." - Charles Betz, Research analyst, Forrester
"For me, the number-one piece of automation to look for is testing software." - Jayne Groll CEO, DevOps Institute
A mid-sized telecom company lost about 25 percent of its customers over a period of three months due to multiple delayed releases of critical software. Clearly, the existing release management process was not effective, and the company had to troubleshoot and arrive at a concrete process to set things straight.
The company formed a new release management team to assess and streamline its existing release management process flow. In the following three months, the new team was able to put out both the pending releases, and two scheduled releases of re-engineered applications. Below, you'll see how the team streamlined their release management process. The steps the team took reflect best practices that can be employed by any IT organization, including yours.
The new team wanted a detailed picture of the current release management process flow. They began with a number of walkthrough sessions with key individuals involved in the software process. In these sessions, the team learned that a CRM software had been pending release for two months after completion.
Based on customer`s requirements, the CRM software included claim settlement. It also included service credits and penalties for telecom customers who faced issues in downtime, service degradation, and transfer of new billing plan. The delay in rolling out the CRM software meant that several of the customers couldn't get their service credits or refunds on time, and that there was no tracking system for updated status requests. About 25 percent of retail customers lost patience in the system, and walked away.
In the existing process, the touch points, validation, and verification process were not seamless due to hardware procurement, test environment, and the absence of an agreed upon release cycle.
In order to deal with this situation, the new release management team followed a few course correction pointers,
as listed below:
Once the team got a picture of the current state of the release management process, they decided to focus on defining and establishing an agreed release cycle. To understand the frequency of release in production, the team had to understand the frequency of the non-functional testing. After a discussion with the engineering teams, the team concluded that the project required regression, performance, and integration testing.
The release cycle was developed to achieve the following:
The release management team started out by experimenting on a weekly cycle, but the plan proved unfeasible. The client's database environment could not be refreshed quickly. The team then tried two-week cycles. There was no immediate objection from the participants, but the two-week cycle failed the first two times. Once they overcame a few environment turnaround bottlenecks and automated some of the tests, the team decided that the two-week cycle was an achievable one.
Finally, the team established a cycle whereby, every two weeks, production-ready code from the engineering team was put into system testing. Two weeks later, they released that code to production.
The release cycle was not about when the customer wanted the release. It was about when the team could deliver it to the market with the desired level of quality. However, when the team engaged the customers, and made them the decision makers, the customers started supporting it!
Lightweight processes are those that do not require lengthy bureaucratic approvals or endless meetings to get agreement. They usually require only the minimum acceptable level of input and output. What they lack in bulk and bureaucracy, they make up for in response to change and popular adoption.
Underpinning this approach was a thorny issue of documentation, however. The team needed to record what was done and how it was done. Otherwise, it would be impossible to review or improve the situation.
The company decided to move towards a documentation standard that would let people (technical and otherwise) read and act upon documents, instead of letting the documents lie on the shelf.
The engineering team chose Confluence, a commercial tool, to collaboratively document their work. They used the software to create minimal but effective documentation of what they agreed to build in every cycle of work. They recorded what they built, how they built it, and what was required to make it work. The new team saw the value in this approach, and rolled it out (both approach and tool) to everyone else involved in the process.
A sequence of tasks was then created to release the software from the engineering teams. The list of tasks included:
A release infrastructure ensures that everything is in place to deploy software and enable use. The release management team took time to develop a configuration management system (CMS), and started by building the tree structure and configuration item (CI) hierarchy.
The team conducted workshops with the infrastructure, application software development, and management teams to decide on the configuration, change, and release processes. The agreed conceptual flow, from a tool perspective, is depicted below.
The release infrastructure covered the hardware, storage, network connections, bandwidth, software licenses, user profiles, and access permissions. Human services and skills were also part of the release infrastructure. For example, if a specialist software needed to be installed and configured, they could not exclude the availability or cost of getting such skills into the infrastructure plan.
It is critical to discover any hidden bottlenecks in procuring the required hardware or missing skills (e.g. configuration of secure networks). The team had to resolve these issues before delivery.
However, that standard was not easy to uphold. The team strove to get the release infrastructure in place as soon as they started on the project. Even after six weeks' lead time, however, they were still waiting on specialized memory and hard drives for the test servers.
The team identified the build and test tasks that could be automated, and came up with the criteria and definitions for normal, standard, and emergency changes. Automation enabled them to perform repetitive tasks without tying up valuable human resources. They also qualified a number of standard changes based on risk, repeatability, and time involved in execution.
Changes were bundled to align with the release cycle of two weeks. Change teams worked with the release and deployment teams synchronize with the release calendar, types of release, and early life cycle support.
Prior to their involvement in the project, the engineering teams manually crafted deployable packages. Due to this, each new package was not guaranteed to be the same as the previous one. In fact, it was not guaranteed to even be the software they had been building, much less guaranteed to work! It often took the tech staff days to create a package with the features, in a structure that could be deployed.
The release management team immediately drew up a structure and service acceptance criteria for the deployable package the engineering team was delivering, and helped them standardize the packaging. This triggered the implementation of automated processes to build the software in that consistent structure for every release point.
As the team had automated the verification process of the acceptance criteria, the execution was guaranteed. For example, code must be unit tested prior to delivery and test deployed to ensure that it could be. As a result, the release management team was able to package, version, test, and deploy the finished code with a single command, in a very short time.
The newly established release cycle meant that a release had to be tested for regression, performance, and integration in just two weeks, for the team to be able to release it into production. The team was able to overcome different types of testing (integration and performance) by having separate environments for each type. However, the challenge of accommodating two months of regression testing into a two-week window seemed impossible. Here's how they did it.
While the combination of configuration management, change control process, and release and deployment management processes were integrated seamlessly, the entire process required the vertex of people in order to be possible.
The release management team backed up this importance by establishing that the designated release manager would expect the software to be ready at the time that the teams agreed upon. The team made an arrangement that the program manager (in this case, the end usercustomer) would explain to the teams why the release was important.
The team requested that engineering teams ensure that the software they delivered conformed to a standard (versioned, tested, documented, and packaged). The team also established that they would request this standard package for every release cycle. This made the automated process easier and more consistent, and the allowed for feedback integration.
The following release management metrics were measured continually to tune the release management process.
During the course of the entire transformation, the biggest asset was the people. The release management team took their perspectives, issues, and challenges in a fair and transparent manner, and informed the management of the same. They even listed a few of the early adopters as change agents. They invested heavily on training, awareness, and communication, thereby instituting a reward mechanism for acknowledging positive behavior and knowledge sharing.
The following workshops were conducted for the configuration, change, and release management teams, without exception. The line managers were also available to validate the effectiveness of the workshops.
The outcome of the five-day workshop was increased clarity for the involved teams on various touch points, deliverables, and overall business impact. A quick cheat sheet was distributed to summarize the key learning points of the training.
Lessons Learned
Based on the release management team's experience in working with others on the project, they realized that good release management takes hard work, resolve, and great communication. However, the greatest skill is the ability to review, learn, and adapt improvements with effective collaboration of people in conjunction with the magic triangle configuration, change, and release and deployment management processes.
ITIL® is a registered trade mark of AXELOS Limited. All rights reserved.