Last updated on: September 19, 2024
IT change management is a structured approach to ensuring that any additions or modifications to IT services and infrastructure are handled efficiently and in a standardized manner. The primary goal of change management is to minimize disruptions to services while implementing necessary changes that improve processes, services, or infrastructure. From routine configurations to critical security updates, each change is vital for staying competitive and secure in a dynamic digital landscape. However, even seemingly minor changes can lead to unscheduled downtime if not managed properly, underscoring the importance of robust change management practices.
Types of change
Change management can be categorized into several types, each tailored to handle different kinds of changes based on their impact, urgency, and complexity. The primary types of change management are:
- Standard changes: These changes are low risk and routine. They are often pre-authorized and executed quickly through an automated or streamlined process.
Example: Routine software or firmware updates.
-
Normal changes: These changes undergo thorough review and approval by the change advisory board (CAB). They have a medium to high impact on services or infrastructure, requiring detailed documentation,
including a change request, impact analysis, risk assessment, implementation plan, and back out plan.
Example: Implementing new features in a critical application.
-
Major changes: These changes are a subset of normal changes and have a substantial impact on business operations, infrastructure, or service delivery. They come with high complexity, high risk,
and broad impact. Major changes require comprehensive analysis and approval, often involving senior management and extensive testing phases.
Example: Data center migrations or large-scale system deployments.
-
Emergency changes: These are changes that need to be implemented urgently to resolve an incident, overcome security vulnerabilities, or prevent an imminent service disruption. They come with high
urgency and potentially high risks. Emergency changes often bypass the standard CAB process, with approvals delegated to an emergency CAB (eCAB). Post-implementation, a review is conducted to assess the change and
its impact.
Example: Applying a security patch to fix a critical vulnerability.
Let's discuss an example of an emergency change, where a leading North American oil pipeline operator navigated a zero-day vulnerability and swiftly applied vital patches using ManageEngine ServiceDesk Plus.
Example of an emergency change
A leading oil pipeline operator in North America received a concerning notification about a zero-day vulnerability for its on-premises enterprise resource planning (ERP) application from the vendor. The oil pipeline operator's ERP system plays a pivotal role in overseeing and managing the essential operations of its pipeline, making it a prime target for a cyberattack.
The identified vulnerability exposed the ERP system to unauthorized access and posed a significant risk of data breaches. The stakes increased with the potential for malicious actors to exploit the vulnerability, and to encrypt critical systems and demand a ransom for their release. Recognizing the potential consequences of such an attack, the oil pipeline company worked closely with the ERP software vendor to expedite patching the zero-day vulnerability. To strengthen its defenses, the company applied patches to its endpoints, aiming to close the vulnerability and protect against unauthorized access or potential data breaches. Unfortunately, delays and an unplanned approach to applying the patches provided intruders with sufficient time to execute a ransomware attack, leading to the shutdown of the pipeline.
Events leading to the ransomware attack
Here's how the sequence of events leading up to the attack unfolded:
- The ERP vendor notified the pipeline's IT team of the vulnerability, prompting the IT team to initiate the patch rollout to all affected endpoints. Subsequently, the head of IT operations created a request for change (RFC).
- The IT team deviated from best practices, but followed a uniform workflow for all types of change requests. Nevertheless, the change manager approved the RFC and forwarded it to the change advisory board (CAB).
- Unfortunately, due to the unavailability of several CAB members on a short notice, a delayed approval occurred that provided intruders with ample time to gain access to the network.
- Now in a rush, the IT team rolled out the patches. But without easy or quick access to the full IT infrastructure, several endpoints were overlooked and were not patched.
- During this intense scenario, the ERP's downtime wasn't announced to the end users, leading to the multiple incident tickets being created that flooded the IT service desk's ticket queue. The IT team was stretched thin from handling the patch rollout, managing the attack, and mitigating the influx of tickets asking about the ERP status.
- These events culminated in the intruders having time to carry out their ransomware attack. The delay in patching meant that the threat actors propagated through the defenses and accessed the entire network of systems, ultimately leading to the shutdown of the pipeline operations.
Unraveling the causes of the chaos
Examining the chain of events, three critical errors emerged as the primary culprits behind this ransomware attack:
- Lack of a dedicated emergency change process: The absence of a specialized process and dedicated CAB for handling emergency changes delayed the approval process, leaving the organization ill-prepared to swiftly address the zero-day vulnerability.
- Inadequate visibility over the affected endpoints: The organization's inability to accurately pinpoint and oversee all affected endpoints due to a siloed configuration management database (CMDB) and IT asset management (ITAM) created a vulnerability gap that intruders exploited.
- Failure to communicate downtime: The failure to communicate the change progress to stakeholders and the ERP downtime to users resulting from the security fix contributed to user exasperation and the high volume of incident tickets.
The pipeline company could have handled the whole situation more effectively if it had followed the best practices or used a platform like ServiceDesk Plus that provides:
- A dedicated RFC template for emergency changes to enable the change requester to initiate a change quickly with all the necessary fields prefilled.
- An emergency change advisory board (ECAB), which is a subset of CAB members available on short notice and outside of normal working hours to speed up the approval process. The ECAB must have the authority to make decisions in an emergency without requiring escalation.
- A lean emergency change workflow that customizes the directional pathway to process change requests with minimal stages and approval.
- An all-in-one solution that helps IT teams track CMDBs and ITAM databases, while implementing the changes.
- A dedicated downtime announcement mechanism.
- Ability to notify stakeholders throughout the entire emergency change process.
Now, let's explore how the oil pipeline company could have ensured business continuity if it had relied on a streamlined change management process and implemented the emergency change with ServiceDesk Plus.
The initial step involves establishing an efficient change process that facilitates the planning, approval, and execution of changes. Here is a recommended change management process that can be adopted to manage changes in the organization effectively.
Emergency change management process in ServiceDesk Plus
The change management process within ServiceDesk Plus consists of eight basic stages as outlined in Figure 1. Each stage includes various statuses. However, ServiceDesk Plus's approach provides flexibility and permits omitting stages with the exception of first and last stages, Submission and Close. This flexibility to customize stages ensures the process can be adapted to meet diverse organizational needs and change scenarios.
Figure 1: The change management process.
The process begins with the Submission stage of the change request, which outlines the proposed alterations. Subsequently, the Planning stage involves thorough preparation, specifying the necessary steps, resources, and potential impacts. This is followed by an Approval stage to ensure alignment with organizational standards and validation from the stakeholders. The execution of changes takes place in the Implementation stage that directly influences the IT infrastructure. The implemented changes undergo testing by selected users during the UAT stage. Following successful testing, downtime is scheduled, and users are informed about the unavailability of services before the changes take effect in the Release stage. The Review stage evaluates the success of the changes and ensures adherence to objectives. Lastly, the Closure stage concludes the change request process by furnishing closing details and assigning a closure code.
To execute this in real time, ServiceDesk Plus uses Change workflows which help in the design of change management processes using a drag-and-drop canvas. By associating these workflows with the RFC, ServiceDesk Plus ensures a structured and controlled approach to handling changes, whether routine or emergency. These workflows provide the ability to customize the sequence of steps involved in handling changes, set specific conditions and criteria, execute time-based actions, send real-time notifications at various stages, obtain approvals, update relevant fields, and run custom functions.
Figure 2 showcases how an emergency change workflow can be configured within the platform. Emergency changes are implemented in critical situations demanding immediate action, and render certain stages unnecessary. In this example, the emergency change process can skip planning and UAT stages, utilizing the flexibility to bypass the stages. This flexibility empowers organizations to efficiently manage a variety of change scenarios by tailoring the workflow to suit the unique characteristics of each change request.
Figure 2: Emergency change workflow.
Let's explore how the zero-day vulnerability security patch could be implemented using the various stages in ServiceDesk Plus' change workflow.
Stage 1: Submission
In response to the zero-day vulnerability, the IT team initiates the patch rollout to all the affected endpoints. The IT team follows a dedicated RFC template (Figure 3) for the different types of changes to record detailed information about the change, such as why the change needed to be initiated, the workflow it should adhere to, the priority, the business impact, and the probable downtime.
Figure 3: Emergency RFC template.
Given the critical nature of an emergency change, the head of IT operations creates a RFC ticket (Figure 4) with fields such as Change Type, and Priority prepopulated, using the Change Template in ServiceDesk Plus.
Figure 4: Emergency RFC ticket.
By utilizing ServiceDesk Plus, the IT team can meticulously oversee its configuration management database (CMDB) and its IT asset management (ITAM) database, ensuring comprehensive coverage of the affected endpoints. The CMDB provides the IT team with complete visibility into organization's IT infrastructure, the dependencies it holds, and the relationships between configuration items. Consequently, it is easier to stay connected with the key components of the organization's digital footprint.
Stage 2: CAB evaluation
The change manager grants approval to the RFC and the change automatically progresses to the next stage for ECAB recommendations (Figure 5). Automated notifications are sent to all ECAB members, prompting them to approve the change. Configuration options allow for rules that specify waiting for either unanimous or any ECAB member's approval.
Figure 5: ECAB approval.
Additionally, timers (Figure 6) can be incorporated to send approval reminders, guaranteeing that approvers remain informed and do not overlook essential emergency approvals. The ECAB assesses change plans and provides the required approval for the emergency change request to move forward with implementation.
Figure 6: Timer action for approval reminder.
Stage 3: Implementation
Upon receiving approval from ECAB, the IT team proceeds with the implementation of the emergency change. Recognizing the significance of timely communication in managing user expectations and mitigating frustration, users are automatically informed of service downtime or disruptions through an announcement (Figure 7). With the end users aware of the ERP upgrade, and the subsequent downtime, the deluge of incident tickets is avoided.
Figure 7: Service downtime announcement.
The IT team now optimizes the integration between ServiceDesk Plus and ManageEngine's Endpoint Central to seamlessly identify endpoints, establish connections, and efficiently distribute patches to affected systems—all through the ServiceDesk Plus platform.
This can also be facilitated by the single-touch workflow automation which initiates the patch deployment either via Endpoint Central, or by executing a PowerShell script triggered by approval from the ECAB serving as the single touchpoint for this process. Improved integration and workflow automation significantly contribute to a more efficient and effective patch management process.
Throughout the change implementation, the IT team closely monitors the process to ensure that it proceeds as expected. Incident tickets that report any issues or unexpected complications post-patch implementation can be associated to the emergency change ticket (Figure 8), and addressed promptly to minimize disruptions.
Figure 8: Incident ticket association post implementation.
Automatic notifications (Figure 9) are employed at each stage to keep relevant stakeholders informed about the progress of the entire emergency change. This ensures the IT team remains focused on only implementing the patch successfully, while ServiceDesk Plus manages the communication, process adherence, and display of information.
Figure 9: Notification.
Stage 4: Review
After the successful implementation of the change, a post-implementation review is crucial to confirm the successful application of patches, followed by an analysis of system performance to ensure there is no unexpected degradation or improvement of functionality. Subsequently, the IT team engages in continuous monitoring, promptly identifying and addressing any emerging issues or disruptions while diligently attending to incident tickets reporting service disruptions. Finally, the option to set the next review date (Figure 10) ensures ongoing assessment and improvement of the change management procedures.
Figure 10: Next review date.
Stage 5: Closure
Following the review of the emergency change, closing the change request is facilitated (Figure 11). ServiceDesk Plus lets you predefine conditions and actions like mandating closure code to streamline the process of concluding the change process. The closed change request, supported by detailed documentation, serves as a reference for future assessments and audits, contributing to the ongoing improvement of change management practices within the organization.
ServiceDesk Plus thus enables any organization to sustain its operations with reduced vulnerability to the zero-day threat.
Figure 11: RFC ticket closure.
Enhance your change management strategy with ServiceDesk Plus
Given the inevitability of zero-day vulnerabilities and risks of unforeseen service outages, emergency change management should be an integral part of any organization's IT strategy—particularly for those operating in critical sectors. By implementing emergency changes swiftly and efficiently, organizations can maintain operational continuity and pursue their strategic objectives with confidence. Well-structured change management practices enable smooth transitions, minimize disruptions, and significantly improve the chances of achieving successful outcomes.
Are you ready to take the next step? Accelerate changes, minimize friction, and enhance your operational efficiency the ManageEngine ServiceDesk Plus way. Get your 30-days, free trial of ServiceDesk Plus to discover how it can enhance your operational efficiency.
Author's bio
With eight years' experience in IT services, Suganya has hands-on experience handling key IT service management (ITSM) practices. As an avid ITSM evangelist, she is also a ServiceDesk Plus product expert. She creates best-practice articles and blogs that can help ITSM practitioners address their everyday challenges with ServiceDesk Plus, the flagship IT and enterprise service management platform from ManageEngine. Besides her passion for writing, she also enjoys trekking, reading books, playing basketball, and stargazing with her daughter.