Making Sense of DevOps

Whether you are keeping your company's infrastructure up and running, supporting customers with their IT issues, building, testing, or fixing software, or keeping your colleagues safe from security threats, you have likely come across DevOps.

After all, the term has been around for 15 years, and its adoption has increased exponentially. However, that does not mean its definition, positioning, and application have been universal. It has led to many semantic corruptions, such as BizDevOps, DevSecOps, AIOps, NoOps, FinOps, etc. This all adds to the confusion about what DevOps is and its relevance to anyone in the digital ecosystem. So, let's take some time to uncover the basics, principles, and hidden truths behind DevOps, shall we?

How it all started

First, let's start with a common definition of what DevOps is. This simple question alone can lead to countless hours of debate, quarrels, or even religious wars. So, by all means, let's return to the source for a solid and reliable answer. This leads us to Patrick Debois, who first coined the term in 2009 by organizing the first DevOps Days in Ghent, Belgium. Debois' frustration as a system administrator, suffering from a lack of collaboration in the technology delivery chain, has been a fire starter for the global DevOps movement. The wall of confusion emerged as a painful metaphor for the lack of empathy, sharing, and effective collaboration, mostly visible between development and operations.

One of his best quotes on the definition of DevOps has been, "DevOps is everything you do to overcome the friction between silos. All the rest is plain engineering".

This rather broad take on DevOps illustrates the holistic view and applicability of DevOps and its underlying philosophy. Although the term itself suggests a narrow scoping towards developers (dev) and operators (ops) alone, the main driver for starting it all was to improve collaboration and flow across all relevant silos, including business, security, quality, or support. Hence, understanding why this underlying desire resulted in the eventual adoption of DevOps as the main carrier for organizational, cultural, and technological change is far more relevant than searching for a better, more comprehensive name.

DevOps is NOT the goal

Now that we know how to define DevOps, we must also understand that DevOps is not the goal itself. Whether we see DevOps as a set of practices, principles, or even a sociotechnical construct, it is merely a means to achieve underlying objectives. The active BVSSH community has provided us with a logical set of primary objectives: Better Value Sooner Safer Happier. In other words, DevOps practices and principles can help us to achieve optimal value, better quality, safer deliveries, sooner feedback, and happier customers and employees. You are not doing it right if you think you are doing DevOps, but it does not lead to any of these underlying goals.

Over the past decade, many DevOps adoptions have failed to bring the desired value. Practice has shown that one of the primary causes why DevOps transformations have failed is because we have called it a DevOps transformation. This urge to label new initiatives has removed our focus from the underlying (business) objectives, such as happier customers or high-quality deliveries.

The Three Ways of DevOps

Introduced in The Phoenix Project book at the beginning of the previous decade, the DevOps promise is built on Three Ways: Flow, Feedback, and Continuous Learning and Experimentation.

1. The First Way: Flow

The First Way focuses on optimizing the flow of work from ideation to production, from development to operations. This means breaking down silos between teams, creating a culture of collaboration, and automating processes to eliminate bottlenecks. The goal is to ensure that work is completed as smoothly and quickly as possible. Strategies for optimizing the end-to-end flow include:

• Batch size: smaller batches of work going through the system. Constructs such as large releases severely hinder optimal flow.
• Work in progress: limiting the number of items a team is working on at the same time.
• Queues: reducing the number of queues and their respective throughput times
• Prioritization: value-driven or economic decision-making supports a focus on the most relevant brought into the flow.

2. The Second Way: Feedback

The Second Way emphasizes creating a fast and reliable feedback loop within, between, and around development and operations. This means implementing continuous testing, monitoring, and customer or user feedback mechanisms. It also means making it easy for teams to share knowledge and learn from each other. This allows teams to receive early feedback on whether their value hypotheses are valid or not. Also, this includes collecting experience data from customers, users, developers, suppliers, or any other stakeholders in the ecosystem. After all, our hypotheses around the potential value of the products or services we deliver can only be validated by measuring the eventual experience once it has been delivered. Feeding sentiment and operational data back into the system stimulates continuous improvement and learning.

3. The Third Way: Continuous Learning and Experimentation

The Third Way emphasizes the importance of continuous learning and experimentation. This means fostering a culture of experimentation, embracing failure as an opportunity to learn, building resilience in the processes, technology, and teams, and using data and metrics to drive continuous improvement. By continuously learning and experimenting, teams can improve their processes and outcomes over time.

By understanding and implementing the Three Ways, organizations can improve collaboration, feedback, and continuous improvement, leading to better outcomes and more efficient processes. In other words, they help to bring true ownership and end-to-end responsibility to the teams. Or, as Amazon CTO Werner Vogels stated, "You build it, you run it." The fewer dependencies a team encounters while creating value, the better outcomes we see in practice.

Keep CALMS, build your DevOps practices

CALMS is a widely accepted acronym used in DevOps that stands for Culture, Automation, Lean, Measurement, and Sharing. When applied together in the right balance, the CALMS pillars provide a solid framework to guide organizations in adopting the Three Ways and integrate the corresponding DevOps practices in their ecosystem:

• Culture refers to creating a culture of collaboration, empathy, trust, and continuous learning within the organization.
• Automation refers to relentlessly automating processes and workflows to improve efficiency and reduce the risk of human error.
• Lean refers to the methodological foundation of Lean (and Agile) ways of working, such as eliminating waste and focusing on customer value and flow.
• Measurement refers to using data (evidence) and metrics to measure the effectiveness of products, services, processes, and practices and identify areas for improvement.
• Sharing refers to promoting knowledge sharing and collaboration between teams and individuals within the organization.

Culture: Fostering Empathy, Collaboration and Shared Responsibility

There is a reason why CALMS starts with Culture. Culture is the very foundation of the DevOps philosophy, emphasizing a shift towards collaboration, shared responsibility, and breaking down organizational silos. This cultural transformation is not confined to development and operations teams but extends to business stakeholders, users, security experts, and suppliers. It comprises a mindset that values communication, empathy, and a collective commitment to delivering value to end-users.

Encouraging a DevOps culture involves fostering transparency and open communication channels between different teams and departments. Business units are integrated into the development process, ensuring that IT initiatives align with organizational goals. The essence of a DevOps culture is to improve the speed, quality, and reliability of product and service delivery by breaking down silos between teams and promoting a culture of shared responsibility.

Focusing on these cultural aspects will enable faster value delivery, improved quality and reliability, enhanced agility in responding to changing business needs, and increased team collaboration and teamwork. Additionally, DevOps culture can help organizations reduce costs and improve efficiency by automating repetitive tasks and streamlining processes.

Automation: Minimizing Manual Work and Streamlining Workflows for Efficiency

Automation is the engine that powers the DevOps machine, enabling organizations to streamline workflows, reduce manual errors, and accelerate delivery timelines. From code development to deployment and monitoring, automation tools play a pivotal role in achieving continuous discovery, integration, delivery, and deployment.

Automated testing ensures that code changes are thoroughly evaluated, reducing the likelihood of introducing defects into the production environment. Continuous Integration pipelines automate the build and integration processes, providing a consistent and reliable foundation for deployments. Automated deployment tools facilitate the seamless and repeatable deployment of applications, minimizing downtime and increasing overall efficiency.

Besides that, countless other technology solutions address relevant tasks such as version control, release management, container orchestration, infrastructure provisioning, knowledge management, experience measurement, monitoring, and observability.

Lean: Focus on Value, Eliminate Waste and Optimize Flow

Lean and Agile are at the heart of the DevOps way of working. The digital organization heavily relies on the Agile and Lean principles and practices to cope with their volatile, unpredictable environment. Whether the teams use Scrum or Kanban or whichever instrument to manage their work(flow), key DevOps working characteristics entail an obsessive focus on value, flow, and resilience.

If you are part of a Scrum team involved in developing a new product for your organization and have difficulty obtaining feedback from the product increments you have put into production already. In that case, you may consider using pair work, observability, or experience measurement to gather the required feedback. In doing so, Lean practices like visual management, value stream mapping, and Kaizen can effectively apply these practices by removing bottlenecks or reducing manual efforts in gathering the required feedback. Similarly, if you are a Service Desk agent dealing with customers daily, you surely would be excited about the efficient collaboration and workflow between the resolution groups working on complex incidents or with continuous feedback loops of experience data used to create happy customers and users.

Essential in applying Lean and/or Agile ways of working is the notion and understanding that not all contexts are equal. If your team operates in a predominantly predictable environment with standard working procedures, your need to apply agile approaches will be quite different from product development teams dealing with high levels of uncertainty and volatility. Consequently, the latter team will more likely benefit from agile principles and frameworks like incremental development, timeboxing, or Scrum. Similarly, teams with a predictable and stable context would be helped more with Lean improvements to support standardization and automation of their repetitive tasks and workflows.

Lean thinking also encourages the use of metrics (evidence) to identify areas for improvement. By analyzing indicators such as customer satisfaction, lead time, and time to recovery, organizations can pinpoint inefficiencies and prioritize process enhancements.

Measurement: Data-Driven Decision Making and Continuous Improvement

Measurement is a cornerstone of DevOps, providing the data needed for informed decision-making and continuous improvement. DevOps metrics help organizations assess the effectiveness of their digital products, services, and processes, identify areas for enhancement, and track progress over time. The continuous collection, interpretation, and application of relevant data and metrics supports better collaboration throughout the value lifecycle, from ideation to operation and leadership to teams.

Most high-performing organizations and teams have fruitfully applied the typical DORA metrics for DevOps: lead time for changes, deployment frequency, failure rate, and recovery time. In addition, organizations have started to explore outcome or value metrics, such as customer retention rate, user satisfaction, and employee happiness. By introducing rigorous measurement of outcomes, experience, and impact, the focus is shifting towards true end-to-end delivery, optimization, and collaboration instead of partial or team results.

In line with the Third Way of DevOps, measurement is also an essential ingredient to a continuous learning culture to support ever-evolving ecosystems. Take zero repeat as one of the innovative indicators to stimulate a safe-to-learn environment ("it's okay to make a mistake") while highlighting the importance of learning from previous errors ("just don't make the same mistake twice").

Sharing: Distributing Cognitive Load and Knowledge Sharing

The Sharing aspect of CALMS emphasizes the importance of communication, collaboration, and knowledge sharing within and across teams. In a DevOps culture, information flows seamlessly between development, operations, and other stakeholders, fostering a shared understanding of goals and priorities.

Sharing also extends to documentation and knowledge transfer. DevOps encourages the creation of necessary (Lean as we are: just enough) documentation that captures the entire development lifecycle. This documentation serves as a valuable resource for onboarding new team members, troubleshooting issues, and ensuring continuity in case of personnel changes.

Following the "you build it, you run it" mantra, organizations strive to bring as much ownership and responsibility to the actual teams. The more autonomously a team can perform its tasks, the more reliable its flow will be, the shorter its feedback loops will be, and the more value it will deliver to its customers. A common pitfall is to expect all tasks in the product lifecycle and delivery chain to converge in one team. The concept of cognitive load has taught us that human limits make it impossible to have all relevant knowledge and capabilities, from simple to complex, combined in one team of 5-9 engineers.

This is why organizations adopt organizational constructs distinguishing product and platform, enabling teams to separate product delivery from platform engineering from specialized support. This allows the product teams to focus on understanding, creating, and delivering digital products for their customers while using standardized services delivered by platform teams and supported by security or test automation experts from enabling teams.

Putting DevOps in practice

This all adds to the discussion if there is anything like a DevOps engineer. From a fundamental standpoint, DevOps is a movement, a way of thinking, a set of practices. Claiming we have a DevOps engineer who embraces all this in one role is like saying we have assigned one person to collaborate. A more practical standpoint would be that we can have engineers throughout any modern organization performing DevOps-related activities and embracing typical DevOps principles and practices. Whether they are part of a product or platform team or have an enabling role, we could use the overarching term DevOps engineer to pinpoint the desired capabilities and behaviors.

DevOps is about removing the friction between silos by applying the Three Ways and balancing the respective CALMS pillars. If we truly understand this, we can leverage its potential for our own organizations. It will lead to ecosystems exhibiting increased empathy, better collaboration, optimized flow, and ultimately more value for our customers.