PEARL XI : DEVOPS (a portmanteau of development and operations) is a software development lifecycle approach that stresses communication, collaboration and integration between software developers and information technology (IT) operations professionals . Many of the ideas (and people) involved in DevOps originated from the Enterprise Systems Management and Agile software development movements.
IT now powers most businesses. The central role that IT plays translates into huge demands on the IT staff to develop and deploy new applications and services at an accelerated pace. To meet this demand, many software development organizations are applying Lean principles through such approaches as Agile software development. Influenced heavily by Lean methodology, Agile methodology is based on frequent, customer-focused releases and strives to eliminate all steps that don’t add value for the customer. Using Agile methodology, development teams are able to shrink development cycles dramatically and increase application quality.
Unfortunately, the increasing number of software releases, growing complexity, shrinking deployment time frames, and limited budgets are presenting the operations staff with unprecedented challenges. Operations can begin to address these challenges by learning from software developers and adopting Lean methodology. That requires re-evaluating current processes, ferreting out sources of waste, and automating wherever possible.
According to the Lean Enterprise Institute, “The core idea [of Lean] is to maximize customer value while minimizing waste. Simply, Lean means creating more value for customers with fewer resources.”
This involves a five-step process for guiding the implementation of Lean techniques:
1. Specify value from the standpoint of the end customer.
2. Identify all the steps in the value stream, eliminating whenever possible those steps that do not create value.
3. Make the value-creating steps occur in tight sequence so the product will flow smoothly toward the customer.
4. As flow is introduced, let customer demand determine the time to market.
5. As value is specified, value streams are identified, wasted steps are removed, and customer-demand centric flow is established, begin the process again and continue it until a state of perfection is reached in which perfect value is created with no waste.
Clearly, Lean is not a one-shot proposition. It’s a reiterative process of continuous improvement.
Bridge the DevOps gap
There are obstacles to bringing Lean methodology to operations. One of the primary ones is the cultural difference between development and operations. Developers are usually driven to embrace the latest technologies and methodologies. Agile principles mean that they are aligning more closely with business requirements, and the business has an imperative to move quickly to stay competitive. Consequently, the development team is incentivized to move applications from concept to marketing as quickly as possible.
The culture of operations is typically cautious and deliberate. They are incentivized to maintain stability and business continuity. They are well aware of the consequences and high visibility of problems, such as performance slowdowns and outages, which are
caused by improperly handled releases.
As a result, there is a natural clash between the businessdriven need for speed on the development side and the conservative inertia on the operations side. Each group has different processes and ways of looking at things.
The result is often called the DevOps gap. The DevOps movement has arisen out of the need to address this disconnect. DevOps is an approach that looks to bring the benefits of Agile and Lean methodologies into operations, reducing the barriers to delivering more value for the customer and aligning with the business. It stresses the importance of communication, collaboration, and integration between the two groups, and even combining responsibilities. Today, operations teams find themselves at a critical decision point. They can adopt the spirit of DevOps and strive to close the gap. That requires working more closely with development. It means getting involved earlier in the development cycle instead of waiting for new applications and services to “come over the fence.” And conversely, developers will need to be more involved in application support. The best way to facilitate this change is by following the development team’s lead in adopting Lean methodology by reducing waste and focusing on customer value.
On the other hand, not closing the gap can have serious repercussions for operations. In frustration, developers may bypass operations entirely and go right to the cloud. This is already occurring in some companies.
Another challenge that operations teams face is in how to take the new intellectual property that the development organizations have built for the business and get it out to customers as quickly as possible, with the least number of errors and at the lowest cost. That requires creating a release process that is fast, efficient, and repeatable. That’s where Lean methodology provides the most value.
DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration betweensoftware developers and information technology (IT) operations professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.
A DevOps approach applies agile and lean thinking principles to all stakeholders in an organization who develop, operate, or benefit from the business’s software systems, including customers, suppliers partners. By extending lean principles across the entire software supply chain, DevOps capabilities will improve productivity through accelerated customer feedback cycles, unified measurements and collaboration across an enterprise, and reduced overhead, duplication, and rework
Companies with very frequent releases may require a DevOps awareness or orientation program. Flickr developed a DevOps approach to support a business requirement of ten deployments per day; this daily deployment cycle would be much higher at organizations producing multi-focus or multi-function applications. This is referred to as continuous deployment or continuous delivery and is frequently associated with the lean startup methodology. Working groups, professional associations and blogs have formed on the topic since 2009.
DevOps aids in software application release management for a company by standardizing development environments. Events can be more easily tracked as well as resolving documented process control and granular reporting issues. Companies with release/deployment automation problems usually have existing automation but want to more flexibly manage and drive this automation — without needing to enter everything manually at the command-line. Ideally, this automation can be invoked by non-operations resources in specific non-production environments. Developers are given more environment control, giving infrastructure more application-centric understanding.
Simple processes become clearly articulated using a DevOps approach. The goal is to maximize the predictability, efficiency, security and maintainability of operational processes. This objective is very often supported by automation.
DevOps integration targets product delivery, quality testing, feature development and maintenance releases in order to improve reliability and security and faster development and deployment cycles. Many of the ideas (and people) involved in DevOps came from the Enterprise Systems Management and Agile software development movements.
The focus of Lean is on delivering value to the customer and doing so as quickly and efficiently as possible. It is flow oriented rather than batch oriented. Its purpose is to smooth the flow of the value stream and make it customer centric.
DevOps incorporates lean thinking and agile methodology as follows:
- Eliminate any activity that is not necessary for learning what customers want. This emphasizes fast, continuous iterations and customer insight with a feedback loop.
- Eliminate wait times and delays caused by manual processes and reliance on tribal knowledge.
- Enable knowledge workers, business analysts, developers, testers, and other domain experts to focus on creative activities (not procedural activities) that help sustain innovation, and avoid expensive and dangerous organization and technology “resets.”
- Optimize risk management by steering with meaningful delivery analytics that illuminate validated learning by reducing uncertainty in ways that can be measured.
The first step for operations in adopting Lean methodology is to understand the big picture. That means not only developing an understanding of the end-to-end release process but also understanding the release process within the overall context of the DevOps plan, build, and run cycle. In this cycle, development plans a new application based on the requirements of the business, builds the application, and then releases it to operations. Operations then assumes responsibility for running the application.
In examining processes, therefore, operations should not only look at the release process itself but also at the process before the release to determine where opportunities lie for closer cooperation between the two groups. For example, operations may see a way for development to improve the staging process for operational production of an application.
Release process management (RPM) solutions are available that enable IT to map out and document the entire application lifecycle process, end to end, from planning through release to retirement. These solutions provide a collaboration platform that can bring operations and development closer together and provide that “big picture” visibility so vital to Lean. They also enable operations to consolidate release processes that are fragmented across spreadsheets, hand-written notes, and various other places.
In examining the release process itself, operations should look for areas to tighten the flow and eliminate unnecessary tasks. The operations group in one company, for example, examined the release process and found that it was re-provisioning the same servers three times when it was only necessary to do so once.
Anything that doesn’t directly contribute to customer value (like unnecessary meetings, approvals, and communication) should be considered for elimination.
Automate for consistency and speed
Manual procedures are major contributors to waste. For example, an existing release process may call for a database administrator (DBA) to update a particular database manually. This manual effort is inefficient and susceptible to errors. It’s also unlikely to be done in a consistent fashion: If there are several DBAs, each one may build a database differently.
Automation eliminates waste as well as a major source of errors. Automation ensures that processes are repeatable and consistently applied, while also ensuring frictionless compliance with corporate policies and external regulations. Deployment automation and configuration management tools can help by automating a wide variety of processes based on best practices. For Lean methodology to really work, processes must be predictable and consistent. That means that simple automation is not enough. The delivery of the entire software stack should be automated. This means that all environment builds — whether in pre- or post-production — should be completely automated. Also, the software deployment process must be completely automated, including code, content, configurations, and whatever else is required.
Automate manual and overhead activities (enabling continuous delivery) such as change propagation and orchestration, traceability, measurement, progress reporting, etc.
By automating the whole software stack, it becomes much easier to ensure compliance with operations and security. This can save vast amounts of time usually wasted waiting on security approval for new application deployments.
It is preferable to automate the time-consuming operational policies like initiating the required change request approvals, configuring performance monitoring, and so on. The mundane manual tasks, like these policies, create the most waste.
Before diving into automation, however, it’s essential for operations to map out and fully understand the end-to-end release process. When you use a release process management (RPM) platform to drive the end-to-end process, the team can review the process holistically to uncover sources of waste and determine where to apply automation tools to best streamline the process, eliminate waste, and accelerate delivery.
Measure success and continually improve Lean is an iterative approach to continuous improvement, and iteration necessitates feedback.
Consequently, operations must establish a means of tracking the impact of adopting Lean
methodology. In establishing the feedback metrics, keep in mind that the primary purpose of Lean methodology is not just to smooth and accelerate the release cycle; it’s also to create more value for customers and do it with fewer resources.
Consequently, operations should measure not only the increase in speed of releases but also the impact of the releases on cost and on customer value. For example, did the release result in a spike in the number of service desk incidents? This would not only increase support costs but also would degrade the customer experience. Or did the lack of capacity planning result in over-taxed infrastructure and degrade end-user performance? Here, it’s important to monitor application performance and availability from the customer’s perspective. Customers are not interested in the performance metrics of the individual IT infrastructure components that support a service.
They care about the overall user experience. In particular, how quickly did they complete their transactions end to end?
Application Performance Management (APM) solutions can track and report on a wide variety of metrics, including customer experience. These metrics provide valuable feedback to both the operations and development teams in measuring the impact of Lean implementation and identifying areas that require further attention. With these solutions in place, operations can operate in a mode of continuous improvement.
Use meaningful measurement and monitoring of progress (enabling continuous optimization) for improved visibility across the organization, including the software value delivery supply chain.
IBM DevOps Platform
IBM provides an open, standards-based DevOps platform that supports a continuous innovation, feedback and improvement lifecycle, enabling a business to plan, track, manage, and automate all aspects of continuously delivering business ideas. At the same time, the business is able to manage both existing and new workloads in enterprise-class systems and open the door to innovation with cloud and mobile solutions. This capability includes an iterative set of quality checks and verification phases that each product or piece of application code must pass before release to customers. The IBM solution provides a continuous feedback loop for all aspects of the delivery process (e.g., customer experience and sentiments, quality metrics, service level agreements, and environment data) and enables continuous testing of ideas and capabilities with end users in a customer facing environment.
IBM’s DevOps solution consists of the open standards based platform, DevOps Foundation services, with end to end DevOps lifecycle capabilities. To accommodate varying levels of maturity within an IT team’s delivery processes
Plan and measure: This adoption path consists of one major practice:
This adoption path consists of one major practice:
Continuous business planning: Continuous business planning employs lean principles to start small by identifying the outcomes and resources needed to test the business vision/value, to adapt and adjust continually, measure actual progress and learn what customers really want and shift direction with agility and update the plan.
Develop and test: This adoption path consists of two major practices:
Collaborative development: Collaborative development enables collaboration between business, development, and QA organizations—including contractors and vendors in outsourced projects spread across time zones—to deliver innovative, quality software continuously. This includes support for polyglot programming and support multiplatform
development, elaboration of ideas, and creation of user stories complete with cross-team change and lifecycle management.
Collaborative development includes the practice of continuous integration, which promotes frequent team integrations and automatic builds. By integrating the system more frequently, integration issues are identified earlier when they are easier to fix, and the overall integration effort is reduced via continuous feedback as the project shows constant and demonstrable progress.
Continuous testing: Continuous testing reduces the cost of testing while helping development teams balance quality and speed. It eliminates testing bottlenecks through virtualized dependent services, and it simplifies the creation of virtualized test environments that can be easily deployed, shared, and updated as systems change. These capabilities reduce the cost of provisioning and maintaining test environments and
shorten test cycle times by allowing integration testing earlier in lifecycle.
Release and deploy: This adoption path consists of one major practice:
Continuous release and deployment: Continuous release and deployment provides a continuous delivery pipeline that automates deployments to test and production environments.
It reduces the amount of manual labor, resource wait-time, and rework by means of push-button deployments that allow higher frequency of releases, reduced errors, and end-to-end transparency for compliance.
Monitor and optimize: This adoption path consists of two major practices:
Continuous monitoring: Continuous monitoring offers enterprise-class, easy-to-use reporting that helps developers and testers understand the performance and availability of their application, even before it is deployed to production.
The early feedback provided by continuous monitoring is vital for lowering the cost of errors and change, and for steering projects toward successful completion.
Continuous customer feedback and optimization:
Continuous customer feedback provides the visual evidence and full context for analyzing customer behavior and pinpointing customer pain points. Feedback can be applied during
both pre- and post-production phases to maximize the value of every customer visit and ensure that more transactions are completed successfully. This allows immediate visibility into the sources of customer struggles that affect their behavior and impact business.
Benefits of the IBM DevOps solution
By adopting this solution to address needs, organizations can unlock new business opportunities:
- Deliver a differentiated and engaging customer experience that builds customer loyalty and increases market share by continuously obtaining and responding to customer feedback
- Obtain fast-mover advantage to capture markets with quicker time to value based on software-based innovation, with improved predictability and success
- Increase capacity to innovate by reducing waste and rework in order to shift resources to higher value activities
Keep up with the future
By adopting Lean methodology, operations teams can catch up with and even get ahead of the large and rapidly increasing amount of new and updated services flowing from Agile-accelerated development teams. And they can do so without increasing costs or jeopardizing stability and business continuity.
In so doing, operations can help increase customer value, which has a direct effect on revenue, competitiveness, and the brand. Moreover, the operations team will have the metrics to demonstrate its contribution to the business. That enables the team to transform its image in the organization from software-release speed barrier to high-velocity enabler.
Traditional approaches to software development and delivery are no longer sufficient. Manual processes are error prone, break down, and they create waste and delayed response.
Businesses can’t afford to focus on cost while neglecting speed of delivery, or choose speed over managing risk. A DevOps approach offers a powerful solution to these challenges.
DevOps reduces time to customer feedback, increases quality, reduces risk and cost, and unifies process, culture, and tools across the end to end lifecycle—which includes adoption path to plan and measure, develop and test, release and deploy, and monitor and optimize.
Pearls of Wisdom 