The Challenge of Processes and Technology

By this point in the book, you should have a good idea of how agile can help companies design great customer solutions, meaning the products and services that provide consumers or business buyers with the value they seek. But every service and every product depends upon processes, the steps and procedures by which companies create and deliver those products. The processes, in turn, are nearly always underpinned by technology, primarily software.

Yet for most companies, existing processes and technology are barriers to great customer solutions, not enablers. Think about RBS’s starting point: its cumbersome processes and systems got in the way of delivering a great service. It’s a common situation. For example, a company’s regional offices or business units might be doing things very differently from one another, making integration and training difficult or impossible. Nobody did that on purpose; it was just the result of thousands of small decisions over the years being made by different people. Or perhaps a company made a series of acquisitions and never effectively integrated them, leaving a host of different and less-than-optimal systems in place. Again, the processes and technology that company employees then have to deal with are apt to engender frustration and inefficiencies.

Sad to say, IT departments and the software they develop are notorious for creating such difficulties. A common issue for some companies is spending millions on customized software when standard off-the-shelf solutions would meet their needs. They develop their software through the traditional waterfall process, with every requirement spelled out in advance and the software inevitably loaded up with features that nobody uses. They thereby create a level of complexity that mere human beings can’t easily cope with. Many IT departments get so backed up with requests for changes or for new products that a sort of shadow IT emerges: executives fed up with waiting create mini–IT departments of their own or seek out external providers. This, of course, only worsens the complexity, creating unnecessarily diverse processes, systems, and technical standards.

Bureaucracies, being bureaucracies, typically stick with the processes and technology they have in place for as long as they possibly can, which in most cases is far too long. Changing them, after all, is going to be expensive. The changes are going to get in the way of day-to-day business. And the results probably won’t be that much better anyway—or at least such is the bureaucratic mindset. When things get too bad or too costly, of course, the senior team will finally decide that it’s time for a change. They go behind closed doors, decide the specs and the budget for the new processes and systems, and then tell their underlings to work it all out. These organizations are chronically unbalanced, alternating between stagnation and occasional fits of ineffective change.

As we saw with RBS, companies pursuing the agile journey go about things differently. They understand that customer solutions should be driven by customer needs. They understand that these solutions should shape processes and that technology should support and automate these processes (see figure 7-1). Agile practitioners also know that solutions, processes, and technology must continually adapt as customer needs change. They believe that agile teams are the tools best suited to develop innovative solutions when what to deliver, how to deliver it, or both, are vague and unpredictable—the typical situation when addressing customer needs. They know that innovation and operations teams must collaborate closely and in some cases should even be merged.

In this chapter we’ll explore innovations in processes and technology and see how they support the creation of industry-leading customer solutions. The chapter primarily addresses processes and systems that underlie service-related customer solutions rather than physical products—but this does include the many services that support physical products.

Working Backward from Customer Solutions

Solution teams innovating processes and technology need to focus on customers, just like any other agile team. Sometimes the relevant customers will be internal. But even then it’s important to consider the needs of the external customers being served by those internal customers. Too often, functions such as IT and finance still reflexively focus inward rather than on the needs of the customers they serve—one reason for the bad rap of many IT departments. Innovations in these areas too often reflect what software engineers or finance specialists think is important rather than what their customers might find most valuable. Agile teams that are changing processes and systems, by contrast, treat customer solution, process, and technology innovations as agile products. Working from the customer backward, their goal is to ensure that all these innovations meet customer needs as simply and effectively as possible.

The starting point should always be the solution—the creation of a particular customer experience, such as applying for a mortgage, or the development of a certain capability, such as verifying customers’ stated incomes and assets. Usually, customer experience is the better tool for solution definition, because it’s typically easiest to link what customers experience with what they value. But capability is the better construct in some situations, such as when a capability underpins many different experiences or when organizing by experience is impractical. Both experiences and capabilities often cut across functional and departmental boundaries, so the most effective teams will draw from all the units that are affected. When solutions are large enough to span more than one team, a company can break the solutions into modular components. That way, each team can test options with customers and proceed relatively independently, giving all the teams the greatest possible control and speed.

In chapter 2, we discussed the benefits of structuring agile teams around a taxonomy of related solutions. This holds true for the customer solutions that drive processes and technology. For instance, a leading US health insurance company developed a taxonomy based on five portfolios: plan members, employers, healthcare providers, benefits brokers, and employees. Each of five portfolio owners acts as the chief product manager for the full set of experiences and capabilities serving that portfolio. A few capabilities, such as claims processing, cut across multiple portfolios and are led by a chief capability owner. This kind of structure allows the company to create road maps for larger solutions that require teams of teams, and it helps manage interdependencies among those teams.

Changes in customer solutions, processes, and technology are highly interdependent, so a team responsible for innovation in any of these areas is frequently empowered to change all three. If that task is too big for one team, multiple teams coordinate closely. Moreover, as we discussed in chapter 5, most companies find that persistent teams are more effective than project teams for solution innovation.

Learning from experiments is fundamental to agile innovation. But experimenting with innovations in customer solutions underpinned by processes and technology presents some specific challenges. Bureaucracies strive for clear, stable, predictable operations. Most traditional operations groups aren’t set up to make small, frequent process changes. It’s the same for traditional IT groups—they can’t easily make small, frequent changes in their systems’ functionality. Larger issues intrude as well. Many companies, especially in regulated industries, have lengthy and elaborate procedures that must be followed before anyone can alter processes or technology. Many lack the analytic and technical skills required to design tests and measure outcomes in a manner that maximizes learning. Executives and managers frequently express concern that failed tests will create significant risk with customers.

Let’s look at these challenges in greater detail.

Process Innovation

RBS understood that changes in customer solutions should drive changes in processes. For example, the mortgage advisor reviewing the accuracy of information in a digital application had to work very differently than the advisor reviewing a paper application. The bank thus needed to redesign the processes followed by its mortgage advisors when it introduced its digital mortgage application.

So how should agile teams go about innovating processes? In some ways, process innovation is much like any other agile innovation. You start with the customer and work backward to solve their needs in an incremental, iterative way. You accomplish this with empowered multidisciplinary teams. In other ways, however, the approach requires a higher level of sophistication. Companies have found two methods particularly useful.

Technology Innovation

Agile has spread fastest among technology innovators, especially software engineers. What makes agile such an effective approach for software development? The problems to be solved are complex and the solutions initially unknown. Product requirements will most likely change. The work can be modularized and performed incrementally. Close collaboration with end users (and rapid feedback from them) is feasible. Testing can be automated. Success rates with traditional (waterfall) methods are low. But the value of success is high, especially with the increasing importance to customers of digital solutions.

We see many organizations with lots of agile technology teams, but we do not see many agile technology organizations. Technology departments have adopted agile software development extensively, yet most are not keeping pace with the changes required in business processes and customer solutions. Some of the reasons for this disappointing disparity will by now sound familiar: not working from the customer backward in deciding what to build; top-down, big-bang mandates leading to halfhearted, inconsistent adoption of agile practices; leaders who talk up agile but don’t change their own management styles; rigid, slow, tedious planning, budgeting, and reviewing; compensation, promotion, and other people policies that undercut agile values. But there are other factors at play that are more specific to software development:

• Architecture.  This is among the most important. While agile software development provides better results than traditional methods regardless of the architecture, monolithic systems can greatly compromise that improvement. If the architecture issues are left unaddressed, the results from agile will continue to feel lackluster.
• Excessive specialization.  Software engineers often have highly specialized skills. We typically recommend that agile teams focus on solving customer problems and, when possible, stay together for months and years. But heterogeneous technologies mean the skills required to address the backlog of a complex experience team will vary. If engineers are too specialized, they will end up with teams that are too large or that consistently need to swap out members.
• Departmental silos.  Traditionally, separate IT units are responsible for different tasks: software development, system maintenance, support, IT operations, and information security. In some industries, still more units are involved, such as legal or compliance. Each of these groups works in its own silo and is accountable for different objectives. They are often at odds with one another, causing progress to slow to a crawl. In an agile technology organization, all of the functions are performed by blended agile teams.

For purely digital products, where by definition there is no physical work involved and processes are fully encoded in software, the agile team can have end-to-end responsibility for the full product. These teams essentially combine innovation and operations. With no need to retrain people to adopt process changes, digital products can be innovated more rapidly. These characteristics explain why agile practices are so common among digital-native companies such as Google, Facebook, Twitter, and Spotify. But the benefits of agile software development are equally compelling for other firms. Moving from traditional methods to mature agile teams typically leads to productivity and speed-to-market improvements of threefold or more. The improvements can be traced to a litany of good things, including a reduction in waiting time for design decisions and approvals, less effort spent creating traditional business cases and waiting for funding, automation of IT-related tasks such as functional and security testing, automation of tasks once provided by other units such as provisioning the development environment, and aligned incentives through end-to-end ownership of the product life cycle. Even more important than these benefits, of course, is the fact that agile teams create the features that are most valuable to customers and to the business. They generally don’t waste time creating features that customers do not value.

An example of a purely digital customer solution is RBS’s Home Agent, which is a key to the bank’s strategy of supporting customers in home buying and ownership, not just home financing. Home Agent allows customers to get assistance on their phone with a wide range of home ownership activities, including setting a budget and shopping for a new home, home financing and refinancing, planning and funding home improvements, and tracking home equity. Employing agile teams that incorporated all the needed skills related to customer insights, software development, and third-party partnering, RBS was able to develop and launch the first version of Home Agent in four months. A customer solution of this complexity and quality would have taken at least three times as long under the previous methods—if indeed it could have been successfully developed at all.

Software development is challenging, and achieving the benefits of agile requires traditional organizations like RBS to make a wide range of changes that go beyond the principles and practices we have described for other uses. A full treatment of the requirements for effective agile software development is beyond the scope of this book, but some of them have been mentioned earlier or are listed here. Readers can find others—and a fuller explanation of all—at the website bain.com/doing-agile-right, including:

• Modular architecture, which allows each agile team to minimize interdependencies with other teams when writing code.
• Improved engineering practices and upgraded technical talent, typically requiring extensive technical training and coaching of both front-line employees and leadership, along with selective hiring to augment (and sometimes replace) current talent.
• Converged backlogs, with each agile team responsible for the development, maintenance, and support of the software that relates to its product. This end-to-end accountability creates greater ownership and is more efficient than dividing these three activities among two or three different groups.
• DevSecOps, the tools and practices that allow agile software development teams to do most of the work to move software from development to production quickly and securely.
• New IT services vendor models, often involving a shift from fixed deliverables to fixed capacity (in agile teams) as well as commitments to low turnover among team members.
• Revised location strategy enabling greater colocation and upgraded talent.
• Transformed support and control functions that succeed in their functional missions while allowing agile development teams to work at pace. One key here is to shift these functional groups’ orientations from correcting work product after the fact, as is often done now, to coaching agile development teams to create compliant product from the start.

Software development is especially well suited to a key requirement of implementing agile at scale: breaking large tasks down into modular components and then reintegrating the workstreams seamlessly. Amazon can deploy software thousands of times a day in part because its IT architecture was designed to help developers make fast, frequent releases without jeopardizing the firm’s complex systems. By contrast, many large companies are limited by inflexible architecture: no matter how fast they can code programs, they can deploy software only a few times a week or a month.

Achieving a modular architecture like Amazon’s can seem like an insurmountable hurdle to many large companies that depend on monolithic legacy systems. It can be done, however, by applying the same customer-focused agile principles we have been discussing throughout this book. Modernizing through small steps, sequencing those steps based on the benefit they provide to customers, and using the agile software development methods described in this chapter will make the journey faster, more affordable, and less risky.

Agile Wars

One more note on technology: there are dozens of agile methods, each with its passionate advocates. The problem is that as tribes within the same company practice different agile religions for extended periods, it gets harder to bring them together. They develop habitual behaviors. They malign competitive frameworks and exaggerate the benefits of their own. This process not only causes confusion, it actually creates animosity among colleagues who should be building balanced, harmonized business systems. Nowhere do these debates grow more heated than among proponents of agile, lean, and product management methods. We have seen fierce arguments nearly erupt into fisticuffs. We do realize that the people who try to break up fights always seem to get punched in the process, but someone needs to bring some sanity to this chaos. So here we go.

Lean is a source of considerable confusion because people apply the term to two very different approaches: lean production systems (also known as Lean Six Sigma), and lean product development (also known as lean start-up).

Lean production systems are tools for running the business, for improving the quality and efficiency of operations. They increase conformance to specifications, minimizing variability and reducing waste. Lean Six Sigma demands that no more than 3.4 out of one million outputs can fail to conform. It improves efficiency by continuously reducing eight sources of waste (defects, overproduction, waiting, nonutilized talent, transportation, inventory, motion, and extra-processing).⁴ We highly recommend lean production methods for improving operations. We do not recommend them for managing innovation. Innovation needs variability—even some inefficiency—to test, learn, and evolve. Some lean zealots continue to prescribe Six Sigma for innovation, but research shows that the better a culture gets at eliminating variability, the worse it gets at innovation.⁵

Lean start-up and product management are both methods for agile innovation. Lean start-up is famous for its widespread and highly publicized adoption by GE.⁶ It combines lean principles with design thinking and agile approaches to foster continuous innovation the way successful start-ups do.⁷ Product management encourages technology developers to think of themselves as product or brand managers who are responsible for developing profitable products that solve important customer frustrations, not just managing projects to deliver predetermined features. If the latter sounds a lot like other agile approaches, that’s because it is quite similar. Two substantive differences, however, are worth noting. First, while agile uses the same approach across all innovation activities, product management concentrates on products that are powered by technology. Second, product management advocates argue (convincingly) that the product owners of too many agile teams are simply managing backlogs and are not acting like true CEOs. They are not taking full responsibility for the results of the products they develop. They are not understanding the full market context, true customer needs, competitor positions, or the financial implications of complicated trade-offs. They are not deploying persistent teams to design and scale profitable solutions.⁸

These distinctions may seem fairly minor and indeed could be helpful to agile. But they can generate a surprising amount of confusion, conflict, and inefficiency. Some advocates contend that all product management teams must be persistent and that a company should never launch temporary teams to attack urgent, short-term problems. Some organizations are adding a new layer of strategic product managers to oversee agile product owners, rather than clarifying and upgrading the product owners’ jobs. Fanatics sometimes argue that product management teams are only for technology-related initiatives, forcing separate job titles, vocabularies, and training programs for technology innovations versus others. In our experience, harmonizing agile and product management methods to develop unified approaches that can be customized to the company culture and shared across the entire enterprise yields far better results than fostering adversarial dissension.

Good processes and technology are at the core of creating cost-effective solutions that customers will value. Moreover, the technologies available to automate and improve processes, such as robotic process automation and machine learning, have been growing at an accelerated pace. But the adoption of these technologies at many companies has been slow. The techniques described in this chapter should help companies to break through these barriers.

FIVE KEY TAKEAWAYS

1. Agile is for innovation, but innovation is more than creating new products and services for customers. Agile methods work equally well for improving the business processes that produce those products and services, and for improving the technology that enables those business processes.
2. Because reliability and efficiency are so important to business processes and technology, bureaucracy does everything possible to minimize variability and change. Agile approaches also encourage strict adherence to standard operating procedures, but regularly innovate to improve those procedures. They then ensure that the new procedures are accepted, incorporated in training, and properly executed.
3. Persistent, cross-functional agile teams are the best ways to improve business processes and technology. As teams gain experience and trust with operators, their development capabilities increase and adoption rates accelerate.
4. Agile teams working on business processes and technology are as customer-obsessed as those developing customer-facing products and services. Sometimes their innovations directly enhance consumer experiences. Other times, their customers are internal customers whose performance is vital to improving consumer experiences.
5. Many companies have a large number of agile technology teams, but few companies have agile technology organizations. The reasons include many of the same deviations from best practices that can inhibit any agile team plus several that are specific to software. Overcoming them involves making architecture more modular, helping engineers to become more versatile, and breaking down functional silos within and outside of IT departments.