Exploring the Unexplored - Adventures in Leading a Large-Scale Agile Transformation

Leading a large-scale Agile transformation is hard. It requires relentless focus and discipline to create an environment that empowers teams to deliver innovation at pace. There are thousands of publications describing how this should be done in theory, yet not so many on how to translate that theory into practice. The majority of those that do exist relate to Agile in the context of software development.

Recently our team at Project4 Learning Lab was involved in a large-scale Agile transformation of a complex hardware product development program in a highly regulated industry within a multinational company. In this post, I want to share with you some of the key obstacles and steps that we went through while leading an Agile transformation.

"We Need to Fix It and Fix It Fast"

After a hard-fought sales campaign for a large hardware deal, we soon realized that the selected product would not meet the customer's needs in their specific environment.

"We need to fix it and fix it fast" came the voice of the business as the contract value was measured in billions. The problem was that there was no quick fix and at that point, there was no guarantee that a solution would even exist.

Nevertheless, the company decided to commit to a redesign of the product in order to tailor it to the specific customer environment. Our job was to ensure that we meet the customer's technical requirements and deliver the solution on time. Otherwise, we were facing significant financial penalties and a considerable risk of reputational damage in the event of late delivery.

Inadvertently, we had found ourselves as the top risk inside the business. Still, we were promised whatever we needed to get the job done.

"Have You Got a Concept Yet?"

The program started with a flourish and within days we had a "Tiger Team" in place, an approach generally reserved for crisis situations. A small and highly skilled cross-functional team with representation from across the business started working on potential options. The team rejected the Tiger Team label, instead, declaring themselves a "Falcon Team" with an intent to fly at high speed and change direction rapidly.

Over the course of the next 10 weeks, the Falcons evaluated thousands of potential technical solutions using simulations in conjunction with a "Design of Experiments" methodology.

The team grew to over 30 people as they down-selected 6 of the most viable concepts. Then they evaluated them for the implications on the existing supply chain as well as an introduction into service operation.

We completed the concept selection in record time and passed the first formal Phase Gate review with the "Technical Audit Team" without any customary dress-rehearsals. The conclusion from the team was endorsed by the "Audit Team" but was not good for the business. The scope of change required was large and the design and development program could take 3 - 4 years based on past capability.

"More People Please"

The pressure for progress was building and more people were joining the team. The good news was that more people meant more capacity. The bad news was that more capacity meant more integration. In turn, this meant more meetings, less progress, and again, the need for more people.

We were following a well-worn pathway and needed a fresh stimulus. Something had to change and without delay, we engaged some specialist consultants to help our thinking.

In our first workshop, we brought the whole project leadership team together with the technical specialists. We developed a broader understanding of requirements through "Quality Function Deployment" before completing a Vertical Value Stream Mapping activity to define the key elements of the Programme Master Schedule.

Shortly afterwards we brought the first 150 people from the team together to a program launch event to develop a new shared understanding of the process. Our keynote speaker was an explorer and one of the few people in history to have completed the polar trilogy. He emphasized the importance of planning, teamwork, risk management, and most importantly - reacting quickly to emerging conditions.

As a result, we had found our theme - Exploring the Unexplored.

Meanwhile, our Phase Gate dates were already scheduled according to our need rather than proven capability and soon we faced the "Audit Team" again to fix the system design. This time the mood could hardly have been more different and 2 days later we accepted our 9 critical corrective actions with disappointment.

"What Don't You Know?"

We had a need for speed and were looking for ways to create flow in product development without sacrificing quality.

We split the program into two phases - the exploration and execution phase. In the former, we were learning about the detailed requirements and potential solutions. In the latter, the entire production system had to be flawlessly delivered.

We knew that to accelerate the program in the execution phase, where the scope of work was clear, we had to identify and eliminate waste to create flow. We concluded that in the exploration phase, where the work required was not clear, we would need to create flow through identifying some key decisions and sequencing them correctly.

This way we could sequence and schedule our work and keep the growing number of teams aligned to the work that was required right now. We were repeatedly asking "What's stopping you from fixing the design today?" or "What don't you know?".

It was clear that speed of learning was going to be critical for us to deliver quality to our customers and successfully implement a large-scale Agile transformation, we would need to make the right decisions, the first time around.

As a whole team, we identified 25 key decisions in the exploration phase. We concluded what we needed to make them with no residual risk and identified what stopped us from accomplishing that - i.e. the things that we didn't know. Eventually, the teams started planning the work that they would need to do to turn the unknowns into knowns and make high-quality decisions.

We had a backlog of work that we would collectively burn through to accelerate the program. At this stage, we had grown to approximately 450 people working in 45 distinct teams and we didn't have an effective way of tracking progress and removing any impediments for the team.

"How Do You Know You Have Had a Good Day?"

We needed transparency and a regular forum for the teams to identify issues that they couldn't resolve themselves. We introduced daily stand-up meetings at three levels in the project - team level, sub-system level, and at the whole product/program level.

This wasn't popular everywhere with some teams noting that there were "not enough changes in a day to make a daily meeting worthwhile".

"Exactly!", came our reply with more than a raised eyebrow. We were convinced that we would start winning when most of us had a good day more often than not. To provide transparency, we were tracking decisions and associated learning points in each team using physical whiteboards.

We needed everyone to know whether or not they had a good day. It quickly became apparent that there were more bad days than good and we couldn't keep the pace that we needed to deliver at.

A brief investigation indicated that the teams were still doing the work that they would ordinarily do in addition to the decisions. In the highly coupled environment, this work was in turn generating more work for other teams, snowballing quickly.

We needed a way of removing unnecessary work and synchronizing the efforts of the teams.

"Don't Make Us Do This"

We decided that we would split the program up into 6-week intervals that, building on the exploration theme, became known as "Expeditions". Each Expedition started with a review of the progress made on the product design and the Master schedule. This was followed by intensive team-based planning ahead of the execution phase.

At the end of the Expedition, each team at each level would complete lessons learned review to identify changes to be made in the next one. In the first Expedition, we scheduled two days for the planning phase and after 6 days the teams hadn't finished.

Our first test of faith had arrived at an emergency meeting with the leadership team. "We have tried our best, but we can't get a plan. No other program plans at this level and we're losing valuable time. We think that we have done enough. Can we stop now?".

We had come too far and urged the teams to finish the planning. Two days later they were done. "Why are you doing this to us?" asked a young engineer in the office one day. "Why do you need to know what we are doing every week?".

"We don't but we need you to know what you are doing every week" was our reply. We didn't make the progress that we needed to in this Expedition but there were early signs that the teams were starting to align with each other and schedule their own work accordingly.

More good days were ahead of us.

"When We Commit to a Deliverable, We're Telling You that We Want to Do It but We Don't yet Know How"

We learned from the first Expedition that we needed more structure in the planning phase to coordinate the work of every team. The situation was complicated by the fact that the teams were globally dispersed (both whole teams and team members) and some team members were working part-time on the program, whilst simultaneously supporting up to 3 or 4 other programs.

We set a schedule that considered all time zones involved, including "speed dating" time slots when each team could meet another and talk about what they required from each other, when and why.

We secured commitment from the managers of part-time team members to enable them to support the planning activity and to guarantee a proportion of their availability for the next 6 weeks. This time the planning was completed in 6 days and the collaboration between teams was growing.

There was a new energy in the program, engagement levels were rising as people proposed and developed their ideas to improve the product. However, this wasn't translating into the pace of progress required. The teams were so engaged that they were committing to deliverables that they had no idea how to deliver! Despite this, decisions were starting to flow and through our daily stand-ups, we were noticing more good days than bad.

"Why Isn't Everyone Doing This?"

12 weeks after starting we launched our third Expedition, determined to limit the amount of work that people were doing whilst focusing on the most important elements. We trialed the idea of a prioritized list of "Golden Threads" - the key deliverables needed in the next six weeks to deliver the program.

Once the arguments in the leadership team had subsided, we had a ranked list that would support the teams as they prioritized their work. Leading the Agile transformation, we also implemented capacity-based planning - each team evaluated the work required to produce the key deliverables and stopped committing when they didn't have available capacity.

What we expected was that the relationship between the committed deliverables and "Golden Threads" was clear. In other words, teams were planning to do the most important things, and that the estimates for the work content were open to scrutiny. If those rules were met and the team could not deliver everything required, we would then try to secure more resources from the wider organization.

Up until now all of the Expedition planning had been done using flip charts, post-it notes, and spreadsheets. Whilst it was OK, it didn't give us the transparency to align and engage the teams effectively at scale.

Now that we had a prioritized backlog at program level, we experimented with a digital Kanban system. We initially selected three teams to complete all of their planning and execution using the system. Throughput improved by around 30% in this Expedition and we had closed all of our critical audit actions.

"Why isn't everyone in the organization doing this?" asked the previously dissenting young engineer.

"What Is It that You Are Doing?"

In the next Expedition, we rolled the digital Kanban system out across all teams for a scaled Agile solution, fuelled by the previous success and convinced of the benefits of transparency and alignment. This made the planning significantly easier and the end results had even more coherence.

By now the engagement had sky-rocketed and we introduced the "Wall of Awesomeness", a physical space where we could display and celebrate the amazing things that people and teams were doing. We were attracting attention from other program teams in the organization and curiosity from our external partners.

"What is it that you are doing?", they asked, closely followed by "Can you show us?". The number of visitors to the daily stand-ups at all levels increased with each passing week as word spread.

Whilst the problems and challenges remained, our ability to detect them and react accordingly developed and we were making more progress with each Expedition. We had built a structure, systems, and routines that were key to our success and simultaneously needed incremental improvement.

"Can We Do It too?"

Next, we designed a system of "Process Confirmations" for leaders at all levels to perform small, frequent tasks to confirm that the framework was being followed and was leading to the expected results.

This encouraged transparency even further and gave leaders permission to go to the place where the work was being done to provide feedback to others. Whilst this felt mechanical and artificial in this Expedition, it soon developed into the new normal and boosted the sense of us as one big team learning together even further.

By now, we were advising other programs on how they might do similar things and our partner was proudly telling us that they were running their program in 6-week increments to align with us too. The program continued running Expeditions for the next 6 months, making an increasing number of small improvements each time.

"What Did You Learn from that?"

Whilst we were busy leading an Agile transformation to successfully deliver the program, the industry that we were working in was undergoing a major change.

In this context, and after the organization negotiated a new, mutually beneficial contractual agreement with the customer, the program was canceled. Whilst this was a crushing disappointment to all of us who had invested so much physically and emotionally to deliver excellence to our customers, we were able to conclude our final lessons learned review with pride and gratitude for the experience.

We learned so much from the teams that we were privileged to lead through a large-scale Agile transformation. Stay tuned for the second part of our post where we will describe our reflections on the experience and what insights we can take from it.