Empirical Control

Expecting the Unexpected

During my studies in electrical and information technologies in the late 90s, I had the opportunity to work as a student assistant with a team of engineers and physicists at IBM. Their main task was the production of new hard-disk generations, manufacturing atomic-scale layers of metals and carbon in a clean room facility roughly the size of two football fields.

I was fascinated by the engineering, but equally surprised by how the team operated when bringing a new disk into production:

First, the team formulated a hypothesis outlining the parameters necessary to achieve the desired properties of the final disk. The hypothesis was grounded in theoretical models, laboratory experiments, and knowledge acquired from past products.

After producing a larger batch of disks under production conditions, they started measuring the characteristics of the disks. Most likely the disks failed to meet the desired properties, but the data gave the team hints where to adapt the parameters, which led them to plan the execution of the next cycle.

At times, it took the team weeks and numerous cycles to achieve their desired results. During this pursuit, they had to eliminate common issues, such as a stray hair or a fingerprint fragment in one of the chambers.

Development is the art of expecting the unexpected.

The team also found themselves needing to reevaluate their initial assumptions. For instance, they came across an issue involving minor but quantifiable deformations on the disks. Initially, the team hypothesized that these deformations were due to the plasma’s extreme temperature, a seemingly logical assumption. However, the real culprit was the forces produced by high acceleration when the disks were inserted into the production machine. As a result, they needed to optimize the loading robot’s movement.

Empirical: Based on what is experienced or seen rather than on theory

At the beginning of the millennium, I became familiar with eXtreme Programming as a programmer. It operates on much shorter but similar cycles to what the IBM team had used. You write a test that fails (red), write the code to pass it (green), then refactor while keeping all tests passing. One cycle complete, ready for the next.

I started to understand the impact of this principle for a larger organization in 2004 at Web.de, at the time one of the largest web portals in Germany. It was there that I experienced Scrum for the first time. Sprints offered us an expanded cycle: from a business concept to done functionality within weeks. What we later learned was that the highest form of “Done” means our users and customers have actively started using the new features. Only then do we gain the insights that serve as the definitive test for our business hypotheses.

Ken Schwaber had a term for this meta-process that he, Jeff Sutherland, and Mike Beedle had implemented into Scrum: Empirical Process Control. He had learned the concept from professionals in the chemical industry (Process Dynamics, Modeling, and Control) and made it the fundamental principle of Scrum.

But individual Teams running Sprints was only the beginning. In 2007, Andy and I helped Bwin scale Scrum across the entire development and IT department. When all Teams adopted the same rhythm, organizational noise and chaos dropped significantly. The structure we built there was later described almost exactly by Large Scale Scrum (LeSS), a pattern for a large number of Teams working on one product.

Looking back, these empirical control loops were not just optimizing the process to develop software. They controlled the entire result we were creating: our product, or in AME3 terms, the Arena Product.

Scrum, XP, and Agile did not invent this. Empiricism itself, as both a scientific and philosophical approach, dates back to the 17th century. In fact, you can find these loops in models, methods, and organizations across every domain:

While these loops differ in terminology and steps, they all share the same underlying pattern. AME3 distills this into three phases.

Anticipate, Advance, Assess

In AME3, there are two defined empirical loops. The first, at the tactical level, is the Match within an Arena. The second, at the strategic level, is the Tournament encompassing the entire Enterprise. They are structured into three phases:

Anticipate

Anticipate refers to the phase where the organization or work system formulates hypotheses and sets expectations for the upcoming cycle. This involves reconsidering strategic doctrines, constraints, and defining the parameters and conditions necessary to achieve the desired outcomes. Decision-making and planning are also integral components of this phase.

Advance

The Advance phase involves executing the planned actions and improvements by adhering to the strategic doctrines, constraints, tactics, and plans outlined during the Anticipate phase. This is where the organization actively engages in the process, whether it is developing a new product, conducting experiments, or implementing changes. During this phase, the organization gathers data and observes the outcomes, making real-time adjustments as needed to stay aligned with their goals and improvements.

Assess

The Assess phase focuses on evaluating the results of the actions taken during the Advance phase. This involves analyzing the data collected, comparing the outcomes against the initial hypotheses and expectations set during the Anticipate phase, and identifying any discrepancies or areas for improvement. The insights gained during this phase inform the next iteration of the AAA-Loop, allowing the organization to refine its strategies and approaches continuously.

The phases are not isolated from each other and can have some overlap. For instance, the IBM team collected a significant amount of data during the Advance phase and drew conclusions from it instantly, which can be considered as the Assess phase. However, in a meeting, they summarized these insights and anticipated future experiments before advancing to the next cycle.

Strategy and Tactics

The Rules of AME3 help implement the AAA-Loops and therefore the Empirical Control doctrine. The organization and facilitation of the Tournament and Match are determined by the System Leads and the particular methods and frameworks they elect to use. However, the shared rules ensure that the loops are interlocked.

As an example, the following image illustrates how the Tournament and Match integrate three frameworks. On the strategic level, the Tournament incorporates Wardley Mapping. On the tactical level, the Match integrates Scrum and a research cycle similar to the one used by the IBM team. However, each Arena likely has its own implementation for a Match, utilizing different methods and frameworks.

Integrating frameworks and methods

On the strategic level, the loop of the Tournament does not have its own Advance phase. This phase consists of the combined Matches of all Arenas and the remaining organization that does not operate within the AME3 framework yet.

This reveals a problem familiar to many executives and C-level managers. They formulate a strategy, but it does not get executed as intended. The reason is structural: strategy cycles do not execute anything. Only people in value creation execute. The key is to connect the strategic loop with the tactical loops so that strategic decisions translate into concrete action within Matches. This is why the function of the System Lead becomes so important. Their focus is to connect these loops with adequate practices and ensure the right methods and tools are in place. Frameworks like Scrum, Kanban, or Flight Levels provide exactly this connection.

Only Teams and individuals who actively collaborate with customers or solve problems for them will advance.

The Perfect Length of a Loop

The shorter the AAA-Loop, the quicker the improvement and optimization process. However, certain constraints may necessitate a longer period:

  • Technological complexity
  • Market complexity
  • Organizational complexity
  • Social constraints

Often, we tend to select the time based on the constraints at hand. For instance, in the development of a new drug, the duration of a clinical test seems to dictate the maximal time for the loop. However, this approach has a drawback: we accept our assumption that we could not accomplish the goal more quickly. As an example, the development of the COVID-19 vaccine by BioNTech demonstrated that perceived constraints were actually unchallenged assumptions, as evidenced by their rapid vaccine development initiative.

By setting a fixed timebox, we flip the problem on its head:

  • We challenge ourselves to break down the problem into smaller, more manageable parts from which we can learn.
  • We adopt a divide-and-conquer strategy to tackle the problem.
  • We concentrate on fewer problems and resolve them in less time, providing us with better data faster for future decisions.
  • We establish a rhythm that reduces noise and chaos in the organization.

Designing the Loop in Loop in Loop

So, designing an AAA-Loop seems to be a trade-off. In larger organizations, these loops need to interconnect and build upon each other. Combining numerous loops can result in confusion and disarray. Fortunately, the Agile community’s decades of experience have shown that three interlocked loops are sufficient for most small and medium enterprises:

  1. An hour to a day long AAA-Loop for tasks handled by individuals and the Team, led by
  2. a week to a month long AAA-Loop for end-to-end improvements involving one or multiple Teams, led by
  3. a quarter to a year long AAA-Loop for the overarching Goals and Ambitions of the Enterprise.

Several frameworks and methods effectively address one or two loops. Others are useful for supporting the loops, but do not define a loop on their own.

The challenge often lies in the fact that different parts of the organization may not operate in sync, which causes issues in communication and collaboration. Moreover, these organizational parts may not follow the same doctrine of empirical control, which is crucial for planning common Goals and integrating data from the lower AAA-Loops.

By elevating Empirical Control to an overarching strategic doctrine, the Enterprise ensures that various subsystems within the organization collaborate effectively within a generic structure.

AME3 aims to maintain these AAA-Loops as flexible as possible, allowing different organizational parts to select the specific processes needed for their situation while providing enough structure to ensure subsystem integration.

Next Level

Empirical Control is prevalent in various models, methods, and organizations. It is a fundamental component of effective product and service development and should, therefore, be the first strategic doctrine of an Enterprise.

In the context of AME3, these cycles are embodied in the Tournament and Match. But running empirical loops is not enough. The loops must serve the right scope. When different parts of an organization each optimize for their own results, the enterprise works against itself. The second strategic doctrine, Overall Optimization, ensures that every improvement benefits the whole.