Organisations become more and more Agile and this transformation takes years and might actually never complete. The entire organisation does not change at once or in lock-step. Some areas might spear-head the transformation because they have to, and traditional ways of software development simply did not work. Others are the laggards, diehards or are indeed successful in their “traditional” way of working because their projects succeed with careful up-front planning. Some need combinations of some of agile software development embedded in über-program-plans, because they are deeply embedded in organisational dependencies.

In part 1 of my blog I have explained the different types of organisational controls according to William Ouchi. The so called “clan based” control is akin to the shared values and beliefs of agile software development where individuals try to work effectively together much more via self-motivation rather than through outside control. To rely on this control mechanism however, we need to measure how complete or effective the agile transformation in our organisation already is.

In this post I explain how you can make agility visible through measurable KPIs (key performance indicators) and link these back to the values and principles of the Agile manifesto. These are not the typical KPIs or metrics agile teams use to monitor their own progress or performance, like burn-down charts or velocity. Instead, they are metrics to evidence the effectiveness of agile transformation and hence the “clan control” as organisational control mechanism.

Once these KPIs show us that the agile transformation has become effective and results in better software delivery, I will argue that this is now the time to relax traditional “bureaucratic” control mechanisms as they become counter-productive and risk to undermine the authority of the Enterprise PMO.

However, as long as there is still “transformation chaos”, agile values are not fully embedded or have not yet resulted in visible progress, we still want to stick to established controls.

The values and principles of the Manifesto for Agile Software Development

Let’s start by looking at the values and principles the original signatories of the Agile Manifesto stipulated:

There are 12 more detailed principles behind these values.

Whilst values and mindsets are difficult to measure in objective, data-driven KPIs, we need to help ourselves by looking at some proxies:

1. Leading indicators that examine the prerequisites for successful Agile software development (like proximity between business and IT that is a prerequisite for customer collaboration)
2. Lagging indicators that analyse how software is being delivered – like increase of release frequency, decrease in lead time or fewer bugs in production

Prerequisites for successful agile software development

The first graphic illustrates possible KPIs and their alignment to the respective agile values and principles.

KPIs to demonstrate agility in software delivery

The second graphic lists some output focussed KPIs that indicate progress in the actual software delivery.

These KPIs are a mere proposal that you should consider depending on the circumstances of your organisation. Some of them I have actually used and reported regularly in one large organisation I worked for and it was agnostic to any specific agile framework. If your organisation is mandating a specific method to implement Agile at scale (like LeSS or SAFe) you can consider adding additional indicators, however you might be tempted to measure compliance of teams with that Agile framework and you are back in the domain of bureaucratic controls.

How to measure, report and (not) set targets

I am strong advocate of sourcing KPI data from golden source systems. Some of them can come straight out of your PPM system (e.g. Planview or Clarity PPM): the team continuity can be derived from the team/resource plan of the project. Release frequency can be obtained from the PPM tasks/milestones. Others can be sourced from your tasks management system for Agile (e.g. JIRA or Rally), like the lead time.

Once you start taking data for new purposes you need to review the data quality as all these systems are used in a subtly different way from team to team so reporting existing data in a new context will reveal such differences.

Goodhart’s Law

According to Goodhart, once a measure becomes a target, it ceases to become a good measure. In organisations, if you confront teams with not meeting a specific KPI target, combined with senior management focus they will adjust their behaviour to meet this target. However, there is a high risk of manipulation and this will make you assume that the underlying performance has actually improved. See more in How to choose the wrong KPIs to measure your project progress?

Instead of setting hard targets you should consider to show directional improvements: A team increasing software releases from yearly to quarterly is demonstrating equally good transformation progress as one increasing from quarterly to monthly.

More health warnings of KPIs

1. KPIs are only a numerical indicator of something that is much more complex in real live. In the best case the indicator is good enough to be directionally correct to report the actual underlying performance. In the worst case it shows the opposite. 
2. The Hawthorne effect [1] showed that the act of measuring (via KPIs) also changes peoples’ behaviour by stating the intentions and priorities of the measurer.
3. If you decide to report on velocity do not try to compare this across teams, at best report changes in velocity
4. Stop reporting certain metrics if they have outlived their usefulness
5. What you start reporting to senior management is seen as important – so make sure it is actually important.
6. Ensure that all stakeholders actually understand the real meaning behind them and get their buy-in. There is a high risk of KPI-manipulation otherwise.

Analyse the data and transform your controls

First, aggregate the KPIs in heat map (red, amber, green colour coding of the values) by team, department or project. You can display directional improvements, RAG against hard thresholds or where teams are in a distribution.

Ideally this will now reveal some patterns in your organisation. Hopefully you can see pockets with good prerequisites of agile software development and also changed software delivery patterns.

Now take a bold move and relax some of your bureaucratic rules to reward those teams! They have established clan control and demonstrated the effectiveness. Put progressive teams onto the fast lane: Teams that have a high coverage of regression testing and automatic deployment (and back out) processes can be allowed to deploy software with fewer sign-offs. Teams that have established effective “Agile rituals” like weekly review committees with active engagement of their business sponsor, are expected to be better self-governed, give them dispensation to log fewer risks and issues in the central PPM system. 

If you do not act and adapt your controls in the Enterprise PMO, teams with effective clan control will start to challenge your authority, eroding the effectiveness of the bureaucratic control form.

On the contrary: teams that are seemingly agile, but have neither established the prerequisites nor can demonstrate better software delivery will need further control forms until their agile transformation has progressed. You now have evidence at hand to explain why you cannot (yet) relax your established controls.

Responding to change over follwing a plan

In part three of this series I will show how the Enterprise PMO itself can become more agile to “respond to change over following a plan”. Watch this space!

References

Sealights. ‘10 Powerful Agile Performance Metrics – and 1 Missing Metric’. Accessed 10 January 2020. https://www.sealights.io/software-development-metrics/10-powerful-agile-metrics-and-1-missing-metric/.

Beck, Kent et al. ‘Manifesto for Agile Software Development’, 2001. https://agilemanifesto.org/.

Butler, Jimmie. ‘Measuring Success in Agile Teams’. Accessed 14 January 2020. https://www.youtube.com/watch?v=o8TmUOAxi58.

Netmind. ‘Do Your Agile Metrics Measure Up? Why Traditional KPIs Won’t Work in an Agile Environment.’, 16 January 2019. https://netmind.net/do-your-agile-project-metrics-measure-up/.

‘Hawthorne Effect’. In Wikipedia, 10 January 2020. https://en.wikipedia.org/w/index.php?title=Hawthorne_effect&oldid=935152433.

---