Organisations become more and more Agile and this transformation takes years and might actually never complete. The entire organisation does not change at once or in lock-step. Some areas might spear-head the transformation because they have to, and traditional ways of software development simply did not work. Others are the laggards, diehards or are indeed successful in their “traditional” way of working because their projects succeed with careful up-front planning. Some need combinations of some of agile software development embedded in über-program-plans, because they are deeply embedded in organisational dependencies.

In part 1 of my blog I have explained the different types of organisational controls according to William Ouchi. The so called “clan based” control is akin to the shared values and beliefs of agile software development where individuals try to work effectively together much more via self-motivation rather than through outside control. To rely on this control mechanism however, we need to measure how complete or effective the agile transformation in our organisation already is.

In this post I explain how you can make agility visible through measurable KPIs (key performance indicators) and link these back to the values and principles of the Agile manifesto. These are not the typical KPIs or metrics agile teams use to monitor their own progress or performance, like burn-down charts or velocity. Instead, they are metrics to evidence the effectiveness of agile transformation and hence the “clan control” as organisational control mechanism.

Once these KPIs show us that the agile transformation has become effective and results in better software delivery, I will argue that this is now the time to relax traditional “bureaucratic” control mechanisms as they become counter-productive and risk to undermine the authority of the Enterprise PMO.

However, as long as there is still “transformation chaos”, agile values are not fully embedded or have not yet resulted in visible progress, we still want to stick to established controls.

The values and principles of the Manifesto for Agile Software Development

Let’s start by looking at the values and principles the original signatories of the Agile Manifesto stipulated:

There are 12 more detailed principles behind these values.

Whilst values and mindsets are difficult to measure in objective, data-driven KPIs, we need to help ourselves by looking at some proxies:

1. Leading indicators that examine the prerequisites for successful Agile software development (like proximity between business and IT that is a prerequisite for customer collaboration)
2. Lagging indicators that analyse how software is being delivered – like increase of release frequency, decrease in lead time or fewer bugs in production

Prerequisites for successful agile software development

The first graphic illustrates possible KPIs and their alignment to the respective agile values and principles.

KPIs to demonstrate agility in software delivery

The second graphic lists some output focussed KPIs that indicate progress in the actual software delivery.

These KPIs are a mere proposal that you should consider depending on the circumstances of your organisation. Some of them I have actually used and reported regularly in one large organisation I worked for and it was agnostic to any specific agile framework. If your organisation is mandating a specific method to implement Agile at scale (like LeSS or SAFe) you can consider adding additional indicators, however you might be tempted to measure compliance of teams with that Agile framework and you are back in the domain of bureaucratic controls.

How to measure, report and (not) set targets

I am strong advocate of sourcing KPI data from golden source systems. Some of them can come straight out of your PPM system (e.g. Planview or Clarity PPM): the team continuity can be derived from the team/resource plan of the project. Release frequency can be obtained from the PPM tasks/milestones. Others can be sourced from your tasks management system for Agile (e.g. JIRA or Rally), like the lead time.

Once you start taking data for new purposes you need to review the data quality as all these systems are used in a subtly different way from team to team so reporting existing data in a new context will reveal such differences.

Goodhart’s Law

According to Goodhart, once a measure becomes a target, it ceases to become a good measure. In organisations, if you confront teams with not meeting a specific KPI target, combined with senior management focus they will adjust their behaviour to meet this target. However, there is a high risk of manipulation and this will make you assume that the underlying performance has actually improved. See more in How to choose the wrong KPIs to measure your project progress?

Instead of setting hard targets you should consider to show directional improvements: A team increasing software releases from yearly to quarterly is demonstrating equally good transformation progress as one increasing from quarterly to monthly.

More health warnings of KPIs

1. KPIs are only a numerical indicator of something that is much more complex in real live. In the best case the indicator is good enough to be directionally correct to report the actual underlying performance. In the worst case it shows the opposite. 
2. The Hawthorne effect [1] showed that the act of measuring (via KPIs) also changes peoples’ behaviour by stating the intentions and priorities of the measurer.
3. If you decide to report on velocity do not try to compare this across teams, at best report changes in velocity
4. Stop reporting certain metrics if they have outlived their usefulness
5. What you start reporting to senior management is seen as important – so make sure it is actually important.
6. Ensure that all stakeholders actually understand the real meaning behind them and get their buy-in. There is a high risk of KPI-manipulation otherwise.

Analyse the data and transform your controls

First, aggregate the KPIs in heat map (red, amber, green colour coding of the values) by team, department or project. You can display directional improvements, RAG against hard thresholds or where teams are in a distribution.

Ideally this will now reveal some patterns in your organisation. Hopefully you can see pockets with good prerequisites of agile software development and also changed software delivery patterns.

Now take a bold move and relax some of your bureaucratic rules to reward those teams! They have established clan control and demonstrated the effectiveness. Put progressive teams onto the fast lane: Teams that have a high coverage of regression testing and automatic deployment (and back out) processes can be allowed to deploy software with fewer sign-offs. Teams that have established effective “Agile rituals” like weekly review committees with active engagement of their business sponsor, are expected to be better self-governed, give them dispensation to log fewer risks and issues in the central PPM system. 

If you do not act and adapt your controls in the Enterprise PMO, teams with effective clan control will start to challenge your authority, eroding the effectiveness of the bureaucratic control form.

On the contrary: teams that are seemingly agile, but have neither established the prerequisites nor can demonstrate better software delivery will need further control forms until their agile transformation has progressed. You now have evidence at hand to explain why you cannot (yet) relax your established controls.

Responding to change over follwing a plan

In part three of this series I will show how the Enterprise PMO itself can become more agile to “respond to change over following a plan”. Watch this space!

References

Sealights. ‘10 Powerful Agile Performance Metrics – and 1 Missing Metric’. Accessed 10 January 2020. https://www.sealights.io/software-development-metrics/10-powerful-agile-metrics-and-1-missing-metric/.

Beck, Kent et al. ‘Manifesto for Agile Software Development’, 2001. https://agilemanifesto.org/.

Butler, Jimmie. ‘Measuring Success in Agile Teams’. Accessed 14 January 2020. https://www.youtube.com/watch?v=o8TmUOAxi58.

Netmind. ‘Do Your Agile Metrics Measure Up? Why Traditional KPIs Won’t Work in an Agile Environment.’, 16 January 2019. https://netmind.net/do-your-agile-project-metrics-measure-up/.

‘Hawthorne Effect’. In Wikipedia, 10 January 2020. https://en.wikipedia.org/w/index.php?title=Hawthorne_effect&oldid=935152433.

KPIs – or Key Performance Indicators are an essential tool to measure whether projects deliver the desired business outcome during the project execution. Chosen badly they can also be used to mislead or even deceive the organisation. In this article I will explain how to select good KPIs and give examples of the impact of bad ones. Finally, I will share some thoughts on reporting KPIs in a portfolio context.

What is a KPI anyway?

According to Wikipedia,

A performance indicator or key performance indicator (KPI) is a type of performance measurement. KPIs evaluate the success of an organization or of a particular activity (such as projects, programs, products and other initiatives) in which it engages.

So the indicator tries to measure performance numerically. It simplifies the original, often abstract problem. E.g. if the reduction of operating expenses is your performance your indicator might be the reduction of staff.

The KPI measures what a project does to the organisation, i.e. the outcome of the project, it is not a measure of the project activities itself like the number of key milestones completed.  

Why KPIs matter

Establishing a good business case is often the hardest part of the project initiation. Reporting frequently the achievement of the business case during the project execution would consume considerable time and effort. KPIs simplify the problem by selecting a few that are key and good indicators of the performance  (i.e. the achievement of the business case). The calculation can be automated or at least repeated manually without too much effort so a project can report weekly, monthly or quarterly on its performance.

Characteristics of KPIs

To define good KPIs we need to look at the different categories. Some are better than others to indicate what we eventually want from them: predict the performance.

Leading and lagging

A lagging KPI is often an accurate indicator of success, but the dial is changed some time after the project success has happened. Most financial indicators fall in this category. An example are maintenance costs after decommissioning of software: you need to wait for the accounting period to close (month, quarter, year) before the actual costs become available.

A leading KPI tries to predict an outcome before it actually happened. E.g. the number of new features released in an app can try to predict product sales. However, as you see with this example this prediction then may or may not materialise.

Ideally you find a leading indicator to demonstrate early that you are on the right track and complement this with a lagging indicator to validate this later.

Relevant

A KPI must be relevant to the organisation and the business outcome of the project. If a particular product should be phased out, reporting on the number of new features will not be relevant any more. A relevant KPI is always aligned to your organisation’s strategy.

Specific and attributable

Projects do not operate in isolation. In most cases they run alongside day-to-day operations. Both the project as well other factors might impact the KPI. E.g. a project to improve the response time of a system might report the reduction in milliseconds to load a specific page. However, high or low system demand of existing users might also impact this KPI.

Ideally the day-to-day impact can be excluded in the definition of the KPI by normalisation (e.g. computing time per request). Alternatively, the impact is eliminated by the measuring method (e.g. measure the improvement of the response time in a sandbox environment rather than production).

Independently verifiable

The actual calculation of the KPI could be left to the project manager, the PMO or could be done by somebody outside the project team. However, it is important that an outsider can at least verify the results independently. This is achieved by ensuring

• Clear, unambiguous definition of the calculation
• Accessible data

The limitations of KPIs

KPIs try to condense the complex world of business outcomes, progress, written strategic objectives into one or few numbers. This inevitably loses information as the real world is much more complex.

How to manipulate KPI – a real live example

The following example is based on real events. A big company wanted to simplify its IT application landscape and reduce the number of applications. A large program – we call it AppExit here – was kicked off to take out hundreds of potentially redundant systems. The business case focussed on the cost reduction of maintaining fewer systems and less integration work for future projects.

As the actual cost reduction is a lagging indicator of success, the number of applications in production (or reduction thereof) was perceived to be the best leading indicator of success for this program. To make this indicator specific the team excluded any changes caused by other projects. Fair enough.

In the detailed preparation of AppExit the management discovered that the inventory of applications was not as good as expected. Some entries were entire families of applications, whilst others had bits and pieces, like interfaces or components registered as unique applications. A separate project AppInventory was initiated to define standards to register applications and conduct a one-off clean-up of the inventory itself.

The main AppExit program had a slow start as taking out applications was harder than expected. The majority of functionality could be transferred easily elsewhere. But the last 5% was difficult so the application lived on.

However, AppInventory was very successful: many entries of the inventory were “consolidated” using the new standard. Components of applications were incorporated into the entry of the main application. Then these entries were set to “decommissioned”. At program-level the total number of applications was reported to shrink steadily and management was happy. However, there was actually no change of the application landscape. Only the inventory was cleaned up. It took over two years for management to realise that AppExit was way behind delivering on its promised and it was eventually scrapped.

Lessons learnt

There was a hypothesis made that fewer applications lead to lower operating costs. Applications in production were counted by number of entries in the inventory. This KPI was

• Leading – as a lower number can be reported before the costs go down
• Relevant – cost reduction was part of the organisation’s strategy
• Clear in the definition and based on accessible data: the inventory was the official place to enquire about the status of an application
• It was also specific and attributable: applications removed by other projects were not counted.

All good? No!

Reducing only the number of entries in the inventory does nothing to the operating costs. Only when real applications are decommissioned and removed from the production landscape costs go down. The main mistake was that the inventory and not the real-world applications were counted.

KPI and portfolio reporting

In the chapters above we have talked mainly about the KPIs that are specific to a single project. In project portfolio management you have the additional challenge to report progress across multiple projects and KPIs provide a good source as they are expressed in numbers.

Many KPIs will be specific to a single project. Trying to aggregate them across different projects would add apples to pears. Project A counts the releases per year of an iPhone app and Project B the releases of your SAP system. Whilst you can total the releases of your portfolio A & B the results will be heavily overweight project A.

You cannot aggregate KPIs that are ratios as the denominator could be different. Price per user of €10 for system A and €20 for system B does not make an average price of €15 if the number of users was different.

Do not try to aggregate KPIs bottom up from different projects. Instead, define KPIs for your organisation and request your projects to include them in their performance measurement. This will help you to align your project portfolio to the strategy. E.g. if the improvement of IT Security is part of your strategy you could count the number of encrypted databases in your stock. Projects that encrypt databases must then report on this KPI. If you want to focus on cloud computing then count the number of applications using some cloud components. If you need to optimise your cost base then ask your relevant projects to provide their contribution to cost reduction.

Conclusion

So in this article you have learnt why KPIs matter in project (portfolio) management, how to classify them and ensure you select good ones and avoid bad ones. You have learnt how to use KPIs to manipulate the reporting of your project progress so you can challenge better any such attempt in the future. Last but not least I hope you have understood the limitations of KPIs. They only condense information into a few numbers. So use them as automatic additional metric to manage your portfolio, but do not expect that everything is perfect if only the KPIs tell you that.

If you are interested in learning more about the destructive power of KPIs I recommend the great book by Cathy O’Neil: Weapons of Math Destruction, now long listed for the New Yorker national book awards.

The featured photo is by Michal Mrozek on Unsplash

The common traps

I like the software as-is, but can you also do….? These questions are asked in the sales pitch. You get a nice demo from the vendor and you like what you see. But then you think about your internal processes and how you do stuff differently in your company. Every vendor will happily answer – of course we can! They know that they compete with other vendors and want to tick all the boxes with their product.
Your vendor might also be a software implementation “partner” who make their money with customisation of 3rd party products. Then later they sell you a maintenance contract to help you through ever more complex release upgrades.

You have written a long list of business requirements and now need to tick the boxes. They are very detailed but focus on implementing your current business process. Quite often they relate to either a manual process or your legacy application you want to replace. The closer the new system can satisfy your current process the easier it is to implement it and roll it out (you think!). However, you have not looked at how the new system would actually solve the business problem out-of-the-box.

Your current processes “suck”! They have evolved over time, maybe there is a good reason to run them that way, maybe not, nobody knows. What you do know is that you need to “tweak” the package a just a bit to run your process as it is. Sometimes you actually have more than one process in your company for doing the same thing. So the software needs a bit of “if … then … else …” coding to do the same step differently depending on where or who is running it. You think this could be harmonised, but this might require agreement with lot of different people, so now is not a good time.

Some other department tells you they need this change, otherwise they cannot possibly sign-off the application. You might sit in IT and the other function is your business. You are their service provider and you want to make them happy. You might think that they could change the way how they run things better, but they tell you that this is none of your business. You should be their partner, but they do not want your advice.

It is not customisation but configuration. There is a fine line between different the two types of changes, often defined by what is technically done to the product. For a customisation you need a developer, for the configuration only an administrator. However, there are heavy configurations that almost duplicate the original functionality. Those are worse than small customisations.

Getting out of the trap

Go back to basics and pull out your project charter document, somewhere in the objectives you have written about simplify the IT landscape, reduce support costs, harmonise the environment. These are your strategic objectives. Your customisations are in direct conflict with these objectives. Customised software is not simpler and does have higher support cost than a plain-vanilla configuration. Start to use business language to describe the impact of customisations. Consider the long-term impact on further development or maintenance, technical documentation and also user training. With this extra information, have a discussion with the right people who have really signed up to the charter and challenge every single request for customisation.

Keep a log of customisations if you really need to do them. Capture why they were required and who signed them off. This will be your log of technical debt, things that ideally should not exist because they get you further away from your target state. Whenever there is an opportunity, try to clear off items of your technical debt.

Stay in control of the decisions and do not leave them to 3rd parties. Vendors have different objectives than your company. They live off your money. So what is complex and expensive for you to build and maintain is their revenue stream for them. Some good vendors will partner with you to do the right thing, but it is important that you keep implementation decisions in-house.

Do not start with a long list of detailed business requirements when you buy a software package. When you build your own the case might be different, but when you buy a package it often comes with a good default configuration. Keep your business requirements as high-level and abstract as possible so that the software package has a chance to fulfil it in the way it was designed. Use the Agile methodology to start with the off-the-shelf configuration as minimum viable product. Really think hard in your use case and acceptance criteria what extra business value a story or epic will add.

Review the stakeholders of your implementation project. If you need to change processes to keep your software simple, do you have the right people at the table who can make these decisions? Establish a forum of like-minded people who can help you make the right decisions for the organisation and fully understand the long-term impact of getting it wrong. In one company I established a Design Authority with appointed process owners for topics like IT Governance, Demand Management, Project Accounting, Benefits Management, Enterprise Architecture etc. who had the authority in the organisation to either request software changes or change their respective processes.

Credits and further reading

• YouTeam.io – Custom or off-the-shelf-software
• Featured photo by Harley-Davidson on Unsplash