Organisations become more and more Agile and this transformation takes years and might actually never complete. The entire organisation does not change at once or in lock-step. Some areas might spear-head the transformation because they have to, and traditional ways of software development simply did not work. Others are the laggards, diehards or are indeed successful in their “traditional” way of working because their projects succeed with careful up-front planning. Some need combinations of some of agile software development embedded in über-program-plans, because they are deeply embedded in organisational dependencies.

In part 1 of my blog I have explained the different types of organisational controls according to William Ouchi. The so called “clan based” control is akin to the shared values and beliefs of agile software development where individuals try to work effectively together much more via self-motivation rather than through outside control. To rely on this control mechanism however, we need to measure how complete or effective the agile transformation in our organisation already is.

In this post I explain how you can make agility visible through measurable KPIs (key performance indicators) and link these back to the values and principles of the Agile manifesto. These are not the typical KPIs or metrics agile teams use to monitor their own progress or performance, like burn-down charts or velocity. Instead, they are metrics to evidence the effectiveness of agile transformation and hence the “clan control” as organisational control mechanism.

Once these KPIs show us that the agile transformation has become effective and results in better software delivery, I will argue that this is now the time to relax traditional “bureaucratic” control mechanisms as they become counter-productive and risk to undermine the authority of the Enterprise PMO.

However, as long as there is still “transformation chaos”, agile values are not fully embedded or have not yet resulted in visible progress, we still want to stick to established controls.

The values and principles of the Manifesto for Agile Software Development

Let’s start by looking at the values and principles the original signatories of the Agile Manifesto stipulated:

There are 12 more detailed principles behind these values.

Whilst values and mindsets are difficult to measure in objective, data-driven KPIs, we need to help ourselves by looking at some proxies:

  1. Leading indicators that examine the prerequisites for successful Agile software development (like proximity between business and IT that is a prerequisite for customer collaboration)
  2. Lagging indicators that analyse how software is being delivered – like increase of release frequency, decrease in lead time or fewer bugs in production

Prerequisites for successful agile software development

The first graphic illustrates possible KPIs and their alignment to the respective agile values and principles.

KPIs to demonstrate agility in software delivery

The second graphic lists some output focussed KPIs that indicate progress in the actual software delivery.

These KPIs are a mere proposal that you should consider depending on the circumstances of your organisation. Some of them I have actually used and reported regularly in one large organisation I worked for and it was agnostic to any specific agile framework. If your organisation is mandating a specific method to implement Agile at scale (like LeSS or SAFe) you can consider adding additional indicators, however you might be tempted to measure compliance of teams with that Agile framework and you are back in the domain of bureaucratic controls.

How to measure, report and (not) set targets

I am strong advocate of sourcing KPI data from golden source systems. Some of them can come straight out of your PPM system (e.g. Planview or Clarity PPM): the team continuity can be derived from the team/resource plan of the project. Release frequency can be obtained from the PPM tasks/milestones. Others can be sourced from your tasks management system for Agile (e.g. JIRA or Rally), like the lead time.

Once you start taking data for new purposes you need to review the data quality as all these systems are used in a subtly different way from team to team so reporting existing data in a new context will reveal such differences.

Goodhart’s Law

According to Goodhart, once a measure becomes a target, it ceases to become a good measure. In organisations, if you confront teams with not meeting a specific KPI target, combined with senior management focus they will adjust their behaviour to meet this target. However, there is a high risk of manipulation and this will make you assume that the underlying performance has actually improved. See more in How to choose the wrong KPIs to measure your project progress?

agile observer effect

Instead of setting hard targets you should consider to show directional improvements: A team increasing software releases from yearly to quarterly is demonstrating equally good transformation progress as one increasing from quarterly to monthly.

More health warnings of KPIs

  1. KPIs are only a numerical indicator of something that is much more complex in real live. In the best case the indicator is good enough to be directionally correct to report the actual underlying performance. In the worst case it shows the opposite. 
  2. The Hawthorne effect [1] showed that the act of measuring (via KPIs) also changes peoples’ behaviour by stating the intentions and priorities of the measurer.
  3. If you decide to report on velocity do not try to compare this across teams, at best report changes in velocity
  4. Stop reporting certain metrics if they have outlived their usefulness
  5. What you start reporting to senior management is seen as important – so make sure it is actually important.
  6. Ensure that all stakeholders actually understand the real meaning behind them and get their buy-in. There is a high risk of KPI-manipulation otherwise.

Analyse the data and transform your controls

First, aggregate the KPIs in heat map (red, amber, green colour coding of the values) by team, department or project. You can display directional improvements, RAG against hard thresholds or where teams are in a distribution.

Sample of heat map

Ideally this will now reveal some patterns in your organisation. Hopefully you can see pockets with good prerequisites of agile software development and also changed software delivery patterns.

Now take a bold move and relax some of your bureaucratic rules to reward those teams! They have established clan control and demonstrated the effectiveness. Put progressive teams onto the fast lane: Teams that have a high coverage of regression testing and automatic deployment (and back out) processes can be allowed to deploy software with fewer sign-offs. Teams that have established effective “Agile rituals” like weekly review committees with active engagement of their business sponsor, are expected to be better self-governed, give them dispensation to log fewer risks and issues in the central PPM system. 

If you do not act and adapt your controls in the Enterprise PMO, teams with effective clan control will start to challenge your authority, eroding the effectiveness of the bureaucratic control form.

On the contrary: teams that are seemingly agile, but have neither established the prerequisites nor can demonstrate better software delivery will need further control forms until their agile transformation has progressed. You now have evidence at hand to explain why you cannot (yet) relax your established controls.

Responding to change over follwing a plan

In part three of this series I will show how the Enterprise PMO itself can become more agile to “respond to change over following a plan”. Watch this space!

References

Sealights. ‘10 Powerful Agile Performance Metrics – and 1 Missing Metric’. Accessed 10 January 2020. https://www.sealights.io/software-development-metrics/10-powerful-agile-metrics-and-1-missing-metric/.

Beck, Kent et al. ‘Manifesto for Agile Software Development’, 2001. https://agilemanifesto.org/.

Butler, Jimmie. ‘Measuring Success in Agile Teams’. Accessed 14 January 2020. https://www.youtube.com/watch?v=o8TmUOAxi58.

Netmind. ‘Do Your Agile Metrics Measure Up? Why Traditional KPIs Won’t Work in an Agile Environment.’, 16 January 2019. https://netmind.net/do-your-agile-project-metrics-measure-up/.

‘Hawthorne Effect’. In Wikipedia, 10 January 2020. https://en.wikipedia.org/w/index.php?title=Hawthorne_effect&oldid=935152433.


Agile software development is here to stay and has quite a dramatic impact on the “traditional Enterprise PMO”, the department that was originally set-up to bring order into the chaos of projects and software development.

In a series of blog posts, I will explain firstly the principles of three organisational control mechanisms: the market-, bureaucratic and “clan-based” control system. Depending on the scenario some work better than others and sometimes only the clan-based system achieves the desired control (hint: this is about Agile).

The Enterprise PMO needs to understand if a particular control system works in a given scenario. I will then give some ideas how the PMO can gather evidence via metrics and KPIs to what extent the outcome of Agile principles can be evidenced to become acceptable as control mechanism.

Finally, I will explain how the PMO can transform itself by adopting agile concepts to stay relevant in the long run.

Overview

Part 1 – Principles of organisational control

Part 2 – Agile transformation is a process, not an end-state

Part 3 – Active Portfolio Management – Agile values at the core of the new PMO (not published yet)

Design of Organizational Control Mechanisms

The Market, the Bureaucracy, the Clan

Before I start with the more practical aspects of control in Agile software development, I would like to introduce some theory behind Organizational Control mechanisms that was originally published in a series of papers by Prof. William G. Ouchi starting in 1979  and has often been cited in works about the theory of Agile software development. I will apply these concepts to the requirements of an Enterprise PMO and Project Portfolio Management.

Ouchi outlines three principal control mechanisms that can be employed to achieve control in an organisation.

The market system

In the context of PPM, the market system can be used as a control mechanism if the project outcomes or benefits can be linked easily to the market environment of the organisation. A project that delivers new products can be controlled by observing the revenues created by these products. If we can measure client satisfaction reliably and relate this back to project deliverables then this gives us external benchmarks of our project outcomes. To solely rely on market control however, we either need short projects (so we can wait until their end to verify the outcome) or the ability to observe progress of the outcome during the project delivery phase

The bureaucratic system

This is often at the heart of many PMOs: there are rules how to submit, plan, budget, manage and close projects; how to deal with risks and issues and how to escalate them. At approval or toll-gates and sometimes during execution there are checks for compliance; and the degree of compliance is seen as the factor of success of the project. How the agile practitioners view this kind of PMO is neatly summarised in “The Agile PMO – Smile, it’s the Process Police

Bureaucratic systems work best when there is a good understanding of the “transformation process”, i.e. there is enough expertise with the project type and evidence that following a prescribed process improves the chances of success or at greatly diminish the likelihood of failure.

A well-researched bureaucratic control system is used in commercial air traffic. Pilots have to go through pre-flight checklists to ensure proper procedures are followed and this has greatly increased aviation safety (and by the way: pilots are being checked regularity that they have completed the checklists).

Frameworks like PMI or Prince2 have helped to improve project success, however only when applied to suitable project types, adapted to the respective environment and within appropriate governance structures (Joslin & Müller, 2014).

Limited understanding of achieving success

Sometimes there is limited understanding how success can be achieved. Ouchi uses the example of the buyer in a women’s boutique as example when following rules cannot guarantee to be successful. Whilst success of the decisions of the buyer can be clearly measured using market control (sales, required reductions, items that are left unsold etc.) it is impossible to prescribe via rules how to be a good fashion buyer, so a bureaucratic control system would not work. It would only create overhead and be intrusive without delivering the form of control that is expected.

The clan system

The “clan system” is based on informal value-based rules that allow innovation and collaboration. Individuals work well together and do the right thing because they share the same values, methods, beliefs. It requires individuals who have completed lengthy periods of socialization to share the values of the clan. Ouchi also explains that clans “cannot cope with heterogeneity nor with turnover”.

The clan correlates with the values and principles of the Manifesto for Agile Software Development, in particular value #1 – “we value individuals and interactions over processes and tools”.

In Ouchi’s last example: a research and development centre, he explains that neither market nor bureaucratic control systems work: several years might have gone by before any research result becomes marketable and following strict rules will not ensure you make new discoveries. You are left with clan control as the only option.

So, only clan control then?

A critique of Ouchi’s model was that he did not argue for a combination of different control forms (see Organizational control and project performance. Paper presented at PMI® Research Conference). So instead of using exclusively one control method over the other I will also recommend combinations if they bring control: if we can link our projects to outcomes – we should do so! This will give us the evidence that they bring us forward. If we care how our projects are delivered, then we should ensure that there is a common understanding of values. If following good governance is a predictor of success then we should ensure it is being followed!

Use cases for control methods

Ouchi has drawn use cases by two dimensions

  1. Can I measure the outcome?
  2. How well do I understand the transformation process?

The chart above shows possible control mechanisms depending on the combination of these two factors with some example type of projects or activities to control. Whilst clan control (i.e. agile methods in our PPM context) can be complementary control methods they might be the only one if we can neither understand nor directly control the transformation process nor measure the outcomes.

So, is there clan control yet?

If we want to – or need to – rely on clan control we still have to ascertain that it actually works. Putting teams into clans, scrums or release trains and announce that they are now agile is not enough. It takes typically several years to embed values of the agile manifesto, together with its principles and also specific implementations (e.g. SAFe, LeSS, Scrum etc.) into the minds and hearts of all affected employees.

To rely on clan control we need to ascertain whether the agile transformation in our organisation is already effective. The adoption of agile values is difficult to measure. However, we can start with measuring the impact of agile transformation on software development, e.g. the frequency of releases should increase, the lead time from request to production decrease etc.  We observe that the agile transformation is changing the way how software is being delivered and has become effective. Being effective we can hence rely more on “clan control”.

In part 2 of this series I will explain how we can make the progress of the agile transformation visible for our Enterprise PMO and will suggest some metrics and key performance indicators sourced from PPM and other systems (watch this space)!

References

Elatta, Sally, and Anthony Mersino. ‘An Agile PMO Transformation’. presented at the PMO, Agile Practises, 23 October 2012. https://www.pmi.org/learning/library/pmo-agile-transformation-6063.

Liu, Li, Philip W. Yetton, and Christopher Sauer. ‘Organizational Control and Project Performance’. 14 July 2010. https://www.pmi.org/learning/library/organizational-control-modes-ouchi-model-6447.

‘Manifesto for Agile Software Development’, 2001. https://agilemanifesto.org/.

Mitchell, Ian. ‘The Agile PMO’. Scrum.org. Accessed 10 January 2020. https://www.scrum.org/resources/blog/agile-pmo.

Ouchi, William G. ‘A Conceptual Framework for the Design of Organizational Control Mechanisms’. Management Science, 1 September 1979. https://doi.org/10.1287/mnsc.25.9.833.

Robert, Joslin, and Ralf Müller. ‘Project Methodologies’ Impact on Project Success in Different Contexts’. Accessed 10 January 2020. https://www.pmi.org/learning/library/project-methodologies-impact-success-contexts-8947.

Scaled Agile Framework. ‘SAFe 5.0 Framework’. Accessed 14 January 2020. https://www.scaledagileframework.com/.

Tikka, Ari, and Ryan Nyman. ‘Scaling Agility or Bureaucracy’. Gosei (blog). Accessed 16 January 2020. http://gosei.fi/blog/scaling-agility-or-bureaucracy/.

In simple terms project management is about doing projects right, Project portfolio management (PPM) is about doing the right projects – and ensuring those are done well. PPM complements successful project management and ensures that not only individual projects become successful, but the entire organisation achieves success through its projects. If you want to know more what this is all about, then read further.

Table of contents

Is PPM just project management on steroids?

PPM consists of various disciplines, some of those use data of individual projects as input for decision-making. Without project management as basis, project portfolio management becomes difficult if not impossible. However, there are additional data points required from “the top of the house”, financial controlling and other departments that determine standards and directions for the organisation to help in the selection process.

According to PMI – Portfolio Management Professional PfMP there are five domains required for a comprehensive PPM. Let’s look at those disciplines or domains one by one:

Strategic Alignment

This domain includes activities required for aligning portfolio components (initiatives, programs, projects) with organisational strategic objectives. For successful strategic alignment it is paramount that the PPM

  • Knows the strategy and the strategic objectives of the firms
  • Is able to express and capture this strategy in a way to align the portfolio to it, e.g. in strategy management plans or strategic objectives

Key tasks are

  • Identify existing and potential projects or project ideas, those must be able to express their individual contribution towards the strategy
  • Create different scenarios (what-if-analysis) and recommend for decision making
  • Determine the impact of decision making on the portfolio (what if we need to reduce our investment budget, what if we can add additional funds, what if we’d exit Russia)
  • Create a high-level portfolio roadmap

Governance

I like to describe this domain as the “key operating procedure (KOP)” for portfolio management. This domain includes activities related to establishing the governance model (how you run projects and the portfolio as a whole), developing a portfolio management plan and approving the portfolio.

Key tasks are

  • Establish PPM standards, rules and best practises. These might be aligned to industry standards for project/program management like PMI, MSP, Prince2 or IPMA.
  • Select, introduce and operate systems to structure the PPM
  • Define processes and procedures for e.g. benefits realisation management, risk and issue management, stakeholder management, resource management, change management, dependency management, assumptions management
  • Create the portfolio governance model, i.e. define roles and responsibilities of project manager, sponsor, SteeringCommittee with regards to escalation procedures, risk tolerances, (monetary) approval thresholds
  • Define check-points for projects, e.g. tollgates or transition check points before projects move into execution or closure.
  • Ensure awareness of stakeholders with portfolio governance, this includes providing documentation, training and information material

Portfolio Performance

In Portfolio Performance we manage the portfolio using the processes defined in the governance domain. There is continuous monitoring and evaluation of the components with heavy focus on structured reporting.

Key tasks are

  • Collect and consolidate key performance metric data, e.g. achievement of milestones, financial performance, progress towards defined KPIs and measure progress against defined strategic goals and objectives
  • Manage and escalate risks and issues and recommend actions to appropriate decision makers
  • Analyse and monitor dependencies between elements of the portfolio or against particular units of the line organisation, vendors or other external bodies
  • Manage portfolio changes using change management techniques
  • Balance the portfolio and optimise resource utilisation (people, vendors, facilities)
  • Provide reporting or make available data so that appropriate stakeholders stay informed
  • Maintain records by capturing portfolio artefacts, like approvals or other decisions and ensure compliance with internal organisational policies, audit or legal or regulatory requirements

Portfolio Risk Management

This domain is closely related to Strategy Management: for every strategic objective there is a risk attached not to achieve it. In total the organisation defines its specific “risk appetite” and then we need to balance the portfolio against the risk appetite. We can recommend not to start or stop a project, analyse and aggregate project risks, establish a financial reserve and promote common understanding and stakeholder ownership of portfolio risks.

Communications Management

This domain is about continuously communicating with stakeholders, understanding their needs and expectations, fostering appropriate stakeholder engagement in portfolio decisions and activities, managing conflicting interests.

What is a PMO?

There are multiple meanings of the term PMO – all somewhat around project management:

  • Project Management Office/Officer: here it is used to describe the guy(s) who is/are in charge of the project admin stuff, typically reporting to the project manager. Not accountable for anything but does all the work. Many projects would not work without the project management officer, she is in charge of the standards and quality and is often the go-to person for all sorts of queries about the project. They can also help with continuity, e.g. familiarise a new joiner-PM to get familiar with the PPM processes of the organisation or train the project team on key project processes.
  • Program Management Office/Officer: same as above, but in charge of the admin work across a program, typically spanning multiple projects.
  • Portfolio Management Office: this term is typically used for the (Enterprise) Project Portfolio Management. This teams brings it all together and runs the portfolio of the organisation. Bigger organisations might have one central PMO and then additional divisional PMOs.

Do I need to be a project manager to run a good PPM?

There is a German saying “you do not need to be a chicken to tell a good egg from a bad one”. So, in principle you do not need to be a project manager to run a PPM. In particular if an organisation has got a bigger PPM office not all members need to be trained/qualified/experienced project managers themselves. However, there are several activities within a PPM where project management expertise is a key factor of success:

Challenging project managers

The governance domain will define various check points where inevitably you will find some “bad eggs”. The higher your personal experience with doing it better, the more likely the stakeholders will buy into your findings and observations.

SWAT Team / Intensive Care Unit

Some organisations establish special teams of highly qualified individuals to step in when extra help is needed, this could be

  • Help jump-start special or challenging projects
  • Step in when things turn sour, e.g. help a failing project to recover
  • Interim management of projects until the first or replacement PM is found

Promoting best practises

More often than not the PPM office is seen as some kind of internal police or audit function. To establish better perception, it is helpful to also promote best practises for project or program management. Effective training is best provided from experienced professionals to professionals.

My organisation does not have a PPM – are we doomed?

First of all, this obviously depends on the size of the organisation or and the complexity of the project portfolio. Small organisations will not need a dedicated PPM or PMO office. Other disciplines, like performance management of the project portfolio could also be covered by (financial) controlling, but when size and complexity grows organisations without PPM loose effectiveness in their change operations.

From my experience there are specific aspects where a professional, dedicated PPM is essential:

  • If you leave financial controlling to the accountants, they will focus on costs only. Underspend is good. In a project context however, underspend could be slow start, under delivery or under-resourcing of projects.
  • Standards in the organisation: a central PPM can define standards (e.g. what is “amber” or “red” in status reporting) that all projects must follow. Only these standards can ensure that management information is objective and comparable.
  • Professionalisation of the project manager profession: some train to become a project manager, some learn it by doing, some are simply appointed. A PPM function can promote minimum standards, require certifications or provide professional training. It can also ensure that the best project managers are put on the most challenging projects.

I thought portfolio management is about investment theory?

True, and there are a lot of similarities between investment portfolio management and project portfolio management

  • Projects are also investments: there is risk (of project failure) and return (if the project delivers its benefits). A balanced portfolio of projects should include some risky, speculative projects, but should not put the entire organisation at risk.
  • In investment and project portfolios the items are by-and-large independent but to some extent correlated. In investment portfolios there is correlation (“beta”) between investments, in project portfolios there are dependencies – so performance of one investment/project to some degree depends on the performance of the others.
  • There is a selection process: you want to get the most bang for your buck by either finding the best investment opportunities or the best project proposals. Funds are always limited so investing in everything or doing every project idea is not possible.

Is it true that every project manager hates the Enterprise PMO?

Yes (mostly). Who wants to be hold to account to their promises from the project start when even their line manager has already forgotten? Who wants to be asked awkward questions about compliance with standards and (not) having provided minimum required information? Who wants to be told that they cannot proceed this toll-gate as their project is not ready for execution?

However, if the PPM / EPMO does its job well, they become respected. They will never be loved (nor should they).

We are becoming an Agile organisation; do I now lose my job?

It depends. When the organisations change, the Enterprise PMO has to change too. If you stick to annual budgeting of projects and focus on milestone tracking and change request approval you will soon be seen as a prohibitor of the Agile transformation of your organisation.

There are lots of challenges to the traditional PPM / Enterprise PMO. With more and more projects becoming Agile product development cycles and the traditional toolset of the Enterprise PMO has to adapt. This means processes like project approval, tracking of project progress, annual demand management, review and approval of change requests need to change. If it doesn’t, the Agile community can rightly claim that they are tracked with the wrong tools.

Change request management has its own challenges when budget and resources and often release dates are fixed, but scope and approach is flexible.

The PPM must evolve and focus on tracking what really matters – achievement of business outcomes rather than project milestones. Help with solving risks, issues and resource requirements the Agile project team cannot solve alone. And translate information from Agile projects into language that senior management understands and can act upon. Project management – or better – the structured development of software – has evolved, so must the project portfolio management.

To keep your job, I recommend the following:

  1. Engage with the Agile community and learn from them. Understand their language so that you know what they are talking about.
  2. Review if any of your processes (or even your language) have waterfall bias and change it.
  3. Agile produces a lot of data that can be used to demonstrate progress and provide transparency. Often the terms used (e.g. burn-down chart, story points, velocity, MVP – minimum-viable-product) are difficult to grasp by senior management (unless they are Agilists themselves). You can help translate this data so your senior management can understand it.
  4. Whilst Agile very much focusses on the availability and capacity of resources the traditional PPM tools and processes can help with the structured management of resources – especially with shared resources (e.g. IT Architects or testers) that are required to work across teams.
  5. Read about how to coordinate large Agile portfolios – e.g. the Scaled Agile Framework (SAFe). Agile teams will do a lot of coordination within their framework, even without your help. Coordination beyond – say with waterfall projects, financial cycles, prioritisation of business value etc. is where the new Agile-supporting Enterprise PMO can add value.

KPIs – or Key Performance Indicators are an essential tool to measure whether projects deliver the desired business outcome during the project execution. Chosen badly they can also be used to mislead or even deceive the organisation. In this article I will explain how to select good KPIs and give examples of the impact of bad ones. Finally, I will share some thoughts on reporting KPIs in a portfolio context.

What is a KPI anyway?

According to Wikipedia,

A performance indicator or key performance indicator (KPI) is a type of performance measurement. KPIs evaluate the success of an organization or of a particular activity (such as projects, programs, products and other initiatives) in which it engages.

So the indicator tries to measure performance numerically. It simplifies the original, often abstract problem. E.g. if the reduction of operating expenses is your performance your indicator might be the reduction of staff.

The KPI measures what a project does to the organisation, i.e. the outcome of the project, it is not a measure of the project activities itself like the number of key milestones completed.  

Why KPIs matter

Establishing a good business case is often the hardest part of the project initiation. Reporting frequently the achievement of the business case during the project execution would consume considerable time and effort. KPIs simplify the problem by selecting a few that are key and good indicators of the performance  (i.e. the achievement of the business case). The calculation can be automated or at least repeated manually without too much effort so a project can report weekly, monthly or quarterly on its performance.

Characteristics of KPIs

To define good KPIs we need to look at the different categories. Some are better than others to indicate what we eventually want from them: predict the performance.

Leading and lagging

A lagging KPI is often an accurate indicator of success, but the dial is changed some time after the project success has happened. Most financial indicators fall in this category. An example are maintenance costs after decommissioning of software: you need to wait for the accounting period to close (month, quarter, year) before the actual costs become available.

A leading KPI tries to predict an outcome before it actually happened. E.g. the number of new features released in an app can try to predict product sales. However, as you see with this example this prediction then may or may not materialise.

Ideally you find a leading indicator to demonstrate early that you are on the right track and complement this with a lagging indicator to validate this later.

Relevant

A KPI must be relevant to the organisation and the business outcome of the project. If a particular product should be phased out, reporting on the number of new features will not be relevant any more. A relevant KPI is always aligned to your organisation’s strategy.

Specific and attributable

Projects do not operate in isolation. In most cases they run alongside day-to-day operations. Both the project as well other factors might impact the KPI. E.g. a project to improve the response time of a system might report the reduction in milliseconds to load a specific page. However, high or low system demand of existing users might also impact this KPI.

Ideally the day-to-day impact can be excluded in the definition of the KPI by normalisation (e.g. computing time per request). Alternatively, the impact is eliminated by the measuring method (e.g. measure the improvement of the response time in a sandbox environment rather than production).

Independently verifiable

The actual calculation of the KPI could be left to the project manager, the PMO or could be done by somebody outside the project team. However, it is important that an outsider can at least verify the results independently. This is achieved by ensuring

  • Clear, unambiguous definition of the calculation
  • Accessible data

The limitations of KPIs

KPIs try to condense the complex world of business outcomes, progress, written strategic objectives into one or few numbers. This inevitably loses information as the real world is much more complex.

How to manipulate KPI – a real live example

The following example is based on real events. A big company wanted to simplify its IT application landscape and reduce the number of applications. A large program – we call it AppExit here – was kicked off to take out hundreds of potentially redundant systems. The business case focussed on the cost reduction of maintaining fewer systems and less integration work for future projects.

As the actual cost reduction is a lagging indicator of success, the number of applications in production (or reduction thereof) was perceived to be the best leading indicator of success for this program. To make this indicator specific the team excluded any changes caused by other projects. Fair enough.

In the detailed preparation of AppExit the management discovered that the inventory of applications was not as good as expected. Some entries were entire families of applications, whilst others had bits and pieces, like interfaces or components registered as unique applications. A separate project AppInventory was initiated to define standards to register applications and conduct a one-off clean-up of the inventory itself.

The main AppExit program had a slow start as taking out applications was harder than expected. The majority of functionality could be transferred easily elsewhere. But the last 5% was difficult so the application lived on.

However, AppInventory was very successful: many entries of the inventory were “consolidated” using the new standard. Components of applications were incorporated into the entry of the main application. Then these entries were set to “decommissioned”. At program-level the total number of applications was reported to shrink steadily and management was happy. However, there was actually no change of the application landscape. Only the inventory was cleaned up. It took over two years for management to realise that AppExit was way behind delivering on its promised and it was eventually scrapped.

Difference between expected and reported application reductions

Lessons learnt

There was a hypothesis made that fewer applications lead to lower operating costs. Applications in production were counted by number of entries in the inventory. This KPI was

  • Leading – as a lower number can be reported before the costs go down
  • Relevant – cost reduction was part of the organisation’s strategy
  • Clear in the definition and based on accessible data: the inventory was the official place to enquire about the status of an application
  • It was also specific and attributable: applications removed by other projects were not counted.

All good? No!

Reducing only the number of entries in the inventory does nothing to the operating costs. Only when real applications are decommissioned and removed from the production landscape costs go down. The main mistake was that the inventory and not the real-world applications were counted.

KPI and portfolio reporting

In the chapters above we have talked mainly about the KPIs that are specific to a single project. In project portfolio management you have the additional challenge to report progress across multiple projects and KPIs provide a good source as they are expressed in numbers.

A list of defined organisation-wide KPIs allows to aggregate the contribution of projects

Many KPIs will be specific to a single project. Trying to aggregate them across different projects would add apples to pears. Project A counts the releases per year of an iPhone app and Project B the releases of your SAP system. Whilst you can total the releases of your portfolio A & B the results will be heavily overweight project A.

You cannot aggregate KPIs that are ratios as the denominator could be different. Price per user of €10 for system A and €20 for system B does not make an average price of €15 if the number of users was different.

Do not try to aggregate KPIs bottom up from different projects. Instead, define KPIs for your organisation and request your projects to include them in their performance measurement. This will help you to align your project portfolio to the strategy. E.g. if the improvement of IT Security is part of your strategy you could count the number of encrypted databases in your stock. Projects that encrypt databases must then report on this KPI. If you want to focus on cloud computing then count the number of applications using some cloud components. If you need to optimise your cost base then ask your relevant projects to provide their contribution to cost reduction.

Conclusion

So in this article you have learnt why KPIs matter in project (portfolio) management, how to classify them and ensure you select good ones and avoid bad ones. You have learnt how to use KPIs to manipulate the reporting of your project progress so you can challenge better any such attempt in the future. Last but not least I hope you have understood the limitations of KPIs. They only condense information into a few numbers. So use them as automatic additional metric to manage your portfolio, but do not expect that everything is perfect if only the KPIs tell you that.

If you are interested in learning more about the destructive power of KPIs I recommend the great book by Cathy O’Neil: Weapons of Math Destruction, now long listed for the New Yorker national book awards.

The featured photo is by Michal Mrozek on Unsplash

As organisations grow, their project portfolio becomes more complex or important projects fail management will set up an Enterprise PMO. Their names are different (e.g. Portfolio Management Office, Portfolio Management, PPM, Multi-Projectmanagement, Corporate Project Portfolio Management etc.), but in essence it will be responsible for the day-to-day oversight of the project portfolio.

I have set up such EPMOs in different organisations with different flavours and objectives. Some worked better than others. Below I would like to share some of my experience and lessons learnt.

Consider where you sit in the organisation

Generally there is no right or wrong place for the setup in the organisation. Some are located within the IT organisation, others in Finance or with the CEO Office. Some positions work better than others for specific aspects the EPMO’s objectives. If there is a choice, consider the 2-3 most important objectives you want to achieve. Then consider from where those objectives can be best delivered.

Below is a list of typical setups. In addition to the main, central setup larger organisations might add divisional sub-PMOs to balance central vs. divisional requirements.

Table 1 – Where should the Enterprise PMO sit?

Senior management backing

To become successful you need authority right from the top. When you set up shop and when things become difficult. As soon as you reject the first project submission because the business case does not stack up authority is of essence. People will try to bypass you or get special treatment using their network. At this point they need to be told to get back into the line.

Run by experts, not by senior management

Whilst you are dependent on senior management backing you need to become the expert to run your shop. Do not let senior management interfere how you exactly do this.

One example is senior management demand for more data that is then pushed without much thinking into the PPM system. A lack of decision-making is often blamed on the lack of (detailed) information provided. Your senior management will always ask for more information without giving much thought into the time and effort to provide this data.

The nice marketing slides of your PPM system provider with shiny charts and reports only work with quality data. You need to balance the effort of asking for extra data with the impact on data quality.

Tools and standards give you power

As EPMO, you typically are in charge of the PPM system. You can define the setup, the data model and the surrounding processes. This will give you access to the organisation via definition of standards, training, rollout and demand management.

“A bit of governance is good – more is better?”

This is a term I heard a few times on a Gartner PPM summit. You need to find the right balance between heavy-handed governance processes and hands-off trust in your people. The initial transparency might be welcome, the more you add the more you could become a bureaucratic monster.

Do not expect to be liked…

…but you should want to be respected. Also, consider yourself as much as serving as policing the individuals you oversee.

Do provide guidance and information throughout. It is really important to be as visible as possible to the “end user” – i.e. the project manager and project PMOs. This can avoid the “ivory tower” trap: you try to please your management, get disconnected from reality and push all requirements to the end user without understanding the work you create.

and do listen

Give your end users a voice by letting them influence how to improve PPM processes and listen to their expertise and feedback.

The common traps

I like the software as-is, but can you also do….? These questions are asked in the sales pitch. You get a nice demo from the vendor and you like what you see. But then you think about your internal processes and how you do stuff differently in your company. Every vendor will happily answer – of course we can! They know that they compete with other vendors and want to tick all the boxes with their product.
Your vendor might also be a software implementation “partner” who make their money with customisation of 3rd party products. Then later they sell you a maintenance contract to help you through ever more complex release upgrades.

You have written a long list of business requirements and now need to tick the boxes. They are very detailed but focus on implementing your current business process. Quite often they relate to either a manual process or your legacy application you want to replace. The closer the new system can satisfy your current process the easier it is to implement it and roll it out (you think!). However, you have not looked at how the new system would actually solve the business problem out-of-the-box.

Your current processes “suck”! They have evolved over time, maybe there is a good reason to run them that way, maybe not, nobody knows. What you do know is that you need to “tweak” the package a just a bit to run your process as it is. Sometimes you actually have more than one process in your company for doing the same thing. So the software needs a bit of “if … then … else …” coding to do the same step differently depending on where or who is running it. You think this could be harmonised, but this might require agreement with lot of different people, so now is not a good time.

Some other department tells you they need this change, otherwise they cannot possibly sign-off the application. You might sit in IT and the other function is your business. You are their service provider and you want to make them happy. You might think that they could change the way how they run things better, but they tell you that this is none of your business. You should be their partner, but they do not want your advice.

It is not customisation but configuration. There is a fine line between different the two types of changes, often defined by what is technically done to the product. For a customisation you need a developer, for the configuration only an administrator. However, there are heavy configurations that almost duplicate the original functionality. Those are worse than small customisations.

Photo by Rosalind Chang on Unsplash

Getting out of the trap

Go back to basics and pull out your project charter document, somewhere in the objectives you have written about simplify the IT landscape, reduce support costs, harmonise the environment. These are your strategic objectives. Your customisations are in direct conflict with these objectives. Customised software is not simpler and does have higher support cost than a plain-vanilla configuration. Start to use business language to describe the impact of customisations. Consider the long-term impact on further development or maintenance, technical documentation and also user training. With this extra information, have a discussion with the right people who have really signed up to the charter and challenge every single request for customisation.

Keep a log of customisations if you really need to do them. Capture why they were required and who signed them off. This will be your log of technical debt, things that ideally should not exist because they get you further away from your target state. Whenever there is an opportunity, try to clear off items of your technical debt.

Stay in control of the decisions and do not leave them to 3rd parties. Vendors have different objectives than your company. They live off your money. So what is complex and expensive for you to build and maintain is their revenue stream for them. Some good vendors will partner with you to do the right thing, but it is important that you keep implementation decisions in-house.

Do not start with a long list of detailed business requirements when you buy a software package. When you build your own the case might be different, but when you buy a package it often comes with a good default configuration. Keep your business requirements as high-level and abstract as possible so that the software package has a chance to fulfil it in the way it was designed. Use the Agile methodology to start with the off-the-shelf configuration as minimum viable product. Really think hard in your use case and acceptance criteria what extra business value a story or epic will add.

Review the stakeholders of your implementation project. If you need to change processes to keep your software simple, do you have the right people at the table who can make these decisions? Establish a forum of like-minded people who can help you make the right decisions for the organisation and fully understand the long-term impact of getting it wrong. In one company I established a Design Authority with appointed process owners for topics like IT Governance, Demand Management, Project Accounting, Benefits Management, Enterprise Architecture etc. who had the authority in the organisation to either request software changes or change their respective processes.

Credits and further reading