When a measure becomes a target

In this issue

A number on a board is a signal. Once pay, rankings or blame depend on it, it also becomes something people steer. This issue is about what happens to a measure when it is made a target, and how a manager can tell a number that improved from work that improved.

Charles Goodhart stated the rule for money in 1975, Donald Campbell for social indicators a year later, and Marilyn Strathern gave it the short form in 1997. In English emergency departments, the official share of patients seen within four hours was 96% in 2004/05; patients in a survey put it at 77%. Gwyn Bevan and Christopher Hood name three kinds of gaming, Wells Fargo shows what sales targets can pay for, and Jerry Muller warns against metric fixation.

Stiven Janaqi, Editor

Cover story

One law, three authors

In July 1975, at a Reserve Bank of Australia conference in Sydney, Charles Goodhart of the Bank of England presented a paper on monetary management in the UK. It holds the first statement of what became Goodhart's law:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

• 1975 Goodhart: a regularity used for control breaks down.
• 1976 Campbell: an indicator used for decisions distorts the work it monitors.
• 1996 Keith Hoskin calls the idea Goodhart's law.
• 1997 Strathern gives the idea its short form.

“When a measure becomes a target, it ceases to be a good measure,” wrote Strathern about audit in British universities: the more a good grade is expected, the worse it tells students apart.

Our reading

Goodhart says the number stops telling the truth. Campbell adds that the work changes too. A manager has to watch both.

The volume is dated 1976, the paper usually cited as 1975; a 1984 reprint is quoted with "on" for "upon". Campbell's journal version is from 1979. Translations are ours.

Sources: Charles Goodhart, 1975; Donald T. Campbell, Evaluation and Program Planning, 1979 (via Dartmouth occasional paper (1976), reprinted in the Journal of MultiDisciplinary Evaluation (2011)); Marilyn Strathern, European Review, 1997

The numbers

Reported and experienced

In the early 2000s England rated its hospitals with stars, and four hours in accident and emergency (A&E) was a key target. Gwyn Bevan and Christopher Hood set the official figures next to independent surveys of patients.

Patients in English A&E seen within four hours: Official figure, 2004/05 96%, Patient survey, 2004/05 77%, Patient survey, 2002/03 69%.

In 2002/03 the official returns showed 139 of 158 acute trusts seeing 90% of patients within four hours. Ambulances had to reach 75% of life-threatening calls within eight minutes; in a third of ambulance trusts, inspectors found times “corrected” to under eight, with a spike at the eight-minute mark.

Our reading

A target that only the measured report on checks their reporting, not their work.

Hospitals reported the official figures, patients answered the surveys: the gap is a warning, not a measure of gaming. Bevan & Hood cite the National Audit Office, the Healthcare Commission and the Commission for Health Improvement.

Source: Gwyn Bevan & Christopher Hood, Public Administration, 2006 (via LSE Research Online; full text in a copy of the published article)

The model

Three ways to hit a number

Bevan and Hood describe gaming as “hitting the target and missing the point”. From studies of Soviet production targets they take three kinds, each with its own warning sign.

• Ratchet. next year's target is built on this year's result, so it pays not to beat it by much
• Threshold. results crowd at the line, and those above it may slip back to it
• Distortion. the target is met at the cost of what nobody measures

“A wise director fulfils the plan 105 per cent, but never 125 per cent,” Alec Nove wrote of Soviet managers in 1958. In an employment office Campbell cites, staff counted by placements took the easiest cases. In one large English hospital, the target for new eye appointments was met by cancelling and delaying follow-ups, which no target counted; a parliamentary committee reported that 25 patients lost their sight over two years as a result.

Our reading

Every target has a shadow: the work next to it that nobody counts. Name it before you set the target.

The three kinds and Nove's line follow Bevan & Hood (2006); the eye clinic case is from a 2003 select committee report they cite, the employment office from Campbell. The reading is the editors'.

Sources: Gwyn Bevan & Christopher Hood, Public Administration, 2006 (via LSE Research Online; full text in a copy of the published article); Donald T. Campbell, Evaluation and Program Planning, 1979 (via Dartmouth occasional paper (1976), reprinted in the Journal of MultiDisciplinary Evaluation (2011))

More in the essay: How we worked with damage

What the research says

When pay follows the number

In September 2016 the US Consumer Financial Protection Bureau fined Wells Fargo. Spurred by sales targets and compensation incentives, thousands of employees had secretly opened accounts customers had not authorised. By the bank's own analysis:

• 1.5 million deposit accounts that may not have been authorised
• 565,000 credit card accounts applied for without authorisation
• $185m in fines from the CFPB, the OCC and the City and County of Los Angeles

In 2020 the bank agreed to pay $3 billion to settle the federal investigations into its sales practices from 2002 to 2016. The pattern is older: in the early 1990s Sears set its auto repair staff a goal of $147 an hour, and staff overcharged and did repairs no one needed. In a laboratory study, Schweitzer, Ordóñez and Douma found people more likely to overstate their results with a specific, challenging goal than without one, above all when they had just fallen short.

Our reading

The risk is highest just below the line. That is where a target needs a check, not more pressure.

Account numbers: the bank's estimate, as reported by the CFPB; 2020: the bank's announcement. Sears and the laboratory study as cited by Ordóñez et al. (2009).

Sources: Consumer Financial Protection Bureau, 2016; Wells Fargo & Company, 2020; Lisa D. Ordóñez, Maurice E. Schweitzer, Adam D. Galinsky & Max H. Bazerman, 2009

How it is measured

Measure the measure

Jerry Muller calls the problem metric fixation: the belief that numbers can replace judgement, that publishing them ensures accountability, and that rewards tied to them motivate best. He is not against measuring, and Bevan and Hood suggest ways to make a target harder to game.

• Compare with a second source. The patient surveys showed a gap the official returns did not.
• Look at the spread, not only the share. A spike just under the line, like the one at eight minutes, is a sign.
• Check where no one expects it. Open afterwards, uncertain in real time: random checks and visits.

Hypothetical example, a dispatch target

• Target: 95% of vans out by 07:00
• Reported: 96%, and a third of departures logged at 06:59
• Second source: late-delivery complaints unchanged

The share rose; the spread and the complaints say why. The numbers are invented.

Muller's three beliefs are from his book and a 2018 essay; the steps are the editors' reading of Bevan & Hood (2006), not a tested method.

Sources: Jerry Z. Muller, Princeton University Press, 2018 (via Publisher's page, table of contents and the author's essay “Against metrics” (2018)); Gwyn Bevan & Christopher Hood, Public Administration, 2006 (via LSE Research Online; full text in a copy of the published article)

Tool of the issue

The target check card

Fill it in with the team before a number becomes a target. The people who do the work usually know the fourth line best: ask them.

1. The number and the target definition, source of the data, the line
2. What it is meant to show the purpose behind it, in one sentence
3. What depends on it pay, ranking, praise or blame, and for whom
4. How it could be hit without the work improving ratchet, threshold, distortion, recording
5. Its shadow the work next to it that nobody counts, and a second number for it
6. The independent check a survey, a sample or a visit; who, how often
7. Review date when we ask whether it still means what it meant

A practice proposed by the editors, after Bevan & Hood (2006), Campbell (1979) and Muller (2018).

Sources: Gwyn Bevan & Christopher Hood, Public Administration, 2006 (via LSE Research Online; full text in a copy of the published article); Donald T. Campbell, Evaluation and Program Planning, 1979 (via Dartmouth occasional paper (1976), reprinted in the Journal of MultiDisciplinary Evaluation (2011)); Jerry Z. Muller, Princeton University Press, 2018 (via Publisher's page, table of contents and the author's essay “Against metrics” (2018))

Open the tool: KPI Diagnostic

Sources and method

Every figure has a source.

The figures in this issue come from the sources below. The year shows how recent each one is.

• Charles Goodhart, “Problems of Monetary Management: The U.K. Experience”, 1975.
• Donald T. Campbell, Evaluation and Program Planning, “Assessing the impact of planned social change”, 1979 (via Dartmouth occasional paper (1976), reprinted in the Journal of MultiDisciplinary Evaluation (2011)). https://doi.org/10.1016/0149-7189(79)90048-X
• Marilyn Strathern, European Review, “‘Improving ratings’: audit in the British University system”, 1997. https://www.cambridge.org/core/journals/european-review/article/improving-ratings-audit-in-the-british-university-system/FC2EE640C0C44E3DB87C29FB666E9AAB
• Gwyn Bevan & Christopher Hood, Public Administration, “What’s measured is what matters: targets and gaming in the English public health care system”, 2006 (via LSE Research Online; full text in a copy of the published article). https://doi.org/10.1111/j.1467-9299.2006.00600.x
• Consumer Financial Protection Bureau, “Consumer Financial Protection Bureau Fines Wells Fargo $100 Million for Widespread Illegal Practice of Secretly Opening Unauthorized Accounts”, 2016. https://www.consumerfinance.gov/archive/newsroom/consumer-financial-protection-bureau-fines-wells-fargo-100-million-widespread-illegal-practice-secretly-opening-unauthorized-accounts/
• Wells Fargo & Company, “Wells Fargo Reaches Settlements to Resolve Outstanding DOJ and SEC Investigations Related to Historical Community Bank Sales Practices (Form 8-K, exhibit 99.1)”, 2020. https://www.sec.gov/Archives/edgar/data/72971/000007297120000214/exhibit991settlementpres.htm
• Lisa D. Ordóñez, Maurice E. Schweitzer, Adam D. Galinsky & Max H. Bazerman, “Goals Gone Wild: The Systematic Side Effects of Overprescribing Goal Setting”, 2009. https://repository.upenn.edu/mgmt_papers/325
• Jerry Z. Muller, Princeton University Press, “The Tyranny of Metrics”, 2018 (via Publisher's page, table of contents and the author's essay “Against metrics” (2018)). https://press.princeton.edu/books/hardcover/9780691174952/the-tyranny-of-metrics

Editorial method

Each figure was checked for its year, its publisher and what exactly it measures. Where the publisher's page could not be opened, the figure was checked against independent summaries and is marked “via”. The editors' interpretation is marked “Our reading”. Figures that could not be confirmed are not in the issue.