Mathew McKie

A longstanding customer of a financial services company accidentally submitted the same application to two different offices. Both were supposed to follow the same guidelines, yet the quotations were substantially different. The customer took their business elsewhere. The company might never have discovered the inconsistency had the application not been submitted twice.

Daniel Kahneman and colleagues described the incident in a 2016 Harvard Business Review article about inconsistent professional judgement. What interests me is how easily the problem could have remained invisible to an organisation that presumably had experienced employees, established procedures and confidence in the way decisions were made.

Organisations invest considerable effort in ensuring that decisions are made by qualified people, through approved processes and with properly controlled technology. Those safeguards matter, particularly where decisions have serious consequences. Yet they establish that decisions have been made in an authorised way, not necessarily that they have been made well.

This is the decision validation gap: the difference between what an organisation can demonstrate about its decision-making processes and what it knows about how well those processes work. The gap matters most when decisions have significant consequences, particularly where their quality is difficult to assess. Outcomes may take years to emerge, depend on circumstances beyond the organisation’s control or be difficult to compare with what might have happened under a different approach.

The decisions we already make

The researchers also audited two financial services organisations, asking professionals to assess identical cases independently. Senior executives expected the judgements to differ by around 5–10%. The actual differences were far greater: assessments varied by an average of 48% in one organisation and 60% in the other. These were measures of inconsistency, not error: the researchers could not say which assessment was better. What they exposed was a degree of variation that management had substantially underestimated.

Consistency is not sufficient either. People can agree on the wrong answer, and a well-founded decision can produce a poor outcome because of events nobody could reasonably have anticipated. To judge whether a decision was good, we need to look at the information available at the time, how it was interpreted, the alternatives considered and, where possible, what happened afterwards. Looking only at eventual outcomes risks confusing bad decisions with bad luck.

Credit risk offers a practical example. Some lenders deliberately approve controlled samples of applicants they would normally reject, then observe their repayment behaviour. That creates evidence the normal lending process would never produce and can reveal whether the policy is unnecessarily restrictive. It cannot tell the lender what every rejected applicant would have done, but it allows an established approach to be challenged with something more useful than assumption.

Many organisations already measure performance rigorously. The question is whether those measures reveal how well their important decisions are being made, or simply record the outcomes of a process whose effectiveness has never been properly examined.

When assurance stops short

This distinction is visible at a much larger scale in government. In a 2021 report, the National Audit Office concluded that inconsistent evaluation left government unable to be confident that spending in many policy areas was making a difference. A 2019 review of £432 billion of major-project expenditure found robust impact-evaluation plans for only 8% of spending and no evaluation plans for 64%. A separate 2025 review found that 34% of projects in the 2023–24 major-projects portfolio had good-quality evaluation plans. The reviews covered different projects and used different measures, so the figures cannot be treated as a trend. Nor do they show that the projects themselves failed.

They do, however, expose how little we sometimes know about whether large sums of public money achieve what they were intended to. Approving a project, controlling its delivery and accounting for its spending are different tasks from determining whether it produced the intended benefits. A project can be carefully approved and managed without anyone establishing whether it delivered the promised benefits.

This may be partly because following a process automatically produces records: approvals, completed checks, exceptions and evidence of who was responsible. Finding out whether a process works takes more effort: examining results, considering what might have happened otherwise and sometimes testing an alternative. That evidence is harder to gather, and it may challenge a process that people already have a responsibility to maintain.

There are genuine limits to what can be evaluated. Results may take years to emerge, a fair comparison may be expensive, unethical or unsafe, and some effects are difficult to measure. The NAO findings do not prove that organisations neglect evaluation because they prefer evidence that procedures have been followed. They do illustrate why an organisation can become very good at demonstrating that a process was followed without becoming equally good at establishing whether it works.

What happens when AI becomes involved?

AI adds another complication: it changes how decisions are made and what we expect people to be responsible for. Consider a radiologist who fails to identify a brain bleed. In a 2025 study published in NEJM AI, US participants were asked to judge hypothetical cases involving that mistake. Without AI, 56.3% sided against the radiologist. When AI had correctly identified the abnormality but the radiologist had not, the figure rose to 72.9%.

In a separate randomly assigned group, participants also learned that half of the AI system’s positive alerts were false alarms. The proportion siding against the radiologist was then 48.8%. The clinical mistake and its hypothetical consequences were the same. What changed was what participants knew about how reliable the AI was. The result is striking because the AI had been right about this particular case, yet evidence of its wider fallibility substantially altered how responsibility was judged.

This measured who participants thought should be held legally responsible, not the outcome of real malpractice cases or the effect on patients. It cannot tell us what participants had originally assumed about AI reliability, or how a real clinician would have behaved. It shows that knowing how reliable a system is can fundamentally change how we judge someone who made a decision using it. That should matter to anyone proposing to retain human approval as the answer to AI accountability.

Many organisations intend to keep a professional in charge of the final decision, partly to preserve oversight and ensure that someone remains accountable. Those are legitimate objectives. But placing a person between an AI recommendation and the final outcome does not establish whether they can recognise mistakes, challenge unreliable advice or improve the result.

A 2024 analysis by Michelle Vaccaro and colleagues brought together 106 experiments comparing people, AI and people working with AI. Human–AI combinations performed better than people working alone on average, but worse than whichever of the two performed best independently. This was particularly true for decision-making tasks. More than 95% of the combinations studied involved a person making the final decision after receiving AI input. Where AI already performed better, human intervention tended to reduce its performance. The studies covered systems tested up to mid-2023, so they do not settle how today’s models would perform.

Human-in-the-loop may not be the silver bullet organisations hope it is. It may, in fact, make decisions worse. The evidence suggests that when AI already outperforms human judgement, introducing a person into the decision can reduce performance rather than improve it. This does not mean human oversight should be abandoned, particularly where safety and accountability are concerned. It means its effectiveness cannot simply be assumed.

Nuclear regulators have recognised that human involvement can create problems in opposite directions. In a 2024 joint paper, the UK’s Office for Nuclear Regulation and its Canadian and US counterparts considered how people might second-guess correct AI outputs or place too much trust in them to verify their work effectively. They also recognised situations where there may not be enough time for someone to check an AI decision. They stressed the need to give people and technology work suited to their strengths and weaknesses, and to make sure people can intervene effectively when needed.

ONR’s own April 2024 policy focuses on whether safety risks are controlled rather than prescribing particular technologies. It calls for safety evidence proportionate to the risks, independent safeguards and clear accountability for nuclear operators. It also warns that assumptions about what regulators will accept can prevent better solutions from being considered. The point is not that a human must be removed from the loop. It is that the whole process must be shown to manage the risks it is intended to control. We need evidence that reviewers can detect errors, intervene when necessary and improve outcomes, rather than relying on the reassuring presence of a human signature.

Evaluate the complete process

The Swedish MASAI breast-screening trial shows what this kind of evaluation can look like. Researchers did not simply pit a model against a radiologist. Normally, two radiologists independently examine each mammogram. In the trial, AI assessed the images first: most were then reviewed by one radiologist, while those flagged as high risk were reviewed by two. The AI also highlighted potentially suspicious areas to help the radiologists identify cancers.

More than 105,000 women took part. The AI-supported process reduced screening reads by 44.2%, and a 2025 analysis reported 6.4 cancers detected per 1,000 women screened, compared with 5.0 under conventional screening. Finding more cancers does not, on its own, establish better long-term patient outcomes. But the subsequent 2026 results found that the AI-supported approach met the trial’s requirement to be no worse on its main measure, cancers diagnosed between screening rounds. The process detected a greater proportion of cancers without increasing the rate of false alarms among women without cancer.

Those results apply to this screening process, not to every use of AI in healthcare. What makes the trial useful beyond medicine is that it tested what happened when the work was organised differently. The researchers measured both clinical results and radiologists’ workload, comparing the processes rather than just the technology.

Where enterprise value comes from

For businesses, the same discipline has a commercial purpose: better information only creates value when it enables better choices and those choices improve outcomes. A 2013 study of an American auto finance company serving a relatively high-risk market estimated that automated credit scoring, alongside changes to lending practices, appeared to increase profitability by approximately $1,000 per loan.

The better risk assessment helped the lender identify which applicants could be offered more attractive terms and where it needed to be more cautious. The financial benefit came from changing lending decisions, not merely producing more accurate scores. The estimate concerns one lender and should not be taken as a general return from credit scoring, but it illustrates the connection between information, the decisions an organisation actually makes and the value those decisions produce.

Consider a retailer struggling with stockouts. More accurate demand forecasts may help, but not if purchasing budgets, supplier contracts or inventory incentives prevent buyers from ordering differently. Conversely, changing purchasing rules will achieve little if suppliers cannot fulfil orders. The business needs to understand whether its constraint lies in the information available, the choices being made or its ability to act on those choices.

Without that understanding, an organisation can invest heavily in improving the technology while leaving the cause of poor performance untouched. It may also discover that the existing technology is perfectly adequate and that the problem lies in its operating model. Either conclusion is useful if it prevents spending money on the wrong intervention.

The responsibility of leadership

The difficulty is that responsibility for important decisions is often spread across different teams. Analytics supplies predictions, operations makes choices, technology maintains the systems and governance checks whether controls have been followed. Each function can perform its role competently while nobody has a sufficiently clear picture of whether the combined process is producing good decisions.

The leader accountable for a business outcome should also make sure the decisions that drive it are evaluated. That does not mean marking their own homework: independent analysis and challenge matter, particularly when results cast doubt on established practice. Nor does it require another governance framework. It requires someone with the authority to act on the evidence.

A useful starting point is an important decision the organisation makes regularly. Examine the information decision-makers actually receive, how consistently they use it, what outcomes can be observed and how the current process compares with a realistic alternative. Where AI is involved, assess how the whole process works, including what people detect, miss, override or accept. The answer might justify better technology, a different role for professional judgement, a change in operating practices or no change at all.

Professional competence, clear procedures and independent assurance remain essential. They help organisations make good decisions, but they are no substitute for finding out whether those decisions are actually good. An organisation can know precisely who made a decision, why they were authorised to make it and which controls were followed, while knowing remarkably little about whether the way it makes those decisions is effective.