The incoming government faces many challenges in seeking to boost economic growth, improve public services and enhance families’ wellbeing. But it has opportunities too – some of them sitting right beneath its nose.
One of the most important is the unrealised potential of the data that public bodies collect on individuals and businesses for their own purposes. By sharing, linking, analysing and communicating this data more effectively, the government could improve policy design and delivery – both directly and by helping outside researchers and analysts scrutinise policy choices and generate new ideas.
The Office for Statistics Regulation (OSR), the UK’s statistics watchdog and part of the UK Statistics Authority, highlights this issue in its 2024 state of the statistical system report. Encouragingly, it found that statisticians across the official system are already linking and using administrative data extensively and imaginatively to respond to new and complex questions posed by government and society. But it saw several barriers to using this critical national asset as effectively as it could.
One is that the official statistical system is under greater strain than it used to be. It faces similar financial constraints to the rest of the public sector, compounded by the fact that that it is becoming increasingly difficult and expensive to maintain the quality of the household surveys that we rely on to understand important parts of the economy and society, because of falling response rates and higher collection costs, a trend seen by other statistics agencies across the world.
The statistical system is also more decentralised than in many other countries. The Office for National Statistics (ONS) is the biggest single producer of official statistics, but most come from other government departments and public bodies. This has the advantage that statisticians work closely with those developing and delivering policy in each area. But the downside is that data and analysis are often siloed. Each organisation holds a piece of the puzzle that, if combined, could offer richer insights.
During the pandemic, the sense of national emergency provided an urgent impetus for sharing data. This enabled valuable insights, such as identifying where to target communication campaigns to increase vaccine uptake. But since then, the system has not fully capitalised on similar opportunities. Many gains remain untapped.
Why is this? The OSR found that while there are pockets of innovative and ambitious data sharing and linkage in government, it is neither widespread nor the norm. The conversation continues to focus more on the risks than the potential benefits – risks to individual privacy and the reputational risks that data owners may feel they face if weaknesses in their own data are revealed or if they are used to reach conclusions or narratives that the institution or its leadership finds inconvenient.
These risks should not be dismissed, but real-life examples show they can be managed and should often be embraced for the greater good. For example, research shows that the public support sharing of their data for research and statistics, so long as they are informed about how it will be used and how it will be kept private. And if sharing data exposes weaknesses in the national data infrastructure or in policy thinking, that should not be regarded as a risk, but rather as a welcome challenge.
after newsletter promotion
Recent independent reviews, such as that undertaken by Prof Denise Lievesley on behalf of the Cabinet Office, or by parliament’s inquiry into transforming the UK’s evidence base, have all quickly come to the same conclusion: it is time to rise to this challenge. Data sharing across government should be the rule, rather than the exception. But given the perceived risks, robust political leadership is vital to make this happen. The decision to concentrate responsibility for data and digital policy in the Department for Science, Innovation and Technology could help bring renewed focus, but it will need the enthusiastic support of No 10, the Cabinet Office and the Treasury.
A new government has a fresh opportunity to set expectations across Whitehall and the broader public sector and to drive a data-sharing revolution. If it is serious about being “mission-led” – defining and pursuing objectives that span departmental boundaries – seizing this great opportunity will be all the more important.
Sir Robert Chote is chair of the UK Statistics Authority, which oversees the Office for Statistics Regulation, the body’s regulatory arm, and the Office for National Statistics, the national statistical institute
