News

Ofsted reliability and consistency: Evidence-base 'far too thin'

Ofsted Whole-school issues
The available evidence regarding inspection reliability and consistency is “far too thin” and Ofsted must do better, a leading researcher has said.
Image: Adobe Stock -

John Jerrim, professor of education and social statistics at UCL Institute of Education and a research associate at FFT Education Datalab, said that in its 30-year history Ofsted has published very little on this key issue.

Prof Jerrim is currently working on a Nuffield Foundation-funded research project into Ofsted reliability and consistency alongside Dr Christian Bokhove and Dr Sam Sims and said that the interest being shown in their work “has been quite overwhelming”.

Writing in a blog for FFT, Prof Jerrim lays blame at the door of Ofsted itself.

“In many ways, Ofsted is in the best position to conduct such research. It has the resources, access to inspectors and a great research team. Yet, throughout its 30-year history, Ofsted has published little on this matter. Why?

“One reason is likely because some give Ofsted a hammering whenever such studies show anything less than 100% reliability.”

Prof Jerrim pointed to the reaction to a 2017 Ofsted study looking at the consistency of its ungraded inspections. Despite 92% agreement across judgements by different inspectors, Ofsted still received criticism for the remaining 8%.

Prof Jerrim acknowledges an “unreasonable expectation” from some quarters for inspections to be 100% reliable and says that Ofsted is “stuck between a rock and a hard place” leading to a reluctance on its part to publish research and a subsequent evidence “vacuum”.

However, independent researchers cannot step in, he warns, because they do not have access to enough data.

He writes: “Sam, Christian and I have had to be really creative in our work – scraping information from inspection reports and collaborating with journalists on information they have obtained via Freedom of Information requests. But this is a time-consuming, sub-optimal process, with the data obtained patchier than it should be. It is, however, the best that any non-partisan researcher currently has. This is an area where Ofsted must do better.”

A recent example of this problem came when the researchers were investigating whether schools that employ an Ofsted inspector get better inspection outcomes. The data was only forthcoming from Ofsted after Ofsted was forced to release it following a journalist’s Freedom of Information request. You can read more about this research here.

Prof Jerrim is hopeful that the current Education Select Committee inquiry into Ofsted’s work with schools will tackle this issue. Dr Sims has appeared as a witness at the inquiry and the researchers have also submitted written evidence.

For example, the written submission cites a recent study by the researchers into whether some inspectors are more lenient than others (Bokhove et al, 2023).

Initial findings identify “small differences in Ofsted judgments between male and female inspectors (males being more lenient than females)” in primary school inspection. It also finds “variations in judgements” between HM inspectors and Ofsted inspectors.

The researchers say that this study “should be viewed as merely the initial step in understanding the issue”.

The submission added: “It is reasonable to assume that Ofsted holds more comprehensive data, enabling a more thorough investigation of these concerns … consequently, obtaining a detailed understanding of the usefulness of school inspections necessitates a comprehensive examination of their consistency and reliability.

“Regrettably, the current body of evidence in this area remains remarkably thin, emphasising the urgency to expand and deepen the available evidence base on this crucial aspect.”

In his blog, Prof Jerrim added: “The evidence available regarding inspection reliability and consistency is far too thin. Ofsted, who are in pole position to generate such evidence, struggle to do so. At least in a way that garners the credibility and acceptance it needs from across the education sector.

Yet independent researchers don’t have access to the data they need to play the role of the ‘honest broker’.”

It comes as new survey data from the Beyond Ofsted inquiry shows that 62% of more than 6,700 teachers asked do not think the outcome of their most recent inspection accurately reflects their school. Even in those schools judged by Ofsted as good or outstanding, 58% did not think the rating was reliable.

The inquiry has been set up by the National Education Union and is due to report later this month. Its aim is to develop a set of principles for underpinning a better inspection system and proposals for an alternative approach.