Hattie’s research: egregious errors

The egregious errors that beset John Hattie’s research are so pervasive as to prove difficult to encompass and thus lay bare – but various insights local and international are at last coming together to achieve just that. This posting argues that this coming together will reveal there is nothing about Hattie’s research that stacks up: the mathematics, statistics, design, curriculum understanding, or presentation.

So influential has Hattie’s research become and Hattie along with it, that to critically examine it, whatever the outcome, if integrity and validity of policy information is valued, should be welcomed by all in school education, in academia, in government bureaucracies, by governments, and by Hattie himself. If readers take a stand (as I have) that the egregious errors are just that, the only path remaining is that Hattie has been astonishingly careless and ignorant in the maths, statistics, research design, understanding of curriculum, and presentation. But what would be beyond egregious is if Hattie, having had the errors pointed out to him, persisted in defending them.

Throughout the posting I will make references to neoliberalism and the connection between that ideology and Hattie and quantitative research. The key way to establish whether neoliberalism is dominant in education is the degree to which the concept of the need to avoid provider capture is at play. The application of the concept, ostensibly to avoid teachers pursuing their ‘selfish’ interests, is really a way to prevent teachers introducing their knowledge of the needs of children. And in the philosophical space made available, the replacing of that with alternative education knowledge – knowledge held by neoliberal governments and certain, favoured quantitative academics. Governments require from quantitatives a dismissive attitude to teacher knowledge and along with that a provision of that alternative knowledge – measurable knowledge to enable governments to control school education for neoliberal ideological ends, particularly, indoctrination for control, utilitarianism to serve the economic interest of the power elite, and a running down of public education for social class advantage.

A difficulty with mounting the argument that the fault of Hattie’s research is egregious error alone, is the determined way he scatters disclaimers throughout his writing and presentations so that when a claim is made against his research it seems he always has a recourse, flimsy as to nothing but, up to now, serving to provide sufficient cover and confusion for temporary relief. For instance, when he is challenged about his hinge-point of 40% on his effect size barometer of fail or succeed (or whatever words he chooses to use) he said ‘that he did not say that we use this hinge-point for making decisions, but rather we use it to start discussions about the effect of teachers on students’.

That leads to another key point: Hattie does not really discuss, present, or defend his curriculum or education conclusions on the basis of the range of known arguments but on what his statistics demonstrate. In this age of the obsession with certainty based on numbers, Hattie has settled on a winning combination, and when presented by a professor of considerable standing, his conclusions are difficult to touch let alone challenge. And with Hattie it is not just numbers but numbers gigantism. He unleashes a flood of numbers tumbling out of statistics and references to over 800 meta-analyses which transmogrify into meta-meta analyses using in excess of 50,000 studies and an estimated number of students to be approximately 236 million.

Imagine a teacher at a conference, in the face of those numbers, contemplating standing up and challenging Hattie, for example, on Hattie’s fail mark to homework, and saying: I send home my six-year-old children with a reader to reinforce their learning and make a connection between home and school – is that wrong? In actual fact, the teacher has hit on a huge part of why Hattie’s research has egregious errors.

Imagine governments and bureaucrats hearing Hattie who, in his research presentations, delivers messages powerfully supportive to their neoliberal purposes. They would have their profoundest beliefs about education and human nature given expression, definition, and validity by an academic of the highest standing. Their view of teachers as misguided naïfs would be confirmed, encouraging them to be even more determined to counter teacher arguments as a step to further control. Ministers of education or their equivalent in nearly all Western countries regularly pronounce, sometimes mysteriously, sometimes boldly, they have special knowledge from a source not always identified, that children’s socio-economic circumstances are of little moment in education achievement – that source is the voice of Hattie or his quantitative local equivalent or, increasingly, both.  Sometimes Hattie breaks away from the neoliberal message but this is often just tactical, serving to maintain a degree of credibility with teachers while making no dent in government and bureaucratic faith in him. Indeed, he is of more value to governments and bureaucracies if he can bring along with him a significant number of the school community.

Then there is the research community. Why haven’t other quantitatives challenged what this series of postings will reveal as some of the worst research design on record? a research design with mathematical clangers, statistical absurdities, and an ugly and destructive hold on the curriculum. My conclusion is that Hattie’s education ideology has not been challenged because that ideology has been found by academics to be the Yellow Brick Road to research funding, publication recognition, appointments, and influence, and if not that, at the very least, in the interests of career security, impolitic to challenge.

Why haven’t teachers and principals, given their everyday experience with reality, been able to declare Hattie’s research as the 76 trombones of school education? Not only did they not challenge Hattie’s research findings, many embraced them. I think I can explain why.

First, as referred to above, there was the flood of numbers. I believe that that flood is an extension of Hattie’s personality and important in his rise to the top of the education research tree and his fixedness there. I don’t intend getting personal with Hattie beyond what I believe is necessary to my story (yes story, my story, and there will be other stories to be listened to from other people, including Hattie’s), but I don’t go along with the academic unreality of the saying: It’s the argument not the person. Do academics believe they are issuing their utterances in some disembodied, transcendental state? For me the argument cannot be separated from the person, to think they can be is to give academics a power beyond their right, an aura they are bound to misuse, a reversion to the shaman, a journey along the Yellow Brick Road to show biz effects. Academic arguments are not delivered by the Archangel Gabriel as though separate from common human impulses and motivations, in fact, so high stakes have academic ideologies become, so vibrant with possibilities for power, fame, and riches that I always approach education academics as being in the same category as politicians. In the list of postings that accompanies this posting (detailing my battle with Hattie) the first posting is about just that.  I believe that the flood of numbers is an extension of Hattie’s personality and that that personality needs to be taken fully into account to understand his research. In my view the flood of numbers acts as an argument in itself and interpreted by me as typical of a charismatic, macho male. How puny the argument of the teacher and homework in comparison with muscle-flexing research involving 236 million students – the teacher may even have a point it might be conceded, but what a trifle in comparison with the flow of education destiny involving 236 million students. There is a sense of the ends justifying the means at play here – a totalitarianism of numbers. Move aside with your little quibble or be swept away – and the same to you other people in education, even other academics.  My research is too big to fall, no matter who you are: ‘Look on my works, ye mighty, and despair!’

Second, the primary school cultural understandings teachers have carried through, that served them well and should still be serving them well, are being destroyed by neoliberal education proponents, particularly by academics like Hattie with unsafe research and belittling attitudes to teachers and teacher knowledge. The bond between quantitative academics and the ideology of neoliberalism has undermined the foundations of decades of humanistic education values and practices, replacing them with utilitarian values and practices designed to serve power elites. In harmony with Orwell’s prediction, the neoliberals in education and government have worked together to control the present to control the past and from there the future. Teachers have had their humanistic ideas and values and their own sense of worth discredited. They are bewildered and at a loss, searching for stability, searching for something to replace that which had been lost and, for many, the certainty of academics like Hattie has been found as something to cling to, to help fill the void.

Hattie, of course, will deny he belittles teachers, or that his education ideology is neoliberal, or that he works with neoliberal governments to neoliberal ends. He will just say that he goes where his research leads him – but that research is defined by decisions that can only end in one ideological positioning. Hattie pours scorn on teachers: approvingly quoting someone who calls teaching ‘an immature profession, one that lacks a solid scientific base and has less respect for evidence than for opinions and ideology.’ This from an academic who, in the next posting, I will demonstrate as making a fantastical mathematical error in his calculations and who uses maths no mathematician uses, statistics no statistician uses, and research design of such error that even his fellow quantitatives don’t use it. And who blames teachers as the ‘devil in this story especially the average teacher who has no idea of the damage he or she is doing.’ This from an academic who omits any education learnings that aren’t visible in the sense of being measurable and then presents his findings as the evidence-based truth of school learning; who bases his research on a meta-meta analysis which results in transgressions of a multitude of research fundamentals, for instance, sometimes he uses effect size to mean compared to a control group and other times to mean as compared to the same students before the study started; who redefines what learning is so that he can claim that good teaching can overcome the effects of poverty and inequality; and who redefines what learning is so that his idea of good teaching can be codified for a particular form of teaching; who belittles teacher organisations as obstructive to change and who ponders not on the ethicality, but the effectiveness of imposing policy on schools.

In the next posting, probably after Christmas, I lay bare the Hattie’s egregious research errors referred to. Merry Christmas.

A selection of writings on John Hattie from my two websites 2009-14

An important posting is missing in which John Hattie, via the Sunday Star-Times, January 4, 2009, displays his quantitative education wares for the inspection of the new minister of education in the new National government. Tolley, in response, in the same front page article, declares that Hattie’s ideas on education would play be an important in her government’s education policies. In the years ahead they had their ups and downs as Hattie balanced his support for national standards, behavioural objectives, reducing teacher organisation role, performance pay, strong control over teachers, larger class sizes with trying to keep on side with teachers – however, any leftward feint to teachers was to prove only that. The contents of the missing posting can be worked out from references in subsequent postings.  From the moment I read the newspaper article I recognised the task that lay ahead. As much as one elderly gentleman living in a provincial town can do, I followed with a steady stream of postings, Hattie’s arc to education and political dominance.  I never felt an Ahab or a Javert but I was fairly determined. The New Zealand Herald called me ‘John Hattie’s nemesis’ – if only. There was a time when Hattie was feeling the pressure to sign an academic petition against national standards, he balked and I was approached to lay off him as an incentive for him to sign. I was happy to oblige because I knew that in a few weeks my quarry would be carrying on as if he had never signed.

Portrait of a professor: 11.02.2009


Instead of Stephen Dedalus spurning the blandishments of Jesuit power, I had this hero spurning the blandishments of professorial power.

The Hattie series Part 1: Eliding on thin ice: 07.04.2009


This is the first of three lengthy parts in which I apply my curriculum knowledge to Hattie’s research to expose its errors and to urge teachers to stand up for their knowledge in the interests of children and their professional status.

The Hattie series Part 2: Eliding on: 06.05.2009


The Hattie series Part 3: Visible Shipwreck: 13.06.2009


Hattie and the Official Information Act: 30.06.2009


I applied under the Official Information Act to find out what I could about Hattie’s status in the ministry of education. I have some fun with the ministry. I give little weight to the minister’s plea to me that she had never even met Hattie. She might not have met him but I saw the relationship somewhat akin to Elizabeth’s to Essex (though an Essex who had far more policy influence). If only we could have found him some education Ireland to engage his misdirected energies and overweening ambition.

Calculated epiphany now: the partial turning of John Hattie: 03.08.2009


Sometimes Hattie diverged from Tolley in an Essex-type sulk, other times he became concerned with the declining regard of teachers for him, but it added up to nothing. He was joined at the hip to Power.

Hattie fails to follow through: 08.11.2009


More bewildering twists and turns of Hattie.

How corrupted is our education system? 26.04.2010


A lengthy examination about how the ideals and practices of New Zealand education had become corrupted, with Hattie’s research central.

A cute little group of like-minded quantitatives: 12.02.2010


I mount an attack on New Zealand’s quantitatives for, amongst other things, not only accepting Hattie’s research, but acknowledging him as their leader.

Alison Gilmore, executive director (quantitatives) replies and I reply to her: 16.02.2010


The head of a group of quantitatives attacks me; in return, I attack her. Sometimes in the gloom of present-day education there can be moments of fun.

Is ‘Visible Learning’ disappearing? 10.05.2010


Hattie set up the ‘Visible Learning Laboratory’ as a commercial entity (with the University of Auckland) which I set out to undermine. He became desperate at the meagre support for it. Hattie even surprised a Northland principal who had expressed support for my postings by popping in and arguing the point. Hattie’s main argument to encourage teachers to support ‘Visible Learning’ was that ‘If you don’t support national standards you’ll get national testing’.

Hattie and Tolley at it again: 31.08.2010


More of Hattie and his dizzying logic.

Horizons, whirlpools, Sartrean secrets, John Hattie and other symptoms of the continuing education tragedy: 18.10.2013


Hattie wrote a major article to support his commercial entity, ‘Visible Learning Laboratory’, which was set up to travel around New Zealand. The article was called, ‘Horizons and whirlpools: The well-travelled path way of national standards’. Hattie had announced with trumpet accompaniment that he had the answer to national standards. I took him not at his word and examined every claim he made. I believe I drove Hattie’s claims into the ground. It was almost sad to discover how pathetic his arguments were. My final thoughts were ‘What a lightweight – why doesn’t he slow down and think?’

The class size issue: riposte from a professor – Wow! 28.02.2012


Professor John O’Neill delivers a telling hit on Hattie’s research. John O’Neill (along with Ivan Snook) has been an academic who has consistently taken Hattie to task.

Hattie makes his move on early childhood: 27.04.2014


With ‘Horizons, whirlpool …’ I had decided to leave Hattie alone for a while, but it was only a brief respite because there he was in a major Listener article. He declared that early childhood education was in disarray and he knew how to put things right. I saw it as the equivalent for early childhood of his play for dominance of primary back in early 2009. Once again, he was not really talking to those in education but to governments. It was another move by the emperor to expand his empire over the top of education institutions. I replied scathingly. The response was astonishing – over 30,000 hits in a day. Hattie was by now in a very powerful position at Melbourne University. Good luck to Australia but, unfortunately, he was still firmly held on to by the government and private education interests in New Zealand.

Auckland Festival of Education is a glittering cover for something very ugly: 19.03.2014


This is a contemporary symptom of Hattie’s malign influence.

The Rock Lady


If you want to explain the holistic, holistic evaluation, an example of the structure of an holistic activity, an holistic question, the ideal of teaching as it used to be, how to be a great teacher, the antithesis of John Hattie’s philosophy – the true story that follows is it. For me, this story from the ‘80s is an icon or is it a parable? If someone asks me: How could I be a better teacher? I say, read this, absorb this, now go forth and teach.

I was a primary school teacher, deputy principal, and principal. For six years I taught at North Shore Teachers College and was fifteen years a senior inspector in the Waikato. I left the education department and the formal education system in 1990 to fight the neoliberal implications of the Tomorrow’s Schools. I began a magazine for primary school teachers (Developmental Network Magazine) and in 2008 set up a website www.networkonnet.co.nz and just recently https://networkonnet.wordpress.com

The inspiration, motivation, and knowledge for my writing comes from 46 years of visiting classrooms as a teachers college lecturer, primary school inspector, and as someone invited into schools to talk with teachers.

To read the review of the four booklets of my account of New Zealand primary education in click: Information on Primary School Diaries

This entry was posted in Hattie and tagged , , , , , . Bookmark the permalink.

4 Responses to Hattie’s research: egregious errors

  1. Reblogged this on Blogcollectief Onderzoek Onderwijs and commented:
    Kelvin Smythe has been critically following Hattie’s work for a long time and is almost his absolute antipode. Smythe follows up on the recent publication of methodological flaws in Hattie’s research. In addition, he is critical of the connection between Hattie’s rsearch and the neoliberal education agenda.
    Several of us on this Blog Collective have been avid critics of neoliberal trends in education. One of the outcomes of that is the book ‘Het Alternatief’ (The Alternative) that has been quite influential in recent education developments in the Netherlands. An international version of The Alternative will be published next Spring.
    Smythe is not mild in his criticism of John Hattie. The reader may decide for themselves if he has a point.

  2. George LILLEY says:

    Prof Higgins and Simpson point out many more major errors in Hattie’s research, e.g., “We argue the process by which this number (effect size) has been derived has rendered it practically meaningless. “(p199)

    The paper can be found in the British Journal of Educational Studies ( i have the paper if you want a copy)

  3. Kelvin says:

    Much appreciated George. As you will have noted I have been fighting this individual for about 15 years. His research is a disgrace and so all those power-holders who wanted to believe him, and all those other quantitative academics who knew but said nothing. I have a two-page Attack! series posting coming out in a few weeks that brings the badness of his research together in, I think, a pretty compelling way. Anyway thanks to you again, and hooray for bringing tides of bad joy to the individual in question. Will get the paper you refer to.

    • George LILLEY says:

      Thanks Kelvin, I would be very interested in your paper. I’m also going through Hattie’s number 1 effect – “self report grades” and comparing it with “worked maths examples” – for maths teacher.
      His research claims that if maths students merely predict their grade, that will advance their achievement by 3+ years (which is crazy) compared to “worked maths examples” is about one third as effective.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.