Hattie’s research is false: Vice-Chancellors asked to investigate Part 1

5 April 2018                                                                                                   

 Pokai Tara                                                                                                      

The Vice-Chancellors                                                                                    

Wellington 6142

Dear Vice-Chancellors

To protect the good name of New Zealand universities, the professional integrity of schools, and the welfare of children, I ask the Vice-Chancellors of New Zealand to begin an investigation into the soundness and authenticity of John Hattie’s research – research involving 800 meta-analyses (now grown in absurdist fashion to 1,400), comprising more than 50,000 studies (now 90,000), and some 146 effect sizes (now 250), which provided the basis for the Hattie’s book, Visible Learning. 

The Vice-Chancellors have the overall responsibility for ensuring academic quality assurance, helped in carrying out that function by the Universities New Zealand’s Committee on University Academic Programmes (CUAP) or the Academic Quality Agency for New Zealand Universities (AQA) which is set up as an independent body.

My concern is that none of the variables in his research are validly isolated or under control, resulting in an academic shambles that, in being left unexposed, has had devastating consequences for teachers and children around the world, and especially New Zealand.

This writing will convey far more opinion and information than is strictly needed for the request to the Vice-Chancellors, that is because it is also being written, as is the overriding intent of all my writing, for teachers and principals as a means of keeping them informed of influences on their professional lives. As well, as regular readers will know, I am serious and sincere in what I write though I usually succumb to a levity that I find hard to resist given my general view of the world. I blame my Shadbolt genes for that element of larrikinism but I still affirm that everything I write remains serious and sincere in purpose. 

As well as writing to the Vice-Chancellors about the falseness of Hattie’s research and asking them to make a declaration as to its soundness and authenticity, I will be writing about the consequences of that research for New Zealand teachers and children and how, in alliance with powerful political and education forces, Hattie came to dominate and, resultantly, harm school education. Adults, of high social status, have acted badly in relation to Hattie’s influence, looking away, it seems, to serve their own interests and, in that way, betray a generation of children. Hattie’s research is so obviously false, so obviously against accepted ideals of the principles, purposes, and practices of universities, it suggests to me that universities are fearful of his exposure as leading to their own. Hattie, in a brochure for 2018 Australian conference, described critics of his ‘evidence-based’ research (in other words, people like me) as producers of ‘fake news’. I am returning the epithet – but in this case validly – and calling Hattie’s research ‘fake’.My assumption, given Hattie’s personality, is that he felt he could get away with it by deus ex machina and he largely has. What was threadbare was seen as gold thread. The combination of the modern obsession with certainty, and the universal striving for power, has drawn in and beguiled academics. 

Hattie, is all over the place in his research, his only consistency being in never getting it right (to be demonstrated throughout). What follows are small examples that, in themselves, should be sufficient to alert that something is afoot, something discreditable. Such examples do not happen by accident – only design. The following is just a supporting act to the full dodgy horror of the Hattie Rocky Horror show.

An academic dug deep into Hattie’s research information (his research details are extraordinarily difficult to locate) and came up with the following (there are dozens more like this):

  • Hattie’s ranking of quality of teaching: The question he put was – how important an influence was it? 

Oh my goodness! What does ‘quality’ mean? You could drive a fleet of buses through it? Such research vocabulary impropriety is a characteristic of Hattie’s research; another is the never asked yet crucial research question – in relation to or comparison with what?

Hattie is producing this research for the supposed edification of school teachers – so what is the status of the people he is asking? university students. And in their wisdom, ‘quality’ was ranked 56th out of 138 in Hattie’s league table.

As many interpretations as there can be about what quality teaching means, surely it is what teaching is about and there can really be only one ranking and that is number one.

I really feel I should say to the Vice-Chancellors, I rest my case.

But want some more little pearlers?

  • The micro-teaching influence was ranked 4th as an influence, but only came from pre-service teachers
  • Professional development was ranked 19th as an influence but drops to 48th when only school research is included
  • Formative evaluation was ranked 3rd as an influence but there were only two sets of results and both to do with special education children
  • Comprehensive interventions were ranked 7th as an influence but they only involved learning disabled children.

 Hattie arrived in New Zealand and, with his coruscating charisma and his ranging research, bedazzled universities with the prospect of boundless income and prestige. If there were any reservations, they were probably allayed with the thought that if his research was good enough for Toronto, it was good enough for them. But what happened at Toronto itself needs investigation – something occurred there in relation to his research that must surely have been highly irregular. The interesting thing is that he came to Toronto with his research well underway: where did he come from and why did he go to Toronto? (Probably baseless suspicion, but I’m puzzled at how Hattie’s research got through the supervision process.)

I participated with an academic science group at Waikato University that had, as one of its functions, examining post-graduate theses in development. Just as in a novel there is always a plot weakness the writer has to find a way to glide over, camouflage, or manoeuvre around; there is always in research an argument weakness the researcher has to deal with, but, in the case of a research, the ethical response is for a researcher to be honest and open and address it according to academic convention and, in that way, give integrity to the whole.

If Hattie’s research had come to our academic science group it would have been laughed out of town. His wasn’t an argument weakness; it was a black hole. Hattie’s research is a battered, articulated truckload of rubbish being presented as a caravan of giant gold and silver carriages transformed from pumpkins.

Hattie based his research on a form of large-scale medical research which deals with only a few variables – always kept under tight control – for instance, the effects of the use of surgical gauze – but in Hattie’s case he just went for the very large-scale aspect, gargantuanly large, and encompassing the totality of school education. He inexorably lost control of the variables, though he freely allocated names to them which, to fantastical confusion, as part of that loss of control, have different meanings and expressions within and across education systems (see above).

Because the research was so confused and impossible of following or understanding, it provided Hattie with considerable imaginary scope for explaining what it meant, giving him the freedom to produce a particular kind of Holy Grail, one for which he was supremely fitted, that of exciting narrative but research rubbish.  He was able to deliver the research as the evidence-based research to end all evidence-based research, as certainty beyond dispute nay even quibble, as without the complexities heretofore considered inherent in education given the otherwise complexities of human behaviour – thereby setting up a generation of mini-Hattie-messiahs in schools and conference halls to deliver the truth.

When I took science with children, I always stressed as part of the investigatory process, that any result, to be valid, needed to be able to be replicated. 

This has never been done with Hattie’s research. 

Perhaps it is post-modern. And if so it might not matter the numbers being wrong. When pursued and pushed (in particular by British academics), he, at last, grudgingly admitted that due to a school level mathematical error, half his figures were wrong. I don’t make a big thing of this because the other half are mathematically correct in accordance with his weird manner of education thinking. But this mathematical error does reveal slack doctoral supervision and a cavalier attitude towards research.

I was to spend weeks going through his writings to try to identify any acknowledgement by Hattie of the nonsense of his research, any reasons he could give for his research having no control of variables. I was to find only one reference:

‘No worry, it all balances out.’

It is post-modern. 

However, I’ll push on.

In 2009, I wrote my first formal comment about Hattie (‘Hattie fails to follow through’):

‘If they are in, they are in for years. John Hattie’s condemnation of national standards in today’s Sunday Star-Times (November 8, 2009) might have been unreserved but national standards are still in. And Hattie would know that: he’s had his predictable bob each way. At the end of it, his university commercial arm will still have its lucrative contracts, and he will still have his relations with the government, at the same time he will be seen as a hero by some.’

‘Not following through on his condemnation was entirely predictable. We all have to get a clear idea about Hattie’s policies and personality, because his vision will be the dominating one in education for the decades ahead. He should be understood and confronted now. His ‘research’ findings, have already had an effect. The head of Treasury has cemented into government policy the idea of no more reductions in class sizes.’ 

Hattie is entirely predictable if you consider his behaviour in the light of a particular personality category. At the height of the national standards debate a large number of academics got together to sign a petition. He agreed on the condition that I stop writing about him (‘attacking’ I think was the word he used). Another academic approached me and I readily agreed because I knew Hattie’s forbearance level would not last a month and it didn’t.

There is a disconcerting sense of the academic huckster about Hattie:

  • It is well-beyond the time when he should have stopped cashing in on the 2008 Times Educational Supplement reference to his research being ‘the equivalent to the search for the Holy Grail [hence my reference above]. It was repeated on the cover of his book and it has been quoted again and again in subsequent years. The reviewer was no doubt carried away by the certainty and sweep of the book and the exciting narrative, but he wasn’t, of course, in a position to ascertain the research’s validity. A Holy Grail would not have half the mathematics wrong and all the research.
  • He should stop boasting about the size of the study. If a study is wrong, the bigger it is, the more wrong it is. I am looking at advertising for the 2018 ACER Research Conference, Hattie has written that his book ‘is believed to be the world’s largest evidence-based study into the factors that improve student learning.’ Oh for pity’s sake!
  • And there is, indeed, something Trumpean about his approach to education, in that same Conference advertising he says: ‘there is ongoing backlash against evidence-based research (fake news)’. The Karmel Oration, for which he is the orator, has the heading ‘The role of educator in the fake news world’. I ask the Vice-Chancellors and readers to adjudge my criticisms (see below) as fake or not.
  • Everything he says points to himself and his sense of entitlement and in doing so he comes across as sulky or trite. In that Conference advertising he says: There is ‘continued debate over inputs to the system without references to outputs.’ Interpreted that means I did that for you with Visible Learning – now I am getting criticism for it. Also: ‘This Oration will explore the notion of expertise and evidence and how expertise is anchored in evaluative thinking.’ That is both sulky, self-centred, and trite. It means you peasants in classrooms do what experts tell you – we know what we are doing so keep your notions to yourself; and how trite: ‘expertise is anchored in evaluative thinking’? Frankly, I think the expertise boat to which his evaluative thinking is anchored is full of holes.

To act on this request for Hattie’s research to be investigated, the Vice-Chancellors would need to be convinced of Hattie’s continuing connections with New Zealand universities. These have reduced but he remains an Honorary Professor of Education at Auckland University and, more directly to the matter of quality assurance in universities is the widespread use of Visible Learning (and accompanying research) in university courses. (The irony has not escaped me that Hattie was once chair of the Performance-based Research Fund.) 

I am a former primary school teacher and principal, teachers college lecturer, and senior inspector of schools – when Tomorrow’s Schools began I resigned from the formal education system to visit schools on invitation, speak to teachers, and to write. I have visited schools in some kind of official or invitational capacity for 51 years. 

I have sent, in association with this communication, a complimentary copy of my latest publication, The File, which sets out my expression of the holistic as against Hattie’s formalistic. I reiterate, that under no circumstances am I advocating that the formalistic should not be a choice in class- and lecture rooms – all I am seeking is fairness. 

When Hattie first came to my attention it was clear that his formal, teacher-directed, and highly structured ideas on education were antithetical to mine. This, in itself, was of no immediate concern to me, simply another quantitative with his or her way of looking at school education. 

But Hattie was a phenomenon with a sense of absolute certainty about the rightness of his ‘evidence-based’ findings and the ideas generated from them. He became central to the education functioning of Auckland University and New Zealand universities generally, the Treasury, the government, the education review office, corporate advisory, the advice many senior teachers and principals found accessible to deliver, and national standards.

None of this, of course, is evidence of any lack of integrity in his research but, on examination by his peers, if his research is found to lack soundness and authenticity then his position as New Zealand’s controlling ideological academic would be reduced to manageable proportions.

Hattie would dispute his research as being ideological because he would describe it as ‘evidence-based’ and therefore, in his thinking, the truth. If some other evidence-based findings appear, there is a propensity to appropriate opposing ideas with the effect of corrupting them. An example is the evidence coming from neuroscience, in reality it produces evidence directly opposed to Hattie’s evidence, indeed, is crunching support for the holistic – but Hattie uses the word ‘mindset’ with accompanying words like ‘growth’ as in ‘growth mindset’. 

Hattie’s research and Visible Learning, and subsequent manic screeds of explanations about effect size and their rankings, comprising slides, presentations, updates, and mountains of ideas in flashy array, serve to overwhelm education systems to Hattie’s slippery advantage. His research, in being about the totality of education, gives, in Hattie’s mind, the license, freedom and say-so, to comment, criticise, elevate, and appropriate the totality of that education. There is a definite bias in what he promotes and demotes – he promotes education ideas that will appeal to conservative governments, for instance, he gives a low rating to smaller classes; and in classroom learning, he promotes practices that are formal, for instance, phonics instruction as against whole language. But the two concepts of class size and whole language, for validity, require a different kind of research, first of all, of course, authentic research, then a kind of research that is patient, extending to the longer term, and is based on the principle that what is going on in children’s minds is not always immediately or directly visible.  

In an article in the Herald (February 6, 2010), Andrew Laxon writes ‘Bill English sought Hattie’s views when he originally developed the party’s national standards policy and Key took the same route, drawing inspiration from Hattie’s advice that a standards-based approach could work wonders in even the poorest schools.’

In that same article Andrew Laxon writes: ‘Hattie’s sworn enemy in all educational matters, former school inspector-turned-blogger Kelvin Smythe is far more forthright.’

‘He believes that Hattie has worked hand in glove with the Government on the system and claims his influence is so great that other academics are too scared to speak up against him.’

I commented: ‘John Key said John Hattie was the architect of the introduction of national standards into New Zealand. He became the overwhelmingly dominant academic supporting government policies. To those of us who wanted truth in education, who wanted progressive policies, who wanted genuine discussion, Hattie was an academic miasma. He was everywhere. His research was the rationale for government policy and in combination with the government and a bewitched media, a devastatingly formidable force to confront.’

But his research, as I hope to convincingly demonstrate, is false.

I know some will think such a charge ridiculous? Hattie, a former University of Auckland professor, producer of the Holy Grail research, dominant with the previous government, still highly influential with education institutions and pervasive throughout the education system, presently head of the Melbourne Education Research Institute, the leading academic voice in Australia, consultant to Pearson, the go-to academic world-wide.

Just as Fukuyama declared the end of history so, in a way, was Hattie’s research declared the end of education research.

But that would be strange wouldn’t it given the name of the book of his research: Visible Learning? A book of research that concerns itself only with that which is visible? A perfect model for the neoliberal commodification of education but an imperfect one for research on children and their learning.

Continued in Part 2

This entry was posted in Hattie and tagged , , , . Bookmark the permalink.

6 Responses to Hattie’s research is false: Vice-Chancellors asked to investigate Part 1

  1. visiblehat says:

    Thanks Kelvin, the best place for the Vice Chancellors to start is to read the 3 Meta-analyses Hattie used for the 2009 version of Visible Learning. These show, Hattie can’t average and gross misrepresentation. I contacted Gene V Glass, the inventor of Meta-analysis and author of the major study Hattie used for Class size, he commented about Hattie’s averaging – ‘Averaging class size reduction effects over a range of reductions makes no sense to me’.

    Summaries of the 3 meta-analyses are here – https://visablelearning.blogspot.com.au/p/class-size.html

  2. Kelvin says:

    Thanks visiblehat. Excellent. If you look at Part 2 you will find your letter and the link included as a component part of the posting. He’s obviously feeling the pressure. I think we might have him.

  3. Excellent. About time this became more widely known.

  4. Barbara Brann says:

    I have always had great respect for John Hattie, but I questioned some of his findings in this research very early in the piece. My questions related to research about PMP which was ranked very low. But when you looked at the research that was used in the meta-analysis, not only was it highly questionable even at the time it was done, but was nearly 40 years old! Then there was the class size – much of that research came from research into secondary and tertiary education – it is very different teaching 5 year olds! Then there is Direct Instruction which resulted in millions of dollars being invested on American Based programmes and training and then implemented in Indigenous Schools throughout remote Australia. It lasted all of 3 years before being tossed. If you look at the research quoted for Direct Instruction, again it is very old – about 40 years, and very dubious in its authenticity. I was in a tiny remote school in the Pilbara in WA where the children could parrot how long the Mississippi River was, but did not understand the concept of distance except in broad times to travel! – long time, sleeping time!!! What waste of time and resources. But because it had come through Hattie’s research it was treated as gospel and implemented without questioning. I could not find anyone in NT education who was remotely interested in debating the original research on which these aspects of the analysis were based!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.