Hattie’s research: Is wrong Part 3 – meta-meta analysis a monster

Dear John oh it is my fate to write

Dear John I must let you know tonight

That patience with your research has died like grass upon the lawn

I tried to let it go but my conscience was stirred and torn.

So I’m back again John, but a little more pointy.

This posting is in two parts. The first part goes directly to the schoolboy mathematical errors in Hattie’s calculations. A key argument related to this is that they and other errors were unavoidable; Hattie had to resort to them because of the inherent faults in his overall research design.

The second part looks at my history with Hattie 2009-11, the philosophical context he was working in then, how is working now, and the central part Hattie plays in our education system. Particular attention is paid to the harm Hattie’s influence is having on children from lower socio-economic environments.

While what follows points out fundamental statistical and mathematical errors in Hattie’s research, I want to emphasise that the central error in Hattie’s research is not in his mathematics and statistics but in his lack of control over the variables. All other errors, such as the mathematical ones that follow, are symptoms. In a bizarre sense, the errors were ‘necessary’ – necessary to cope with the massive lack of control of variables.

The mathematical and statistical errors to be described might seem unbelievable but if viewed in the Miltonic shadow of the lack of control of variables, inevitable. Was there a Mephistophelean moment for Hattie, or did he just rush around patching up the dyke? Is he as incompetent as this research reveals him or was he just trying to make the best of a bad job? Did he fall victim to a lack of education imagination in how to cope with the piles of data his research ambition had delivered?

I have always been able to ‘read’ Hattie and here I’m reading him backward.

Here is a highly ambitious, hugely energetic, and charismatic young man wanting to do the research of everything.  He had this idea of congregating a great pile of data from all over the place and on all manner of things. But what to do then? How to get the data under some kind of control? How to provide some kind of cover for the inherent lack of variable control? The first step would have been easy because it is a key part of the quantitative philosophy and very much in line with what politicians want to hear – that step was to eliminate any data that wasn’t visible and measurable; the second step of eliminating socio-economic influences on learning would have been slightly more tricky – so he took that step by saying it was too difficult to include and, anyway, it was nowhere as important as the qualitatives claimed; once again, the elimination of this variable is very much in line with what politicians want to hear; the third step would have been to construct some dazzling mathematical, statistical , and presentation techniques to cope with the lack of control of his research variables. Oh, and from whatever motivation, it might be a good idea to keep politicians on side by being anti-teacher union, pro-performance pay, and anti-reducing class size.

Hattie may have become trapped by the ‘success’ of his book – an immature researcher thinking big, catching a political wave to wild success (dreamt of but not to that extent), then having to protect the research’s  reputation by playing superficial, using his newly acquired status as an argument in itself, and scrambling for dear cover. Oh, and one has to mention his matinee idol good looks and accompanying charisma as factors.

For me, as ever, literature holds the key to understanding Hattie’s actions. I see his obsessive pursuit of a form of knowledge that he can call his own as producing a Frankenstein. Hattie’s research, pushed beyond accepted research limits, has proved as destructive to children – the declared beneficiaries of that research – as it may turn out for him. As Frankenstein was to Victor his creator, so meta-meta analysis may be to Hattie. The Frankenstein of meta-meta analysis based on visible knowledge is an education monster. Perhaps the just-in-time self-realisation of Robert, will, in this case, be played out by academics quantitative and qualitative, and spur them to slay that monster and, perhaps, going against the story, save its creator.

Whatever, the crucial point is that it is the fundamental fault in the research design that is the problem; one inherent in the meta-meta analysis, any education meta-meta analysis.  So I’m not getting carried away with any of the symptoms. There are a number of reasons why I keep stressing the research design and lack of control of variables as central to the problem of Hattie’s research: first, to provide an analytical tool to help the reader cope with the sheer scope of the furore; and second, in ‘reading’ him backwards, a concern that he might be contemplating a revised version of his book as a way of shucking off the challenges to it. Hattie’s book cannot be shucked; it is rotten to the core.

Teacher who have confidence in their judgements about children and classroom learning and haven’t been taken in by Hattie’s razzle-dazzle know that the research is wrong. We already know from earlier postings that the design is hugely wrong now we are going to learn that his mathematics is too.

I am indebted to website writers Leafstrewn and ollieornage2 amongst others for details of this in-your-face-evidence of Hattie’s errors.

One of the central ways Hattie expresses effect size is with a mathematical concept, Common Language Effect (CLE). Mathematics may not be your area as it isn’t mine but stay with me. CLE is the probability that a random child in a treatment group will outperform a child in a control group. In using CLE, Hattie is basing his research on probabilities which means that there can’t be negatives. That is an everyday fundamental of secondary school mathematics but Hattie has used negatives. This is either cataclysmic incompetence or bravura, I think bravura.

In my view, what has drawn Hattie into this error is those out-of-control-variables, and one in particular, Hattie’s inability, because of the research design, to determine whether there was a control group in place, or whether the effect size was calculated as compared to the same children before the study began.

And from there it is one small but error-ridden step to his barometer with the swingeing hinge – oh the razzle-dazzle. If the hinge for an effect falls below 0.40 it is more-or-less a case of why bother, if above, all hail. In a later posting, I will detail how the wonder of the barometer was used to deliver via Hattie to the treasury – the bureaucracy at the heart of bureaucracies – the most appalling nonsense with stone-chiselled certainty.

I am reminded for some reason of Charlie Chaplin tremulously hanging on to the hands of a big clock.

The homework findings of which Hattie makes much in his book is given as 0.21. This means that a child who didn’t do homework would very likely do better than the one who did. Aha! no more sending junior children home with their little readers to share with their parents. But the hinge having swung swings on to further 21st century education enlightenment.  The hinge has ended up pointing to reading making more difference than the use of calculators – now there is a pedagogical comparison to go to bed with, so build that into your programme. But hold on, oh goodness, but less of a difference than outdoor programmes, now there’s a dilemma – what to do? I know … how about children doing their reading outside? And even more challenging news for reading, it makes much less difference than vocabulary programmes. Well there you go; I’ve often thought reading an unnecessary interruption to children learning words.

I have written extensively of John Hattie (see list in Part 1) mainly to caution teachers about him, all the time waiting for his quantitative academic colleagues to see what I see – presentation enormities clambering over research absurdities – and from there gathering the courage to break through the aura of fear his status engenders, disregarding the career enhancing spaces his work message provides, and, as with the emperor with no clothes, shout out the obvious in a public way.

A metaphor for me is seeing a car on the wrong side of the road broadsiding another car and disoriented children and teacher clambering out; for quantitatives though, it is a car parked at a an intriguing angle and position, with passengers exiting hurriedly so as to gain all they can from the experience so kindly provided.

My postings on Hattie from 2008-11 were persistent and detailed but I didn’t push them to their logical conclusion. My intention was to provide teachers with another perspective on Hattie’s entrepreneurial campaigns, and some protection from the classroom effects of his education message. From time-to-time, a whinge from Hattie was communicated to me (from other academics) and I was warned to be careful. How Hattie presented me to other academics was made clear in a comment reported from a very well-known holistic primary school advocate who attended a course by Guy Claxton from England. At this course, Guy Claxton sympathetically to Hattie’s position relayed Hattie’s whinge that he was being harassed through the internet. You can judge the validity of this description by reading my postings at the time (the links are listed at the conclusion of Part 1 of this series). To me, the postings were a back-to-the-wall defence of primary’s holistic curriculum written in opposition to the depredations of a very different curriculum written by a powerful academic enjoying a shower of hosannas and shekels.  A notable moment was when Hattie was caught in between his occasional tactical rhetoric criticising national standards, and his close association with the government to which he had suggested them. When Hattie was invited to sign a petition against national standards, I was asked to lay off him as an encouragement for him signing it. I was happy to agree because I knew that within a few weeks, like a moth to the light, he would be back again supporting the government and national standards.

I was described in the New Zealand Herald as his ‘nemesis’, which, unfortunately, I wasn’t – just his irritant. However, for a number of reasons he did leave New Zealand. But it was only to Australia which only served to increase his influence not only in Australia but in all Western education systems; in respect to New Zealand his direct influence on teachers reduced but on politicians and bureaucrats probably increased. Hattie’s appeal to the governing elites was his definition of education as that which was visible, and by that characteristic, measurable. Hattie had caught in spectacular fashion the wave of obsession with certainty.

Any debate with Hattie about his visible learning research should start with him explaining why only visible learning qualifies as learning. His recourse, of course, is to acknowledge that there is other learning; indeed, that he makes a statement to that effect in his book. However, it is important to pause to ponder what is going on here – important because it as this juncture, if the reader isn’t careful, Hattie slips through as somehow normal his research and education claims issuing from it. You see, when Hattie is selling his visible learning results, while there might be a disclaimer here and there, the results of his visible learning research are received by his audience, and understandably so, given the charismatic and definitive way it is presented, as ranked, absolute and complete. Hattie’s research should be dismissed on its title alone but because he is so upfront about it, the argument can slip past without the reader taking it in. It is sleight of mind by effrontery.

Are we, in the 21st century, really allowing a group of apparently narrowly functioning quantitative academics to hijack learning to be restricted to not only the visible but also the immediately measurable? What kind of stunted and deformed education is that? This alliance of powerful quantitative academics with Western political and bureaucratic elites has been devastating to school education. It needs to be appreciated that the source that correctly allows one to describe those academics powerful has, to a significant degree, not been assigned by those in education but by those in the political and bureaucratic elites. And the research they produce for those political and bureaucratic elites is very much of the present, based on artificial research set up for immediate result – a kind of fast food education. The immediacy of that result being a central determinant of it.

Quantitative academics are inheritors of the Aristotelian analytical tradition? A tradition that has, of course, changed and developed to the scientific method as we understand it today. But does this group of academics pay genuine heed to the education philosophy of scientific reasoning in a proper scientific sense? Where is the way that testing an hypothesis always leads to a profusion of other hypotheses in continuous process? And crucial to this, if the search for ‘truth’ is the central purpose, would be the inclusion of not just hypotheses from those academics’ specialised field but the affective field as well. The affective can, indeed, in some sense be measurable if done in combination with non-measurable constructs and based on long term research. And the more researchers work through hypotheses as part of this process, the less certain researchers will be, and the more ‘truths’ revealed and the more hypotheses to choose from. This would lead to hyper-caution in presenting results and a refreshing consequential plummeting in appeal to political and bureaucratic elites.

Children’s learning isn’t divided; why is research?

Hattie and his associated quantitative academics have sidelined the affective side of education in their research to devastating effect on Western education systems. Their research is leading to children who can read but not the inclination to do so; who can write but find no satisfaction in it; who are asked questions but not challenged to think; who are hardly taught any science or social studies and when they are, find it lacking in heart and connectedness; who do a fair amount of art but in a stereotyped way; who hardly do any drama or dance; who are taught to a national standards timetable not a personal development one; and who, in general, find learning, boring, lacking challenge, and engagement.

In New Zealand, children from low socio-economic areas, for whom Hattie’s definition of education is particularly aimed, have now been the target of that kind of education for 25 years. It has been a failure. In the most recent secondary school university exams, when secondary school children were exposed to exams slightly more genuine than before, in other words, not internal jack-ups, the results were disastrous, especially for children from low socio-economic areas. This was made even more disastrous by the fact that huge numbers of those children had been diverted from even trying out for UE to soft internal options. Hattie’s definition of education, his opposition to smaller classes, his downplaying of the effect of socio-economic status on learning, provide a way out for governments not to do more to reduce poverty and to increase education expenditure. In combination, this has had a harmful effect on all children and a devastating effect on some.

Intellectual challenge and affective engagement is not an optional extra to be delayed pending the 3Rs being attained. Learning the 3Rs and intellectual challenge and flexible thinking are not mutually exclusive – just the reverse, they are reinforcing and, for anything like a satisfactory education, imperative. The emphasis on the 3Rs is a return to the past, condemning many children to a second-class education and given the demands of contemporary life, and access to university, akin to certain groups of children being allocated to gardening skills. It is a policy doomed to failure in the guise of one deemed for their salvation.

Education is and should be a matter of eternal debate. The education of children is problematic and value-laden. For the integrity of the education system, the various groups within it need to be free, willing, and able to argue and even, at times, obstruct the ideas and actions of other groups. The powerlessness of the young, the fact of them being young, makes school-children tempting targets for those who want to turn schools into battleground for competing visions of what society ought to be. There never has been and never will be a set of aims and related processes that have met, or will meet, the needs of all children within a system, or be agreed to by all those within a system. I am referring here to democratic societies, the conditions for authoritarian societies will, by definition be different.

At a level that matters, in a way that matters, there have been severe restrictions on education debate in most Western democratic societies in a sense of a debate amongst equals. Neoliberal education, on the basis of avoiding provider capture by teachers, has led to teachers being excluded from decision making and their ideas poopooed in debate. That has resulted in power being distributed upwards and undemocratically. Isn’t it ironic, and with deeply dangerous implications to democracy, that the effect of avoiding provider capture is absolute capture elsewhere? An effect on education is that a certain set of values has become inflexibly held amongst those who administer education. Those who administer education are looking for a kind of school education they can understand; that has certainty to it; that can be measured for control. Hattie’s definition of education, his research, and education philosophy has been of central recourse to those leading the neoliberal revolution of education in New Zealand and elsewhere. So central has it been, and as to be detailed in subsequent postings so faulty, I am calling for his visible learning research to be subject to an enquiry.

This entry was posted in Education Policy, Hattie and tagged , , . Bookmark the permalink.

5 Responses to Hattie’s research: Is wrong Part 3 – meta-meta analysis a monster

  1. Kelvin says:

    A comment of a reader: Hi Kelvin

    Thanks again for your revealing post – the fact that you persevere and hold your ground on this is great encouragement for others.

    Some relevant thought of others.
    Lakota Elder Dottie LeBeau: ‘When we approach teaching with one worldview…we create systems of failure in our schools.’

    Martin Luther King: ‘Our lives begin to end the day we become silent about things that matter.’

    Bertrand Russell: ‘Almost all education has a political motive: it aims at strengthening some group, national or religious or even social, in the competition with other groups. It is this motive, in the main, which determines the subjects taught, the knowledge offered and the knowledge withheld, and also decides what mental habits the pupils are expected to acquire. Hardly anything is done to foster the inward growth of mind and spirit; in fact, those who have had the most education are very often atrophied in their mental and spiritual life.’

  2. kellyned says:

    Hi Kelvin.
    Keep it up.
    Keep in mind that in previous generations and under previous secondary systems a huge percentage left school at the end of year 11 (5th Form) and went into the trades or other practical types of employment. We (at our school) have virtually all of these students continuing on to Year 13. They are not an academic bunch but gain meaningful credit for courses they choose which lead them into a range of other areas that don’t require University qualifications.
    This has to be better than having them still leaving at Year 11 without a pathway into what is often service type industries.
    Certainly Hattie’s ‘research’ (if it can even be called that now) has done real education no favours. Bean counters will continue to love it because it appears to quantify the unquantifiable. As educators we just need to be able to match the, be prepared to stand up against it at every opportunity and – as you do – keep banging the drum for real education of the whole child.
    Kia kaha

  3. Kelvin says:

    Thanks Kellyned: It is the high stakes’ use of results being interpreted that charter schools are doing well and the education system is doing well by all children. If that is the case there will never be any change. All best. Thanks for your regular contribution.

  4. It certainly is difficult to separate the Edu wheat from the Edu chaff these days. In most cases the solution is NOT to throw more of each into the hopper to find just the wheat. I find Hattie’s work to be somewhat useful but hardly a definitive guide. So often “Interventions” depend on what students, what location, the methods employed, synergy, desired outcomes, etc. etc. Many “Interventions” are far more effective on the “educationally disadvantaged” learner than the “well supported more privileged” learner.

    Consider “Class Size” (low effect size) and “Grade Level Retention” (negative effect size) …. In Florida (a historically low performing state) both were implemented across the entire state as part of a program to increase the competence of students. An organized program of instruction that included “lower class size” and “Grade Level Retention” at grade three for poor readers produced a resounding success. The percentage of students retained each year has dropped as schools have improved reading instruction. NAEP data for Florida in subsequent years confirms this as an ongoing success and effecting student performance in grades far above the primary grades. Yet strictly applying Hattie’s numbers would tell me “dollars would be better spent” on something other than reducing class size and that “Grade Level Retention” is a counterproductive practice.

    It also seems that concerns for the “affective domain” of students is rarely a part of any education discussion in State Legislatures. There should be much more to decision-making than Hattie’s effect-size numbers.

  5. Kelvin says:

    Thank-you again Danaher. Hattie sometimes articulates sound ideas but they are not based on sound research, because none of his research is. The ideas just happen to be sound. Your chaff metaphor is a spot on. Keep in touch.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s