Ben Goldacre, The Guardian, Saturday 21 August 2010
Pass rates are at 98%. A quarter of grades are higher than an A. This week every newspaper in the country was filled with people asserting that exams are definitely getting easier, and then other people asserting that exams are definitely not getting easier. The question for me is always simple: how do you know?
Firstly, the idea of kids getting cleverer is not ludicrous. The Flynn Effect is a term coined to describe the gradual improvement in IQ scores. This has been an important problem for IQ researchers, since IQ tests are peer referenced: that is, your performance is compared against everyone else, and the scores are rejigged so that the average IQ is always 100. Because of the trend to greater scores, year on year, you have to be careful not to use older tests on current populations, or their scores come out spuriously high, by the standards of the weaker average population of the past. Regardless of what you think about IQ tests, the tasks in them are at least relatively consistent. That said, there’s also some evidence that the Flynn effect has slowed in developed countries more recently.
But ideally, we want to address the exams directly. One research approach would be to measure current kids’ performance on the exams of the past. This is what the Royal Society of Chemistry did in their report “The Five Decade Challenge” in 2008, running the project as a competition for 16 year olds, which netted them 1,300 self-selecting higher-ability kids. They sat tests taken from the numerical and analytical components of O-level and GCSE exams over the past half century, and performance against each decade rose over time: the average score for the 1960s questions was 15%, rising to 35% for the current exams (though with a giant leap around the introduction of GCSEs, after which score remained fairly stable).
There are often many possible explanations for a finding. Their result could mean that exams have got easier, but it could be that syllabuses have changed, and so modern kids are less prepared for old style questions. When the researchers delved into specific questions, they do say they found some things that were removed from the GCSE syllabus because they’d moved up to A level, but that’s drifting – unwelcomely – into anecdote.
Another approach would be to compare performance on a consistent test, over the years, against performance on A levels. Robert Coe at Durham University produced a study of just this for the Office of National Statistics in 2007. Every year since 1988 they’ve given a few thousand children the Test of Developed Abilities, a consistent test (with a blip in 2002) of general maths and verbal reasoning skills. Scores on this saw a modest decline over the 1990s, and have been fairly flat for the past decade. But the clever thing is what they did next: they worked out the A level scores for children, accounting for their TDA scores, and found that children with the same TDA score were getting higher and higher exam results. From 1988 to 2006, for the same TDA score, levels rose by an average of 2 grades in each subject.
It could be that exams are easier. It could be that teaching and learning have improved, or teaching is more exam focused, so kids at the same TDA level do better in A levels: this is hard to measure. It could be that TDA scores are as irrelevant as shoe size, so the finding is spurious.
Alternatively, it could be that exams are different, and so easier with respect to verbal reasoning and maths, but harder with respect to something else: this, again, is hard to quantify. If the content and goals of your exams change, then that poses difficulties for measuring their consistency over time, and it might be something to declare loudly (or consult employers and the public over).
Our last study thinks more along those lines: some people do have clear goals from education, and they can measure students against this yardstick over time. “Measuring the Mathematics Problem” is a report done for the Engineering Council and other august bodies in 2000, analysing data from 60 departments of maths, physics and engineering who gave diagnostic tests on basic maths skills to their new undergraduates each year. They found strong evidence of a steady decline in scores on these tests, over the preceeing decade, among students accepted onto degree courses where they would naturally need good maths.
Sadly they didn’t control for A level grade, so we can’t be sure how far they were comparing like with like, but there are a few explanations. Maybe maths syllabuses changed and were less useful for maths and engineering degrees. Maybe the clever kids were doing English to become lawyers instead. Or maybe exams got easier.
If you know of more research, I’d be interested to see it, but the main thing that strikes me here is the paucity of work in the field. There’s a man called Rupert Sheldrake who believes that pets are psychic: they know when their owners are coming home, that sort of thing. Obviously we disagree on a lot, but we chat, and are friendly, and once when we were talking he came out with an excellent suggestion: maybe 1%, maybe 0.01% of the total UK research budget could be given to the public, so that they could decide what their research obsessions were. Obviously most of the money would get spent on psychic pets, and which vegetables cure cancer, but I’d like to think that some of it, possibly, might get spent on good quality robust research to find out whether exams are getting easier.