This post reviews Sandra Martin-Chang, Odette N. Gould, and Reanne E. Meuse, “The Impact of Schooling on Academic Achievement: Evidence from Homeschooled and Traditionally Schooled Children.” Canadian Journal of Behavioural Science 43, no. 3 (July 2011): 195-202.
The authors of this study of 74 children, half homeschooled, half institutionally schooled, conclude that structured homeschooling is best, public schooling next, and unstructured homeschooling worst at producing high levels of academic achievement. They begin with a nice summary of the limits of previous studies of academic achievement, especially of the famous Rudner study of 1999, which we have had several occasions to mention on this blog. They wanted to overcome the sampling biases of this and many other HSLDA-funded studies of academic achievement.
To do this they sent out ads in every medium possible to recruit public and homeschooled students between ages 5 and 10 in Canada. They chose 37 public schooled and 37 homeschooled children from the recruits. These kids were then matched up so as to create pairs of demographically similar children. Later, the homeschoolers were further subdivided into “structured” and “unstructured” once researchers figured out there was a real distinction here in how homeschoolers went about their daily education. There were 25 “structured” homeschoolers and 12 “unstructured.”
Parents filled out a demographic questionnaire and children took the Woodcock-Johnson Test of Achievement. Both homeschoolers and conventionally schooled children were given the test in the same way by the same people, thus avoiding one of the most common problems with most academic achievement studies.
The results were that the structured homeschoolers scored higher than the public schoolers on every part of the test, even though their families reported slightly lower incomes and their mothers had a bit less formal education.
Unstructured homeschoolers, on the other hand, consistently scored lower than institutionally schooled students, and in four of the seven categories they performed below grade level. The authors are quick to note that their sample of 12 is inadequate to make compelling generalizations, but they find this “exploratory analysis” to be intriguing.
The authors end with a discussion of some of the limitations of their study. They note that the study’s reliance on volunteers likely contributed to an inflation of homeschooler scores, but in their case (unlike all other studies) the institutionally-schooled students were ALSO volunteers. They too scored above grade level, just not as much as the structured homeschoolers. The researchers also note that the small sample size did not allow for more nuance about level of structure homeschoolers employ, and they wonder if over time maybe the unstructured kids would catch up.
I have three points to make about this article:
1. Part of me is skeptical about the whole thing. Here’s why. The authors say that at the outset of their study they were only planning on comparing the results of homeschoolers to those of traditionally schooled children. But that’s NOT what they ended up doing. They say that they later came to figure out that there was a distinction between structured and unstructured homeschooling, but it is very convenient for their findings that this division has allowed them to pull out the low scoring homeschoolers from their sample. The authors never end up giving us the actual comparison between their 37 public schoolers and their 37 homeschoolers. Instead, we get a comparison between the 37 public and the 25 best scoring homeschoolers, and then a separate comparison between the 37 public and the 12 worst scoring homeschoolers.
It is impossible to tell from the data provided in the article itself what the results of a straight-ahead comparison between the 37 and 37 would have given, but it looks to me like if you combined the two groups of homeschoolers you’d end up with scores that were pretty close to the average obtained by the public schooled students.
The reason this bothers me is that I’ve recently been reading Ben Goldacre’s very funny Bad Science: Quacks, Hacks, and Big Pharma Flacks, which chronicles several ruses pharmaceutical companies use to cook the books to make their drugs look good. One of their best tricks is to start a study looking for a result in the general population, but when the data doesn’t given them a good result for everyone, they look for any sub-population on which it seemed to work. If they find that the drug works on, say, women of Chinese ancestry, they rework the entire study into a trial about the results of their drug on Chinese women.
Now, Martin-Chang, Gould, and Meuse do not seem to have any political agenda driving their disaggregation of the homeschoolers into a high-performing group and a low-performing group, so I don’t want to accuse them of intentionally cooking the books, but it does seem very strange to me that they don’t even report at all their findings on the original question that inspired their study.
2. I found the distinction between “structured” and “unstructured” homeschoolers a bit unsatisfying, especially given how significant it is in this analysis. Here’s why. As any veteran homeschooler will tell you, there’s no hard-and-fast distinction here. There’s a continuum. Furthermore, researchers have consistently found that over time families tend to move along that continuum from structure to unstructure. This paper’s facile distinction misses both of those points. The paper also fails to consider length of time homeschooling among its subjects. Had the “structured homeschoolers” been homeschooling their entire lives? For just a few months? We don’t know, though the researchers themselves pointed out the significance of this question in their opening discussion of Rudner.
3. Finally, despite these significant misgivings, I want to applaud the study for trying to right some of the flagrant wrongs of nearly all previous studies of academic achievement. They’re not comparing a select group of homeschoolers to national averages like Rudner did and like Ray has done and continues to do (for a long treatment of Ray’s oevre see here and here). They’re not comparing public school kids who take their tests under stressful conditions in school, proctored by outsiders, to homeschoolers who take them in their living rooms with mom looking over their shoulder. They control for demographics. All of this is wonderful. If they had only given us the generic data comparing all homeschoolers to all public schoolers in their sample and given us data comparing the results of longtime homeschoolers with those who have only just begun, the results would be much more valuable. Of course larger sample sizes would be nice too, as would (pipe dream) a longitudinal follow-up in 10 years to see if the results from these young children remained consistent over time. But for all its failings, this study provides a great model future studies could build upon to get us some real data on homeschool/public school comparisons.