Archived InformationThe Quality of Vocational Education, June 1998
Standardized tests do not measure student achievement perfectly, but they do a good enough job for people to base decisions on them. School counselors use test scores in guidance and placement; colleges use them in admissions; and the public uses them to gauge the accomplishments of schools. Most people are pleased when test scores are high, and they worry when they are low.
Vocational students do not perform as well on standardized tests as students in college programs do, and that worries some educators. Boyer (1983), for example, has concluded that vocational programs shortchange students academically. Oakes (1985) doubts that students in vocational programs get adequate preparation in basic subject areas. Lotto (1986) has concluded that vocational programs provide an inadequate preparation in the basic skills.
The Coleman report of 1966 ushered in the modern era of research on curricular tracks and student achievement (Coleman et al., 1966). Coleman's Equal Educational Opportunity Survey (EEOS) examined the relations between many individual and social factors and school learning, and it did not mince words about curricular tracks. "Tracking," Coleman wrote, "shows no relation to achievement" (p. 314). Coleman compared schools that track students with schools that do not track students, and he found no difference in test scores at the two types of schools. He concluded therefore that tracking did not make a difference in student achievement.
Coleman's focus was on differences between schools, however. He found that average test scores were very similar in schools with and without curricular tracks, but he also found that there was a great deal of variation in student achievement within schools of both types. It was this finding that stimulated a new generation of survey research on tracking. Researchers speculated that track membership might explain some of the variation in achievement within schools. Researchers therefore began putting track membership into equations predicting the achievement of individual students within schools.
Some of the resulting studies tell us little if anything about curricular effects, but other studies are far more informative. In this chapter, I examine the variety of available studies. I first describe the studies and look at some of the differences among them. I then turn to the study findings and their implications.
The studies I review in this chapter came from two sources: (1) a computerized search of the data base of the Educational Resources Information Clearinghouse (ERIC); and (2) studies referred to in reviews located through the ERIC search. I first searched the full text of citations and abstracts in the ERIC data base from the years 1982 through September 1993 for the terms secondary education, vocational education, and academic achievement. I located a total of 74 abstracts that contained the three terms. Relatively few of the documents, however, contained relevant quantitative findings. Reviews by Weber et al. (1982) and by Mertens et al. (1980) turned out to be very useful for finding earlier studies.
Through direct database searching and branching, I located 10 studies with relevant findings (table 4.1). The studies have some things in common. Each covers either a national or state-wide population of young people. Each reports on student achievement as measured by broad tests administered near the end of high school, and each contains a quantitative description of average performance and variation in performance in vocational and nonvocational programs. The studies were not uniform in design, however. They differed in (a) method for identifying vocational students; (b) the groups to which vocational students are compared; and (c) method of analysis.
Identification of vocational students. Alexander et al. (1978) and Echternacht (1975) used student transcripts to identify the curricular programs of students. All other researchers relied on student self-categorizations. Evans and Galloway (1973), Hilton (1971), and Jencks and Brown (1975) used self-categorizations that students made at the beginning of secondary school; Alexander and Cook (1982) and Alexander and McDill (1976) relied on categorizations that students made at the end of high school. Gamoran (1987) and Vanfossen, Jones, and Spade (1987) used self-categorizations made both early and late in high school to place students into tracks. Although some of their analyses involve students who switched tracks between grades 10 and 12, they estimate the size of tracking effects from the test scores of students who stayed in the same track over the period of the study: track-stayers (about 60 percent of the sample) rather than track movers.
Comparison groups. The most valuable studies for our purposes are those that report test scores separately for academic, general, and vocational groups. Echternacht (1975), Evans and Galloway (1973), Gamoran (1987), Hilton (1971), and Vanfossen et al. (1987) conducted studies of this sort. Other researchers reported on academic versus nonacademic groups and did not distinguish between students in vocational and general programs (i.e., Alexander et al., 1978; Alexander & Cook, 1982; Alexander & McDill, 1976; Jencks & Brown, 1975). Trent's (1982) study compares test scores of vocational and nonvocational students and does not distinguish between those in academic and general tracks.
Method of analysis. Echternacht (1975), Evans and Galloway (1973), Hilton (1971), and Trent (1982) carried out simple descriptive analyses of test scores. Alexander and McDill (1976) carried out a regression analysis using cross-sectional data. Alexander and Cook (1982), Alexander et al. (1978), Gamoran (1987), Jencks and Brown (1975), and Vanfossen et al. (1987) carried out regression analyses using longitudinal data. The descriptive analyses present a simple statistical description of test performance by group. The goal of the regression analyses is to determine whether apparent differences among groups are actually program effects. I discuss these differences in research design as I present the findings from the various analyses. It is necessary now to note only that all the results that I present are calculated from statistics presented in the reports. I used standard statistical equations to translate results from each study into a common metric of standard deviation units. I also used normal curve areas to convert the resulting z-scores into percentile scores.
I have divided the studies of curricular effects on student achievement into four main types. The first type of study reports on the performance of academic, general, and vocational students in terms of national norms. The second type of study applies regression analysis to cross-sectional data in order to compare the performance of academic, general, and vocational students who are similar at the end of high school in measured aptitude and in other characteristics. The third type of study applies regression analysis to longitudinal data in order to compare the end-of-school performance of academic, general, and vocational students who were similar in aptitude and background at the beginning of high school. A fourth type of study also uses regression analysis and longitudinal data, but studies of this type compare performance of students in different curricular programs who are similar not only in background characteristics, but who also take a similar number of advanced courses in core high school subjects.
Comparisons with national norms. Weber et al. (1982) wrote an authoritative review of studies examining performance of vocational students on standardized tests. They concluded that the scores of vocational students on standardized tests fall about 0.5 standard deviations below national norms. Students in vocational programs thus fall between the 35th and 40th percentile on standardized tests. Weber and his colleagues also noted that this performance level was typical for vocational students both at the beginning and at the end of the programs.
Table 4.2 is based on results in studies cited by Weber et al. (1982). It is obvious that at the end of high school there is an achievement gap between students in academic and nonacademic programs. Students completing vocational programs score on the average 0.43 standard deviations below the national norms; students completing general programs score 0.42 standard deviations below the norm; and students completing academic programs score 0.57 standard deviations above the norm. Test scores of vocational and general students fall at the 34th percentile; test scores of students completing academic programs fall at the 71st percentile. It is also obvious that there is an achievement gap at the start of high school between students who elect different programs. The percentile score of each group at the beginning of high school is nearly the same as the group's percentile score at the end of high school.
The similarity suggests that students in the three curricular groups grow academically at the same rate. This consistency in rate of growth was an important finding of the Academic Growth Study (Hilton, 1971). Students in the Academic Growth Study took standard achievement tests in grades 5, 7, 9, and 11. Figure 4.1 shows the relationship over time between test scores in mathematics and curricular group membership as determined by self-report in grade 11. The pattern of results is the same for other tests used in the Academic Growth Study. Two points are worth noting about the figure. First, the lines for the academic, general, and vocational groups are nearly parallel during the junior and senior high school years. This means that none of the groups falls behind or gets ahead during the period in which students were taking vocational, academic, and general courses. Second, the lines for general and vocational groups are nearly indistinguishable from the earliest points of measurement. This similarity in pretest scores suggests that students in general programs may be a good comparison group for students in vocational programs. Comparison of academic and vocational groups, on the other hand, are harder to justify on methodological grounds (Grasso & Shea, 1979; Slavin, 1990a; Woods & Haney, 1981).
Figure 4.1 Mean Standardized Scores on STEP Mathematics by Year and Curriculum. (Based on Hilton, 1971)
Although academic, general, and vocational groups show the same growth patterns on academic tests, it is worth noting that the groups do not show parallel growth in all areas of knowledge. Hilton (1971) has provided graphic evidence that rates of growth are very different for academic and vocational students on a test of knowledge of industrial arts (Figure 4.2). Academic and general students hardly increase at all in their knowledge of industrial arts during the middle and high school years. Vocational students, on the other hand, learn a significant amount about industrial arts during the high school years.
Figure 4.2 Mean Standardized Scores on Industrial Arts Scale by Year and Curriculum. (Based on Hilton, 1971)
Regression analyses with cross-sectional data. The evidence from simple descriptive studies is far from conclusive, however. To draw firmer conclusions, we need studies in which researchers measure background, curricular, and outcome variables on the same students. We also need statistical analyses in which researchers are able to make separate estimates of the importance of these factors. One approach that yields such estimates is regression analysis. Table 4.3 presents the results of such analyses along with the results of the simple descriptive analyses that I have already reviewed.
Alexander and McDill (1976) used regression analysis with cross-sectional data to estimate the importance of curriculum when background factors are held constant. Their data came from a survey conducted by Johns Hopkins University researchers in 1964 and 1965. The survey covered 3,700 seniors in 18 public high schools. Alexander and McDill assumed that a number of factors influenced achievement, including background factors (e.g., socioeconomic status, number of siblings, and gender); academic aptitude; peer characteristics (e.g., the academic aptitude, socioeconomic status, and educational expectations of the student's friends); and differences in the schools that the students attended. Alexander and McDill's goal was to find out whether curricular track had an effect over and above the effect of such factors.
They found that their entire set of variables accounted for 48 percent of the variance in mathematics achievement. Academic ability was a major direct determinant of achievement, but track membership was almost as important a factor. Students in the academic track scored 0.80 standard deviation units higher than students of comparable ability and background in nonacademic tracks. The effect is a large one by almost any standard. The result suggests that moving a typical student from a nonacademic to an academic track would raise the student's mathematics test score by 0.80 standard deviations, or from the 34th percentile to the 66th percentile. In other words, nonacademic students would perform at a much higher level if switched to an academic track.
Later and better analyses of survey and test data have not supported the results of Alexander and McDill's study, however. The basic problem with Alexander and McDill's analysis is its use of aptitude data collected concurrently with the outcome data. To measure scholastic aptitude, Alexander and McDill used a 15-item multiple-choice test measuring ability to find logical relationships in patterns of diagrams. The test may have been the best one available to the investigators, but it was not good enough for this kind of analysis. For one thing, the reliability of the test was between .60 and .65, or not very high. For another, the academic aptitude measure correlated about .50 with the outcome measure of mathematics achievement. Subsequent studies have shown that the reliability and validity of this measure are too low for work on track effects. More recent longitudinal studies have used pretest scores to predict achievement outcomes, and the investigators who have used such scores have reported much larger correlations with outcome measures. Jencks and Brown (1975) and Gamoran (1987), for example, reported correlations in the .80s between measures of achievement made at the beginning of high school and those made during later high school years. The moral is clear. Aptitude tests such as those administered in the senior year in Alexander and McDill's study do not adequately reflect the capacities that students have when they enter high school.
It is also difficult to defend logically the use of end-of-program measures of ability as control variables in studies of tracking. A basic problem is that both aptitude and achievement scores change with education. Aptitude scores on tests administered at the end of high school may therefore make good outcome variables, but they cannot serve as proxies for scores on aptitude tests administered at the beginning of high school. Studies that use end-of-school aptitude scores as predictor variables are likely to produce misleading results.
Regression analyses with longitudinal data. Longitudinal designs overcome this basic limitation of cross-sectional studies, and most investigators have therefore used longitudinal data in their regression analyses (table 4.3). Jencks and Brown (1975) carried out one of the first of these longitudinal analyses. They examined data from 91 predominantly white comprehensive high schools throughout the United States that had tested their students for Project Talent in the 9th grade and then had retested them in the 12th grade. Some of the students reported in the 9th grade that they were in academic programs, and others identified themselves as being in nonacademic programs. Jencks and Brown showed that academic and nonacademic students who were initially similar on pretests and in background would also be similar on outcome tests at the end of high school. The academic students averaged only 0.06 standard deviation units higher on achievement tests than did comparable nonacademic students. The effect is a trivial one by almost any standard, and Jencks and Brown concluded therefore that curricular tracks do not have much effect on students' test scores.
Alexander, Cook, and McDill (1978) examined the influence of track placement on scores in Educational Testing Services' Academic Growth Study, and they reported more substantial effects. They classified students into college and noncollege tracks based on their reported course work and self-reported curricular track. Predictor variables in their regression equations were in addition to curricular program, socioeconomic background, gender, race, academic aptitude, and educational plans. Dependent variables were verbal and quantitative scores on the Scholastic Aptitude Test (SAT). Alexander and his colleagues found that students in the academic track scored about 16 points higher than similar nonacademic students on the SAT verbal (about 0.14 standard deviations) and 47 points higher on the SAT quantitative (about 0.36 standard deviations). Differences between academic and nonacademic students therefore averaged 0.25 standard deviations.
Alexander and Cook (1982) reanalyzed the data from the Academic Growth Study, using student self-report data alone to identify a student's curricular track. Predictor variables were similar to those used by Alexander et al. (1978), and dependent variables were test scores in history and English and PSAT-M and PSAT-V scores. Alexander and Cook carried out several regression analyses, and one of these examined effects of curricular track with background and aptitude factors controlled. Alexander and Cook's results were very similar to Alexander et al.'s (1978) results. They found that on the average test, the academic track raised performance by 0.17 standard deviations. >Both Gamoran (1987) and Vanfossen, Jones, and Spade (1987) used HSB data in analyses similar to those of Jencks and Brown (1975) and Alexander et al. (1978). The HSB data set came from a survey of approximately 30,000 high school sophomores and seniors surveyed initially in 1980 and again in a 1982 follow-up. Both Gamoran and Vanfossen and her colleagues estimated effects of curricular tracking on students who stayed in the same high school tracks for the two-year period between sophomore and senior year in high school. Both of the research studies were based on the assumption that factors other than curricular program influenced student achievement: socioeconomic background, race, sex, and educational expectations in the 8th grade; 10th-grade social-psychological variables (friends' plans to go to college, educational expectations) and 10th-grade academic characteristics (grades so far, courses completed in the subject area of the dependent variable). The researchers therefore formed regression equations that allowed them to specify the effects of track membership with these pre-existing characteristics held constant.
Vanfossen et al. (1987) analyzed scores on a composite achievement measure based on tests of vocabulary, reading, and math. Their regression equation predicts that a student who is average on all background factors (z-scores = 0.00) would score 0.15 standard deviations above the population mean on achievement tests if placed in the academic track, 0.08 standard deviations below if placed in the general track, and 0.20 standard deviations below the mean if placed in the vocational track. This implies that on a nationally normed test the student would score at the 56th percentile if placed in the academic track, at the 47th percentile if placed in the general track, and at the 42nd percentile if placed in the vocational track.
Gamoran (1987) analyzed scores on six achievement measures (mathematics, science, reading, vocabulary, writing, and civics). His results were similar to the findings of Vanfossen and her colleagues. According to Gamoran's regression equation, a student who was average on all background factors would score 0.10 standard deviations above the population mean on achievement tests if placed in the academic track, 0.06 standard deviations below if placed in the general track, and 0.13 standard deviations below the mean if placed in the vocational track. On a nationally normed test the student would score at the 54th percentile if placed in the academic track, at the 48th percentile if placed in the general track, and at the 45th percentile if placed in the vocational track.
Regression analyses with course work as a predictor variable. Why do students learn less in the vocational track? There are two factors to consider. First, students in vocational programs are more likely to be in the lower level of core courses that all students take. That is, they are unlikely to be in the elite sections of stratified core courses. Second, students in the vocational track take fewer advanced courses. Compared to academic students, for example, they are less likely to take advanced math courses, advanced science courses, foreign languages, and so on.
Alexander and Cook (1982) and Gamoran (1987) carried out further regression analyses to determine whether the achievement differential for vocational and academic students could be explained by the second factor, the number of advanced courses that students take in core areas (table 4.3). They developed regression equations in which they were able to hold constant students' prior background and subsequent course work while investigating the effects of curricular track alone. The analyses complemented Alexander and Cook's and Gamoran's analyses in which only background variables were held constant.
Alexander and Cook (1982) used data from the Academic Growth Study in their analysis. They found that effects of curricular track were reduced when students were compared who took the same number of advanced courses in an area. In fact, academic, general, and vocational students all performed at the same level when both background factors and advanced courses were held constant. Gamoran (1987) used the HSB data set in his analysis. Gamoran's analysis covered six different outcome tests. He found that on the average achievement test, academic students scored 0.08 standard deviation units higher than comparable general students who had taken the same number of advanced courses (table 4.3). General students scored 0.08 higher than comparable vocational students who had taken the same number of advanced courses.
These analyses suggest that curricular programs produce most, but not quite all of their effects by prescribing different numbers of advanced courses for students. Non-academic students usually take fewer advanced courses in subjects like mathematics and this affects their performance on mathematics tests. If vocational students elected as many advanced courses in mathematics as academic students did, the gap between vocational and academic students would be narrowed.
Test scores of high school students completing academic and vocational programs are clearly different. Academic students usually score at the 71st percentile on standardized achievement tests given at the end of high school (or about 0.56 standard deviations above the mean); vocational students usually score at the 34th percentile (or about 0.41 standard deviations below the mean). The achievement gap at high school graduation is therefore large. The question is, What causes it?
Regression analyses suggest that the most important cause of the achievement gap is student self-selection into academic and vocational programs. If the same students enrolled in the two types of programs, graduates of the two programs would differ very little in test scores at the end of high school. A second factor contributing to the achievement gap is the different number of advanced courses in core subjects taken by academic and vocational students. Academic students take more of these advanced courses. If vocational students were as academically strong as college-prep students at the beginning of high school and they took as many advanced courses in core areas as college-prep students do, their test scores would be nearly indistinguishable from those of college-prep students at the end of high school.
It is possible to quantify these results. The difference in test scores of academic and vocational students on standardized tests at the end of high school is equal to about 1.0 standard deviation. Regression analyses suggest that the gap would be about 0.2 standard deviations if similar students enrolled in academic and vocational programs. Thus, 80 percent of the difference in test scores of academic and vocational students at the end of high school appears to be due to the difference in aptitude of the students who enter the programs. In addition, regression analyses suggest that 10 percent of the achievement gap is due to the different number of advanced courses in core subjects taken by academic and vocational students. If vocational students were similar to academic students in aptitude and took the same number of advanced courses in core subjects, the achievement gap between academic and vocational students would be no more than 0.1 standard deviations. The remaining 10 percent of the gap is due to other curricular and program factors.
Regression analyses, therefore, suggest that moving a student from a vocational to an academic program would raise a students test scores on academic achievement tests. The increase in scores might be as much as 0.2 standard deviations (if the student took a heavy load of advanced courses in mathematics, English, and so on) or as little as 0.1 standard deviations (if the student avoided advanced courses). Two questions naturally arise about these regression results. First, how important are these differences? Second, how trustworthy are the analyses that produced the results.On the question of importance of these differences, two points are worth noting. First, an increase in test scores of 0.1 to 0.2 standard deviations is a trivial to small effect. Cohen (1977) has reviewed the educational and psychological literature on effect sizes. He concluded that an effect of 0.8 standard deviations is large, an effect of 0.5 standard deviations is moderate in size, and an effect of 0.2 standard deviations is small. Moving a student from a vocational to an academic program will have at best a small effect on the students test scores in academic subjects.
In addition, the difference is found on standardized tests in academic subjects and such tests do not measure all the things that students learn in high school. Standardized tests give a lot of weight to skills that are useful for survival in college; they give less weight to skills and knowledge that are useful in jobs and careers. Although academic and vocational students appear to grow at the same rate in academic knowledge, they apparently grow at different rates in job knowledge. Hilton (1971), for example, has provided graphic evidence that vocational students acquire industrial arts knowledge at a quicker rate than academic students do. In addition, specific vocational programs may prepare students very well in specific academic areas. In fact, some vocational programs may outdo academic programs in specific areas. Ramey (1990), for example, has reported that business students increase their verbal skills at a faster rate in business programs than they would in an academic or general program.
A more critical question is whether regression comparisons of vocational and academic students are trustworthy. Slavin (1990a) has argued forcefully that they are not. He believes that such comparisons are untrustworthy because academic and vocational students differ too much in aptitude and in too many other ways at the start of high school. For regression results to be trustworthy in such situations, measures of relevant initial characteristics would have to be both complete and completely reliable. According to Slavin, they never are. It is difficult to know therefore what conclusions to draw from regression comparisons of academic and vocational students. Even if academic and vocational tracks had identical results on students, Slavin has noted, the studies comparing the achievement of academic and vocational students would still show higher achievement for the academic track.
We are on sounder ground with regression comparisons of students in vocational and general tracks. The students in these tracks do not differ greatly in aptitude initially, and regression problems are therefore less severe in comparisons of students in the two tracks. The regression analyses suggest, however, that general and vocational programs have roughly the same effects on student achievement. General and vocational students score at nearly identical levels on standardized achievement tests given both at the beginning and at the end of high school. General students score on the average at the 34th percentile (or about 0.41 standard deviations below the mean); vocational students score on the average at the 32nd percentile (or about 0.47 standard deviations below the mean). Regression analyses suggest that program effects are trivial. If the same students enrolled in general and vocational programs, their test scores would differ by less than 0.1 standard deviation, a trivial amount, at the end of high school.
My overall conclusion therefore is that academic and vocational programs may differ slightly in how well they prepare students in the broad academic skills needed in modern society. Academic programs may provide slightly better academic preparation. Requiring vocational students to pursue a college-preparatory curriculum might raise their sores on tests of academic skills by 0.2 standard deviations, but we cannot be sure. This estimate may be inflated by two methodological artifacts: imperfect reliability in the measurement of predictor variables and incomplete measurement of factors influencing student achievement. General and vocational programs, on the otherhand, seem to have equivalent effects on student achievement. Moving a student from the vocational track to the general track would have no measurable effect on the students overall achievement level.