REAP Study on Question Types (Summer 2006)
REAP Study on Question Types
|Contributors||Maxine Eskenazi, Alan Juffs, Michael Heilman, Lois Wilson|
|Study Start Date||May, 2006|
|Study End Date||July, 2006|
|Learnlab Courses||English Language Institute Reading 4 (ESL LearnLab)|
|Number of Students||42|
|Total Participant Hours (est.)||195|
|Data in Datashop||no|
This study, described in further detail by Heilman and Eskenazi (2008), examined the differences between four types of assessment question types for measuring knowledge of individual vocabulary items. The four types of questions were cloze, synonym, sentence production, and self-assessments. Twenty words were tested using all question types on pre-, post-, and retention-tests. Ten of the words (the same set for all students) were practiced during an eight week period between pre- and post-tests.
Self-assessment: The act of a learner informing the tutor about the degree to which a particular knowledge component is known. In this study, the REAP tutor asked students for each word X, "Do you know the word X? Yes/No". A finer grained self-assessment question with multiple levels would probably be preferable.
Cloze Question: A fill-in-the blank question. A target word is replaced with a blank in a sentence containing that word. The student must select the target word from a list of distractors.
What relationships exist between the four types of vocabulary assessment types?
Assessment performance on each question type on pre-, post-, and retention-tests.
Question type, Whether a word was practiced or unpracticed.
It was found that self-assessments are accurate when the learner claims that a word is NOT known. Self-assessments are not accurate, however, when the learner claims that a word IS known. The accuracy of self-assessments was measured by performance on cloze exercises given after a pre-test but prior to practicing the word for the first time. Learners claiming that a word was unknown INCORRECTLY answered a subsequent exercise on that word over 90% of the time (5% chance of guessing). Learners claiming that a word was known, however, CORRECTLY answered a subsequent exercise about 30-40% of the time. Learners appear overconfident in terms of their vocabulary, and their self-assessments should only be used to determine a list of unknown words or when asking if further practice on a word is necessary.
Further analysis of the relationships between question types is pending.
In this study, students were given practice exercises (cloze and sentence production) before readings rather than after readings as in previous studies. This is actually not preferable according to teachers and students.
Heilman, M., Eskenazi, M. (2008). Self-Assessment in Vocabulary Tutoring. Ninth International Conference on Intelligent Tutoring Systems. (poster)