- Assessment Purposes and Types
In 2009, the National Assessment Governing Board (NAGB) adopted a multifaceted program of research aimed at providing validity evidence for reporting preparedness on 12th grade NAEP. The program of research was formulated and recommended by a panel representing a variety of technical expertise related to preparedness for postsecondary education and job training. Judgmental standard-setting studies were recommended as one of several methods to identify points on the NAEP scale to reference preparedness for college credit-bearing, entry-level courses of the type that would fulfill general education requirements or preparedness for placement in a job training program for one of several occupational areas to be specified for this research (NAGB, 2009). In 2011, twelve operational standard setting studies will be implemented for the 2009 grade 12 NAEP in reading and mathematics. In this symposium, the steps in this innovative set of judgmental standard-setting studies will be traced and some preliminary results will be presented.
The first two presentations will provide the context for NAGB's 12th grade preparedness research studies. First, the role of these studies in NAGB’s 12th Grade NAEP Preparedness Research program will be presented. Second, information on the meaning of preparedness in the NAEP context will be presented and compared with other definitions currently under consideration.
The third presentation will describe the process for developing performance standards used for the judgmental standard setting and highlight key procedural results. To develop preparedness performance standards, panelists will be assembled online for webinars with the goal of drafting performance standards that they will subsequently finalize during the onsite standard setting panel meetings.
The fourth presentation will focus on the judgmental standard setting process to set the cut scores using the bookmark method. Several computer-based innovations of the method will be implemented in six judgmental standard setting workshops. A demonstration of some elements of the Computer Aided Bookmark (CAB) is included in the symposium. Preliminary results will also be presented.