Types of Tasks and Items
Allowing students to demonstrate the wide range of knowledge and skills detailed in the NAEP Technology and Engineering Literacy Assessment targets will require a departure from the typical assessment designs used in other NAEP content areas. Thus students will be asked to perform a variety of actions using a diverse set of tools in the process of solving problems and meeting goals within rich, complex scenarios that reflect realistic situations. Consequently, this assessment will rely primarily on scenario-based assessment sets that test students through their interaction with multimedia tasks that include conventional item types, such as selected response items, and also monitor student actions as they manipulate components of the systems and models that are presented as part of the task.
The following sections describe in detail the scenario-based assessment sets and the sets of discrete, conventional items that will be developed for the NAEP Technology and Engineering Literacy Assessment. Note that the examples of items shown are drawn from what is available at the time of developing this framework (spring 2010). It is expected that, as the use of such innovative items expands and technology advances, these examples will become dated, but they still illustrate the principle components that are important in this assessment.
The assessment will be administered to a nationally representative sample of students to report on student achievement at the group level. The assessment is not designed to measure the performance of any individual student or school. To obtain reliable estimates across the population that is tested, a large pool of assessment items will be developed. That pool of items will be too large to give to any individual student, so subsets of items will be selected to administer to each individual student. The NAEP Technology and Engineering Literacy Assessment will be given in approximately 50 minutes, with additional time for background questionnaire completion. The assessment sets that will be developed for the NAEP Technology and Engineering Literacy Assessment are described below.
Scenario-Based Assessment Sets
There will be two types of scenario-based assessment sets, one long and one short. The long scenarios will take students approximately 25 minutes. The short scenarios will take students about 12 to 15 minutes to respond. The two types of scenarios have common characteristics, but they differ in complexity and in the number of embedded assessment tasks and items to which a student is asked to respond. Long scenarios will provide about 10 to 15 measures of performance, and the short scenarios will capture approximately 5-10 measures of performance. Measures will include innovative measures of a student's interaction with aspects of the scenario as well as conventional selected response items and short constructed-response items. The different measures are discussed later in this chapter.
Discrete Item Sets
One of the challenges for this assessment is that the use of scenario-based assessment sets reduces the number of independent measures in the assessment as a whole. Because of their capability to replicate authentic situations examinees may encounter in their lives, scenarios have the potential to provide a level of authenticity other types of assessment tasks cannot provide which, in turn, may contribute to the validity of the entire assessment. At the same time, however, the choice to use these complex tasks reduces the number of measures that can be included in any one test and causes many of the measures to be interdependent because they are related to the same scenario. To counteract this interdependency and ensure reliability, the NAEP assessment of technology and engineering literacy will also include sets of discrete items that produce independent measures. Discrete item sets will include conventional selected response items and short constructed response items.