Effects of Task Goal On Learning to Use an Instructional Computer Simulation
cognitive tasks, such as learning to use a computer simulation, can be
difficult to analyze because of the number of factors that can affect an
individual's learning and performance outcomes (e.g., previous education,
prior computer experience, motivation, simulation usability and learnability).
Prietula, Feltovich and Marchak (2000) recently proposed a new conceptual
model to facilitate the analysis of this class of complex cognitive tasks.
Their model is adapted from one originally developed by Jenkins (1979)
to characterize the kinds of variables that influence laboratory studies of
human memory and guide researchers in planning research and interpreting the
results. Prietula et al.
suggest that four categories of task factors may influence the analysis of
complex cognitive tasks: task
goals, knowledge characteristics, task materials, and strategies. Task goals are defined as what the user is trying to
accomplish in the task. Knowledge
characteristics are defined as the particular kind of knowledge demanded
by the task and depend on the user's background and experience.
Task materials are defined as characteristics of the materials
the user has to work with, including the characteristics of the problem (e.g.,
well-structured or ill-structured) and its structure, environmental
constraints, and available tools (e.g., the interface design).
Finally, strategies are defined as the goal-directed activities
that the user applies to solve the problem (e.g., allocation of attention,
pattern recognition, knowledge access, and interaction with task materials).
The model has been used to explain a variety of aspects of cognition (Hoffman,
Feltovich & Ford, 1997; Honek & Temple, 1992) and serves as the
conceptual framework for the research reported here.
In this paper, we describe the results of research evaluating the
interaction of two of the model's components, task materials (computer
displays) and task goals.
In previous research, Effken and colleagues developed several complex computer simulations of hemodynamics using design principles derived from ecological psychology’s theory of perception and action (Effken, 1993; Effken, Kim, Kadar & Shaw, 1992; Effken, Kim & Shaw, 1994, 1997). Subsequent empirical tests showed that the etiological potentials display (EPD) consistently improved the speed and accuracy of novice and expert clinicians’ problem identification and treatment when compared with their performance using an integrated balloon display (IBD) or a traditional strip-chart display (TSD). In a recent study, Effken & Doyle (in press) investigated the effects of cognitive style on nursing students' performance outcomes with each display. As in the previous studies, variation was observed in both learning and subsequent performance with the displays across participants and experiments that could not be explained by gender, age, prior education, experience or cognitive style. For example, qualitative differences in performance were seen when the experimenter-defined goal varied across different experiments.
As any teacher or researcher will attest, not all learners readily adopt the instructor's goal, nor do all experiment participants follow the specific directions of the experimenter. Even when they adopt the same overall goal, individual learners may intentionally or unintentionally adopt very different sub-goals, or ways of fulfilling their intentions (Biggs, 1987; Howie, 1996). In previous studies, we sometimes asked participants to solve the problems as quickly as they could, while using as little drug as possible. Novices tended to try to optimize speed, at the expense of drug usage. Only experts were able to respond to both goals simultaneously. In those experiments, the differences were simply treated as "noise" or variability in the data. The purpose of the present research was to investigate systematically how task goal interacts with interface design to affect student nurses' abilities to learn to use the a computer simulation and their subsequent problem-solving performance using the simulation.
hypotheses. The general question to be answered in this study was: To
what extent are differences in students’ abilities to learn to use an
instructional computer simulation presented via two different interface and
their subsequent problem-solving performance with the simulation moderated by
treatment goals? To assess
performance, we measured diagnostic accuracy and treatment speed,
operationalized as time to initiate treatment, time to reach target range,
percentage of time in target range, number of scenarios corrected, and number
of drugs used. Based on our
previous results, we generated the following hypotheses:
Diagnostic accuracy and treatment speed of participants using EPD will
exceed that of participants using IBD.
Experimenter-defined goals will affect performance with the displays,
particularly in terms of time to initiate treatment and number of drugs used,
but performance differences will be mediated by display type.
Sixteen undergraduate students
(15 females and 1 male) enrolled at the University of Arizona College of
Nursing were recruited to participate in the experiment.
Participants were required to have normal or corrected-to normal visual
acuity (self reported) and normal color vision (tested by asking participants
to point to red and green areas of the display).
Participants were randomly assigned to one of two display groups, IBD
This assignment resulted in the IBD group including 1 pre-nursing
student, 5 first year and 2 second year nursing
students and the EPD group including 2 pre-nursing students, 3 first year and
3 second year nursing students.
presented the simulations on a WinNT workstation with a 17” color monitor. Guyton’s (1980) computer model of hemodynamics, which shows the
intrinsic constraints on pressure and flow, serves as the basis for the
simulations. The model is simple, but includes the components thought to be most
critical by clinicians (preload, afterload, and contractility). In addition, the physiological relationships in the model have been
well validated clinically by Guyton and his colleagues. To create a scenario (“illness”), the experimenter changes the
values of the control parameters in the Guyton model (e.g., contractility,
resistance and fluid volume). The simulations depict four output variables of the Guyton model
(cardiac output, and arterial, venous and right atrial pressure) in two
different visual formats, an integrated balloon display (IBD) and an
etiological potentials display (EPD).
interface uses two windows. The
simulated patient parameters are shown in the main window and ‘drug’
options in another window. Drugs
are the same for both displays. Drugs
act directly on the etiological factors (resistance, heart strength and
volume). No drugs affect more than one factor. Participants select a
desired drug and dose (high, medium low), then press a mouse button to give
discrete ‘drug’ doses.
In IBD, mean arterial pressure, right atrial pressure and central venous pressure are represented by three balloons that expand and contract horizontally (Figure 1) as pressures increase or decrease. Increased or decreased cardiac output is shown by changes in the vertical dimension of a "bellows-like" object. Small bar graphs superimposed over the balloons and the bellows show normal and high and low danger areas for each. The student's objective is to administer drugs until all pressures and cardiac output are within the normal range. The lower part of the main window contains a reflected bar graph showing the overall status of the system (computed as the absolute value of the distances of the three pressures and cardiac output from normal). When participants select a drug and a dose, the name of the drug and the dose are shown on the "bottle" at the upper left of the screen. For a more detailed description, see Effken et al, 1997.
EPD (Figure 2), the three pressures and cardiac output are indicated by the
corners of a four-sided object (box). The
box moves through a space defined by two large red bars representing
resistance (the horizontal dimension) and contractility (the vertical
dimension). The central crossing
point for the two bars represents the optimal value for each.
A shrinking or expanding "box" shows fluid volume changes.
Very small bar graphs over the corners of the box indicate target
ranges for each pressure and cardiac output. Green areas represent the target
area; high and low danger areas are black.
When all pressure and cardiac output values are normal, the box
approximates a square of a particular size positioned at the intersection of
the red bars.
The computer records values for the three pressures and cardiac output,
heart strength, resistance and volume, as well as the amount and strength of
each drug used and the number of key presses made, at 1-second intervals.
type was controlled between subjects and task goal, scenario and trial within
subjects in the 2 (display type) x 3 (goal) x 3 (scenario) x 2 (trial) mixed
experimental design. Two displays were compared, IBD and EPD. Three goals were assigned by the experimenter: 1) correct the problems
as quickly as possible (speed), 2) correct the problems with as little drug
therapy as possible (drug), and 3) correct the problems (correct). Students
were presented with three test scenarios developed previously with the help of
clinical experts to simulate common hemodynamic problems (hypertension, heart
failure and hypovolemia). Each scenario was presented twice in each goal condition, in fully
randomized order. There were three sets of experimental trials, one for each
goal. The first set of trials was always with the instruction simply to
correct the problems. The order of the second and third sets of trials (speed and drug goals)
After reading and signing a disclaimer, participants were given written instructions that explained the purpose of the experiment, gave a brief description of hemodynamics, and described the display. The experimenter then initiated the display and demonstrated the kinds of changes the model can undergo. After reading a description of the drugs, participants practiced using them (with the display in a normal state) until they stated they felt comfortable using these “tools.” Participants then were given two practice problems that were repeated until each had been solved twice. Participants were tested individually, in the presence of the experimenter.
analysis. Means and
standard deviations were computed to summarize group scores for each of the
dependent variables. Separate
mixed design ANOVAs were used to compare the effect of goal, display type,
scenario, and trial on: 1) time to initiate treatment, 2) percentage of
problems solved, 3) number of drugs used, and 4) percentage of time the
patient system was maintained within target range.
Percentage of problems solved. The results are summarized in Table 1. Overall, participants solved 97% of the problems they viewed with EPD, but only 73% of those viewed with IBD. In addition, there was much more variability among the students using IBD than those using EPD (Mean SDs = .36 and .05, respectively). Students solved 83% of the problems in the baseline "correct problems" condition, 86% in the "as fast as possible" condition, and 88% in the "minimize drug" condition. A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial) ANOVA found only a significant main effect of display, F1 = 14.32, p < .01.
Time to initiate treatment. Mean
treatment initiation time differed little across the two display groups (M
= 7.9 s for IBD and M =
7.3 s for EPD). In the "as fast as possible" goal condition (M
= 6.81 s), participants started treatment 1 s (IBD) to 2 s (EPD) earlier than
in the baseline "correct" condition (M = 8.26 s).
In the "as little drug as possible" condition, students
initiated treatment about the same time or even later (M = 8.38 s) than
in the baseline condition. A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial)
ANOVA revealed only a main effect of goal, F2 = 10.67, p
= .00. When the
"correct" condition was eliminated from the ANOVA, an interaction of
display type and goal emerged, F1 = 5.05, p < .05.
A post hoc Tukey test showed that in the "speed" condition,
students using EPD were significantly faster at initiating treatment than
students using IBD. In the
"drug" condition, the two display groups did not differ.
Percentage of time the system was maintained in target range. Students using IBD maintained the patient in target range considerably less time (M = 46%) than students using EPD (M = 80%). In the baseline "correct" goal condition, students maintained the patient in target range 57% of the time, while in the "as fast as possible condition" and "minimize drugs" conditions they maintained the patient's parameters in target range 68% and 64% of the time, respectively. Because the baseline condition was always presented first, the difference between that condition and the test conditions, which were presented in counterbalanced order, is most likely a learning effect. Students were faster, on average, on the second trial with each scenario, further evidence of a learning effect. A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial) ANOVA revealed significant main effects of display, F1 = 30.31, p = .00; goal, F2 = 7.5, p< .01; and trial, F2 = 6.47, p < .05.
Number of drugs used. Students using IBD used more drugs to correct problems (M
= 3.31) than students using EPD (M = 1.77). In the baseline "correct problem" condition,
students used more drugs (M = 3.01) than in either the "as fast as
possible" (M = 2.39) or "minimize drugs" (M =
2.23) conditions. The difference in the baseline and test conditions is again
most likely a learning effect. In
the "drug" condition, IBD users decreased their drug use from 3.88
in the baseline condition to 2.92 while EPD users went from 2.13 (baseline) to
1.54. Both groups also reduced
their drug use in the "speed" condition, but not by as much.
A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial) ANOVA revealed
significant main effects of display type, F1 = 12.34, p
< .01; goal, F2 = 9.83, p = .001; and scenario, F2
= 5.30, p < .05. The
latter is expected because different amounts of drugs are required for solving
each problem (scenario).
The major effect of experimenter-imposed goals to maximize speed or
minimize drug use was on how quickly treatment was begun.
In the "speed" condition, participants started treatment 1-2
seconds earlier, whereas in the "drug" condition, they initiated
treatment about the same time or even later than in the baseline
"correct" condition. When only the "speed" and "
drug" conditions were considered, the ANOVA revealed a significant
interaction of display type with goal.
The lack of difference in drug use in the two experimenter-assigned
goal conditions is somewhat surprising. It may be that the students in this
experiment had already internalized a goal of using few drugs, either on the
basis of their previous education or as a constraint imposed by both displays.
Another possible reason for the lack of effect of goal on other
performance measures is the fact that learning to use IBD requires
considerable practice. Students
using IBD solved only 73% of the problems and varied widely in their
performance levels (SD = .38). Although
students had to meet a performance criterion (solving two problems with the
display) before beginning the experimental trials, this may not have ensured
that they had enough skill to vary their performance to meet specific goals.
In contrast, EPD requires very little practice to learn; as a result,
students using EPD solved over 97% of the problems with little variability (SD
= .05). Because they had
already developed a degree of comfort with the display, students using EPD had
sufficient skill to be able to optimize different goals.
A second potential reason for the lack of effect on number of drugs
used is that the two goal conditions are not truly independent.
That is, minimizing drug use in these problems can also improve the
speed of solution. Each problem
can be solved most efficiently by using only one drug.
As a result, by adopting a goal of using fewer drugs, students could
also improve their speed.
Conclusion and Implications
Our previous studies showed effects of a number of user characteristics
(gender, level of expertise, and cognitive style) on students' learning to use
a computer situation. However,
that research also showed variability in performance that was unexplained by
gender, level of expertise, age or cognitive style.
The goal of this study was to investigate explicitly one possible
source of that variability: task goal.
We investigated the
effects of task goal on students' abilities to learn to use a computer
simulation. The simulation presented fundamental principles of hemodynamics in
two different visual formats: IBD and EPD.
Students used a mouse to select and administer simulated
"drugs" to treat simulated clinical hemodynamic problems.
Our manipulation of
goal was at the level of sub-goal. The
overall goal was always to solve each problem.
The experimenter-defined sub-goals simply constrained how the overall
goal was to be attained, that is, as quickly as possible, or with as little
drug therapy as possible.
did not affect the number of problems solved, but did affect the strategies
students used to solve them. When
asked to optimize speed, students initiated treatment earlier; when asked to
limit drugs, they tended to start treatment later and used fewer drugs
(although not significantly fewer than in the "correct the problem"
condition). The results also
showed that display made a difference on how well students could meet the
experimenter's goals. Because
students learned to use EPD so easily, their performance in the
"correct" condition already was very high.
In the first set of 2-minute trials, they quickly (M = 42 s)
corrected 96% of the problems and used an average of 2.13 drugs (an optimal
solution requires one drug). Because
of this, these students had little margin for improvement (a ceiling effect).
Still, the students took only about half as long (M
= 24 s) to solve the problems in the speed condition. In contrast, some of the students using IBD were still
struggling to master the display throughout the experiment, as shown by their
solving only 73% of the "speed" and 77% of the "drug"
trials. Because the students were
still learning, they lacked the flexibility to respond to additional
experimenter goal constraints. Even
so, IBD users cut their time for solving problems from 87 s in the control
condition to 63 s in the speed condition and to 73 s in the drug condition.
The difference between the speed and drug conditions represents an 11%
change for IBD users (based on their solution time in the control condition)
in contrast to a 33% variation shown by the EPD users.
Using the Cognitive Task Model as a framework, we investigated the
interaction of task materials (interface design) and task goals, while
controlling knowledge characteristics and found an effect on the fourth factor
in the model (strategies). Task materials (specifically, the computer interface design)
affected all performance variables. This
is consistent with our previous research, as well as with findings in studies
involving a variety of complex cognitive tasks (Hoffrage & Gigerenzer,
1996; Larkin & Simon, 1987; Payne, Bettman & Johnson, 1988; Roy &
Lerch, 1996; Zhang, 1997; Zhang & Norman, 1994).
Task goal affected how quickly treatment was initiated. This is
consistent with research reported by others (e.g., Baron & Galizio, 1983;
Pilgrim & Johnston, 1988) showing that even subtle orienting instructions
can cause substantial performance differences.
Speed was also moderated by interface design.
two goal conditions used were not completely independent.
To the extent that the drug used is the drug needed to solve the
problem (a minimizing drug use solution), the problem is likely to be solved
faster because additional unneeded drugs simply create additional disturbances
that then have to be corrected.
We looked at the effect of goal in one computerized simulation task
with two interfaces only. In
addition, our sample size was relatively small and was restricted to novices.
Thus we cannot generalize the results beyond the conditions of this
Many human-computer interaction models have been proposed (e.g., Card, Moran, & Newell, 1983; Clarke, 1986; Jacgodzinski & Clarke, 1988). We found the Model of Cognitive Tasks (Preitula et al., 2000) provided a useful, generic framework for organizing the analysis of data in a complex cognitive task, learning to use a computer simulation. The model has some similarity to the Nurse-Computer Interaction Framework (Staggers & Parks, 1995), but does not include context and is not a developmental model. Because it includes only four categories, it has intuitive appeal for the researcher looking for a simple model. However, each of those categories requires subcategories to operationalize it, so the model is not as simple as it initially seems. It is likely that the model can provide a framework for a variety of informatics-related research. In any applied research study there are many degrees of freedom to be controlled if analysis is to be possible. It seems likely that any research dealing with complex computer activities will have to take into account the relevant degrees of freedom pertaining to each factor in the model: task materials, strategies, knowledge characteristics, and goals.
This research was supported by a grant to the first author from the Emmons Foundation at the University of Arizona College of Nursing and a dissemination grant from Beta Mu Chapter, Sigma Theta Tau.