If you’ve conducted evidence-use studies in the past, you likely know how tricky it can be to capture in situ references to research, data, and other types of information. More so, even, when the goal is to understand how that evidence may influence decision-making processes. One of our teams at NCRPP is reflecting on ways we recently extended our typical methodologies in order to better interpret the use of evidence in district-level conversations.
Like most researchers, we value time-tested and rigorous qualitative methods. During this study, however, we felt that our data set allowed us—and our research questions implored us—to expand beyond our typical analytic toolbox. We wanted to understand how district actors employed evidence in long-term deliberations, and in what ways that information may have influenced their groups’ ultimate decisions. What’s more, we hoped to accomplish this primarily using extensive observational fieldnotes, rather than participant self-reports. Doing so obliged us to experiment with new techniques during analyses. Here, I share two ways that we pushed the boundaries of our methodological comfort zone: the first we call decision trajectories, and the other mapping.
A brief background on the study
To build deeper understanding of evidence use in internal district deliberations, we conducted two years of observations of school district meetings, amassing over 300 hours of fieldnotes. We were interested in instructional decisions related to the roll out of the Common Core State Standards in mathematics (CCSSm), investigating how evidence was leveraged in these conversations, and what influence it had on decision making over time, if any.
Pulling the common threads from long-term deliberations: Decision trajectories
Imagine one conversation topic that comes up in numerous meetings over the course of several years. Picture, for instance, that you work in a school district and your team is tasked with developing or selecting new mathematics curricula. You would not likely reach a conclusion in one moment or even in one meeting. Rather, evidence for and against particular approaches would accrete over time (Weiss, 1980), with participants influencing each other’s opinions throughout the process. People could intermittently invoke evidence, such as data to identify a problem or research to bolster an opinion. These conversations would be interspersed with others about professional development, or budgeting, or the many other subjects that comprise district meetings.
We needed to pull out that common thread—about new math curricula in our hypothetical—from across the many hours of meetings. To do so, we identified every exchange that implicitly or explicitly attended to the subject of developing or adopting math curricula. We benefited from our extensive data set, but inspecting hundreds of hours of fieldnotes from end to end would be unproductive. Using that approach, a conversation about curricula in one meeting might not appear to link to a conversation on the same topic in a meeting several months later, although the two discussions could actually play off of one another in important ways.
To identify the links across meetings, then, we tagged all portions of dialogue that touched upon the same topics. We identified three important threads of deliberation that occurred throughout our two years of observations, then read through fieldnotes to select any interaction which surrounded one of the topics. Compiled, each set of conversations became its own unit—a concentration of the back-and-forth moments that defined each deliberation over two years. We called these decision trajectories, and they became central to our analyses.
Visualizing the arc of a deliberation over time: Mapping
Another example of experimentation can be seen in our foray into mapping. We wanted an approach that would provide visual representation of the deliberation, offering a new way to view the dynamics at play.
Our mapping process was undergirded by frame theory (Snow, 1988), a theory of social mobilization which posits that individuals frame problems and solutions in ways that convince others to agree with them, often providing reasons to support their frames. After identifying the frames and reasons that comprised each decision trajectory, we wanted to visualize the arc of the deliberation over time—the build up toward an agreed-upon solution (or continued disagreement).
We devised a system in which each frame and reason was represented by a shape (e.g., squares represented problem frames; circles represented solution frames), and could be linked by a line with some significance (i.e., a solid line represented a frame or comment that supported its predecessor; a dashed line meant that the comment was challenging the previous one).
In these maps, we highlighted when data, research, and other forms of information were used, allowing us to see how these forms of evidence influenced the deliberation. For instance, if we noted that there was an invocation of data followed by group agreement around a topic, this could indicate that the data were influential to the group’s decision. We found this innovative approach useful in making meaning of the back-and-forth that occurred in our trajectories over time.
Interested in more?
We found that our analyses, while deeply rooted in tried-and-true qualitative methods, benefitted from some exploration around the margins. We hope the brief insights here prove useful to researchers seeking new ways to analyze evidence use and decision making, and are always interested to hear more about the processes adopted by fellow researchers. If you’d like to discuss—or hear more about our methods for investigating evidence use—we invite you to attend our presentation at the American Educational Research Association conference this month, titled Locating Data Use in the Microprocesses of District-Level Deliberations: A Methodological Approach.