1.1 Summary of the chapter: objectives and key results
The aim of this chapter is to provide some practical advice on the design and implementation of questionnaires to evaluate teaching and learning in economics. The structure of the chapter is as follows:
- The next section, 1.2, offers some Top Tips: key ideas as to good practice in the administration and analysis of questionnaires and their use in evaluation.
- Section 1.3 discusses in a general way the role and purpose of questionnaires, identifying the particular strengths of questionnaires in comparison to other means of evaluation.
- Section 2 identifies the key stages in implementation of questionnaires from the initial design stage to the process of using questionnaire results to improve teaching practice. Each stage is discussed in turn. This section draws largely although not entirely from generic literature.
- Section 3 reviews the practice of questionnaire implementation in a random sample of (anonymous) economics departments in the UK, in light of the good practice guidelines discussed in section 2. As stated, the aim of the chapter is to provide practical advice that can support departments in the design and refinement of their evaluation procedures. Where there are potentially useful questions or procedures, these are identified and reproduced in detail.
- Section 4 discusses the use of electronic questionnaires, the frequency of questionnaires and issues related to confidentiality of questionnaire responses.
- Section 5 reproduces substantial parts of three questionnaires that have interesting features.
Key ideas and tips on good practice are concisely summarised, sometimes in note form, using bullet points.
Summary results – some thoughts on questionnaires and staff morale
Questionnaires and their use in academic departments are a controversial issue. Questionnaires typically contain ranked questions that are used to measure the perceived quality of specific aspects of a module and its teaching staff. Where the scores are low, this has potential to be extremely damaging to the morale (and possibly to the careers) of staff. In addition, most questionnaires contain ‘open’ questions that allow students some freedom to express their opinions about a module or tutorial programme. In a minority of cases, this is used irresponsibly and lecturers have been subjected to personal abuse. More generally, in their comments, students tend to focus on negative aspects of a module or its staff and do not necessarily evaluate the module according to the appropriate criteria, i.e. the extent to which it supports and facilitates learning.
In the way that we design and particularly in the ways that we use questionnaire results, we need to be aware of these issues. This is discussed fully in the subsequent sections, but a number of key points emerge. First, staff and students need to be clear as to the purpose of questionnaires – questionnaires comprise part of a multifaceted process whose goal is constructively to support teachers in making improvements in teaching and learning, where appropriate. They are not a mechanism for assessing the performance of members of staff, and should not be used in that way.
The practice of comparing scores across staff is totally inappropriate, and it should be made clear to staff that questionnaire results will not be used in this way. As suggested, scores are sensitive to non-appropriate criteria, and have been shown to be highly correlated to factors outside of the control of the teaching staff, such as the type of module, the background, level and year of the students, whether the module is optional or core, and exactly when in the module the questionnaire is implemented.
It is standard practice for students to submit their responses to questionnaires anonymously. It is argued that this approach increases the rate and quality of response. In this chapter, it is suggested that departments might consider relaxing the confidentiality of questionnaires, and oblige or request students to put their name to at least some of their responses. It is argued that anonymity may induce disingenuous responses that ultimately threaten the whole process and the objective of improving the teaching and learning experience. Positive effects of removing anonymity are that students are encouraged to articulate their concerns and ideas in a constructive and open manner, and there is a basis for dialogue and feedback after the questionnaire is submitted.
As stated, the purpose of questionnaires is to improve teaching and learning. To achieve this, teachers should receive some possibly informal training in how to read, interpret and respond to questionnaire responses. This is particularly relevant to inexperienced staff.