A question of questions

What mix of question types is optimal for an assessment?

Now and again, teachers are asked to construct ‘set-piece’ assessments, like an end of year test. These assessments are often intended to establish students’ learning of a domain of content - perhaps what they have been taught this year to see how well it has stuck.

We have previously written about the first two decisions to be made when constructing such an assessment: what should be in scope and how the assessment should sample from this domain. These steps help determine what knowledge the assessment will assess. The next question is how to assess this knowledge - what to ask students to do.

Set-piece assessments can take many forms: a written paper, a performance, or a project, for example. We may ask students to answer questions or we may set them tasks to do. The mode of assessment depends largely on the nature of the subject and what knowledge or skill is valued. For example, we wouldn’t assess a student’s ability to dance by setting multiple-choice questions as we wouldn’t be able to make valid inferences about whether students have mastered the thing we want them to master. This sounds obvious, yet schools can mistakenly begin by fixing the mode of assessment (such as end of year written papers taken under controlled conditions) then asking subjects to fall into line. This is often driven by misguided attempts to be robust and rigorous, at the expense of assessments being authentic and meaningful.

But assuming we have more control over the nature of the assessment, what should we consider?

Task openness

In selecting tasks, we need to understand a particular feature of assessment items that affects difficulty: task openness.

A task is considered more open if it allows for a wider range of acceptable responses from the student, whereas a closed question has only one correct answer. This openness-closedness continuum has infinite variations, but we can think of them as falling into one of four broad categories:

A very closed task such as the question, ‘New Delhi is the capital of India: true or false?’

A somewhat closed task could be, ‘State two tools that might be suitable for making a tongue and groove joint’.

A somewhat open task might ask, ‘Why did the Great War come to an end?’

A very open task would be, "Tell me as much as you can about Buddhism” or “Create a piece of art which represents your personality”

Closed tasks include true/false questions, multiple-choice questions, matching tasks, ordering tasks, labelling tasks, short answer questions, or fill in the blanks questions, to name a few. Closed tasks are a tool for precision - for working out whether students have retained specific knowledge. They can be used as part of a random sampling approach to ‘spot check’ the breadth of a student’s knowledge or as part of a cluster sampling method to dig deeper into a specific, critical aspect of understanding.

Open-ended tasks aren’t necessarily longer, broader or more difficult. They are defined by their lack of specificity about what should be in the response. Open tasks reveal not just what students know, but how they think—what knowledge they draw on, how they organise it, and how they apply it in loosely cued situations.

There are four reasons we may choose open tasks:

1. To test knowledge transfer when cues are weak

2. To allow for multiple valid answers

3. To assess synthesis and flexible thinking

4. To invite creative or subjective responses.

For more guidance on why and how to use open-ended tasks, see this previous post.

Balancing task types

The effectiveness of an assessment significantly hinges on the balance of task types used, but it is not always the case that a greater mix of task types will assess a greater mix of knowledge. Certain task types are suited to particular knowledge domains so that the assessment you are writing may concentrate on a single task type or may include a mix to reflect different types of knowledge and depth of understanding required.

Assessments often try to link questions and tasks, either because they share contextual prompts, or simply because the response to one question is used in the response to a later question. We often do this during an assessment to help scaffold students towards answering more complex and demanding questions. Creating questions that are conceptually connected can encourage students to think critically and apply knowledge across different but related topics. This approach tests the depth of their understanding and their ability to synthesize information. Employing thematic links or a gradient of complexity in questions can offer a structured yet challenging assessment experience. However, it is important to consider the potential for compounded errors — if a student misunderstands an early concept, this misunderstanding may negatively impact their performance in subsequent, linked questions.

When students ask: “What’s going to be in the test?”, you might want to consider what you will tell them in advance about the types of questions they will answer. Intuitively, it might sound helpful to tell them what types of tasks they will encounter so that that they can prepare more effectively for the assessment. This transparency may also reduce test anxiety. However, research suggests that student beliefs about the type of test questions they will face affects their revision strategies. In particular, a belief that the test will include multiple-choice and other recognition questions may encourage less intense study. Given that belief about assessment tasks seems to inform the type of test preparation that students complete, it seems prudent to lower expectation about exactly how they will be tested. This means that not only should you not pre-announce question types, but you also should try to mix up the types of tasks used where you regularly assess students in a class.

For example, if you regularly include vocabulary tests in modern language lessons, you might not always want to stick to single word recall assessments. It can help students’ study preparation if they believe you will switch between different short-answer methods, such as:

· Direct two-way translation (e.g. a mix of French-English and English-French translation)

· Aural questions in addition to written ones

· Multiple-choice in addition to open-text short answer questions

· Drawing lines between words

· Cloze passages, asking students to fill-in-the-blanks in a short passage

· Use of key vocabulary words in a sentence creation exercise

· Find the error in a sentence

· Asking for opposites, synonyms or antonyms

· Picture description or picture-word matching exercise

The desirability of question type mix presents a dilemma if you wish to heavily use online assessment platforms since they often use one type of assessment question more than others. When selecting the platform to use, do look at the diversity of assessment types that students will encounter. It is fine to use a platform that has a single question type, provided it is only used alongside other tools.

When to set the same question repeatedly

Although we are mostly looking at the act of creating a one-off assessment in this post, it is worth thinking about how useful it is to set an assessment task more than once. There are good reasons for this: each time you set it, some students will not have responded correctly and so setting it again gives you a chance to review whether they now know the answer. Furthermore, even for those who did respond correctly, some will have forgotten the answer. The act of retrieval within a quiz will in itself improve the chances that the information can be a retrieved successful again in the future.

The online quizzing platform, Carousel Learning, has a framework for helping teachers decide whether or not to include a question already asked in their next assessment. Even if you don’t use their platform, the framework is a useful one. They say it should be informed by three things:

1. Student prior performance on that assessment task, which in turn is likely to be related to strength of the original teaching episode in that knowledge domain and the difficulty of the task itself.

2. The number of times it has been asked in the past.

3. The amount of time elapsed since it was last asked in an assessment since this will inform how useful the retrieval exercise is in strengthening their chances of future retrieval.

The right mix

In this post, we have touched upon the complexity of selecting the right mix of assessment tasks and questions. The key considerations are as follows:

1. What degree of task openness is required?

2. What tasks suit the knowledge domain?

3. Should tasks and questions be stand-alone or linked?

4. What behaviours are we trying to incentivise in students?

5. Should we repeat tasks from previous assessments?

Having considered scope, sampling and task selection, there is one more thing to think about: the order of tasks. We will discuss this in a future post.