A matter of time
How long should we spend assessing students?
We have all heard it said that we should spend more time fattening the pig, and less time weighing it. It is a phrase which is often used to critique heavy accountability systems in schools. But the advice might equally apply to assessment.
On the one hand, we need to gather information about how well pupils are mastering the curriculum. On the other, if we spend too much time in this endeavour we crowd out the time to teach them anything!
How do we manage this trade off?
The answer is that our assessments must be efficient. Efficiency in relation to assessment measures the relationship between the time and effort expended and the value of the information gained. An efficient assessment is one that helps us learn as much as possible about what the students know and can do, in as little time as possible.
We like this idea of efficiency because it reflects the fact that assessment in schools is never perfect. It must be an act of compromise where we cannot know everything about attainment that we want to know. This is because attainment is such a complex construct, because time is limited, and because we usually need to assess an entire class at one time.
In this post, we will not give you the answer to how much time in total you should spend assessing pupils (there is no right answer to this). Instead, we will look at how you make an individual assessment, like a test, as efficient as possible. This begins by deciding how much time should be committed to this particular assessment.
Length matters
In schools, the length of an assessment is critical but often non-negotiable. Teachers frequently find themselves within the constraints of institutional schedules that dictate how long an assessment can be. Despite these restrictions, it is important to understand the interplay between the duration of an assessment, the breadth of the knowledge domain it covers, and the reliability of the inferences we draw about student attainment.
If we could choose the length of an assessment, we would consider three factors:
The span of the domain we wish to measure
The granularity of the inferences we seek
The level of confidence we desire in our evaluation of attainment.
Put simply, assessing larger domains requires more time, seeking more granular insights requires more time, and achieving high validity conclusions requires more time. In the ideal scenario, where the length of a test can be determined by the teacher, a fundamental principle holds: a test with more items tends to yield more reliable results. This is because, on a short test, each incorrect answer disproportionately impacts the overall score. Conversely, on a more extended test, the effect of a few wrong answers is mitigated by the sheer number of items, allowing for a more nuanced view of a student's capabilities.
However, a lengthy test runs the risk of fatigue, where students, bombarded by repetitive tasks, may lose concentration and thus compromise the accuracy of their responses. We can try to counteract this, for example by including a variety of task types to maintain engagement.
Of course, in a class all students will typically have the same length of time to complete an assessment, so the question of how many assessment items we should have is one of compromise. We want there to be sufficient time for most students to be able to demonstrate what they know and can do. Running out of time should not be the norm.
So, while the length of a test is often beyond our control, the design within that framework is not. A well-structured assessment, mindful of the trade-offs between length and engagement, and crafted with clear instructions, can still offer a reliable measure of student attainment within the constraints we face.
Insights into what is not assessed
When faced with a test, students think that the point is to demonstrate what they know about what is in the test. However, teachers are also interested in what students know about what is not in the test. This is the case not just for tests, but for assessments more generally, including projects, performances, and coursework.
A student’s performance in an assessment is a clue as to how well they have mastered the domain. For this reason, teachers do not preannounce exactly what will be in a test as they want students to study the whole of the content in scope. Teachers will make inferences about whether students have learnt what might have been in the test; they will generalise from the specific.
There is an art and a science in deciding what aspects of the curriculum to test in an assessment and what to leave out. This is known as sampling and we will consider this in a future post. Sampling is about ensuring what is being tested provides information that enables you to generalise about the pupil’s attainment across the whole domain. For now, we will remain focused on the idea of assessment efficiency in relation to question and task selection (or assessment items, as they are known).
It is quite easy to write assessment items that are useful in helping learn what students know. Your subject will have different needs for different types of tasks, from multiple choice questions, through to essays, through to observation tasks. The assessment items you pick will depend on the part of the knowledge domain you are measuring and how you hope the assessment might promote learning. However, the task becomes much more difficult when time constraints are introduced as we must decide which assessment tasks are more helpful than others.
We can take any assessment we have written and ask the assessment items to ‘compete’ for a place in our assessment, by thinking about which give us the most useful information about what students know. Assessment items must meet the efficiency test as well as the assessment as a whole.
Time invested beyond the test itself
We should also think about the time involved in assessments in the broadest sense, not just how much time is available for the assessment to be completed. For teachers, how much time will it take to create the assessment items in the first place, how much administration is associated with running the assessments, and how long will it take them to mark and record attainment?
Assessment weighs more heavily on teacher time than teaching. Teaching a one-hour lesson usually requires far less than an hour to plan and prepare the lesson. Conversely, the relationship is flipped for assessment. To administrate a one-hour test is likely to require far more than one-hour of additional work for the teacher - marking a pile of papers can take many hours. It is therefore essential that we question the value of the information we receive from the assessment and ask ‘is it worth it?’.
Broader notions of efficiency
Throughout this post, we have been guilty of taking a uniquely teacher-perspective on the idea of efficiency. At the start, we defined efficiency as ‘the relationship between the time and effort expended and the value of the information gained’. This definition assumes that the purpose of assessment is to gather information upon which valid inferences of attainment can be made.
But, as we wrote about here, assessment has a much wider purpose: to promote learning. Perhaps we should be defining efficiency more broadly as the relationship between time and effort expended and its impact on learning. This broader concept helps us question the efficiency of assessment for pupils who are really not concerned with our pre-occupation with inference validity.
So, for the pupils, we should reflect on how much time is taken preparing for an assessment in relation to the learning gains they receive. The act of preparing (or revising) for an assessment will benefit learning as the expectation of inference motivates effort. So too will sitting the assessment (due to the testing effect - the benefits gained through retrieval and application of information stored in memory). Pupils may also learn something about where the gaps in their knowledge are and may fill some of these gaps following the feedback they receive from their teacher.
Given all the above benefits, we can conclude that assessment, if done well, is an efficient and effective learning tool. Arguably, it is a more effective learning tool than instruction, although there will be diminishing returns if over-used. We should bear this in mind when considering the burden of assessment on teachers - although assessment may appear inefficient in a narrow sense, from a teacher perspective, we may consider it worth the cost when we examine pupil learning gains.
These matters are not the concern of psychometricians, which is why we cannot leave assessment to the assessment specialists. Teachers must consider not only the technical arguments for assessment efficiency but the pedagogical ones too. Weighing the pig also fattens it in terms of assessment and learning.
There is a complex dynamic playing out when we assess pupils and it requires the insight and expertise of the teacher to ensure we squeeze as much value out of assessment as possible. After all, time is precious.