The scope of an assessment
The hidden logic behind what should go in and what should be left out
One of the things schools try to find out is how much pupils have recently learnt within a particular subject - perhaps over the last few months, or a year. This often requires a big ‘set piece’ assessment which covers a substantial domain of content.
On the face of it, this is relatively straightforward: identify the content to be tested, write some tasks and questions, put the pupils in a room and tell them to get on with it. However, it is not quite so simple. In this post, we discussed the challenges faced in subjects where there is a fractured notion of attainment i.e., mastery of one aspect of the subject tells us little about competency in other areas. Fractured subjects make it difficult to decide what should be in scope for an assessment. The assessment must sample widely from across the subject and often multiple question types are required to make valid inferences about whether a pupil is ‘good’ at the subject.
In this post, we will explore other challenges in making decisions about what to include in such assessments - what is in scope.
What’s in, what’s out?
There are four reasons that decisions about what content should be in scope are difficult.
First, a curriculum domain contains all kinds of implicit judgements about the relative importance of different elements in relation to curricular aims. For example, consider the distinction in a subject like history between the core knowledge required and the ‘hinterland’ which comprises the context and richness of historical stories (the vehicle by which the core knowledge arrives).1 It is only within the assessment that we are forced to make all these judgements explicit.
Second, learning in most subjects, most of the time, builds on what has been learnt before, therefore we may want to assess earlier learning to check it is still secure and to help us understand why students may be struggling with the more recent content. Writing an assessment to test pupils’ knowledge of what they have been taught this year or this term, therefore, is not possible without testing some of what was taught before. For example, a GCSE-level understanding of the role greenhouse gases play in climate change will depend on pupils grasping a simplistic model of the greenhouse effect learnt in KS3 or even earlier. An assessment may first test understanding of the simple model before going on to test more sophisticated understanding.
Third, some knowledge will be more important than other knowledge in terms of students learning what comes next; for example, Newton’s laws of motion as a basis for the study of mechanics. Teachers may want to focus on this knowledge to ensure it is secure.
When deciding what content is essential to have in scope for an assessment, your subject is likely to have its own language for describing critical ideas and concepts. You might call them something like ‘hinge concepts’ – crucial points or ideas in a learning sequence that enable the progression from one stage of thinking to another. Or you might call them ‘threshold concepts’ – core ideas that, once understood, transform a student’s perception of a subject. Whether you like to think about these crucial parts of the curriculum as ‘hinges’ in a door or as a portal that the student passes through, it should be obvious that you will attend closely to how they are included in the scope of your assessment.
Fourth, the assessment will signal to students what being ‘good at’ the subject means. The content of the assessment should reflect the content of the curriculum and the nature of the knowledge which is most valued in the subject or the disciplinary tradition from which it derives. Defining a domain with clear boundaries within which the relative value of knowledge is clear is not a simple task.
The Year 8 end-of-year test
Imagine we are writing an end-of-year assessment for Year 8 which will provide information about students’ attainment in our subject. Now, suppose you are a Year 8 maths teacher who hopes that their students have learnt, understood, and remembered all maths in the curriculum from age 5 to 13. However, you care more about some deficits in their knowledge than others, and this will affect your choice of what to assess. You will likely feel very differently about a Year 8 student that cannot remember how to manipulate Roman Numerals than you will about a Year 8 student that cannot remember how to manipulate fractions!
In other subjects, earlier parts of the curriculum are not intended to be memorised and constantly reassessed through further study, but instead their appearance may ebb and flow. For example, students may not have revisited the idea of a sonnet as a form of poetry in Year 8, having learnt this in Year 7 (or before). However, in Year 9 they will study a Shakespeare play and the teacher may wish to check students’ understanding of the term.
Now, suppose you are an art teacher deciding whether the art history that was studied during Year 8 should appear in the end-of-year exam. Whether or not it does will depend on a variety of judgements about what knowledge is valuable, what signals you want to send about what it means to be ‘good at’ Art, and the purpose of art history within the curriculum. If your primary aim is to help students create brilliant pieces of art, you may prioritise this over assessing their cultural capital, which may be useful for them in their future lives, but not critical in terms of developing students’ artistic talent.
The above examples illustrate the point that value judgements matter when it comes to deciding what is in scope for an assessment. Scope decisions are underpinned by customs, traditions, and beliefs about what the subject’s purpose is. What we include is at least in part dependent on what we care about.
But it is impossible to make value judgements about what should be in an assessment without having a view on knowledge architecture and learning progression in your subject. The nature of knowledge architecture means that in some subjects we intentionally want to keep checking what is remembered from years before. For other subjects, this may be less important, and for some subjects such as maths it is simply impossible to avoid re-assessing past learning since it is integral to the curriculum.
One size does not fit all
If we include knowledge learnt prior to Year 8 as ‘in scope’, we must be careful later when we interpret assessment data. For example, we should avoid making claims about how well students have mastered Year 8 material if a significant proportion of the assessment required them to draw on prior learning. Your school may work on the assumption that a Year 8 end-of-year assessment is testing only Year 8 material, or it may have a policy that subjects should sample earlier taught content. It makes no sense to adopt generic policies such as these as the needs of each subject will differ. Either way, being clear about the scope of the assessment is important for ensuring later inferences are valid.
In or out? These are not decisions to be taken lightly. We should first ask ourselves:
What is the purpose of this subject?
What knowledge is most valued?
What does it mean to get better at this subject?
What knowledge do I expect to have been retained?
Christine Counsell says the core is “like a residue – the things that stay, the things that can be captured as proposition”, whereas the hinterland is the rich curricular content that is present in the delivery but may not be discernible in long-term memory. What is interesting about Counsell’s concepts of core and hinterland is that assessment often prioritises the propositional over the hinterland, but Counsell argues that it is no less important. Ref: the dignity of the thing | Christine Counsell's blog