This is such a good explanation of why assessment quality matters in all assessments where we make inferences that impact learning. It all goes back to the purpose of the assessment. There are naturally some assessments that focus more on measuring learning at the end of a programme of study. Ultimately, whilst these are used for accountability and certification, the end goal for a student is always learning and measurement of it. I remember being incredibly frustrated as a 15 year old maths student because my teacher would not explain why a formula mattered. She told me I just needed to learn it for the exam. I told her I didn't agree. The whole point of being at school was to learn and apply. As a learner, I became frustrated and disengaged. Definitely not all about the exam!
Thought it was just me thinking this, as far back as my NQT, when I had to mark 7x 30ish books and tests every fortnight. They told me the same things, yet I had to create more, tailored questions on green paper for students to ‘retry’ the things they couldn’t do… at the time I thought it made more sense to re-teach them and correct misunderstandings but I was so scared of failing my NQT I didn’t question it. I’m glad things moved on, and so did I.
Your views seem similar to Michael Scriven's original introduction of F and S. His distinction between them was about timing. Formative is before the end, Summative is at the end.
1. National exams obviously can't be used formatively, can they? Or have I missed something? This is important because they way we use/consume these exams informs how we think about replicating their structures. That is to say, a GCSe past paper can be used for formative purposes but it's unlikely to work well even if the person runing the assessment really understands the limitations.
2. Possibly a more useful distinction is to talk about assessment being mastery or discriminatory. SATs, GCSEs, A levels are all used to discriminate between pupils' performance but isn't just a feature of the inferences drawn from these assessments; their stucture and form actually works against them beign used to improve instruction. A mastery assessment needs to designed to allow all students to answer all items correctly with the aim that we are then able to draw inferences about how will we have taught: if they have got a question wrong it would suggest it wasn't taught well (or was poorly written, but that's a different issue.) But, in order for assessments to be used like this it isn't just a matter of whether we intend to use them formatively or summatively, it's a basic question of how test items are written.
I agree that the summative/formative distinction is an important area for exploration, for exactly the reasons you've stated—the way that we will now be able to assess student work using AI models naturally changes the scope of what it's possible to achieve with assessment. We're still talking in human-scale, when we will have '1000 ECTs' helping every teacher in the background...
This in turn leads to a world where there is an assessment 'stream', which is itself a.) of a consistent robustness, b.) has the detail to steer for granular support for students and intervention for teachers, whilst c.) probably providing a better metric of progress than so-called summative tests do as it stands.
Pragmatically speaking, I think the terms summative and formative still hold meaning—one of the more important distinctions in practice is the 'high stakes-ness' of the assessment environment and the effect that has on the reliability/validity of the result. You mentioned Sparks maths as a sort of longitudinal data aggregator, but some kids will complete their homework using a textbook, google window, and a calculator, whereas some will do it on the bus using none of these. Neither a reflection of absolute mastery nor a relative representation of their place in the cohort!
Looking forward to seeing how educators respond to the new world of high volumes *and* high quality of assessment data as our use of AI models continues to improve.
This is such a good explanation of why assessment quality matters in all assessments where we make inferences that impact learning. It all goes back to the purpose of the assessment. There are naturally some assessments that focus more on measuring learning at the end of a programme of study. Ultimately, whilst these are used for accountability and certification, the end goal for a student is always learning and measurement of it. I remember being incredibly frustrated as a 15 year old maths student because my teacher would not explain why a formula mattered. She told me I just needed to learn it for the exam. I told her I didn't agree. The whole point of being at school was to learn and apply. As a learner, I became frustrated and disengaged. Definitely not all about the exam!
Thought it was just me thinking this, as far back as my NQT, when I had to mark 7x 30ish books and tests every fortnight. They told me the same things, yet I had to create more, tailored questions on green paper for students to ‘retry’ the things they couldn’t do… at the time I thought it made more sense to re-teach them and correct misunderstandings but I was so scared of failing my NQT I didn’t question it. I’m glad things moved on, and so did I.
Your views seem similar to Michael Scriven's original introduction of F and S. His distinction between them was about timing. Formative is before the end, Summative is at the end.
Two points:
1. National exams obviously can't be used formatively, can they? Or have I missed something? This is important because they way we use/consume these exams informs how we think about replicating their structures. That is to say, a GCSe past paper can be used for formative purposes but it's unlikely to work well even if the person runing the assessment really understands the limitations.
2. Possibly a more useful distinction is to talk about assessment being mastery or discriminatory. SATs, GCSEs, A levels are all used to discriminate between pupils' performance but isn't just a feature of the inferences drawn from these assessments; their stucture and form actually works against them beign used to improve instruction. A mastery assessment needs to designed to allow all students to answer all items correctly with the aim that we are then able to draw inferences about how will we have taught: if they have got a question wrong it would suggest it wasn't taught well (or was poorly written, but that's a different issue.) But, in order for assessments to be used like this it isn't just a matter of whether we intend to use them formatively or summatively, it's a basic question of how test items are written.
What do you think?
Thanks for this, both.
I agree that the summative/formative distinction is an important area for exploration, for exactly the reasons you've stated—the way that we will now be able to assess student work using AI models naturally changes the scope of what it's possible to achieve with assessment. We're still talking in human-scale, when we will have '1000 ECTs' helping every teacher in the background...
This in turn leads to a world where there is an assessment 'stream', which is itself a.) of a consistent robustness, b.) has the detail to steer for granular support for students and intervention for teachers, whilst c.) probably providing a better metric of progress than so-called summative tests do as it stands.
Pragmatically speaking, I think the terms summative and formative still hold meaning—one of the more important distinctions in practice is the 'high stakes-ness' of the assessment environment and the effect that has on the reliability/validity of the result. You mentioned Sparks maths as a sort of longitudinal data aggregator, but some kids will complete their homework using a textbook, google window, and a calculator, whereas some will do it on the bus using none of these. Neither a reflection of absolute mastery nor a relative representation of their place in the cohort!
Looking forward to seeing how educators respond to the new world of high volumes *and* high quality of assessment data as our use of AI models continues to improve.