Highlights from The Exam Man podcast, Season 2 Episode 13
This episode took a close look at school performance measures and the role of exams in arriving at the scores for individual schools: hugely useful information for anyone working in exams, including Exams Officers, teachers and school leaders, but it’s also key for any parents investigating secondary schools for their children.
So a lot of Exams Officers are also Data Managers, and people might not always understand why those two roles are connected. They are connected because part of the role of the Data Manager is dealing with school performance data, which is all around how students do in the exams. So there is a connection on that level, which explains why a lot of Exams Officers are also Data Managers, but you don't have to be an Exams Officer to be a Data Manager and vice versa. There is a way in which the skills overlap, which might explain it too. So being methodical, being systematic, all that kind of stuff overlaps.
Pre-2016, what was the way that schools were measured overall?
Can I take you back to the 1990s? Everyone's probably heard of school league tables, they’re kind of controversial. But this is the idea that schools are accountable for how their students do in their exams. In the early 90s the idea of school league tables was introduced, and there obviously then had to be some way of measuring how schools were doing. And so what they arrived at was this measure of five A* to C grades, including English and Maths. It was all about getting everyone to get Cs.
What you wanted as a school was to get a high percentage of students who had got five A* to Cs, including English and Maths. That was your big target. If you got a very high percentage of that, if you were up in like around 90%, you were going to be very high up the league tables. And if you were around 40% or lower, then it looked like you were struggling big time as a school. So it was all about trying to get students those five GCSEs.
So John, tell us why it changed. It was 2015, wasn't it?
So in 2015, they piloted, or semi-introduced, the new system. When you start to think about five GCSEs as a measure, it is quite blunt. Obviously, the people who designed did so with good intentions to try and find some way that they felt would be fair to measure schools. But I think over time, it became clear that it wasn't really fair or particularly conducive to a good education to have this as the key measure. And there were a few reasons for that. So firstly, five subjects incentivises you narrowing down the curriculum as you get closer and closer to the GCSEs. So you would end up with lots of students who by the end were just doing English, Maths, maybe a couple of sciences and another subject or one science and two other subjects.
It felt really good as a French teacher, I have to say!
Did you find French getting cut quite a lot? So that was one of the issues - this sort of narrowing down of the curriculum. The second issue is that it's just an attainment measure. So it treats all schools as if they're the same. And obviously anybody who works in the sector knows schools are in very different contexts. To judge them all on exactly the same parameters is a little bit unfair. I think probably the third thing was this issue of the C grade being so important. Trying to get students to a C grade meant that there was a particular focus on a particular group of students, those who were on the C/D borderline to try and get those students up over into the C grade category.
Which had its merits though, didn't it? Because that was probably the majority in most schools.
Yes, that would have captured a lot of students, but it would have meant that students who were struggling more were ignored more than they should have been. This isn't because people don't care, it's just that this is what the system's incentivising.
And the same at the other level as well - at the top.
And also at the top end, yes. Perhaps students are not being pushed far enough, because what's the point in putting loads of energy into a student who might get 8 A* if getting 5 A* is good enough?
So what did they come up with?
They came up with a few measures, a few new ways of measuring schools. And it's interesting to think about which is the most important. Progress 8 was the one that was game changing. What Progress 8 seeks to do is look at schools within their context and to look at not how highly the students are achieving, but to look at how much progress they're making.
It's what you would have expected of them to do since primary.
Exactly. It baselines students when they come out of primary school. So it uses the SATs exams that we talked to Dan Walton-Smith about. It takes those as a baseline. And then it says at year 11, how far have students come? Have they progressed well from when they came into the school? If you think about a school in a more deprived, challenging area, where maybe the students are coming with lower SATs results. On a 5A* to C measure, that school is not going to look very good because it might only get 40%, 50% 5A* to C. But that school might make incredible progress with students. So it might take students who have low SATs results and by year 11 have them at a really decent standard. What Progress 8 seeks to do is to measure that progress.
It really captures that, doesn't it?
I think sometimes people in schools talk about attainment and progress, and it's not that they don't know what they're talking about, but they sort of forget what they're talking about. I once did a training session, and one of the trainee teachers on it came up with a great analogy for it, which I'm going to use now, and I use now in the training, which is to think of it like a ladder. Attainment tells you what rung of the ladder you are on, and a progress measure tells you how many rungs you've climbed. I doubt you are listening, Chris. But thank you so much because it is such a good way of thinking about it! If I tell you that I'm on rung number eight, you'll be able to tell instantly what I can do. It's a great thing to know. But what it doesn't tell me is how far I've come, what sort of journey have I been on, where am I going? It doesn't tell me that I might be stagnant - I might be able to do this thing, but I've been able to do it for years, and I'm not going anywhere. Whereas the progress thing, like the moving up the rungs, this person's like on a journey, they're getting better at this thing. A pure progress measure tells you that a person's climbed three rungs but it doesn't tell you where they are, how good they are. Which is why they brought in multiple measures- so now, you have two big kind of key measures, which are Attainment 8 and Progress 8.
That you get pretty much straight away, don't you?
Attainment 8, you can get once you have the exams results. You can calculate your Attainment 8 straight away and you can tell the world how well your students are attaining.
But Progress 8 is a bit more of a wait for schools, isn't it?
What the 8 means is basically, there are 8 subject areas that you need to cover for a student to essentially fulfil the measure. It broadens the curriculum. So whereas you had the 5 before, you've now got 8. But just one little quirk is that a double weighting is given to English and Maths. Wheras in the old measure, you had to have English and Maths in there, in the new measures, you obviously have to have English and Maths again, but also they get a double weighting to reflect their importance as subjects. Therefore, if you get like a Grade 6 in Maths, that gets counted twice, so it’s a 12.
English is interesting because there's obviously English language and English literature. And the way that works in the Attainment 8 and Progress 8 is you take the better grade out of those two and then double weight it. So there's a big incentive for putting students into both English language and English literature, because that's what gives you the double weighting, and then you can choose the better grades. So English language or English literature, that's the one that gets factored in to the Attainment 8 and Progress 8 figures. So Attainment 8 just looks at a student's achievement across all those eight areas. And then you get an average score for your school. So all the students go into that, you take an average, and that's the Attainment 8 score for your school. It’s a pure attainment measure. It's a bit like the 5A to C, but it's just a bit broader than that.
Progress 8 is a bit different. Progress 8 is slightly more complicated, in that what it does is it looks at your students' SAT scores from primary school, and then it says, these are students who got 100 in their SATs, and then it compares them to other students who got 100 in their SATs across the country. And it looks at how well those students did in their GCSEs to see how much progress they've made relative to their peers across the country.
It's quite a complicated, convoluted system. You can't really calculate your Progress 8 figure yourself. You have to wait until all of that data is in place. And then in October, so a couple of months after you've got the exam results, the DFE then publishes their kind of provisional data. And then in January, you find out what your final Progress 8 score is. You'll get a decent idea from your Attainment 8 and just a feeling of how well your students have done. The difference, I guess, is that Attainment 8 is like this absolute measure, and it tells you absolutely how well your students have done and are doing right now. Progress 8 is like a relative measure. It tells you two things - how far your students have come, but also that journey in relation to all the other students in the country as well who are at the same level. So it's very different. It's more complex.
So the two big measures are now Attainment 8 and Progress 8. But interestingly, one measure that has slightly survived from the old system is the one looking at English and Maths specifically and saying, what percentage of students in your school have got a pass in both English and Maths? And that becomes a really shorthand, quick way for schools to look at their results and get a sense of how well they've done. I know it's the first thing I always look at as a Data Manager. Whenever we get our students predicted grades in or their mock exam grades, you go straight to that figure and you look at English and Maths.
We tend to look at that as the percentage of students who've got a four or above in both, and then the percentage of students who've got a five or above in both, because four is a pass and then five is like a strong pass. So you look at both of those to try and work out what percentage of your students are picking up both those key skills at a decent level.
There's also the EBACC as well. The EBACC is kind of related to Progress 8 and Attainment 8, but what it does is it specifies the subjects that students must do. So it says you have to do English, Maths, science. You've got to do one of the big humanities, history, geography, etc. And you have to do a language as well to get the EBACC.
So schools vary on how committed they are to the EBAC. Some schools are really committed to it, others aren't. You can still get a good Progress 8, Attainment 8, without necessarily committing that hard to the EBAC. But obviously, I think Ofsted probably values the EBAC quite well.
It's got this flexibility within it though, hasn't it? Which is good.
Yes, a bit of flexibility. that's how the system changed really in 2016. what Progress 8 does, more than anything, is take into some account the context of schools. So if you imagine a school in a deprived area, they'll have students who come in with low Key Stage 2 results potentially, a lot of them. It'd be very hard for them to get really high Attainment 8 scores. But what they can demonstrate is that they've made a lot of progress with their students. So when they look at other students across the country who had similar SAT scores, they can say, our students did a lot better, they made a lot more progress. That allows schools which have lower attainment to say, actually, we're doing a really good job, because you can't have the same expectations of us in terms of our attainment.
But you can see quickly what interventions they're doing with the students that they have.
You can see they're making a difference to their students, which I think is a good thing. And Progress 8 gets quite a lot of criticism.
What are the main criticisms?
Firstly, is that thing of it being a relative measure. As a school there's no objective standard where you can say, if we do X, Y and Z and we get to this level, then we'll get a good Progress 8 score, because your Progress 8 score is dependent on how all the other schools in the country do as well. There's no objective standard that you can work towards for Progress 8. It's always going to be dependent upon how other schools also are doing. So I guess that's one thing. The other thing is that the way Progress 8 is structured as a measure has some problems. Zero or above is where schools want to be. You don't want to have a minus progress score.
People talk a lot about positive P8.
It creates this weird jeopardy around zero. So a school that has a Progress 8 score of minus 0.01, and one that has a Progress 8 score of plus 0.01 would be virtually indistinguishable in terms of the amount of progress they're making with students.
It becomes a focus, doesn't it?
It feels very different. it's that feeling that you're behind the curve, or in front of it. that's to do with the design of the measure, I think. It could have been designed better so that it didn't create this kind of weird jeopardy.
It's almost like the positive/minus in the measure makes anything sound really negative if it's got a negative mark in it.
I think another thing is probably to do with the reliability of the Key Stage 2 data - so whether or not the SAT scores are giving you a reliable starting point for the students. We know that primary schools are incentivised for their students to do very well in their SATs, and it's not unusual in secondaries to find students who have very good SAT scores, but then when you actually start to work with them, you find that maybe they're not quite at that level. The Progress 8 measure is based on the student's starting point, which is their SAT scores, and if those SAT scores aren't entirely reliable, then that again can distort whether or not the Progress 8 is actually reliable as well.
I think the other thing is whether or not it's caught on. I think that within the education sector, Progress 8 is used, and it's considered carefully in terms of thinking about how schools are doing, how school leaders are doing. But I don't know how much traction it has with the public as a measure.
Yeah, because it is quite complex. And unless you work within education in some way, you probably aren't as familiar. It's trickier to explain quickly what that means.
I also don't think it's just because of the complexity though. I think that there might be something that maybe people would be less willing to acknowledge, which is that a lot of parents are more concerned about the attainment levels at schools than the progress levels. They just want to know, is this a good school that gets good results? Not is this a school that makes excellent progress with its students compared to the rest of the country?
One of the things about progress, is that it's actually easier to make progress with students who come in at a lower level, because you've got more room to progress into. Whereas if you've got students who have come in at a very high level, the room for progress is actually less, because there's limitations, if you like, because the measures set these limitations. You can only make a certain amount of progress with a student who's come in with the very, very best SAT scores.So in some ways, it's easier to make progress with students who have much lower attainment at primary school. But that in itself might be an indicator of a school where the attainment is lower.
But then I guess parents have that attainment measure to use.
Yes, absolutely. But I guess if we just talk about Progress 8 and its effectiveness, and the extent to which it's caught on, I'm not sure that there'd be many members of the public who would be able to tell you what Progress 8 is.
No, and it's good for people outside of the education sector to realise that this is the measure that schools are judged on, not just the attainment measure. So John, tomorrow, you're in charge of performance measures for the country. What would it look like?
Do you know what? I'm going to make a confession here, which is I actually think Progress 8 is quite good. It gets a lot of flak, but if you compare it to what we had before, I think it's a massive, massive improvement.
Would you add anything on, though? Obviously, we're not talking about all the ways that schools are measured right now. We're just talking about them in relation to exams. But is there another measure that you think you could add on that would either bridge that gap between people in education who understand this and the wider public who need to understand how schools are and how they perform?
One of the big problems with Progress 8, and this gets commented on quite a lot, is that if you think about the SAT scores, those are in English and Maths. We know from where we've talked about this, that it's quite narrow what students are ultimately assessed on in primary school - it's their competence in English and Maths. Part of the problem with Progress 8 is it's looking at a broad curriculum. So you're looking at subjects potentially like music and drama and art, and design technology and PE and all these sorts of things. But you're using the students’ performance in English and Maths to determine the amount of progress that students have made in those subjects. I think that that is a real weakness. I don't necessarily think that whether a student is good at English or Maths can really tell you that much about whether or not they're going to be good at art. It might tell you a bit, but probably not a huge amount.
But because of the way the measure is structured, what we do is set targets for students in art and music, based on how they did in English and Maths at primary school, because we're trying to maximize their progress score. If I were to slightly re-engineer the system, I think I would look to move some subjects towards a different measure. Try to keep that breadth, try to keep it compulsory, but maybe looking at different ways in which you could judge performance in subjects that aren't as clearly linked to English and Maths as others, so that you could capture students' abilities a bit better. But I don't know exactly what that would look like.
One thing I just wanted to do really quickly was to illustrate the different effects that Progress 8 and Attainment 8 have, and how looking at things in terms of attainment and in terms of progress can give you some interesting results. So I'm not going to name any names here, but I'm just going to talk about the local authority that I work in.
I'm just going to look at the year 2022, which happens to be a year that the school I work in did rather well.
Is that why you use it every year?!
Which is why I use it all the time. I work at a school which has another school just a hundred yards up the road. So we are sort of friends and competitors at the same time. It's quite an interesting relationship. I think it's just a historical thing, but we have slightly different intakes. our intake tends to be more disadvantaged than this other school.
Typically, our students will come with lower SATS scores than at this other school. And in 2022 the other school had a better attainment score than we did, but our progress score was massively better than their progress score. It's quite interesting, but it just illustrates how these measures work together. So you need to look at both of them. You look at Attainment 8 and Progress 8, because if you just looked at our attainment score, you'd say, oh, that school is not doing as well as the one up the road. But then if you just looked at the progress scores, you'd say, that school is actually doing way better than the one up the road. you have to look at them together to get a real sense and a real picture of how the school is performing with students in their exams.
Before we finish, John, let's have a chat about some of the top performing schools in the country, just in terms of their measures.
Can we talk about schools that have the best progress? Because I think if we get into attainment schools, some of the answers to that will be quite obvious. It'll be some of the top private schools and things like that. But if we talk about progress, it's quite interesting, when you look at the list of the top 10 schools nationally. I've got 2022 and 2023 here- a lot of the names on these lists appear on both years, so there are clearly some schools who are doing very, very well at Progress 8.
The school that I taught at in London is going to be on there, isn't it? Where Marina works, who did our first Exams Officer interview.
Yes, Sacred Heart School in London typically gets a very, very high Progress 8 score. It's often in the top 10 of schools nationally for Progress 8. Some other famous schools that you might recognise- Michaela Community School in 2022 and in 2023 was the top performing school for Progress 8. It had a Progress 8 of +2.27. Any Progress 8 score above 2 is crazy.
I mean, incredibly unusual, isn't it?
Another school that appears regularly is the Steiner Academy in Hereford- in both 2022 and 2023 it had a Progress 8 score above 2, +2.
There is a bunch of schools performing very, very well on this measure. I thought it would be interesting to look at what those schools have in common and to try and understand why it might be that certain schools are doing well in terms of making very high levels of progress with their students. I actually did a bit of research into this and I found some common factors that link the schools that do very well on Progress 8.
The first thing is that they're disproportionately in London. London's had a lot of funding pushed into its schools. A lot of schools radically improved between about 2000 and 2010. I'm sure there are other reasons why London performs well for that measure, but funding would probably be one of those. They are also disproportionately single-sex schools, in particular, girls' schools do very, very well on Progress 8.They are disproportionately schools with smaller cohorts, so less than 150 students in year 11.
That makes sense, though, doesn't it?
That does. And when you look at a free school like Michaela, that's probably a good one of the reasons why it performs so well. It's a smaller school.
Sorry to interject, but probably a lot of that is just that with smaller cohorts, that ability to really track progress over time, really track data, is much easier with smaller numbers, isn't it?
What I see sometimes is we're having to make tough decisions about, okay, this group of students really needs the extra focus rather than this group. It's about resources and how you're going to deploy them. So maybe if you've got slightly smaller cohorts.
A small number of large multi-academy trusts also feature disproportionately. What the data doesn't show is that being in a multi-academy trust makes you more likely to have a good Progress 8 score. It doesn't show that. But if you're in one of certain multi-academy trusts, you are more likely. Now, I'm not going to name any names here, but let's just say that some of them are the big ones that you may have heard of. Those are disproportionately more likely to feature in the top Progress 8 scores.
The final point here that I think is perhaps the most interesting is that the schools who are the top performing Progress 8 schools disproportionately have a religious ethos. When you look down the list, you'll see that very, very clearly- it's not hard to spot. A lot of them are faith schools of one denomination or another. What's quite interesting about this is that I would even go a bit further than that. I would say that these are all schools that have a strong ethos of some kind. It might not necessarily be a religious ethos, but it has a strong culture. If you think about like Michaela Community School or the Steiner Academy, these are both schools that have a very particular philosophy. They have very clear parameters about how they want their students to conduct themselves, how they want to run their lessons, what they want their curriculum to be like.
Schools with very strong cultures on the ground implement this from Year 7 straight away as well, rather than just closer to the exam years: it's embedded. And obviously Progress 8 is a measure that's formed across the years.
If we were to leave it with one potential thing that you could say about the schools that make the most progress with their students, I think that maybe it’s this notion of a consistent ethos. It doesn't necessarily have to be a particular ethos, but it's well thought out, it's consistent and it's delivered consistently to the students. Potentially this is something which seems to have quite a big, big impact. It has its downsides, obviously- a strong ethos can also be a rigid ethos. Any ethos that doesn’t leave any room for flexibility can feel hard line. But this is data you cannot deny. I think it's quite obvious the schools that have this very strong culture do get very good progress in terms of their students’ exam results.
To listen to and read more interviews like this, subscribe to The Exam Man - the #1 podcast about exams and assessment
To find out more about Examscreen please contact us: Examscreen