FROM THE SECURE ROOM: Exams and school performance

[00:00:15] You ready?

[00:00:16] Yeah, let's go.

[00:00:19] So, Saturday night and I had a couple of glasses of wine because...

[00:00:24] Can't say that.

[00:00:26] Well, sorry, I have.

[00:00:27] It's true, it's true.

[00:00:28] Because this evening we're talking about data.

[00:00:31] You often say when you do this presentation on measuring performance of schools as part of your data role in your job that people love it.

[00:00:42] So, we thought it is actually quite an interesting topic when it comes to exams.

[00:00:48] So, tonight we thought we're going to, you know, just tell you a little bit more about the purpose of exams,

[00:00:53] not just in terms of obviously assessing students at the end of their time at school, but also as a measure for school performance.

[00:01:03] Yeah, absolutely.

[00:01:04] And I think this is really, really interesting for obviously people who work in exams.

[00:01:10] Teachers will know about this school, leaders, it's really key for...

[00:01:14] And it's also really interesting for parents who will have seen Progress 8, for example, as a measure being floating around when you're talking about which school to send your child to, for example.

[00:01:26] Yeah.

[00:01:26] But actually how many people really understand it?

[00:01:29] Yeah, yeah, yeah.

[00:01:29] I'm not sure anyone understands that.

[00:01:31] I mean, I'm not sure I understand Progress 8.

[00:01:33] I'm not sure what you do because you should do because I'm going to ask you lots of questions about it in a minute.

[00:01:36] Yeah.

[00:01:37] But I think it is one of those things you've said before, haven't you, that you've had feedback from some teachers when you've done this training.

[00:01:42] Yeah.

[00:01:42] And they've said that they actually feel a bit embarrassed to ask for someone to explain because you feel like you should know.

[00:01:49] And I think, you know, if you work in any way in education, you feel like you should understand this.

[00:01:53] Definitely.

[00:01:53] So, yeah, I obviously know a lot about this because I work, I recruit school leaders in my job.

[00:02:01] So you've got to explain Progress 8.

[00:02:03] Well, I'm handing over to you.

[00:02:04] I'm just going to ask you lots of questions, John, to explain these measures.

[00:02:10] Okay, yeah.

[00:02:10] So first thing I just would say is that I think that the idea that people love the training that I do on this might be a little bit strong.

[00:02:18] But having said that, I think it's right.

[00:02:21] They haven't fallen asleep.

[00:02:22] Like, I think what's true is that no one's ever fallen asleep.

[00:02:26] No, no one's ever fallen asleep.

[00:02:27] No, no.

[00:02:29] But I definitely think what's true is that, like, there are a lot of performance measures now and some of them are quite complex.

[00:02:36] And so I do think that sometimes people are a bit embarrassed to say I didn't really get what that was about or what it was measuring.

[00:02:44] So, yeah, I think that people like having this explained is like it is a good, good thing for people just generally working in schools.

[00:02:55] But also the other thing I think is that a lot of exams officers are also data managers and sort of like people might not always understand why those two roles are connected.

[00:03:06] Why are they?

[00:03:07] But there is quite a...

[00:03:08] Why are they connected?

[00:03:08] Well, they are connected because part of the role of the data manager is dealing with this school performance data, which is all around how students do in the exams.

[00:03:18] So there is a sort of like a connection on that level.

[00:03:22] I mean, you don't have to be an exams officer to be a data manager and vice versa.

[00:03:26] But there is a kind of connection between the two things at some level, which sort of explains why a lot of exams officers are also data managers.

[00:03:37] I mean, I think there's also a kind of, you know, there's sort of a way in which the skills kind of overlap as well, which might explain it too.

[00:03:48] So being methodical, being systematic, all that kind of stuff overlaps as well.

[00:03:54] But yeah, I'm very excited to talk about this topic.

[00:03:56] I enjoy it a lot.

[00:03:58] I know you do.

[00:03:59] So pre-2016, what was the way that schools were measured overall?

[00:04:07] Yeah.

[00:04:08] Okay.

[00:04:08] So, well, can I take you back to the 1990s?

[00:04:11] Oh, God.

[00:04:11] How long have we got?

[00:04:12] This is supposed to be a very quick episode.

[00:04:14] But just to say that everyone's probably heard of school league tables.

[00:04:19] Kind of controversial.

[00:04:20] But this is the idea that like schools are accountable for how their students basically do in their exams.

[00:04:28] And so in the early 90s, the idea of school league tables was introduced.

[00:04:35] And there obviously then has to be some kind of way of measuring how schools are doing.

[00:04:41] And so what arrived at was this measure of five A star to C grades, including English and maths.

[00:04:49] And the including English and maths is very important.

[00:04:52] And it was all about getting everyone to get C's, wasn't it?

[00:04:56] Yeah, basically.

[00:04:56] So what you wanted as a school was to get a high percentage of students who had got five A star to C's, including English and maths.

[00:05:07] And so that was your big target.

[00:05:09] And if you got a very high percentage of that, you know, if you were up in like around 90%, you were going to be very high up the league tables.

[00:05:16] And if you were around sort of 40% or lower, then you were, you know, you look like you were struggling big time as a school.

[00:05:26] So it was all about trying to get students those five GCSEs.

[00:05:32] So, John, tell us why it changed.

[00:05:35] Okay.

[00:05:36] 2015, wasn't it?

[00:05:37] So 2015, they piloted or kind of semi-introduced the new system.

[00:05:44] But I mean, there was some pretty, when you start to think about five A star to C as a measure, like it is quite blunt.

[00:05:50] Obviously, you know, the people who designed it, designed it with good intentions to, you know, try and find some way of that they felt would be fair to measure schools.

[00:06:00] But I think over time, it became clear that it wasn't really fair or particularly conducive to a good education to have this as the key measure.

[00:06:10] And a few reasons for that.

[00:06:12] So firstly, five subjects.

[00:06:14] So obviously, that incentivizes you narrowing down the curriculum as you get closer and closer to the GCSEs.

[00:06:21] So you would end up with lots of students who by the end were just doing English maths, maybe a couple of sciences and another subject or one science and two other subjects.

[00:06:30] It felt really good as a French teacher, I have to say.

[00:06:33] Did you find French getting cut quite a lot?

[00:06:34] Very much wasn't that.

[00:06:35] Yeah, exactly.

[00:06:37] So that was one of the issues was this sort of narrowing down of the curriculum.

[00:06:41] The second issue is that like it's a very, it's just an attainment measure.

[00:06:47] So it sort of treats all schools as if they're the same.

[00:06:49] And obviously, we know anybody who works in the sector knows the schools are in very different contexts.

[00:06:55] So to judge them all on exactly the same parameters is a little bit unfair.

[00:07:00] As you say, it's a blunt instrument.

[00:07:02] It's very blunt.

[00:07:02] Yeah.

[00:07:03] And then I think probably the third thing was that this issue of the C grade, the C grade being so important, trying to get students to a C grade meant that there was a particular focus on a particular group of students.

[00:07:16] So those who were kind of on the CD borderline to try and get those students up over into the C grade category.

[00:07:24] Which had its merits though, didn't it?

[00:07:26] Because that was probably the majority in most schools.

[00:07:30] Yeah, that probably is.

[00:07:32] That would have captured a lot of students.

[00:07:34] But it would have meant that students who were struggling more were ignored more than probably they should have been.

[00:07:41] And by the way, this isn't because people are like, you know, not they don't care or they're not whatever.

[00:07:46] It's just that this is what the system's incentivizing.

[00:07:49] And so obviously people.

[00:07:50] And the same at the other level as well at the top.

[00:07:53] And also at the top end.

[00:07:54] Yeah.

[00:07:54] Perhaps students not being pushed far enough because what's the point in putting loads of energy into a student?

[00:08:00] Who, you know.

[00:08:02] Particularly in schools where resources are more scarce.

[00:08:04] Who might get like eight A stars if getting five A stars is good enough.

[00:08:08] Do you know what I mean?

[00:08:08] So it's, yeah.

[00:08:11] So what did they come up with?

[00:08:13] Well, they came up with.

[00:08:14] I have to say I found it really complicated at first.

[00:08:17] Yeah.

[00:08:17] Yeah.

[00:08:17] So they came up with a few measures, actually, a few new ways of measuring schools.

[00:08:23] And it's interesting to think about which is the most important.

[00:08:27] You know, it's kind of like.

[00:08:29] Progress 8.

[00:08:30] Progress 8 was the one that was kind of game changing.

[00:08:34] In the.

[00:08:35] What Progress 8 seeks to do is look at schools within a kind of their context.

[00:08:43] And to look at not how highly the students are achieving.

[00:08:48] But to look at how much progress they're making.

[00:08:52] So.

[00:08:52] It's what you would have expected of them to do since primary.

[00:08:55] Exactly.

[00:08:55] So it baselines students, basically, when they come out of primary school.

[00:08:59] So it uses the SATS exams that we talked to Dan Walton-Smith about.

[00:09:03] It uses the.

[00:09:04] That we keep referencing.

[00:09:05] We keep talking about Dan.

[00:09:07] Literally every episode.

[00:09:07] Thanks, Dan.

[00:09:08] Yeah.

[00:09:09] So it takes those as a baseline.

[00:09:12] And then it says at year 11, how far have students come?

[00:09:16] Yeah.

[00:09:16] Like have they progressed well from those where they came into the school?

[00:09:20] So obviously then if you think about, you know, a school in like a more deprived, challenging area.

[00:09:27] Where maybe the students are coming with lower SATS results.

[00:09:30] On a 5A to C measure.

[00:09:32] Those students, that school is not going to look very good.

[00:09:35] Because it might only get 40, 50%.

[00:09:38] 5A star to C.

[00:09:40] But that school might make incredible progress with students.

[00:09:44] So it might take students who have really like low SATS results.

[00:09:48] And by year 11 have them, you know, at a really decent standard.

[00:09:53] And so what Progress 8 seeks to do is to measure that progress.

[00:09:56] It really captures that, doesn't it?

[00:09:59] Yeah.

[00:09:59] Wow.

[00:09:59] So I think sometimes people in schools talk about attainment and progress.

[00:10:03] And it's not that they don't know what they're talking about.

[00:10:05] But like they sort of forget what they're talking about.

[00:10:08] And I once did this training session.

[00:10:10] And one of the trainee teachers who was on it came up with a great analogy for it.

[00:10:16] Which I'm going to use now.

[00:10:17] And I use now in the training.

[00:10:18] Which is to think of it as like a ladder.

[00:10:21] And that attainment tells you what rung of the ladder you are on.

[00:10:27] And a progress measure tells you how many rungs you've climbed.

[00:10:31] Nice.

[00:10:31] Okay.

[00:10:32] So, yeah, it's nice.

[00:10:33] Well done, Chris.

[00:10:34] I was going to say who it is.

[00:10:35] I doubt you are listening, Chris.

[00:10:37] But thank you so much.

[00:10:39] Because it is such a good way of thinking about it.

[00:10:41] So, and obviously you can see the limitations of both.

[00:10:44] So if I tell you that I'm on rung number eight, you can tell, you'll be able to tell instantly like what I can do.

[00:10:51] So if that person is at that level, you can say, right, that person is able to do.

[00:10:56] Which is a good measure to have.

[00:10:57] It's a great thing to know.

[00:10:58] Great thing to know.

[00:10:59] But what it doesn't tell me is how far I've come.

[00:11:02] Like what sort of journey have I been on?

[00:11:04] Where am I going?

[00:11:05] It doesn't say like this person, it doesn't tell me I might be stagnant.

[00:11:09] Like I might be able to do this thing.

[00:11:10] But like I've been able to do it for years and I'm not going anywhere.

[00:11:14] You've had no benefit from being in that current school.

[00:11:17] Whereas the progress thing, like the moving up the rungs.

[00:11:20] This person's like on a journey.

[00:11:23] They're getting better at this thing.

[00:11:33] Yeah.

[00:11:35] Three rungs, but it doesn't tell you where they are, like how good they are.

[00:11:39] It just tells you the number of rungs.

[00:11:40] Which is why they brought in multiple measures.

[00:11:43] Yeah.

[00:11:44] So now you have two big kind of marquee measures, I think, which are attainment eight.

[00:11:50] That you get pretty straight away, don't you?

[00:11:52] So attainment eight you can get once the exam results, you get them, you can calculate your attainment eight straight away and you can tell the world how well your students are attainment.

[00:12:02] But progress eight is a bit more of a weight for schools, isn't it?

[00:12:06] Yeah.

[00:12:06] So what the eight means is basically there are kind of like eight subject areas that you need to cover for a student to essentially fulfil the measure.

[00:12:21] So that probably doesn't make it.

[00:12:23] So it broadens it out, doesn't it?

[00:12:24] It broadens the curriculum.

[00:12:25] So whereas you had the five before, you've now got eight.

[00:12:28] But just one little quirk is that a double weighting is given to English and maths.

[00:12:34] So whereas in the old measure, you had to have English and maths in there.

[00:12:38] In the new measures, you obviously have to have English and maths again, but also they get a double weighting to reflect their importance as subjects.

[00:12:48] So if you get like a grade six in maths, that gets counted twice.

[00:12:54] Effectively, it's like a 12, you know, because it gets counted twice.

[00:12:59] English is interesting because there's obviously English language and English literature.

[00:13:04] And the way that works in the attainment eight and progress eight is you take the best grade out of those two and then double weight it.

[00:13:11] So there's a big incentive for putting students into both English language and English literature, because that's what gives you the double weighting.

[00:13:18] And then obviously you can choose the better grades.

[00:13:21] So English language or English literature, that's the one that gets factored into the attainment eight and progress eight figure.

[00:13:28] So attainment eight just looks across, looks at a student's achievement across all those eight areas.

[00:13:36] And then you get an average score for your school.

[00:13:39] So all the students go into that.

[00:13:41] You take an average and that's the attainment eight score for your school.

[00:13:45] So that's a pure attainment measure.

[00:13:46] So it's a bit like the five A to C, but it's just a bit broader than that.

[00:13:51] And then progress eight is a bit different and progress eight is slightly more complicated in that.

[00:13:57] What it does is it looks at your students sat scores from primary school.

[00:14:04] And then it says, right, these are students who got like a hundred in their sats.

[00:14:09] OK, and then it compares it to other students who got a hundred in their sats across the country.

[00:14:15] And it looks at how well then those students did in their GCSEs to see how much progress they've made relative to their peers across the country.

[00:14:24] It's quite a complicated, convoluted system.

[00:14:26] And actually, you can't really calculate your progress eight figure yourself.

[00:14:31] You have to wait until all of that data is in place.

[00:14:35] And then in October, so a couple of months after you've got the exam results, the DFE then publishes their kind of provisional data.

[00:14:45] And in January, isn't it?

[00:14:46] And in January, you find out like what the final your final progress eight score is.

[00:14:51] You'll get a decent idea from your attainment eight and just a feeling of how well your students have done.

[00:14:57] So, yeah, so the difference, I guess, is that attainment eight is like this absolute measure.

[00:15:03] And it tells you absolutely like how well your students have done.

[00:15:08] And doing right now.

[00:15:09] Yeah.

[00:15:11] And progress eight is kind of like a relative measure.

[00:15:14] So it says it tells you two things.

[00:15:16] How far your students have come, like the journey that they've been on,

[00:15:20] but also that journey in relation to all the other students in the country as well who are at the same level.

[00:15:27] So it's, yeah, it's very different.

[00:15:30] It's a more measured measure.

[00:15:30] It's very different.

[00:15:31] It's more complex.

[00:15:32] Yeah.

[00:15:32] Definitely way more complex.

[00:15:34] It's a complex beast, isn't it?

[00:15:35] Yes, it is.

[00:15:37] It is.

[00:15:37] So, yeah, so I think the two big measures are now attainment eight and progress eight.

[00:15:42] But interestingly, one measure that sort of has kind of slightly survived from the old system is kind of looking at English and maths specifically and saying, right,

[00:15:56] what percentage of students in your school have got a pass in both English and maths?

[00:16:01] And that becomes a really shorthand, quick way, I think, for schools to look at their results and sort of get a sense of how well they've done.

[00:16:08] I know it's the first thing I always look at as a data manager whenever we get our students predicted grades in or their mock exam grades.

[00:16:17] You go straight to that figure and you look at, OK, what's our English and maths?

[00:16:21] Yeah.

[00:16:22] And so we tend to look at that as like the percentage of students who've got a four or above in both and then the percentage of students who've got five or above in both.

[00:16:31] Because four is like a pass and then five is like a strong pass.

[00:16:34] So you kind of look at both of those to try and work out, you know, how what percentage your students are picking up both those key skills at a decent level.

[00:16:46] So that's another measure.

[00:16:48] There's also the EBAC as well.

[00:16:49] The EBAC is kind of related to Progress eight and Attainment eight.

[00:16:53] But what it does is it sort of specifies the subjects that students must do.

[00:16:57] So it says you have to do English, maths, science.

[00:17:00] You've got to do one of the big humanities, history, geography, et cetera.

[00:17:04] And you have to do a language as well to get the EBAC.

[00:17:08] So schools vary on how committed they are to the EBAC.

[00:17:11] Like some schools are really committed to it.

[00:17:13] Others aren't.

[00:17:14] You can still get a good Progress eight, Attainment eight without necessarily committing that hard to the EBAC.

[00:17:21] But obviously, I think Ofsted probably values the EBAC quite well.

[00:17:23] It's got this flexibility within it though, isn't it?

[00:17:25] Which is good.

[00:17:26] Yeah, a bit of flexibility.

[00:17:27] Yeah.

[00:17:28] So yeah, so that's how the system changed really in 2016.

[00:17:31] And what Progress eight does more than anything, I think, is sort of takes...

[00:17:38] It doesn't do this like amazingly in the best way possible.

[00:17:42] But what it kind of does is take into account the context of schools.

[00:17:46] So if you imagine a school in a deprived area, they'll have students who come in with like low Key Stage two results, potentially a lot of them.

[00:17:59] And it'd be very hard for them to get a really high Attainment eight score.

[00:18:03] But what they can demonstrate is that they've made a lot of progress with their students.

[00:18:07] So when they look at other students across the country who had similar SAT schools, they can say, our students did a lot better.

[00:18:15] They made a lot more progress.

[00:18:17] And what that does then is allow schools who have lower attainment to say, actually, we're doing a really good job.

[00:18:27] Because, you know, you can't have the same expectations of us in terms of our attainment as you would have of the school and the leafy...

[00:18:33] But you can see quickly what interventions they're doing with the students that they have.

[00:18:38] You can see they're making a difference to their students.

[00:18:40] Yeah, definitely.

[00:18:41] Which I think is...

[00:18:43] That's a good thing.

[00:18:44] And like Progress eight gets quite a lot of criticism.

[00:18:47] Well, the main criticism is...

[00:18:49] Okay, so some of the main criticisms, I guess, of Progress eight are...

[00:18:52] Firstly, it's like that thing of it being a relative measure.

[00:18:56] So as a school, if you think about it, right, there's no objective standard where you say, right, if we do X, Y and Z and we get to this level, then we'll get a good Progress eight score.

[00:19:06] Because your Progress eight score is dependent on how all the other schools in the country do as well.

[00:19:10] So there's no like objective standard that you can work towards for Progress eight.

[00:19:15] It's always going to be dependent upon how other schools also are doing.

[00:19:20] So I guess that's one thing.

[00:19:24] The other thing is that the way Progress eight is structured as a measure is, I think, has some problems.

[00:19:30] So basically zero is like where schools are kind of want to be, like they want to be at zero or above.

[00:19:37] Yeah.

[00:19:37] You don't want to have a minus progress score.

[00:19:40] But the trouble is that that creates this kind of...

[00:19:43] We talk a lot about positive PAs.

[00:19:45] Yeah, we've got a positive Progress eight score.

[00:19:47] And that's it.

[00:19:47] But it creates this weird jeopardy around zero.

[00:19:51] So a school that has a Progress eight score of minus 0.01 and one that has a Progress eight score of plus 0.01 would be virtually indistinguishable in terms of the amount of progress they're making with students.

[00:20:03] But it becomes a focus, doesn't it?

[00:20:04] But it feels very different.

[00:20:05] So it's like that feeling that you're behind the curve or in front of it.

[00:20:10] Yeah.

[00:20:10] And I think that's one of the ways...

[00:20:12] And that's to do with the design of the measure, I think.

[00:20:15] It's just that could have been designed better so that it didn't create this kind of weird jeopardy.

[00:20:20] It's almost like the positive minus in the measure makes anything, you know, sounds really negative if it's got a negative mark in it.

[00:20:29] Yeah, yeah, yeah.

[00:20:31] I think another thing is probably to do with...

[00:20:34] I mean, again, to reference Dan's interview, is to do with the reliability of the Key Stage 2 data.

[00:20:45] Yeah.

[00:20:45] So whether or not the SAT scores are giving you a reliable starting point for the students.

[00:20:51] So, you know, we know that primary schools are incentivized for their students to do very well in their SATs.

[00:20:57] And it's not unusual in secondaries to find students who have very good SAT scores.

[00:21:02] But then when you actually start to work with them, you find that maybe they're not quite at that level.

[00:21:06] And so the fact that it's basing all...

[00:21:10] The Progress 8 measure is based on the student's starting point, which is their SAT scores.

[00:21:14] And if those SAT scores aren't entirely always reliable, then that, again, can distort, you know, whether or not the Progress 8 is actually reliable as well.

[00:21:25] So there's that.

[00:21:26] And I think just the other thing of like as well of whether or not...

[00:21:29] And this might sound a bit bizarre, but like whether or not it's caught on.

[00:21:35] So I don't know.

[00:21:36] I think within the education sector, Progress 8 is well, you know, kind of is used.

[00:21:43] And it's considered carefully in terms of thinking about how schools are doing, how school leaders are doing and what have it.

[00:21:50] I don't know how much traction it has with the public as a measure.

[00:21:54] Yeah.

[00:21:55] Because it is quite complex.

[00:21:57] And unless you work within education in some way, you probably aren't as familiar as it is.

[00:22:03] It's trickier to explain quickly what that means.

[00:22:08] So I can see that, which is why I think it's quite helpful.

[00:22:11] I also don't think it's just because of the complexity, though.

[00:22:14] I think that there might be something also that maybe people would be less willing to acknowledge.

[00:22:21] What's that?

[00:22:21] Which is that I think a lot of parents are more concerned about the attainment levels at schools than the progress levels.

[00:22:26] Right.

[00:22:27] So like as a shorthand, like you just want to know, is this a good school that gets good results?

[00:22:32] Not is this a school that makes excellent progress with its students?

[00:22:36] Compared to the rest of the country.

[00:22:38] So, for example, like one of the things where you think about it, about progress, is that it's actually easier to make progress with students who come in at a lower level because you've got more room to progress into.

[00:22:50] Whereas if you've got students who have come in at a very high level, the room for progress is actually less because there's sort of limitations, if you like, because the measures set these limitations.

[00:23:01] Like you can only make a certain amount of progress with a student who's come in with the very, very best SAT scores.

[00:23:06] Yeah.

[00:23:07] So in some ways, it's kind of easier to make progress with students who have much lower attainment at primary school.

[00:23:15] But that in itself might be an indicator of a school that, you know, where the attainment is lower.

[00:23:22] Yeah.

[00:23:22] So it's, I think...

[00:23:24] But then I guess parents have that attainment measure to use.

[00:23:28] Yes, absolutely.

[00:23:28] Absolutely.

[00:23:29] But I guess if we're just talking about progress eight and its effectiveness and it's kind of the extent to which it's caught on, I'm not sure that there'd be many members of the public who would be able to tell you what progress eight is.

[00:23:42] No, and it's good for people outside of the education sector to realise that this is the measure that schools are judged on.

[00:23:51] Not just the attainment measure.

[00:23:53] It's important to know that.

[00:23:54] Yeah, totally.

[00:23:54] So, John, tomorrow you're in charge of performance measures for the country.

[00:24:02] What would it look like?

[00:24:04] Oh, God.

[00:24:05] Well, do you know what?

[00:24:07] Like, if we're going to have the same...

[00:24:09] I'm going to make a confession here, which is I actually think progress eight's quite good.

[00:24:13] Yeah.

[00:24:14] Like, I...

[00:24:15] You know, it gets a lot of flack.

[00:24:16] But if you compare it to what we had before, like, I think it's a massive, massive improvement.

[00:24:23] Yeah, totally.

[00:24:23] Would you add anything on, though?

[00:24:26] Would there be other...

[00:24:28] I mean, obviously, we're not talking about all the measures that schools are...

[00:24:32] All the ways that schools are measured right now.

[00:24:34] We're just talking about them in relation to exams.

[00:24:38] Yeah.

[00:24:38] But is there another measure that you think you could add on that would either bridge that gap between people in education who understand this and the wider public,

[00:24:49] who need to understand how schools are and how they're performing?

[00:24:54] Yeah.

[00:24:55] What would you think?

[00:24:55] So the one thing I think I would do is that I think one of the big problems with progress eight,

[00:25:00] and this gets commented on quite a lot, and I think it's a really fair criticism, actually,

[00:25:05] is that if you think about the SAT scores, those are in English and maths.

[00:25:09] Like, we know from where we've talked about this that it's quite narrow what students are ultimately assessed on in primary school.

[00:25:18] So it's their competence in English and maths.

[00:25:21] And part of the problem with progress eight is it's looking at a broad curriculum.

[00:25:26] So you're looking at subjects like potentially like music and drama and art and design technology and PE and all these sorts of things.

[00:25:35] But you're using the student's performance in English and maths to determine the amount of progress that students have made in those subjects.

[00:25:43] And I think that that is a real weakness.

[00:25:46] Like, I don't necessarily think that whether a student is good at English or maths can really tell you that much about whether or not they're going to be good at art.

[00:25:54] It might tell you a bit, but probably not a huge amount.

[00:25:57] But because of the way the measure is structured, obviously what we do is we'll set targets for students in art and music that are based on how they did in English and maths at primary school,

[00:26:09] because we're trying to maximise their progress score.

[00:26:12] So I think if I were to slightly re-engineer the system, I think I would look to maybe move some subjects towards a different measure.

[00:26:24] Like keeping that, by trying to keep that breadth, keep it compulsory, but maybe looking at different ways in which you could judge performance in subjects that aren't maybe as clearly linked to English and maths as others.

[00:26:42] Just so that, I don't know, just so that you could capture students' abilities a bit better, I think, by doing that.

[00:26:55] But I don't know exactly what that would look like, but I think that would be one thing.

[00:26:58] That's right, I haven't asked you to design it in this half an hour as well.

[00:27:01] It's just in my head right now.

[00:27:03] Yeah.

[00:27:04] One thing I just wanted to do really quickly was to illustrate the different effects that Progress 8 and Attainment 8 have and how looking at things in terms of attainment and in terms of progress can give you some interesting kind of results.

[00:27:19] So I'm just going to talk about, and I'm not going to name any names here, but I'm just going to talk about the local authority that I work in.

[00:27:25] And I'm just going to look at the year 2022, which happens to be a year that the school I work in did rather well.

[00:27:31] Is that why you use it every year?

[00:27:32] Which is why I use it all the time, yeah, yeah.

[00:27:35] My PowerPoint is ridiculous.

[00:27:37] It's still talking about 2022.

[00:27:40] But so I work at a school, and it's quite a strange thing because we have a school like literally just 100 yards up the road, another school.

[00:27:51] So we are sort of, you know, friends and competitors at the same time.

[00:27:56] It's quite an interesting relationship.

[00:27:58] But we, I think it's just a historical thing, but we have slightly different intakes.

[00:28:04] So our intake tends to be more disadvantaged than this other school.

[00:28:10] So, you know, kind of typically our students will come with lower SATS scores than at this other school.

[00:28:17] And in 2022, it was quite interesting, but it just illustrates how this can work.

[00:28:24] So the other school had a better attainment score than we did.

[00:28:28] But our progress score was massively better than their progress score.

[00:28:35] So that just illustrates the point about how, you know, how these measures kind of work together, if you like.

[00:28:42] So you need to look at both of them.

[00:28:43] You look at attainment eight and progress eight, because if you just looked at our attainment score, you'd say, oh, that school's not doing as well as the one up the road.

[00:28:51] But then if you looked at, just looked at the progress scores, you'd be like, oh, that school's actually doing way better than the one up the road.

[00:28:57] So you kind of have to look at them together to get a real sense and a real picture of how schools performing with students in their exams.

[00:29:07] So before we finish, John, give us, let's have a chat about some of the top performance scores in the country just to, in terms of their measures.

[00:29:19] OK, so can we talk about schools that have the best progress?

[00:29:24] Because I think if we get into attainment scores, you're going to, like, some of the answer to that will be probably quite obvious.

[00:29:30] Some of the top, like, you know, private schools and things like that.

[00:29:34] But if we talk about progress, it's quite interesting.

[00:29:37] I think this is quite interesting.

[00:29:38] So, yeah, when you look at the list of, like, the top 10 schools nationally for progress 8.

[00:29:48] So I've got here, I've got 2022 and I've got 2023.

[00:29:52] A lot of the names on these lists appear more, you know, appear on both years.

[00:29:58] So there are clearly some schools who are doing very, very well at progress 8.

[00:30:02] The school that I taught at in London is going to be on there, isn't it?

[00:30:05] And Marina.

[00:30:05] Marina, who was our first interview.

[00:30:07] First exams officer interview.

[00:30:09] So Sacred Heart School in London typically gets a very, very high progress 8 score.

[00:30:15] It's often in the top 10 of schools nationally for progress 8.

[00:30:20] So some other famous schools that you might recognise.

[00:30:23] Michaela Community School in 2022 and in 2023 was the top performing school for progress 8.

[00:30:31] So I had a progress 8 of plus 2.27.

[00:30:34] Yeah, I mean, that's...

[00:30:36] Any progress 8 score above 2 is, like, is crazy.

[00:30:39] I mean, incredibly unusual, isn't it?

[00:30:42] Yeah.

[00:30:42] Another school that appears regularly is the Steiner Academy in Hereford.

[00:30:47] So both in 2022 and 2023 had a progress 8 score above 2, plus 2.

[00:30:55] So there is a bunch of schools that are performing very, very well on this measure.

[00:31:01] So I thought it'd be interesting to just sort of look at what those schools have in common

[00:31:07] and to try and understand why it might be that certain schools are doing well

[00:31:13] in terms of making very high levels of progress with their students.

[00:31:17] So I actually did a bit of research into this and I found some common factors that link the schools

[00:31:25] that do very well on progress 8.

[00:31:27] So the first thing is that they're disproportionately in London.

[00:31:32] So a lot of these schools, when you look down this list, yeah, are disproportionately in London now.

[00:31:38] London's had a lot of funding pushed into its schools.

[00:31:43] And radically improved.

[00:31:44] Yeah, a lot of schools radically improved between sort of 2000 and 2010.

[00:31:50] So I think that's kind of maybe unsurprising.

[00:31:55] I'm sure there are other reasons why London performs well for that measure,

[00:32:00] but funding would certainly probably be one of those.

[00:32:04] They are also disproportionately single-sex schools, in particular girls' schools,

[00:32:09] do very, very well on progress 8 as well.

[00:32:12] Progress 8, interesting.

[00:32:13] Yeah.

[00:32:15] They are disproportionately schools with smaller cohorts, so less than 150 students in year 11.

[00:32:22] That makes sense though, doesn't it?

[00:32:23] That does.

[00:32:23] And when you look at a free school like Michaela, that's probably a good, one of the reasons why

[00:32:30] it performs so well, is it's a smaller school.

[00:32:36] Now, this is quite an interesting one.

[00:32:38] So a small number.

[00:32:41] Sorry to interrupt that, but probably a lot of that is just with smaller cohorts as well,

[00:32:45] is that ability to really track progress over time.

[00:32:49] Yeah.

[00:32:49] You know, really tracking data.

[00:32:52] Yeah, yeah, yeah.

[00:32:52] It's much easier with smaller numbers, isn't it?

[00:32:55] And maybe as well, like not having so much to, like what I see sometimes is like we're having

[00:33:00] to make tough decisions about, okay, this group of students really needs the extra focus rather

[00:33:08] than this group.

[00:33:09] And it's like resources and how you're going to deploy them.

[00:33:12] And maybe if you've got slightly smaller cohorts.

[00:33:15] It's naturally going to be smaller numbers in those groups, isn't it?

[00:33:18] Yeah, yeah, potentially.

[00:33:22] So this one's quite interesting.

[00:33:24] So a small number of large multi-academy trusts feature disproportionately.

[00:33:28] Okay.

[00:33:30] So what the data doesn't show is that being in a multi-academy trust makes you more likely

[00:33:36] to have a good Progress 8 score.

[00:33:38] It doesn't show that.

[00:33:39] So you're just as likely if you're in a, but if you're in one of a certain multi-academy

[00:33:45] trust, you are more likely.

[00:33:46] Now, I'm not going to, I'm not going to name any names here, but let's just say that some

[00:33:50] of them are the big ones that you may have heard of.

[00:33:52] Yeah.

[00:33:52] Um, so those are disproportionately more likely to feature in this, um, in the top Progress

[00:34:00] 8 scores.

[00:34:01] The final one here that I think is, is kind of perhaps the most interesting is that, um,

[00:34:06] the schools who are the top performing Progress 8 schools are disproportionately have a religious

[00:34:12] ethos.

[00:34:14] So when you look down the list, you see that very, very clearly.

[00:34:17] Um, it's not hard to spot.

[00:34:18] Um, a lot of them are faith schools, um, of one denomination or another.

[00:34:23] Um, and, um, but I, I think what's quite interesting about this is that I would slightly

[00:34:29] potentially even go a bit further than that.

[00:34:32] And I would say that these are all schools that have a strong ethos of some kind.

[00:34:38] So it might not necessarily be a religious ethos.

[00:34:41] Strong culture.

[00:34:42] But it has a strong, yeah.

[00:34:43] So if you think about like Michaela Academy or the, uh, or sorry, Michaela Community School

[00:34:49] or the Steiner Academy, these are both schools that have a very particular philosophy.

[00:34:57] Very clear ethos.

[00:34:58] Yeah.

[00:34:58] And they sort of have, um, very clear parameters about how they want their students to conduct

[00:35:05] themselves, how they want to run their lessons, what they want their curriculum to be looked

[00:35:10] like.

[00:35:10] So schools with, um, very strong cultures on the ground tend, that's implemented from

[00:35:18] year seven straight away as well, rather than just when, you know, uh, closer to the exam

[00:35:24] years.

[00:35:24] So it's embedded and obviously progress eight is a measure that's across the years.

[00:35:29] So I think that's like, if we were to leave it with one potential thing that you could sort

[00:35:34] of say to schools about, like, is there a lesson in like the schools that make the most progress

[00:35:39] with their students?

[00:35:40] I think that maybe this notion of like, um, that you have a kind of consistent ethos and

[00:35:50] it doesn't necessarily have to be a particular one.

[00:35:53] Yeah.

[00:35:53] But that it is, it's, it's well thought out, it's consistent and it's delivered consistently

[00:36:00] to the students, um, is potentially something which seems to have like quite a big, big impact.

[00:36:09] Um, it has its downsides, obviously.

[00:36:12] Like a strong ethos can also be a rigid ethos.

[00:36:15] It depends on the ethos.

[00:36:17] Yeah.

[00:36:17] Yeah.

[00:36:17] To some extent, but also like any ethos that's, that's, you know, kind of doesn't leave in

[00:36:23] a room for flexibility can, can feel kind of hard line and what have you.

[00:36:28] But it does seem, you know, you can't deny that this is data that you cannot deny.

[00:36:32] Like, I think it's quite obvious the schools that have this very strong culture do get very

[00:36:39] good progress in terms of their exam results out of their students.

[00:36:45] So was that fun?

[00:36:46] I really enjoyed that actually, John.

[00:36:50] I think it was maybe more fun in my head than it was in practice.

[00:36:54] No, it is, it is interesting.

[00:36:57] And I said, I know a lot about this, but it's still every time I, I like to be reminded of

[00:37:02] some of those things about it.

[00:37:04] It's really, it is interesting.

[00:37:05] And if you don't know that much about it and you've just heard these words,

[00:37:09] sort of bandied around, then it's great to have a nice little overview.

[00:37:14] Yeah.

[00:37:15] And I hope that like you found our explanations of things useful.

[00:37:19] We haven't always gone into like the most detail that we could and we tried to keep

[00:37:24] it as accessible as possible.

[00:37:26] So, so, you know, if you're a real expert in this area, then we apologize if it was not,

[00:37:33] you know, as detailed as you might have liked.

[00:37:37] But, but hopefully, you know, a lot of people will be able to get something from it.

[00:37:45] Thank you so much for listening to the exam man podcast.

[00:37:47] We really, really appreciate your support.

[00:37:50] Remember that you can access it on all the major podcast platforms.

[00:37:54] Give us a rating, give us a follow, and we will catch you next time.