Precious Cargo- the journey of exam papers from pens down to results days

[00:00:23] [SPEAKER_01]: The oddest thing I ever saw in an envelope, and this is probably going back five or six

[00:00:26] [SPEAKER_01]: years, was a £10 note. Absolutely no reason.

[00:00:31] [SPEAKER_01]: Thank you to the exam group.

[00:00:33] [SPEAKER_01]: Again, we have a process so Head of Centre was contacted.

[00:00:43] [SPEAKER_04]: Did you make that decision, John?

[00:00:44] [SPEAKER_02]: Yeah, yeah. In discussion with Head of Centre, we'll make... well it's the Head of Centre's

[00:00:50] [SPEAKER_02]: decision. Getting above my station.

[00:01:02] [SPEAKER_02]: Welcome everyone to this summer special edition of The Exam Man podcast. We've been having

[00:01:08] [SPEAKER_02]: a little bit of a break, haven't we?

[00:01:09] [SPEAKER_04]: Yeah, to be honest, it's really tragic, but we've really missed doing this, haven't

[00:01:12] [SPEAKER_04]: we John?

[00:01:13] [SPEAKER_04]: Yeah, we have.

[00:01:14] [SPEAKER_04]: And then kind of itching on ourselves.

[00:01:16] [SPEAKER_02]: When we finished the last season, I thought we thought, oh, we're going to go on a

[00:01:19] [SPEAKER_02]: massive break now, but it's actually it's been about three or four weeks, isn't it?

[00:01:22] [SPEAKER_02]: And actually we've been doing loads of stuff on it.

[00:01:26] [SPEAKER_02]: But we hope you've all had a good summer. I know that everyone's probably sort of either

[00:01:30] [SPEAKER_02]: been back doing results or is gearing up for results this week.

[00:01:36] [SPEAKER_02]: We have been sort of busy over the last few weeks putting together this special

[00:01:41] [SPEAKER_02]: episode, which is all about what happens to exam papers when the students finish

[00:01:46] [SPEAKER_02]: their exams and we send them off.

[00:01:48] [SPEAKER_02]: So it's that sort of period of time where I think most people who work in schools

[00:01:53] [SPEAKER_02]: maybe don't know that much about what happens.

[00:01:56] [SPEAKER_04]: Well, I think everyone's just relieved, aren't they? So students are relieved to

[00:01:59] [SPEAKER_04]: just finish their exams.

[00:02:01] [SPEAKER_04]: Yeah.

[00:02:01] [SPEAKER_04]: Exams teams certainly are as well. Teachers, everyone's just like, right, we're off.

[00:02:06] [SPEAKER_02]: Yeah, no one really wants to think about exams.

[00:02:07] [SPEAKER_04]: You don't really want to think about it. Yeah. I've not really thought about it until we started looking into it.

[00:02:11] [SPEAKER_02]: So the fact that we've basically just ignored the fact that you don't want to think about it.

[00:02:17] [SPEAKER_02]: Yeah, and we're forcing it.

[00:02:19] [SPEAKER_02]: We're forcing it down your throat.

[00:02:22] [SPEAKER_02]: So yeah, so we've been making an episode about the journey of exam papers basically.

[00:02:28] [SPEAKER_02]: And we're going to start this episode actually with the people who are affected most

[00:02:33] [SPEAKER_02]: by this process. And that is, of course, the students who have sat the exams.

[00:02:38] [SPEAKER_02]: So we managed to catch up with a couple of students from a school in London who we asked what they thought happened.

[00:02:48] [SPEAKER_02]: You know, with no prior knowledge, what do they think happens to their exam papers

[00:02:52] [SPEAKER_02]: once they've been sent off by the school? And these are the responses that they gave.

[00:02:57] [SPEAKER_00]: Well, I think that they send it to a teacher. Yeah, they send it to a teacher.

[00:03:03] [SPEAKER_00]: The teacher marks it and then sends it to the board of exams.

[00:03:10] [SPEAKER_00]: Then they do their little investigation to see if there's similar results or something like that.

[00:03:16] [SPEAKER_00]: And then they scan it and send it back.

[00:03:19] [SPEAKER_08]: I don't know much, but in terms of what I think happens,

[00:03:23] [SPEAKER_08]: they take your papers and then they're taken to exam centres.

[00:03:29] [SPEAKER_08]: And then they're processed online and people scan them online.

[00:03:34] [SPEAKER_08]: And then examiners, who I think are mostly teachers, they mark it and then they do their results.

[00:03:42] [SPEAKER_08]: And then the examiners go through like moderations, every 10 exams or so.

[00:03:49] [SPEAKER_08]: Maybe one or two of them will be moderated, cross-examined to check that they're accurate.

[00:03:53] [SPEAKER_08]: And then I don't know how they determine your grades or the grade boundaries.

[00:03:58] [SPEAKER_08]: But then I think after that, I think the papers, they get the results for the students.

[00:04:06] [SPEAKER_08]: I think they throw away the papers because there's no reason to keep them.

[00:04:09] [SPEAKER_08]: So I think they throw away their papers and then yeah, that's the end of your result, I guess.

[00:04:13] [SPEAKER_04]: I mean, surely that second student had been given some coaching or something

[00:04:17] [SPEAKER_04]: because that was quite an answer, wasn't it?

[00:04:19] [SPEAKER_02]: Yeah, I have a feeling that student might be quite close to what actually happens.

[00:04:25] [SPEAKER_03]: Impressive.

[00:04:26] [SPEAKER_02]: Yeah, do you know what?

[00:04:26] [SPEAKER_02]: It reminded me of the exams in 2022 when students were given prior knowledge

[00:04:33] [SPEAKER_02]: of what might be in the exam paper.

[00:04:35] [SPEAKER_02]: I feel like that student might have been given prior knowledge of the question we were going to ask.

[00:04:40] [SPEAKER_04]: So impressed with their knowledge.

[00:04:42] [SPEAKER_02]: Yeah, I mean, this information, the information about what happens is available.

[00:04:46] [SPEAKER_02]: I'm sure it is out there online if you dig for it hard enough.

[00:04:49] [SPEAKER_02]: But we sort of wanted to piece it all together to give a really succinct version of what happens.

[00:04:57] [SPEAKER_02]: So you may remember that when we spoke to Michael Turner from AQA in one of our episodes in Season 1,

[00:05:04] [SPEAKER_02]: he invited us to go to Milton Keynes, didn't they?

[00:05:08] [SPEAKER_02]: To visit their processing centre.

[00:05:10] [SPEAKER_04]: Don't know how serious he was about that but we sort of jumped on it and were like, oh my god.

[00:05:14] [SPEAKER_02]: I think he might have just, it might just be like one of those polite things that people say

[00:05:18] [SPEAKER_02]: and they don't actually expect you to say yes.

[00:05:22] [SPEAKER_02]: But we just dove straight into me and we went to Milton Keynes.

[00:05:26] [SPEAKER_02]: But we had a fantastic day there and we met with Mark Collins who is AQA's head of operations

[00:05:33] [SPEAKER_02]: and he gave us a tour of the exam paper processing centre.

[00:05:39] [SPEAKER_04]: Which was incredible.

[00:05:41] [SPEAKER_02]: Yeah, it was phenomenal.

[00:05:42] [SPEAKER_02]: It was phenomenal.

[00:05:43] [SPEAKER_02]: I don't want to say too much because I think it really comes across in the interview.

[00:05:50] [SPEAKER_02]: He started off by telling us actually about, you'll know as exams officers,

[00:05:55] [SPEAKER_02]: you stick those yellow labels on the packets when they get sent out.

[00:05:59] [SPEAKER_02]: And he started off by actually by telling us all about the postcodes.

[00:06:03] [SPEAKER_02]: And that the postcodes are actually important.

[00:06:05] [SPEAKER_02]: They're used to sort of pre-sort the exam papers when they arrive with AQA from Parcelforce.

[00:06:13] [SPEAKER_02]: So that was an interesting little fact wasn't it, that I didn't know about.

[00:06:18] [SPEAKER_02]: But I'm going to lead explaining the rest of the process to him

[00:06:21] [SPEAKER_02]: because he does such a brilliant job and he'll take us through

[00:06:25] [SPEAKER_02]: basically from the point of the arrival of the papers at the AQA centre in Milton Keynes

[00:06:31] [SPEAKER_02]: all the way through to the point where they're ready to be marked.

[00:06:35] [SPEAKER_04]: So firstly Mark I just want to say a huge thank you for showing us around today.

[00:06:39] [SPEAKER_04]: As someone who can't keep their kids socks together,

[00:06:42] [SPEAKER_04]: I feel quite staggered by the scale and just how impressive it is.

[00:06:48] [SPEAKER_04]: The attention to detail and everything that goes into an operation of this size is incredibly impressive.

[00:06:55] [SPEAKER_04]: So thank you for that.

[00:06:57] [SPEAKER_04]: We're really really interested in basically what the transferable skills are between your previous industries

[00:07:02] [SPEAKER_04]: and kind of working for this in education.

[00:07:05] [SPEAKER_04]: It must be quite a different beast in some ways to other operations roles.

[00:07:10] [SPEAKER_01]: It is very much a different beast.

[00:07:12] [SPEAKER_01]: So I've been in operations for most of my adult life I guess and 12 years as part of the AQA group.

[00:07:18] [SPEAKER_01]: The biggest challenge that we have is the product that we deal with.

[00:07:22] [SPEAKER_01]: So anybody in operations that has a similar background to me,

[00:07:26] [SPEAKER_01]: so I worked for Amazon for a while, I've worked for Tesco's, I've worked for Iceland,

[00:07:29] [SPEAKER_01]: backfilling stores, that type of thing,

[00:07:31] [SPEAKER_01]: we'll know all about having to move boxes in, through and out.

[00:07:37] [SPEAKER_01]: And you've got deadlines to hit and transport offices to keep happy

[00:07:40] [SPEAKER_01]: and a number of issues that you face on a day-to-day basis.

[00:07:44] [SPEAKER_01]: The difference between what we do here and what I've done before is the product that we deal with.

[00:07:50] [SPEAKER_01]: We're dealing here with exam papers and those exam papers represent the journey of a student

[00:07:56] [SPEAKER_01]: that may have taken two, three years to get to that point.

[00:07:59] [SPEAKER_01]: And we have to be so careful that we treat them with the respect they deserve.

[00:08:04] [SPEAKER_01]: We never let a learner down and we have to account for every single page.

[00:08:08] [SPEAKER_01]: And it takes some time to kind of get your head around the difference and the scale of what we do.

[00:08:15] [SPEAKER_01]: But I would say that's probably the biggest difference between my past and my present.

[00:08:22] [SPEAKER_02]: So because you mentioned to us a couple of times on the way round,

[00:08:24] [SPEAKER_02]: I think about what you're really optimising for here is quality rather than efficiency.

[00:08:31] [SPEAKER_02]: And obviously, you've got deadlines to hit.

[00:08:33] [SPEAKER_02]: But I guess maybe in other industries, is there more of a focus on efficiency than you've noticed?

[00:08:40] [SPEAKER_01]: In my experience, there is.

[00:08:42] [SPEAKER_01]: And I think, again, the difference here is for a number of reasons.

[00:08:45] [SPEAKER_01]: One is that we're in the exams industry and that's very different.

[00:08:48] [SPEAKER_01]: And I think the other part is that being a charity, as AQA is,

[00:08:55] [SPEAKER_01]: we reinvest the money that we make into education,

[00:09:00] [SPEAKER_01]: both the physical process of education and research for education.

[00:09:04] [SPEAKER_01]: And so we're able to really look at how can we best serve our customers

[00:09:11] [SPEAKER_01]: and ultimately the learners in the education system.

[00:09:14] [SPEAKER_01]: And so, again, it's a slightly different mindset.

[00:09:17] [SPEAKER_01]: But certainly we have more of a focus on quality than where I've been before.

[00:09:28] [SPEAKER_01]: I think the first thing that we do when the deliveries arrive is we sub-sort.

[00:09:34] [SPEAKER_01]: And we're always sub-sorting it to component level.

[00:09:37] [SPEAKER_01]: So for its onward journey, for the life of those scripts that are with us,

[00:09:40] [SPEAKER_01]: they're all together in relevant components.

[00:09:44] [SPEAKER_01]: So when we ultimately scan stuff,

[00:09:46] [SPEAKER_01]: we'll do multiple centres but the same component at the same time.

[00:09:51] [SPEAKER_01]: At any given point in the series,

[00:09:54] [SPEAKER_01]: we'll get between five and six deliveries a day,

[00:09:58] [SPEAKER_01]: which could be up to 50,000 packets of scripts.

[00:10:02] [SPEAKER_01]: On average in every packet, there's 21 scripts.

[00:10:06] [SPEAKER_01]: And again, that will vary depending between centres,

[00:10:08] [SPEAKER_01]: but that's the average that we get.

[00:10:10] [SPEAKER_01]: And at any given point in the peak of the series,

[00:10:14] [SPEAKER_01]: we will have a work in progress of around three million scripts.

[00:10:20] [SPEAKER_01]: So the first thing we're doing then is sorting to component level.

[00:10:24] [SPEAKER_01]: That's all staged.

[00:10:26] [SPEAKER_01]: We have a team of people, flow managers,

[00:10:30] [SPEAKER_01]: that will determine what it is we need to process and in which order.

[00:10:35] [SPEAKER_01]: So we have our internal SLAs,

[00:10:37] [SPEAKER_01]: but ultimately we're looking to keep the marking teams as busy as possible

[00:10:41] [SPEAKER_01]: and have the right level of work available for them.

[00:10:45] [SPEAKER_01]: So they'll go through and they'll select the next shifts work

[00:10:48] [SPEAKER_01]: that they've got to process.

[00:10:49] [SPEAKER_01]: And that's a rolling programme.

[00:10:51] [SPEAKER_01]: So night shift we'll look after, day shift and vice versa.

[00:10:54] [SPEAKER_02]: So you do some prioritisation at that stage.

[00:10:57] [SPEAKER_02]: So for example, you were saying earlier,

[00:10:59] [SPEAKER_02]: if the markers in a particular subject are getting low,

[00:11:03] [SPEAKER_02]: then we need to make sure that they've got more stuff to mark.

[00:11:06] [SPEAKER_02]: So we might prioritise that component.

[00:11:08] [SPEAKER_01]: Correct, yeah.

[00:11:09] [SPEAKER_01]: So although we have a plan that we work to,

[00:11:13] [SPEAKER_01]: there are times where we'll liaise,

[00:11:15] [SPEAKER_01]: coordinate with our colleagues in the marking department.

[00:11:18] [SPEAKER_01]: And they may say there's a particular component

[00:11:20] [SPEAKER_01]: where they're running a bit short.

[00:11:21] [SPEAKER_01]: Could we push them to the front of the queue?

[00:11:24] [SPEAKER_01]: And we do that in a number of ways.

[00:11:26] [SPEAKER_01]: But ultimately the first thing we do is investigate the system

[00:11:30] [SPEAKER_01]: to see where that product is.

[00:11:33] [SPEAKER_01]: And it may be that there's some sitting waiting at scan

[00:11:35] [SPEAKER_01]: and it will naturally flow through in the next couple of hours anyway.

[00:11:38] [SPEAKER_01]: But it may be that it's further back

[00:11:40] [SPEAKER_01]: and we will push some of that to the front

[00:11:42] [SPEAKER_01]: because it's all about that collaboration really.

[00:11:45] [SPEAKER_01]: Before we scan, we need to determine the different journeys

[00:11:51] [SPEAKER_01]: for the different types of product that we get inside the envelopes.

[00:11:54] [SPEAKER_01]: And so that's done in an area called logging.

[00:11:57] [SPEAKER_01]: And essentially what we're doing in logging is we're opening the envelope.

[00:12:00] [SPEAKER_01]: We are logging the receipt of that envelope onto our system

[00:12:06] [SPEAKER_01]: and we are separating the different types of material that are in it.

[00:12:09] [SPEAKER_01]: And so the main types of material you'll get are batches of scripts

[00:12:13] [SPEAKER_01]: that can essentially just be scanned.

[00:12:16] [SPEAKER_01]: You've got scripts with additional materials.

[00:12:19] [SPEAKER_01]: You've got additional materials on their own.

[00:12:21] [SPEAKER_01]: You've got attendance registers which need scanning and filing.

[00:12:24] [SPEAKER_01]: And then you'll have what we call white mail.

[00:12:26] [SPEAKER_01]: So that's non-candle specific stuff like seating plans

[00:12:30] [SPEAKER_01]: that may have been put in and included.

[00:12:31] [SPEAKER_01]: And so the team of people are subsorting all of that

[00:12:34] [SPEAKER_01]: and logging on the system at the relevant point.

[00:12:38] [SPEAKER_01]: Now one of the key points that they do is they use the data

[00:12:41] [SPEAKER_01]: on the outside of the envelope

[00:12:44] [SPEAKER_01]: to log the number of scripts inside the envelope.

[00:12:46] [SPEAKER_01]: And that's really important for reconciliation.

[00:12:49] [SPEAKER_01]: There's a smile on everyone's face.

[00:12:51] [SPEAKER_01]: I was making this point quite strongly downstairs.

[00:12:53] [SPEAKER_01]: You were, Mark.

[00:12:55] [SPEAKER_01]: And so we use that data along with we count the number of scripts

[00:12:59] [SPEAKER_01]: in that envelope at the point we log it.

[00:13:00] [SPEAKER_01]: And then we count them again when it's scanned

[00:13:02] [SPEAKER_01]: or processed in another way.

[00:13:05] [SPEAKER_01]: So what is it really helpful if schools and centres do

[00:13:09] [SPEAKER_01]: with their envelopes?

[00:13:11] [SPEAKER_01]: On the front of the envelope there is a grid

[00:13:14] [SPEAKER_01]: that asks for multiple pieces of information.

[00:13:16] [SPEAKER_01]: And actually we're currently redesigning the envelope

[00:13:18] [SPEAKER_01]: so we'll only be asking for the number of scripts

[00:13:21] [SPEAKER_01]: because some of the information currently is duplicated

[00:13:23] [SPEAKER_01]: on the label.

[00:13:24] [SPEAKER_01]: Right, yeah.

[00:13:25] [SPEAKER_01]: And so the piece of the puzzle that was missing

[00:13:27] [SPEAKER_01]: that we'd really like all centres to help us with

[00:13:29] [SPEAKER_01]: is the number of scripts contained in that envelope.

[00:13:32] [SPEAKER_04]: Which John hasn't been doing religiously.

[00:13:36] [SPEAKER_04]: But he will from this point on.

[00:13:38] [SPEAKER_02]: He's promised for you.

[00:13:40] [SPEAKER_02]: I will, I promise.

[00:13:41] [SPEAKER_02]: I now absolutely understand the value of doing that.

[00:13:44] [SPEAKER_02]: So yeah, I've given myself a slap on the wrist.

[00:13:48] [SPEAKER_02]: From now on we will do that.

[00:13:49] [SPEAKER_01]: Thank you.

[00:13:51] [SPEAKER_01]: So once everything's logged and we understand

[00:13:52] [SPEAKER_01]: where it's going to go and how it's going to be processed,

[00:13:54] [SPEAKER_01]: it moves through its journey.

[00:13:56] [SPEAKER_01]: And at all points through the journey

[00:13:57] [SPEAKER_01]: we're tracking it either in cages of work

[00:13:59] [SPEAKER_01]: or trolleys of work.

[00:14:02] [SPEAKER_01]: The next natural place for things to go

[00:14:05] [SPEAKER_01]: will be either attendance registers

[00:14:06] [SPEAKER_01]: will go off on their own journey

[00:14:07] [SPEAKER_01]: and they'll be scanned and then uploaded into our system

[00:14:10] [SPEAKER_01]: so that we can deal with them later on.

[00:14:12] [SPEAKER_01]: I'll get to that in a minute.

[00:14:14] [SPEAKER_01]: But the main bulk of the scripts go on to

[00:14:16] [SPEAKER_01]: what we call the guillotine.

[00:14:18] [SPEAKER_01]: And that's essentially some machinery we've got

[00:14:21] [SPEAKER_01]: that take the spine from the booklets

[00:14:23] [SPEAKER_01]: and they convert exam booklets into sheets of paper

[00:14:28] [SPEAKER_01]: because we need them to be individual sheets

[00:14:30] [SPEAKER_01]: so that we can scan them.

[00:14:32] [SPEAKER_04]: And it just looked like the most physical job ever.

[00:14:36] [SPEAKER_04]: Quite impressive for the people working on that day in day out.

[00:14:39] [SPEAKER_01]: We saw two guys who were very experienced, didn't we?

[00:14:42] [SPEAKER_01]: Certainly one of them was very experienced.

[00:14:45] [SPEAKER_01]: So both Richard and Bob have been with us a long time.

[00:14:46] [SPEAKER_01]: They're very experienced.

[00:14:47] [SPEAKER_01]: We do put temporary workers on there.

[00:14:50] [SPEAKER_01]: Some people really settle in that role

[00:14:53] [SPEAKER_01]: and others will ask to be moved.

[00:14:55] [SPEAKER_01]: And of course we'll use people to the best,

[00:14:58] [SPEAKER_01]: put them in the best place we can.

[00:15:00] [SPEAKER_01]: So we don't have to move around.

[00:15:01] [SPEAKER_04]: They do.

[00:15:03] [SPEAKER_01]: But each of those guillotines can cut around 11,000 scripts an hour

[00:15:07] [SPEAKER_01]: and we've got three of them so it's 33,000 scripts an hour.

[00:15:11] [SPEAKER_01]: Just another stat.

[00:15:12] [SPEAKER_01]: On average, a script will have 13.1 sheets in it.

[00:15:18] [SPEAKER_01]: So we generate enough work for us to scan between 3.5

[00:15:22] [SPEAKER_01]: and 4 million sheets of paper in a 24-hour period.

[00:15:26] [SPEAKER_01]: And we do that consistently over the summer

[00:15:30] [SPEAKER_01]: which means ultimately we'll scan about 146 million sheets of paper

[00:15:35] [SPEAKER_01]: for an average summer series at the moment.

[00:15:37] [SPEAKER_01]: And we have to do that using a number of people

[00:15:42] [SPEAKER_01]: and over a number of shifts.

[00:15:44] [SPEAKER_01]: So for summer 2024, we have about 1,000 people,

[00:15:49] [SPEAKER_01]: the vast majority of which are temporary workers in the building.

[00:15:52] [SPEAKER_01]: We work six days a week, Monday to Saturday,

[00:15:55] [SPEAKER_01]: and we run three shifts depending on which function you're fulfilling.

[00:15:59] [SPEAKER_01]: We have day shift and night shift which are the main processes.

[00:16:03] [SPEAKER_01]: So right from the back door through to the scanning.

[00:16:05] [SPEAKER_01]: And then we have day shift and twilight shift

[00:16:09] [SPEAKER_01]: for the more electronic processes that we saw upstairs.

[00:16:12] [SPEAKER_01]: We tend not to do electronic processes at night

[00:16:15] [SPEAKER_01]: because again, we know that quality is a challenge

[00:16:18] [SPEAKER_01]: for that type of thing when people are tired at 2, 3 and more.

[00:16:21] [SPEAKER_04]: It was so interesting to think.

[00:16:23] [SPEAKER_04]: And it's six weeks, isn't it?

[00:16:25] [SPEAKER_04]: You were saying that you have about that number of employers in the building.

[00:16:29] [SPEAKER_01]: It varies slightly depending on exam dates when they exactly start.

[00:16:34] [SPEAKER_01]: It's about six to eight weeks is probably about the right period of time.

[00:16:39] [SPEAKER_04]: What are your main challenges?

[00:16:40] [SPEAKER_04]: Obviously one of them was getting the number of employees

[00:16:43] [SPEAKER_04]: that you need to do the work.

[00:16:45] [SPEAKER_04]: What are the other challenges, do you think?

[00:16:47] [SPEAKER_01]: I think there are a number of challenges that come with scale.

[00:16:51] [SPEAKER_01]: When you've got that number of people in a building at the same time,

[00:16:57] [SPEAKER_01]: keeping them all busy and focused is always going to be a challenge.

[00:17:03] [SPEAKER_01]: I am really lucky that the team that surround me that work here permanently

[00:17:08] [SPEAKER_01]: have over the years become experts at that.

[00:17:11] [SPEAKER_01]: They really know how to manage and motivate these teams of people.

[00:17:16] [SPEAKER_02]: Could you talk a bit about how you prepare people to do these roles as well?

[00:17:20] [SPEAKER_02]: The training, you were saying something about

[00:17:22] [SPEAKER_02]: you don't necessarily expect people to be productive immediately.

[00:17:26] [SPEAKER_02]: You have a process of getting them to that.

[00:17:28] [SPEAKER_01]: Again, that will vary depending on the exact task they're asked to fulfil.

[00:17:32] [SPEAKER_01]: In broad terms, we will bring supervisors in first and early.

[00:17:38] [SPEAKER_01]: We will train the supervisors to the relevant level.

[00:17:42] [SPEAKER_01]: Quite often they're the returners.

[00:17:44] [SPEAKER_01]: They may have been an operator last year and they've come back as a supervisor.

[00:17:47] [SPEAKER_01]: They've already got that basic understanding which is great for us.

[00:17:50] [SPEAKER_01]: On day one for an operator, we wouldn't really assume they're productive at all

[00:17:54] [SPEAKER_01]: because we're bringing them in doing the induction.

[00:17:57] [SPEAKER_04]: That's nice for them as well, isn't it?

[00:17:59] [SPEAKER_04]: To give them time to really understand the process.

[00:18:03] [SPEAKER_04]: It is.

[00:18:04] [SPEAKER_01]: We instil on them in that first day the importance of what they're being asked to do.

[00:18:10] [SPEAKER_01]: Everybody's asked to look upon these exam scripts as if they've done them

[00:18:13] [SPEAKER_01]: and family members have done them, friends have done them.

[00:18:16] [SPEAKER_01]: These are irreplaceable documents.

[00:18:19] [SPEAKER_01]: They're also told on day one that quality is the absolute focus of what we do.

[00:18:26] [SPEAKER_01]: We want them to slow down, take their time.

[00:18:30] [SPEAKER_01]: We don't expect people to be fully productive until about day eight.

[00:18:34] [SPEAKER_01]: It can be a ramp as long as eight days.

[00:18:38] [SPEAKER_01]: Consistently we will tell people we would much rather they stopped,

[00:18:42] [SPEAKER_01]: put their hand up if they weren't sure, wait for their supervisor.

[00:18:46] [SPEAKER_01]: Nobody's going to get in trouble for asking the right questions.

[00:18:49] [SPEAKER_04]: I think you can see why so many come back every year as well

[00:18:52] [SPEAKER_04]: because that's just such a nice culture to work in, isn't it?

[00:18:55] [SPEAKER_04]: Knowing that if you're unsure, you ask.

[00:18:58] [SPEAKER_04]: You're not going to be penalised for not being quick enough.

[00:19:01] [SPEAKER_04]: That's quite something, isn't it?

[00:19:03] [SPEAKER_02]: Do you find that you need to refresh people as well?

[00:19:06] [SPEAKER_02]: Obviously because there can be quite long gaps, can't there, I suppose, between...

[00:19:09] [SPEAKER_04]: And changes, all those incremental changes you were saying about the feedback of the session season.

[00:19:15] [SPEAKER_01]: We are really conscious that those people come back and they remember last year's processes.

[00:19:20] [SPEAKER_01]: Whilst we don't necessarily fundamentally change the process,

[00:19:23] [SPEAKER_01]: we do drive quality and efficiency through continuous improvement programmes.

[00:19:28] [SPEAKER_01]: You saw downstairs, we look at absolutely everything, even to the point,

[00:19:33] [SPEAKER_01]: I think I mentioned at the guillotine that the team had highlighted a challenge

[00:19:37] [SPEAKER_01]: with the way the papers drop into the guillotine

[00:19:39] [SPEAKER_01]: and there's a small hole for treasury tags in the top of the paper

[00:19:43] [SPEAKER_01]: which by chance lined up with the way we need to add compressed air to process it.

[00:19:49] [SPEAKER_01]: By moving that we got a 27% increase in productivity.

[00:19:52] [SPEAKER_01]: Which is just huge.

[00:19:53] [SPEAKER_01]: But it's just enormous.

[00:19:55] [SPEAKER_01]: So we bring people back in and the reason we start slowly

[00:20:01] [SPEAKER_01]: is because everybody, regardless of their history with us, is retrained and refreshed.

[00:20:07] [SPEAKER_01]: We make sure they're all following the processes as closely as they can.

[00:20:11] [SPEAKER_04]: I love that attention to what people are saying on the floor.

[00:20:15] [SPEAKER_04]: This actually could be a bit better and then it being taken by you guys

[00:20:20] [SPEAKER_04]: to make sure that next year you do that.

[00:20:22] [SPEAKER_04]: That's really empowering for the workers, isn't it, as well,

[00:20:24] [SPEAKER_04]: to know that if they can see a way of making things a little bit better

[00:20:28] [SPEAKER_04]: that they'll be listened to and things taken on board.

[00:20:31] [SPEAKER_01]: I think it's a really valuable point.

[00:20:33] [SPEAKER_01]: The whole lessons learned, post summer review,

[00:20:37] [SPEAKER_01]: constant continuous improvement reviews that we do,

[00:20:39] [SPEAKER_01]: it's really important to us here in Milton Keynes

[00:20:41] [SPEAKER_01]: but it's also important across the whole business.

[00:20:43] [SPEAKER_01]: This isn't just a Milton Keynes thing, this is an AQA thing.

[00:20:48] [SPEAKER_01]: We talked about a little bit downstairs around the impact on exams officers

[00:20:53] [SPEAKER_01]: and I was lucky to spend some time with Gemma Miller yesterday

[00:20:55] [SPEAKER_01]: at a centre in Leicestershire.

[00:20:59] [SPEAKER_01]: We want to understand how our processes work side by side with exams officers

[00:21:05] [SPEAKER_01]: and what we can do to improve.

[00:21:06] [SPEAKER_01]: So it's really worthwhile listening to what others have to say in that regard.

[00:21:13] [SPEAKER_04]: John's going to do everything much better now after this.

[00:21:16] [SPEAKER_02]: Yeah, I'll definitely write my numbers on my labels.

[00:21:19] [SPEAKER_02]: I was also thinking, as an exams officer coming here,

[00:21:23] [SPEAKER_02]: what a valuable experience it is as well and how interesting it is for me.

[00:21:27] [SPEAKER_02]: I think a lot of exams officers, they're almost like running

[00:21:30] [SPEAKER_02]: their own mini logistics operation.

[00:21:33] [SPEAKER_02]: That's kind of what they do.

[00:21:35] [SPEAKER_02]: And then to come here and see such a large logistical operation,

[00:21:40] [SPEAKER_02]: there are things that obviously we can take from each other.

[00:21:46] [SPEAKER_02]: And logistics really is about refining processes.

[00:21:49] [SPEAKER_01]: And it's really interesting because yesterday's conversation in Leicestershire

[00:21:52] [SPEAKER_01]: primarily was on logistics and the themes were time and space.

[00:21:58] [SPEAKER_01]: Essentially they were the two big ones.

[00:22:00] [SPEAKER_01]: But it's all logistical.

[00:22:01] [SPEAKER_01]: How do I get candidates in and out of the right rooms

[00:22:04] [SPEAKER_01]: and how do I spread my invigilators far enough

[00:22:06] [SPEAKER_01]: and how far in advance should I be getting papers from the award

[00:22:10] [SPEAKER_01]: and organisations versus where am I going to store them?

[00:22:12] [SPEAKER_01]: It was a whole conversation.

[00:22:14] [SPEAKER_01]: And that's why we've kind of recently recruited Gemma Miller,

[00:22:17] [SPEAKER_01]: who I mentioned a moment ago.

[00:22:19] [SPEAKER_01]: She's our new exams officer engagement manager.

[00:22:25] [SPEAKER_01]: And she's got now a team of regional engagement officers

[00:22:28] [SPEAKER_01]: that are going to spend much more time out in the centres

[00:22:31] [SPEAKER_01]: and liaising with exams officers.

[00:22:33] [SPEAKER_01]: And so they should provide support, guidance, training, those types of things.

[00:22:36] [SPEAKER_01]: But also be the voice of the customer for AQA.

[00:22:39] [SPEAKER_01]: So she'll be bringing back with her team information that helps us.

[00:22:43] [SPEAKER_01]: And I think what's really interesting about that team

[00:22:45] [SPEAKER_01]: is they've got a wealth of knowledge

[00:22:47] [SPEAKER_01]: because all of them are previously exams officers.

[00:22:49] [SPEAKER_01]: So they know some of that pain already.

[00:22:52] [SPEAKER_01]: And they're already starting to make some changes.

[00:22:54] [SPEAKER_01]: Some stuff in the background is already changing that you'll start to see.

[00:22:59] [SPEAKER_01]: Broadly the company will share more information

[00:23:01] [SPEAKER_01]: about what Gemma's team will do with schools in the new academic year.

[00:23:04] [SPEAKER_01]: So we'll kind of do a bit of a launch on what's happening.

[00:23:08] [SPEAKER_01]: But yeah, I think it again,

[00:23:11] [SPEAKER_01]: it's an investment in trying to drive ultimately the service we provide.

[00:23:17] [SPEAKER_01]: That's what we want to do.

[00:23:20] [SPEAKER_04]: Can everyone feel part of something bigger as well?

[00:23:22] [SPEAKER_02]: Yeah, and all part of the same process and the same system.

[00:23:28] [SPEAKER_02]: Rather than feeling like you've got the schools over here

[00:23:29] [SPEAKER_02]: and the exam boards over here. It's like that.

[00:23:31] [SPEAKER_04]: All got the same aim.

[00:23:32] [SPEAKER_02]: Yeah, that connection together that we'll all try

[00:23:34] [SPEAKER_02]: to ultimately achieve the same thing.

[00:23:44] [SPEAKER_01]: So I think if we start with security, you're both wearing badges.

[00:23:49] [SPEAKER_01]: You'll see across the site you can't get through doors

[00:23:53] [SPEAKER_01]: unless you've got the right access.

[00:23:55] [SPEAKER_01]: So we are really careful with security.

[00:23:57] [SPEAKER_01]: Everybody that works here permanently has ID badges that do the same.

[00:24:01] [SPEAKER_04]: I think one thing that I'm particularly impressed by here as well

[00:24:08] [SPEAKER_04]: is that security element, but also in terms of the building.

[00:24:12] [SPEAKER_04]: The security of those exam papers because I think you obviously showed us

[00:24:18] [SPEAKER_04]: the fire safety, flood defences, et cetera here, which are just astonishing.

[00:24:24] [SPEAKER_04]: And I think for exams officers, no, but schools to know

[00:24:28] [SPEAKER_04]: and also students in exams to know that the care that is taken

[00:24:33] [SPEAKER_04]: over each piece of paper that they write anything on,

[00:24:36] [SPEAKER_04]: I find quite emotional to be honest because it's so important

[00:24:40] [SPEAKER_04]: and to think that so much money is invested back in

[00:24:44] [SPEAKER_04]: to make sure that they are secure.

[00:24:46] [SPEAKER_02]: It's something that honestly when you started talking about it

[00:24:50] [SPEAKER_02]: to us down there, it hadn't even occurred to me.

[00:24:56] [SPEAKER_02]: But of course it's obvious, isn't it?

[00:24:57] [SPEAKER_02]: Could you just talk us through, Mark, the resilience that you've built

[00:25:01] [SPEAKER_02]: within the building to ensure that you keep the exam papers as safe as possible?

[00:25:08] [SPEAKER_01]: So obviously the exam papers are absolutely key within this process.

[00:25:15] [SPEAKER_01]: And we're really conscious around fire, flood, anything actually that will affect them.

[00:25:22] [SPEAKER_01]: As with all businesses, the people within this building are our primary concern.

[00:25:27] [SPEAKER_01]: So safety first, absolutely.

[00:25:29] [SPEAKER_01]: But we can't just go and reorder exam papers.

[00:25:34] [SPEAKER_01]: And so some of the resilience we've got in here,

[00:25:36] [SPEAKER_01]: we've put in here a fire suppression system,

[00:25:41] [SPEAKER_01]: which works by adding a naturally occurring gas

[00:25:45] [SPEAKER_01]: which doesn't harm humans, paper, equipment or the environment.

[00:25:49] [SPEAKER_01]: They're all naturally occurring into a particular zone or space

[00:25:54] [SPEAKER_01]: should there be a fire.

[00:25:55] [SPEAKER_01]: And all it does is it reduces the oxygen level from about, I think,

[00:26:01] [SPEAKER_01]: 15% to about 11%.

[00:26:03] [SPEAKER_01]: At that point, combustion can't happen.

[00:26:04] [SPEAKER_01]: So we put any fire out.

[00:26:07] [SPEAKER_01]: But humans can still breathe.

[00:26:10] [SPEAKER_01]: So we've got that and we know that that works.

[00:26:12] [SPEAKER_01]: It's been fully tried and tested.

[00:26:15] [SPEAKER_01]: We've also got two completely separate server rooms in this building,

[00:26:19] [SPEAKER_01]: two completely separate internet lines coming into the building.

[00:26:23] [SPEAKER_01]: We've got a generator.

[00:26:25] [SPEAKER_01]: So at the front of the building, there's a box which is a plug and play really

[00:26:29] [SPEAKER_01]: for a huge generator.

[00:26:31] [SPEAKER_01]: And we have that on contract.

[00:26:33] [SPEAKER_01]: So within four hours of us losing power,

[00:26:36] [SPEAKER_01]: we would be back up and running because again, we can't move deadlines.

[00:26:40] [SPEAKER_01]: We've got to get this work done.

[00:26:41] [SPEAKER_04]: It's basically the place to be with this apocalypse.

[00:26:44] [SPEAKER_01]: I mean, genuinely when trying to design this building,

[00:26:49] [SPEAKER_01]: we tried to look for every possible place where we would have a problem,

[00:26:54] [SPEAKER_01]: even to the point where I said we were on the first floor at the moment

[00:26:57] [SPEAKER_01]: and where we've got kitchens, toilet facilities, those types of things.

[00:27:00] [SPEAKER_01]: Underneath there, there are flood detection points.

[00:27:04] [SPEAKER_01]: And if there is a leak from a sink, from a toilet,

[00:27:09] [SPEAKER_01]: then the flood detection system will stop all the water on the first floor

[00:27:13] [SPEAKER_01]: so that it won't flood the ground floor where all the scripts are.

[00:27:17] [SPEAKER_01]: And so we've gone to that level to make sure we never let anybody down.

[00:27:23] [SPEAKER_02]: That is absolutely fantastic.

[00:27:25] [SPEAKER_02]: Can I ask one more question?

[00:27:28] [SPEAKER_02]: Because obviously you have this very efficient process

[00:27:32] [SPEAKER_02]: and that's going to deal with the vast majority of everything that comes through.

[00:27:37] [SPEAKER_02]: But what are some of the biggest sort of anomalies that you'll experience here

[00:27:42] [SPEAKER_02]: and how do you deal with some of those anomalies?

[00:27:45] [SPEAKER_01]: So as with all processes, we are proud to be the biggest awarding organisation in the UK

[00:27:52] [SPEAKER_01]: and we've talked about the millions and millions of scripts that we deal with.

[00:27:56] [SPEAKER_01]: But we do have anomalies in things like the completion of the name

[00:28:03] [SPEAKER_01]: and centre date and candidate date on the front of the scripts.

[00:28:06] [SPEAKER_01]: So we have an automated process that will match that script

[00:28:11] [SPEAKER_01]: to the official enrolment data that we upload into our system

[00:28:15] [SPEAKER_01]: that's come from the entry data of the centres.

[00:28:18] [SPEAKER_01]: If the candidates don't fill them in properly, we have to manually intervene in that.

[00:28:22] [SPEAKER_01]: And so that can be for a number of things.

[00:28:24] [SPEAKER_01]: It can be preferred name, not legal name.

[00:28:27] [SPEAKER_01]: It can be candidate number written incorrectly, really poor handwriting.

[00:28:32] [SPEAKER_01]: There's a number of reasons that would cause that.

[00:28:34] [SPEAKER_01]: So help and support in getting that done correctly would make a huge difference here.

[00:28:43] [SPEAKER_01]: Word process scripts are treated not as an anomaly,

[00:28:49] [SPEAKER_01]: but they have their own particular route through here.

[00:28:52] [SPEAKER_01]: And so they take up, I guess, compared to a normal exam paper or script

[00:28:56] [SPEAKER_01]: a disproportionate amount of time.

[00:28:58] [SPEAKER_04]: And as you showed us, the exam papers that might have been vomited on during the exam.

[00:29:05] [SPEAKER_04]: Yeah, I mean they're contaminated papers.

[00:29:07] [SPEAKER_04]: Hopefully we don't have too many of those.

[00:29:09] [SPEAKER_01]: So we do get some odd things occasionally.

[00:29:14] [SPEAKER_01]: We call them contaminated.

[00:29:15] [SPEAKER_01]: You're going to get that in a contaminated place.

[00:29:17] [SPEAKER_08]: Scrypts of teenagers are going to get to be trusted.

[00:29:19] [SPEAKER_01]: You'll sometimes get damaged scripts come through.

[00:29:22] [SPEAKER_01]: You'll sometimes get contaminated scripts come through.

[00:29:25] [SPEAKER_01]: You'll occasionally get odd things come in an envelope.

[00:29:28] [SPEAKER_01]: I mean the oddest thing I ever saw in an envelope,

[00:29:33] [SPEAKER_01]: and this is probably going back five or six years, was a ten pound note.

[00:29:38] [SPEAKER_01]: Absolutely no reason.

[00:29:39] [SPEAKER_05]: Thank you to the exam prior.

[00:29:42] [SPEAKER_01]: Again, we have a process.

[00:29:44] [SPEAKER_01]: So the centre was contacted with various bits and pieces.

[00:29:48] [SPEAKER_01]: But you'll be surprised.

[00:29:49] [SPEAKER_01]: It's those little odd things that really slow the process down.

[00:29:53] [SPEAKER_01]: Really slow the process down.

[00:29:56] [SPEAKER_04]: It's quite nice, a ten pound note rather than what it could have been though.

[00:30:00] [SPEAKER_04]: Like my went to different places.

[00:30:02] [SPEAKER_01]: On attendance registers when they're returned, they're two part.

[00:30:06] [SPEAKER_01]: The idea of being two part is that one part is retained by the centre and more types returned.

[00:30:12] [SPEAKER_01]: But if the centre returns both parts we have to have people separate it.

[00:30:15] [SPEAKER_01]: So you're getting it slowly down.

[00:30:17] [SPEAKER_01]: I've definitely done that quite a few times.

[00:30:18] [SPEAKER_04]: Have you done that job? It's all coming out.

[00:30:20] [SPEAKER_02]: Well it's not a job, you're just working quickly.

[00:30:23] [SPEAKER_04]: Now you know the impact.

[00:30:25] [SPEAKER_04]: Someone else for your mistakes has had to deal with that.

[00:30:28] [SPEAKER_04]: Thank you so much.

[00:30:30] [SPEAKER_01]: That's alright, you're welcome.

[00:30:31] [SPEAKER_04]: Your time and wisdom.

[00:30:34] [SPEAKER_04]: Fascinating.

[00:30:36] [SPEAKER_02]: That was a lot cooler than we thought it was going to be, wasn't it?

[00:30:38] [SPEAKER_02]: It's awesome.

[00:30:39] [SPEAKER_02]: Yeah, it was really good.

[00:30:42] [SPEAKER_02]: The thing that we were really impressed by was what had been built into the building

[00:30:48] [SPEAKER_02]: to stop anything going wrong and to stop the exam papers getting destroyed.

[00:30:54] [SPEAKER_02]: That was the...

[00:30:55] [SPEAKER_04]: Yeah, the level of protection for the exam papers basically like a second only to human life.

[00:31:00] [SPEAKER_02]: I thought it was just brilliant.

[00:31:02] [SPEAKER_02]: Yeah, it was fabulous.

[00:31:04] [SPEAKER_02]: Fascinating.

[00:31:05] [SPEAKER_02]: That process for getting the papers and then kind of taking them apart, scanning them,

[00:31:12] [SPEAKER_02]: that's now a kind of universal process across the exam boards.

[00:31:17] [SPEAKER_04]: Yeah, with some differences but broadly.

[00:31:19] [SPEAKER_02]: Yeah, yeah.

[00:31:21] [SPEAKER_02]: And the benefit really comes for the next stage as well which is the marking stage

[00:31:26] [SPEAKER_02]: because it means that all the marking is then taken online.

[00:31:30] [SPEAKER_02]: So I think this whole process really does produce an amazing mix of kind of the physical with the digital.

[00:31:38] [SPEAKER_02]: So from this very physical stage that we just witnessed,

[00:31:44] [SPEAKER_02]: we move through the scanning process into an entirely digital process and that is the marking.

[00:31:52] [SPEAKER_04]: Yeah, it didn't have the foggiest how this works either.

[00:31:55] [SPEAKER_02]: Well, I've seen mark schemes so I know such a thing exists so that when an examiner is marking

[00:32:00] [SPEAKER_02]: they've got a kind of standard that they're meant to mark too.

[00:32:03] [SPEAKER_04]: So we thought we'd speak to someone who knew a little bit more about the process than we did.

[00:32:09] [SPEAKER_04]: So we were delighted to speak to Ruth Bradbury.

[00:32:12] [SPEAKER_04]: Ruth is not only an experienced English examiner but also an experienced head teacher

[00:32:18] [SPEAKER_04]: and finance director of Multi Academy Trust so she really knows her stuff.

[00:32:23] [SPEAKER_04]: What's particularly interesting is that Ruth is also an examiner for two of the awarding bodies

[00:32:28] [SPEAKER_04]: so she's really got that breadth, hasn't she, and depth of experience in this area

[00:32:32] [SPEAKER_04]: and we had lots of questions for her and yeah, she taught us a lot.

[00:32:37] [SPEAKER_02]: She handled them well.

[00:32:45] [SPEAKER_07]: I've done the bullet point list in preparation.

[00:32:51] [SPEAKER_07]: But also for the first time this year I marked for two boards

[00:32:55] [SPEAKER_07]: and the difference between the approaches of the two boards is really, really fascinating.

[00:33:02] [SPEAKER_07]: Oh wow, that's interesting.

[00:33:04] [SPEAKER_07]: So yeah, I'll talk about Borde and Bordby.

[00:33:07] [SPEAKER_07]: Oh yeah, good.

[00:33:08] [SPEAKER_02]: I was about to say please don't...

[00:33:11] [SPEAKER_02]: We've been instructed that examiners shouldn't talk about the boards that they mark for

[00:33:16] [SPEAKER_02]: so I'm glad that you've taken this approach, Ruth.

[00:33:19] [SPEAKER_07]: From the description, Borde and probably Bordby will easily recognise themselves though

[00:33:25] [SPEAKER_07]: but that's their problem.

[00:33:29] [SPEAKER_07]: So Borde, which is one that I have been examining since it started

[00:33:35] [SPEAKER_07]: and the reason that I am an examiner for it is that...

[00:33:42] [SPEAKER_07]: I'm going back and rambling a bit now but at studio school I only had two English teachers.

[00:33:48] [SPEAKER_07]: One of them I had to suspend and one of them was on long term six

[00:33:52] [SPEAKER_07]: so I basically taught all year 11 English in the studio school for half a year

[00:33:58] [SPEAKER_07]: and I thought well the best way of getting to understand this is to do the training

[00:34:04] [SPEAKER_07]: to become a marker for the specification.

[00:34:07] [SPEAKER_07]: So when you're a new person and I've been doing it like I say

[00:34:10] [SPEAKER_07]: since this specification began, you get your sort of training.

[00:34:17] [SPEAKER_07]: There are lots of videos.

[00:34:18] [SPEAKER_07]: There's some training that is mandatory, the safeguarding training

[00:34:22] [SPEAKER_07]: and there's the GDPR training which everybody has to do

[00:34:27] [SPEAKER_07]: and they won't let you mark until you've done those.

[00:34:31] [SPEAKER_07]: Because I've been marking it for so long, I'm pretty clued up in terms of how to mark that exam paper

[00:34:38] [SPEAKER_07]: but obviously we don't just let you do that and go on and mark.

[00:34:42] [SPEAKER_07]: So what happens with Borde is that there's a period of online standardisation

[00:34:47] [SPEAKER_07]: where you log in to a kind of dummy marking environment

[00:34:54] [SPEAKER_07]: and first thing I say is the marking for both Borde and Bord B is electronic

[00:35:00] [SPEAKER_07]: so there are no papers being delivered anywhere which is wonderful.

[00:35:06] [SPEAKER_07]: It makes it an awful lot easier.

[00:35:09] [SPEAKER_07]: So what happens on the online standardisation for the first board

[00:35:14] [SPEAKER_07]: is that you have a window of about 48 hours where you have to go onto the system

[00:35:20] [SPEAKER_07]: and mark usually eight scripts.

[00:35:26] [SPEAKER_07]: There are five questions in total.

[00:35:29] [SPEAKER_07]: So you have to mark eight of each of the five questions

[00:35:33] [SPEAKER_07]: and enter your marks and then it's a bit like a computer game.

[00:35:39] [SPEAKER_07]: It's quite exciting.

[00:35:40] [SPEAKER_07]: You then click and see how many you've got close to what the chief examiner and colleagues

[00:35:50] [SPEAKER_07]: who will do the standardisation together and the team leaders

[00:35:54] [SPEAKER_07]: and then they between them will allocate a score or a mark to each question

[00:36:00] [SPEAKER_07]: and then you do the standardisation and it shows how close you've got to what the allocated mark is.

[00:36:10] [SPEAKER_07]: You then have a conversation with your team leader.

[00:36:13] [SPEAKER_07]: You're notified of your team leader usually a few weeks before the exam season starts

[00:36:19] [SPEAKER_07]: and once you've done the online standardisation you have to fill them up

[00:36:24] [SPEAKER_07]: and you talk through each one where you have diverged significantly from what the kind of the awarded grade is

[00:36:37] [SPEAKER_07]: and then I mean I'm usually quite close.

[00:36:41] [SPEAKER_07]: I don't usually get for this paper.

[00:36:43] [SPEAKER_07]: I don't usually get very much wrong.

[00:36:46] [SPEAKER_07]: Then they clear you to mark.

[00:36:49] [SPEAKER_07]: Then the marking which they have a schedule so you'll get a kind of itinerary

[00:36:56] [SPEAKER_07]: which says when marking starts, when the deadline is to do 25% of your allocation,

[00:37:01] [SPEAKER_07]: when the deadline is to do 66% allocation and when the deadline is to have done all of your allocation.

[00:37:10] [SPEAKER_07]: So you get that when marking then they email you to say when it's live.

[00:37:16] [SPEAKER_07]: They have their own software which you log into and you use the mark scheme for this example Borde.

[00:37:27] [SPEAKER_07]: You have to annotate them with extracts from the mark scheme that clarify which level.

[00:37:36] [SPEAKER_07]: We don't use grades so it's not a 9 to 1.

[00:37:42] [SPEAKER_07]: It is levels and you'll know this if you've looked at mark schemes in the past.

[00:37:48] [SPEAKER_07]: So for English a lot of the answers because if you've got your kind of normal distribution curve,

[00:37:55] [SPEAKER_07]: a lot of the answers are around the top of level 2 and the bottom of level 3.

[00:38:01] [SPEAKER_07]: This is by far where most of them are.

[00:38:04] [SPEAKER_07]: But you have to be really clear about the annotations.

[00:38:08] [SPEAKER_07]: You can drag and drop mark scheme quotes onto the exam paper in the marking software

[00:38:16] [SPEAKER_07]: so that you can clearly demonstrate exactly why you have come to the decision about the grade that you've awarded.

[00:38:25] [SPEAKER_02]: So that's to ensure that the marking is always firmly aligned with the mark scheme.

[00:38:32] [SPEAKER_02]: That there's no deviation.

[00:38:34] [SPEAKER_02]: Yes, that's right.

[00:38:36] [SPEAKER_07]: And in terms of making sure you don't go bog or maz or whatever,

[00:38:43] [SPEAKER_07]: they have what they call seeds.

[00:38:47] [SPEAKER_07]: So when you first log on, you mark question by question.

[00:38:50] [SPEAKER_07]: So I'd say, you know, I'm going to spend this morning doing question four and then they call them clips.

[00:38:58] [SPEAKER_07]: You basically go through clip by clip and market.

[00:39:02] [SPEAKER_07]: But when you first log on to mark a particular question, there are some qualifying seeds at the start.

[00:39:10] [SPEAKER_07]: I don't know why they're called seeds.

[00:39:11] [SPEAKER_07]: I didn't know what they were for a long time and it really confused me.

[00:39:15] [SPEAKER_07]: It does conjure up quite an image, doesn't it?

[00:39:18] [SPEAKER_07]: Yes, they are pre-marked questions that have been marked by the chief examiner and or other senior people.

[00:39:30] [SPEAKER_07]: And if you are outside of the tolerance and depending on the number of marks available for the question,

[00:39:38] [SPEAKER_07]: there is a different level of tolerance.

[00:39:41] [SPEAKER_07]: So, for example, if a question were eight marks, you could be one mark out with a seed and it's fine.

[00:39:48] [SPEAKER_07]: But if you're two marks out and if you get two, if you fail two seeds and you don't know which the seeds are,

[00:39:57] [SPEAKER_07]: what you do if you've marked a thousand of the damn things because you start recognising them.

[00:40:03] [SPEAKER_07]: Initially, you don't know who the seeds are.

[00:40:06] [SPEAKER_07]: And if you fail two, then you get stopped from marking that question and you have to have a conversation with your team leader.

[00:40:15] [SPEAKER_07]: You then need to go on, look at your answer, compare it to the examiner's answer and talk through the differences.

[00:40:27] [SPEAKER_07]: First time I did it, I got stopped loads.

[00:40:29] [SPEAKER_07]: This year I got stopped once on three questions and not at all on two questions.

[00:40:40] [SPEAKER_07]: And I actually did over a thousand of one particular question and get sort of stopped once.

[00:40:47] [SPEAKER_07]: Yeah, I know. I think they don't do medals but it would be nice if they did that.

[00:40:51] [SPEAKER_06]: Yeah, you should. Yeah.

[00:40:53] [SPEAKER_03]: Right, we'll suggest that for you.

[00:40:55] [SPEAKER_02]: Would you put that down to your success rate there, Ruth?

[00:40:56] [SPEAKER_02]: Would you put that down to purely, is that down to experience?

[00:41:00] [SPEAKER_02]: Like you've done this so many times that you just now it's kind of like become like a second nature to you to know what to do.

[00:41:08] [SPEAKER_07]: Yes. And I think largely experience there is one question that I,

[00:41:16] [SPEAKER_07]: some people absolutely hate and can never get their head around it.

[00:41:20] [SPEAKER_07]: But I absolutely love it because it is, you know,

[00:41:26] [SPEAKER_07]: it's one that once you understand how to market, it's very easy to define the level.

[00:41:33] [SPEAKER_07]: And it's also worth a lot of bark.

[00:41:36] [SPEAKER_07]: It's also very easy at the same time to determine which level it sits in and which mark it should get.

[00:41:44] [SPEAKER_07]: So that's why.

[00:41:44] [SPEAKER_02]: Ruth, so if you're assigning, say it's like an eight mark question and you're assigning a certain level to it.

[00:41:52] [SPEAKER_02]: How does it, how do we then get to a point where that question, that student's answer gets awarded a mark out of eight?

[00:41:58] [SPEAKER_02]: How does that work?

[00:41:59] [SPEAKER_07]: That's the bit where you use judgment, I suppose.

[00:42:03] [SPEAKER_07]: Because yeah, you know, that's the bit where you say, well actually this, you know, this level is 13 to 15 marks.

[00:42:09] [SPEAKER_07]: So I think it's one that's just grade 10.

[00:42:11] [SPEAKER_07]: So I'll give it 13 rather than 15.

[00:42:13] [SPEAKER_02]: Got it. Got it. Got it. OK, that makes good sense.

[00:42:16] [SPEAKER_02]: OK, so then what once you've once you've done your marking, what kind of happens then?

[00:42:23] [SPEAKER_02]: Is that the end of the process or is there more more to do?

[00:42:27] [SPEAKER_07]: What happens then is that when all when we meet the deadline for 100 percent,

[00:42:35] [SPEAKER_07]: there are always loads left because some people drop out.

[00:42:40] [SPEAKER_07]: I think they don't necessarily allocate all of them.

[00:42:44] [SPEAKER_07]: Then they have the day after marking the 100 percent deadline,

[00:42:49] [SPEAKER_07]: they then have what is called cap removal, which is where basically anyone who's a marker can log on

[00:42:57] [SPEAKER_07]: and choose which questions to mark and mark them.

[00:43:02] [SPEAKER_07]: And then you get paid for those.

[00:43:04] [SPEAKER_07]: So it's a bit of a free for all this year.

[00:43:07] [SPEAKER_07]: It wasn't as crazy as it has been in previous years.

[00:43:10] [SPEAKER_07]: But it was still I mean, I finished marking last week, even though the deadline was about three weeks before that.

[00:43:19] [SPEAKER_07]: Because and the exam board will email you every day with how many or what proportion of each question has been marked

[00:43:26] [SPEAKER_07]: and suggestions for which ones to focus on so that they get cleared.

[00:43:31] [SPEAKER_07]: It seemed less frantic this year than it has in previous years.

[00:43:36] [SPEAKER_02]: Yeah. Yeah. OK.

[00:43:37] [SPEAKER_02]: Could you talk us through board B then, Ruth?

[00:43:41] [SPEAKER_07]: Yes, that is a completely different culture.

[00:43:47] [SPEAKER_07]: I think partly because it is a board that has fewer entries historically.

[00:43:53] [SPEAKER_07]: OK. So so rather than the online standardization,

[00:43:59] [SPEAKER_07]: what they do is that there's a full day meeting of everyone who is marking the paper on teams

[00:44:10] [SPEAKER_07]: where they go through the standardization scripts and individual team leaders have marked them

[00:44:21] [SPEAKER_07]: and discussed them with colleagues.

[00:44:23] [SPEAKER_07]: They talk through the grading and it's a very long day.

[00:44:28] [SPEAKER_07]: But at the end of it, you feel like you really clearly understand what's needed.

[00:44:34] [SPEAKER_07]: Then when you mark, there's no annotations required.

[00:44:39] [SPEAKER_07]: You have to read the whole question and then give a mark based on how you feel where it sits in the mark scheme.

[00:44:50] [SPEAKER_07]: So whereas board A is requiring you to justify every grade, presumably in case there's a remark request,

[00:44:57] [SPEAKER_07]: board B absolutely doesn't do that.

[00:45:01] [SPEAKER_07]: The only thing you have to do is type in a number and then there aren't any seeds.

[00:45:08] [SPEAKER_07]: So again, it's a very different feel to it.

[00:45:12] [SPEAKER_07]: But what happens is your team leader will dip into your marking.

[00:45:18] [SPEAKER_07]: So they will talk through with you.

[00:45:21] [SPEAKER_07]: They'll send you a message saying it looks like you're going too high on this particular question

[00:45:27] [SPEAKER_07]: or you're going too low on this particular question.

[00:45:30] [SPEAKER_07]: And I'm intrigued how that works with remarks because basically all there is is a script and a mark.

[00:45:40] [SPEAKER_07]: Whereas with the first boards, you could come if you've got an audit trail or whatever you want to call it.

[00:45:46] [SPEAKER_07]: And then board B, that's the same in terms of having deadlines for percentages.

[00:45:53] [SPEAKER_07]: And then it's the same having the cap removal as well where you then end up marking free for all for whatever.

[00:45:59] [SPEAKER_07]: So there was a point when I was doing the free for all for board B, whilst also doing the standard marking phase for board A.

[00:46:08] [SPEAKER_04]: That is so interesting, particularly about the differences between the boards' work and the process.

[00:46:14] [SPEAKER_02]: Do you think that there's a bit of that to do with the question of scale?

[00:46:17] [SPEAKER_02]: So you talked about one board handling a lot more than the other.

[00:46:20] [SPEAKER_02]: Do you think the processes differ because of the scale involved?

[00:46:24] [SPEAKER_07]: I do think scale does have something to do with it.

[00:46:26] [SPEAKER_07]: I mean, it's obviously easier to review and remark if you've got 40,000 entries rather than half a million or whatever it is.

[00:46:35] [SPEAKER_02]: The boards are very careful now aren't they as well with post results to make sure that it's called a review of marking and not a remark.

[00:46:44] [SPEAKER_02]: So if you imagine board A, what they're doing I guess is reviewing the processes aren't they?

[00:46:49] [SPEAKER_02]: They're reviewing that all the processes were followed and if all the processes were followed then.

[00:46:55] [SPEAKER_07]: And I suppose I mean the paper I mark for board B is perhaps a slightly more subjective paper with a slightly less prescriptive mark scheme.

[00:47:06] [SPEAKER_07]: Than the one I mark for board A as well.

[00:47:09] [SPEAKER_02]: Interesting, interesting.

[00:47:10] [SPEAKER_04]: So, Rosie you obviously explained a little bit about why you came to be an examiner.

[00:47:16] [SPEAKER_04]: Obviously a really creative way of understanding when you were having to teach English under those circumstances when you were a head teacher.

[00:47:26] [SPEAKER_04]: In terms of the challenges, obviously there's definitely challenges about working for two boards there and the differences.

[00:47:32] [SPEAKER_04]: What are the things that you love about the role and any other challenges as well that have come up?

[00:47:37] [SPEAKER_04]: Be interesting to hear.

[00:47:39] [SPEAKER_07]: I think what I love about it is the fact that you can do it at home on the sofa in your pajamas or anything.

[00:47:51] [SPEAKER_07]: That you get into a rhythm and I've always, it's not mindless because you're thinking but it's a very, if you've been ahead or you've been a finance director,

[00:48:02] [SPEAKER_07]: you have these kind of massive complicated tasks that will take ages and have so many different elements to them.

[00:48:10] [SPEAKER_07]: And there's something really satisfying about just clicking the mouse a few times and typing in a number and submitting it.

[00:48:18] [SPEAKER_07]: Which, you know, you're still giving it intellectual effort but it's not the exhausting level that you get in school leadership I suppose.

[00:48:27] [SPEAKER_07]: I wouldn't say it's a break but I would say that it's a nice compliment to it.

[00:48:33] [SPEAKER_07]: Yeah, I can see that.

[00:48:34] [SPEAKER_02]: It's a very tangible process isn't it? You start at the beginning, you do a set of tasks and you complete that task.

[00:48:41] [SPEAKER_02]: And I guess most jobs as a head aren't really like that are they?

[00:48:44] [SPEAKER_07]: Curve balls coming from all angles.

[00:48:47] [SPEAKER_07]: No, it's completely different so I think that's one of the reasons why I like doing it.

[00:48:51] [SPEAKER_07]: And when I was a head and when I was teaching, another reason was so I, you know, I enjoyed seeing what the standard was.

[00:49:01] [SPEAKER_07]: You don't know which school they're from, you don't know the child's name and because you're only getting them question by question,

[00:49:08] [SPEAKER_07]: those who say things like, as you can see in my answer to question three, well no sorry I can't see your answer to question three.

[00:49:15] [SPEAKER_07]: So that's probably worth something that the kids do need to know because there's always a few that do that.

[00:49:21] [SPEAKER_07]: Oh I forgot to say as well, both exam boards are very hot on safeguarding referrals.

[00:49:27] [SPEAKER_07]: And so, you know, I did the question for the exam board A paper that I marked this year,

[00:49:38] [SPEAKER_07]: lent itself to things that could have been safeguarding referrals.

[00:49:43] [SPEAKER_07]: So I did more this year than I ever have in the past.

[00:49:47] [SPEAKER_07]: And I very much hope that the board in question think about that when they set their questions next year.

[00:49:56] [SPEAKER_02]: I'm just going to explain a bit on that Ruth.

[00:49:58] [SPEAKER_02]: So as exams officers we'll often receive emails from the exam board asking us if we can look into something that a student has written in their exam paper.

[00:50:08] [SPEAKER_02]: And they will then give us kind of secure access to something, a portion of what a student has written.

[00:50:15] [SPEAKER_02]: And then we can then investigate that from a safeguarding point of view and then we report back to the board whether or not we, you know, we consider that it's fine.

[00:50:24] [SPEAKER_02]: There's no issue or whether we're investigating further.

[00:50:28] [SPEAKER_02]: Well yeah in discussion with the head of centre, well it's the head of centres decision.

[00:50:33] [SPEAKER_02]: I'm getting above my station.

[00:50:36] [SPEAKER_02]: Yeah, they will obviously speak to the head of centre about it and try and reach and safeguarding team and try and reach determination.

[00:50:46] [SPEAKER_03]: How often do you have that?

[00:50:48] [SPEAKER_02]: It happens, it definitely happens every exam season.

[00:50:52] [SPEAKER_02]: You know, you definitely get one or two and it does tend to be GCSE English tends to be the one.

[00:51:00] [SPEAKER_02]: Thank you so much.

[00:51:01] [SPEAKER_02]: It's been so insightful.

[00:51:03] [SPEAKER_02]: We understand it a bit more now, I think.

[00:51:05] [SPEAKER_02]: Yeah, we're getting there.

[00:51:06] [SPEAKER_03]: We're getting there.

[00:51:07] [SPEAKER_03]: That was definitely a big piece that was missing wasn't it?

[00:51:11] [SPEAKER_03]: In terms of our understanding.

[00:51:13] [SPEAKER_02]: My big takeaway from that was to sort of understand the pro like it's quite a methodical process.

[00:51:19] [SPEAKER_02]: And that explains to me a bit about the some of the things around what happens after results in terms of like asking for reviews of marking and what that actually then mean.

[00:51:30] [SPEAKER_02]: What a review of marking might actually look like because I think I've always been not too clear on that.

[00:51:35] [SPEAKER_02]: So that's really interesting.

[00:51:38] [SPEAKER_02]: Thank you so much for your time.

[00:51:40] [SPEAKER_04]: Thanks Ruth, that was brilliant.

[00:51:41] [SPEAKER_04]: Thank you.

[00:51:43] [SPEAKER_04]: That was amazing, wasn't it?

[00:51:45] [SPEAKER_04]: Yeah, fantastic.

[00:51:45] [SPEAKER_04]: For what we need for the episode.

[00:51:46] [SPEAKER_04]: That was brilliant.

[00:51:47] [SPEAKER_02]: Ruth, did you see the board B?

[00:51:50] [SPEAKER_07]: I know it's not.

[00:51:59] [SPEAKER_04]: So I think we've got a pretty good picture now haven't we of the of the broad process.

[00:52:06] [SPEAKER_04]: Understanding how exam papers are collected and how they're processed and scanned and then marked.

[00:52:14] [SPEAKER_04]: But what happens next John?

[00:52:16] [SPEAKER_02]: Well so now we've got to a stage where students have got marks for their individual papers.

[00:52:22] [SPEAKER_02]: So obviously all the question marks are aggregated together for a paper that gives them a mark from the paper.

[00:52:28] [SPEAKER_02]: But obviously that's not the thing that the students are looking forward to receiving on results day.

[00:52:33] [SPEAKER_02]: They're looking forward to getting their grades.

[00:52:35] [SPEAKER_02]: So the next process is to translate the marks that they get from papers into grades.

[00:52:41] [SPEAKER_02]: This is a part of the process that I think can be a bit misunderstood.

[00:52:45] [SPEAKER_02]: We are not experts at it but we're going to have a go at explaining it in this kind of the simplest terms possible.

[00:52:53] [SPEAKER_02]: Obviously every year exam papers are different so kids don't sit the same exam paper year after year.

[00:52:59] [SPEAKER_02]: That would be madness.

[00:53:01] [SPEAKER_02]: So the exam papers have to change every year and the exam boards obviously will try and do their best to ensure that the level of difficulty of the papers stays as constant as possible.

[00:53:13] [SPEAKER_02]: That's however not an easy thing to achieve and the difficulty levels will inevitably go up and down as the years go by.

[00:53:22] [SPEAKER_02]: So the remedy to that in order to try and keep the standards the same is to move the grade boundaries.

[00:53:29] [SPEAKER_02]: So for example if a student did an English exam this year that was considerably more difficult than the exam that took place last year

[00:53:39] [SPEAKER_02]: then the grade boundaries will come down a little bit in order that the standard for say a grade five is basically the same.

[00:53:47] [SPEAKER_02]: So students achieving that standard are performing at the same level this year as they did last year.

[00:53:53] [SPEAKER_02]: Does that make sense?

[00:53:53] [SPEAKER_04]: Yeah, that does. Definitely.

[00:53:57] [SPEAKER_02]: So the way these grade boundaries are determined for each exam paper is by the chief examiners from the exam boards.

[00:54:04] [SPEAKER_02]: So they make the determination about where the grade boundaries should sit and they do that by looking at exam papers from previous years.

[00:54:15] [SPEAKER_02]: And they use some other data as well.

[00:54:17] [SPEAKER_02]: For example, some of you may be familiar with the national reference test or your school may have done it this year.

[00:54:22] [SPEAKER_02]: So data from that is also used to try and locate where the right levels are for setting grade boundaries.

[00:54:30] [SPEAKER_02]: And then the process is overseen by Ofqual who are the qualifications regulator to make sure that it's fair across all the exam boards.

[00:54:40] [SPEAKER_02]: And then obviously the students marks are then translated into grades by the exam boards.

[00:54:45] [SPEAKER_02]: And then it's the process where we as exams officers get back involved again because a day before the students get their results

[00:54:53] [SPEAKER_02]: the results files with all the students grades are sent to us in digital format.

[00:54:59] [SPEAKER_02]: We download them. You stay up all night waiting for them.

[00:55:03] [SPEAKER_02]: Yeah, they come out. They do come out at midnight.

[00:55:06] [SPEAKER_02]: So if you want to, if you're particularly keen, then you can stay up and download them.

[00:55:12] [SPEAKER_02]: So one part of this process as well, which I think I should emphasize, is that obviously these exam grades are confidential, but we're getting them a day early.

[00:55:22] [SPEAKER_02]: So what we do as exams officers is we place an embargo on them.

[00:55:26] [SPEAKER_02]: And what that means is that when we download them into the school system, we can restrict who is able to power you.

[00:55:35] [SPEAKER_02]: So typically members of the senior management team and the exams officer are the only people who would be able to view those results on that day before the students get them.

[00:55:49] [SPEAKER_02]: I have heard stories of exams officers that are quite like this, who set their embargo up but failed to include themselves in it.

[00:56:00] [SPEAKER_02]: So they download all the results and then they can't see any and go into a mad state of panic.

[00:56:05] [SPEAKER_02]: So yeah, so if you're an exams officer this week, just make sure you include yourself in the embargo.

[00:56:13] [SPEAKER_02]: Thank you so much for listening to the Exam Man podcast. We really, really appreciate your support.

[00:56:17] [SPEAKER_02]: Remember that you can access it on all the major podcast platforms.

[00:56:22] [SPEAKER_02]: Give us a rating, give us a follow and we will catch you next time.