EXAMS IN THE NEWS: Digital Exams 2030?

[00:00:14] So John, exams have been in the news quite a lot this week, haven't they, again?

[00:00:19] Yeah, they have. And it's all related to digital exams and the transition from paper-based exams to digital exams, which we've spoken to some of our guests about.

[00:00:30] In particular, we talked to Michael Turner at AQA about it when we interviewed him.

[00:00:35] So it's been a bit of a confused picture this week, actually, though, although it hit the news on, I think it was Tuesday or Wednesday,

[00:00:42] so there was a front-page article in the i newspaper which basically said digital exams were going to be coming by 2030.

[00:00:49] But then there was an article later in the week which featured in Schools Week, which is a kind of sector publication,

[00:00:56] which basically said that there were some issues and that certain exam boards had been pulling their proposals to introduce digital exams.

[00:01:06] So it seems like there's quite a lot going on, really.

[00:01:10] But it's a hot topic nonetheless.

[00:01:11] It is a very hot topic and it's one that, like, it causes quite a lot of consternation amongst exams officers as well.

[00:01:19] Well, you're on the front line.

[00:01:20] We are on the front line. We would have to deliver whatever new things are introduced.

[00:01:26] So I would say, you know, I see a bit online of what exams officers think about digital exams.

[00:01:33] I would say they're not wholly enthusiastic about the prospect.

[00:01:38] If I had to summarise it, I would say something along the lines of people understand, you know, the arguments for it and the rationale behind it.

[00:01:49] But I also think they think that...

[00:01:51] Stop there. What are the, like...

[00:01:54] Oh, OK.

[00:01:54] The broad arguments for digital exams.

[00:01:57] OK. So, I mean, I guess it depends what perspective you're looking at.

[00:02:00] So you can look at it from the perspective of the student.

[00:02:02] You can look at it from the perspective of the exam boards and the system as a whole, I suppose.

[00:02:08] So from the student's point of view, I guess if you've got more students who...

[00:02:11] Increasingly, students who are used to using digital technology at school.

[00:02:16] They tend to type more than they write.

[00:02:19] Certainly for students with special educational needs, there are things that you can build into a digital assessment.

[00:02:25] So, you know, like things like being able to increase font size, change colour backgrounds, accessibility tools effectively you can build in.

[00:02:35] So, yeah, from that point of view, for the student, there are some benefits.

[00:02:44] From an exam board point of view, I think the benefits are really obvious.

[00:02:49] So at the moment they have this whole massive nationwide operation which involves trucks of papers going to schools.

[00:02:56] It involves scanning.

[00:02:58] The logistics.

[00:02:59] That's the scanning that we saw in Milton Keynes.

[00:03:01] It's a massive logistical operation which I imagine is very expensive.

[00:03:06] And obviously a lot of things that in the past were very cumbersome from that point of view have been digitalised and huge efficiency savings have been made.

[00:03:19] But digital exams will be really expensive to implement as well.

[00:03:23] Well, and what I was going to say is from the school's perspective, I think that's where it starts to get a bit murky.

[00:03:28] It's because there's obviously, you know, the way the system works currently, if you imagine, say, GCSE English exam at a school like ours,

[00:03:40] you've got 200, 250 students sitting down all at once to sit the same paper at the same time.

[00:03:46] And the security of the entire system is based on the idea that they do that.

[00:03:52] To deliver that digitally means that you have to have 250 devices, all of which are set up to have the right levels of security and all of which you can deliver power to that can't run out of power.

[00:04:09] So and then obviously the infrastructure then to be able to then send the completed exam to the exam board as well.

[00:04:15] So I think the big concern that comes from exams officers and that comes from schools generally is that is the practicality of it.

[00:04:23] Basically, it's like we would have to invest a lot of money to be able to deliver this in the current as the system currently works.

[00:04:31] And it doesn't really feel practical, I think, from a school's point of view at the moment.

[00:04:38] So those are the kinds of the sorts of issues that are there.

[00:04:44] You know, I think there are some benefits.

[00:04:46] But also the other thing I would say about the benefits, obviously, I said that, you know,

[00:04:50] an increasing number of students are working like this using digital technology.

[00:04:54] But that is quite patchy.

[00:04:56] Yeah.

[00:04:57] So if you were to move now, tomorrow, if we were to move entirely to digital,

[00:05:03] forget all the logistical problems for a second.

[00:05:06] But if we were to do it, there would be a group of students within the country who are used to using digital all the time,

[00:05:14] effectively in their learning.

[00:05:16] And then there will be some students who use it some of the time.

[00:05:20] And then some students who don't use it that much.

[00:05:24] And so if you're looking, ultimately, exams have to optimise for fairness.

[00:05:29] So if you're looking at a system which is kind of fair across the board,

[00:05:33] then that would obviously create a bit of an issue.

[00:05:37] But then I suppose, on the other hand, you could argue, well,

[00:05:40] is paper-based fair for students who are learning at the moment?

[00:05:44] Exactly.

[00:05:45] It's complex.

[00:05:46] Digitally, so it's complex.

[00:05:47] Complex.

[00:05:48] And then you get into the idea of a hybrid system where you run both.

[00:05:52] You basically give schools the option,

[00:05:53] which seems to be the language that the exam boards are using quite a lot at the moment.

[00:05:56] That makes sense.

[00:05:57] They're iterative, isn't it?

[00:05:58] Well, yeah, but I'd say there are two problems with that.

[00:06:01] One of which is that from the exam boards point of view,

[00:06:05] that actually becomes very costly.

[00:06:07] Because they're running two times.

[00:06:08] Because you're running two parallel systems.

[00:06:10] Yeah.

[00:06:10] And then also, from the point of view of the system as a whole,

[00:06:19] it starts to maybe not feel quite as fair and equitable

[00:06:23] because it doesn't feel as sort of standardised.

[00:06:25] So, you know, it might be the case that you start to feel that the students

[00:06:30] who are using the digital route have an advantage to some extent

[00:06:34] or the paper-based students have an advantage.

[00:06:36] And you're running two different systems to try and achieve what is meant to be like,

[00:06:39] you know, a kind of perfectly standardised thing.

[00:06:43] So, like, basically it's really complicated.

[00:06:47] And I think that the exam boards have been,

[00:06:51] some of them have been very cautious in the way they've talked about it.

[00:06:54] Some have been quite bullish about it.

[00:06:57] Which you understand as well.

[00:06:58] I can understand both.

[00:06:59] Because if you're putting in a, you know, a very different system,

[00:07:05] then it is quite hard to do that in a piecemeal, iterative way.

[00:07:09] Yeah, yeah.

[00:07:10] But then the costs attached to do it in a whole skill while, yeah.

[00:07:13] Yeah, and also because we have multiple exam boards,

[00:07:16] there's a problem of coordination as well.

[00:07:18] But the other thing is I can sort of understand both approaches.

[00:07:22] So I can understand the cautious thing

[00:07:23] because that demonstrates a real understanding of, like,

[00:07:27] the logistical issues and the practicalities.

[00:07:29] You can see that, can't you, in things that are coming out now.

[00:07:32] That is very much, I think it is really reflective of the examples.

[00:07:36] But I can also understand the bullish approach.

[00:07:40] Because when you want to change a system,

[00:07:44] and one that's difficult to change,

[00:07:46] sometimes setting very ambitious targets

[00:07:49] is the only way that you can get the thing moving.

[00:07:53] So 2030, probably,

[00:07:56] if you had the idea that by 2030,

[00:07:58] all exams would basically be done digitally,

[00:08:01] I think from a school's perspective right now,

[00:08:05] that feels impractical, overly ambitious.

[00:08:10] But I guess if you set that deadline,

[00:08:13] then there is the thing of, like,

[00:08:14] well, it's kind of galvanising.

[00:08:17] So, yeah, what you run the risk of with that, though,

[00:08:20] is when you set very ambitious targets,

[00:08:22] is sort of discrediting the whole process

[00:08:25] if you fail and fail badly.

[00:08:28] So, yeah.

[00:08:29] Which could be due to so many different factors in something like this.

[00:08:33] Yeah, absolutely.

[00:08:35] And the government's got to decide as well about,

[00:08:37] you know, what are the priorities in education as well.

[00:08:40] Well, that's another big thing.

[00:08:41] So, Ofqual, interestingly, said about this.

[00:08:46] And they've said this a couple of times.

[00:08:49] I noticed it a couple of years ago,

[00:08:51] coming out of the pandemic,

[00:08:52] that Ofqual,

[00:08:52] because there was a lot of noise around that time around digital exams.

[00:08:56] And I remember the chair of Ofqual

[00:09:00] saying that it wasn't the job of Ofqual, effectively,

[00:09:03] to be evangelists for technology.

[00:09:05] Like, their job was to make sure that

[00:09:06] there was a good assessment system that was fair,

[00:09:09] not to push technology for its own sake.

[00:09:14] And I think one of the things that has to be really considered here,

[00:09:19] and we got our guest on today,

[00:09:21] and this is one of the things that we spoke to him about.

[00:09:24] And I think it's something that I see creeping into these discussions more and more.

[00:09:29] It hasn't really been talked about before,

[00:09:30] but I see it creeping in more and more.

[00:09:32] But it's something that I've thought about for a long time,

[00:09:34] and it's something that some of our guests have also mentioned as well.

[00:09:38] Well, this idea of students doing all their exams together at the same time

[00:09:44] makes digital exams really difficult to implement.

[00:09:47] A different system might make them easier to implement.

[00:09:51] So a system where exams were more on demand.

[00:09:53] Like, for example, we have this with Pearson,

[00:09:56] run a set of functional skills,

[00:09:58] where you basically, you can take the test at any time

[00:10:00] when you feel the students are ready.

[00:10:02] And delivering digital like that is much easier

[00:10:05] because you're not doing all the students all at once.

[00:10:07] And you're not needing all the, you know, number of plug sockets.

[00:10:11] Yeah, exactly.

[00:10:12] You don't need the enormous levels of investment

[00:10:16] to be able to achieve that.

[00:10:18] So I think there is an argument that you can do digital assessments

[00:10:23] and that digital assessments would be good,

[00:10:25] but perhaps not with the assessment system as it currently exists.

[00:10:30] Now, the problem with that is that then you start...

[00:10:32] That's quite a change.

[00:10:33] Then you go down a massive rabbit hole

[00:10:35] because then you're talking about the whole basis of our system

[00:10:40] and how it works.

[00:10:41] You know, you can imagine students finishing their GCSE,

[00:10:44] you know, exams in like, you know, February or whatever

[00:10:48] because they've reached the point where they can sit the exams

[00:10:51] and they want to take them and they've done them.

[00:10:52] And it's like, well, what do we do with them until June?

[00:10:54] And some of them have finished and some of them haven't.

[00:10:56] And then that becomes like a real practical...

[00:10:59] And it also brings in the question of what's the purpose of school

[00:11:01] other than assessment?

[00:11:03] And, you know, there's the working towards something,

[00:11:05] all being together.

[00:11:07] Yes.

[00:11:08] Yeah, that is a much better question.

[00:11:09] Yeah, exactly.

[00:11:10] It's really revolutionary that, like,

[00:11:12] it would be something that looked completely different.

[00:11:16] And you've also got to think about assessment,

[00:11:18] which has come from many of our conversations,

[00:11:20] is about how it reflects the world of work as well.

[00:11:25] And as that changes, as students finish school and go into work,

[00:11:29] that on-demand exam stuff is quite interesting

[00:11:33] when you look at the kind of, the way people are working,

[00:11:36] like nine to five, not being as popular as before.

[00:11:41] Lots of industries go into it.

[00:11:43] I have to say, I know what you're saying,

[00:11:46] but I have to say I'm less, that argument I'm less convinced by.

[00:11:49] So the argument of, like, because this is how people do things at work,

[00:11:54] this is how.

[00:11:55] So because I think as long as students are...

[00:12:01] Because working and learning are not exactly the same thing for a start.

[00:12:05] No, not at all.

[00:12:06] So I think it's, you know, people say, well, kids don't sit writing at work,

[00:12:13] but also kids don't sit, you know, necessarily with Bunsen burners.

[00:12:18] And, you know, there's lots of things you do at school.

[00:12:20] Yeah, it's a school that you never do again.

[00:12:21] Because you're in the process of learning that you don't.

[00:12:23] But I think one thing that is kind of interesting as well is, like, whether or not...

[00:12:30] So one of the things that the exam boards talk about all the time as, you know,

[00:12:35] kind of being...

[00:12:36] And JCQ talk about it as well as being, like, the kind of baseline for fairness

[00:12:39] is that the way students do their assessment should reflect their normal way of working.

[00:12:44] So my personal view is I'm, I would say, in favour of digital assessments ultimately

[00:12:51] because also just because I think it is kind of inevitable in the end, you know,

[00:12:55] the way that all of this technology is going.

[00:12:57] It's like it will be mad probably in 15 years if kids are still there in examples,

[00:13:02] you know, kind of writing.

[00:13:04] So I do think that it is the right move.

[00:13:07] But I also think that there's a little bit at the moment that feels a bit of, like,

[00:13:13] putting the cart before the horse.

[00:13:15] It's like get the digital learning right first and across the board.

[00:13:21] And then following from that comes the digital assessment, which reflects the way of working.

[00:13:26] What we're trying to do at the moment is sort of, like, reverse engineer the system a bit

[00:13:32] so that you put the assessment bit, the bit at the end, which is meant to reflect the way of working first

[00:13:38] and then try and adapt everybody's way of working afterwards.

[00:13:42] It would make more sense to focus more energy on transitioning, on the digital transition in learning

[00:13:48] and then follow that up with the digital transition in assessment

[00:13:53] rather than what seems to be happening at the moment,

[00:13:55] which is sort of, like, quite almost chaotic, I would say.

[00:13:59] Like, it's random, certainly, in terms of where the focus is.

[00:14:03] Like, you could have a much more focused approach.

[00:14:05] Well, it's where you've got skin in the game, isn't it?

[00:14:07] So if you're working in assessment, then you're going to be really focused on that.

[00:14:11] Exactly. So the example is going to do it and you can understand why they want to do it.

[00:14:13] But that's why the curriculum assessment review is good,

[00:14:15] because it's both together at the same time, which I thought that...

[00:14:19] And it might be the curriculum assessment review.

[00:14:21] I mean, people have...

[00:14:22] The people we've spoken to said that it will probably be quite cautious in what it recommends.

[00:14:26] But if it does, you know, look to introduce something that might move the system

[00:14:35] into something that looks a bit different from what we've got now,

[00:14:38] then there might be more opportunities there to implement digital assessments

[00:14:42] in a way that would please everybody rather than sort of create the sort of

[00:14:48] slightly antagonistic situation that we have at the moment.

[00:14:52] Yeah.

[00:14:52] But anyway, we did speak to someone who knows a lot more than us.

[00:14:56] We spoke to an actual expert.

[00:14:57] I mean, we're just spitballing here, but we...

[00:14:59] Pontificating.

[00:15:00] Pontificating, yeah.

[00:15:01] But we did...

[00:15:01] We spoke to someone who's been working in this area for a very long time.

[00:15:05] Very long time, yeah.

[00:15:06] So Graham Hudson is an international special advisor on assessment.

[00:15:12] He's worked in this area for 35 years.

[00:15:15] He is the chair of the E-Assessment Association, and he's also on the board for Qualifications Wales.

[00:15:23] He's also worked for awarding bodies, hasn't he, in the past as well.

[00:15:27] Yeah.

[00:15:27] So he knows...

[00:15:28] Oh, yeah.

[00:15:28] He knows what's great about Graham is that he really does understand this from...

[00:15:32] Historical.

[00:15:33] ...lots of different angles.

[00:15:35] And, yeah, like you say, the historical...

[00:15:37] Yeah.

[00:15:37] ...the journey of this, which has actually been going on for a lot longer than we realise.

[00:15:40] Yeah, so you give that really interesting perspective of we all think this is, you know,

[00:15:45] the first time that people have had big conversations about looking at assessment, but absolutely not.

[00:16:03] So the problem is, John and Sophie, that I just have to catch myself saying,

[00:16:09] well, we did that before, or we tried that and it didn't work 30 years ago.

[00:16:14] The trick is to take an old thing and repackage it.

[00:16:18] Repackage.

[00:16:18] And set it as something new.

[00:16:20] Let's give it a new name.

[00:16:22] Yeah.

[00:16:23] Absolutely.

[00:16:24] Could we talk to you, Graham, a bit about e-assessment generally and the use of digital for examinations?

[00:16:34] One thing that I'm interested in is whether or not you think that our current system, how we run exams,

[00:16:42] can be adapted to digital or whether or not there has to be some changes in the system to accommodate digital.

[00:16:52] Do you see where I'm going with that?

[00:16:54] Absolutely.

[00:16:56] Do you know, I think that's a real conundrum.

[00:16:58] I have sort of thought about that lots, many, many times.

[00:17:05] I think in your interview with Michael, some of these sort of factors emerged.

[00:17:10] But I've felt for a long time that, you know, students are using technology in schools.

[00:17:17] Yeah.

[00:17:18] So it seems lunacy for us to take them and say, sit and write for two hours.

[00:17:24] Just at a base level, you think this is not right because that's not what they do all the time.

[00:17:30] I'm not saying that they shouldn't learn to write.

[00:17:34] But it's as Michael, I think, said, you know, people's usual way of working.

[00:17:37] Yeah.

[00:17:38] So as a starting point, that seems a bit crazy.

[00:17:44] But in the environment that we are, where everybody sits down and you have cohorts of 400,000 students

[00:17:50] all sitting down to take English at one time, I find it difficult to see how that could be supported through online tests.

[00:18:01] And then there is an element, which I think, again, emerged in previous discussions that you have on your show,

[00:18:06] which is, is that always fit for purpose?

[00:18:11] And sometimes it may not be.

[00:18:15] Yeah.

[00:18:15] So if you look at other initiatives, the initiative that's been running in Wales for a number of years for primary school children,

[00:18:22] which is the adaptive testing, which is more of a formative type assessment,

[00:18:25] those can be done at any point that the teacher wishes and provides, you know, ready feedback to the students.

[00:18:33] So in that context, that works quite well.

[00:18:37] But I do wonder whether we'd have to move away from everybody has to do the test at the same time

[00:18:44] to make this practical across the board.

[00:18:47] Now, that isn't to say that there aren't some areas where it is possible now.

[00:18:51] And AQA have taken a particular line with the way that they've done it.

[00:18:54] But if you look at what we're doing in Wales through Qualified for the Future and the new GCSEs,

[00:19:01] it's being done through some of the syllabuses there, but in a slightly different context.

[00:19:05] So they're taking things like performing arts and doing it in that context,

[00:19:09] which is a slightly different tack from what AQA did.

[00:19:13] So it is a bit of a conundrum.

[00:19:15] I'm not sure I really have an easy answer unless there is a bit of a change.

[00:19:21] No, and I do think we would have to change the way that we assess.

[00:19:26] And then the next question is, which I wanted to ask the Secretary of State when she was on a call yesterday that I was saying,

[00:19:33] is are you going to think of 11 to 19 education or are you going to continue to think of 11 to 16 and 16 to 19?

[00:19:39] Yeah, yeah, yeah.

[00:19:40] Yeah.

[00:19:41] Because I think there are so many interesting questions in this because I agree with you about the thing about, you know,

[00:19:48] 400 kids all at the same time.

[00:19:51] I think schools look at that at the moment and are like, how the hell would we deliver that?

[00:19:56] And I mean, some schools would be in a position to do it.

[00:19:58] But for example, ours, it would require large capital investment to be able to do that.

[00:20:04] And then I think the, yeah, the other question is about, because sometimes you hear conversations about GCSEs, don't you?

[00:20:12] And whether or not GCSEs are an appropriate kind of qualification, whether they're relevant nowadays.

[00:20:18] But of course, the whole school system is sort of built 11 to 16 and 16 to 18.

[00:20:23] So, for example, in Brighton, we don't really have any sixth forms left.

[00:20:27] We have secondary schools and then a couple of enormous sixth form colleges.

[00:20:34] Colleges, yeah.

[00:20:35] Yeah.

[00:20:35] So that idea of continuous 11 to 19 education, it doesn't really fit with the bricks and mortar.

[00:20:43] No.

[00:20:44] So there's some really massive questions actually, aren't there?

[00:20:49] It shouldn't actually stop us from continuing down this route and looking to see how it can be done.

[00:20:56] Yeah.

[00:20:58] But it would, I mean, what has happened in Wales is that the Welsh Government has invested in infrastructure there.

[00:21:05] Yeah.

[00:21:05] So, you know, all schools do have good connectivity and they also have invested in the hub, which is accessible to students and, you know, wider audience.

[00:21:18] And what we've done in creating the whole Qualifier for the Future is had a specific team that's looking at what we call modernising assessment.

[00:21:29] You know, so that has been, it's focused on what benefits can we actually bring in the most sensible manner into the new GCSEs as they start to emerge.

[00:21:40] Do you think, Graham, obviously we touched on a couple of the kind of major kind of challenges for thinking about exams and assessment, but also recognising that they're real opportunities as well.

[00:21:51] Like most of these things, it's not just a challenge, is it?

[00:21:55] It's an opportunity as well.

[00:21:56] What are the other opportunities and challenges for assessment do you see, taking into account, you know, all your experience and also what you're seeing kind of internationally at the moment as well?

[00:22:10] Well, what encourages me very much is what we see through the e-assessment association.

[00:22:19] And you'll probably be aware that we run, we've now run for eight years a series of international awards, which are under a number of categories.

[00:22:29] And organisations put forward case studies and projects for an award.

[00:22:39] And the range of what we see is amazing.

[00:22:43] And some of the innovation that we see is amazing.

[00:22:48] And I've judged that myself for all of that time.

[00:22:53] And all those people who get involved in judging say what an amazing experience it is because of what they see.

[00:23:00] So, you know, across the piece, not just in terms of sometimes people are awarded because they do a big thing, you know, like TCS introduced a system for recruiting students into universities.

[00:23:13] And you can imagine in India, you're talking about hundreds of thousands, millions of students.

[00:23:19] So that was a big thing.

[00:23:20] But then there's others who do more niche things.

[00:23:23] We had some language assessments related to the European Union, for example.

[00:23:30] And also some work in Sky in Italy on helping people transition from different types of jobs supported through digital means.

[00:23:43] So the first thing that I'd say is that there's a huge amount of talent and innovation out there.

[00:23:48] And that's very encouraging.

[00:23:49] So, you know, if there's opportunities.

[00:23:51] Yeah, they're there.

[00:23:52] People are there actually starting to drive that.

[00:23:55] So, you know, that would be the first thing.

[00:23:56] Then I think that the growing interest in things like adaptive assessment is very encouraging.

[00:24:03] It's maybe not fit for everything, but it certainly does have a place.

[00:24:09] Graham, for our listeners, can you just explain the concept of adaptive assessment for us?

[00:24:14] Yeah, I mean, in straightforward terms, in a regular test, you might have 20 questions.

[00:24:22] All students complete the 20 questions.

[00:24:24] And those questions might be graded in terms of difficulty.

[00:24:28] And you add up what they get at the end and you give them a grade based on what they achieve and how far they actually achieve on questions of different difficulty.

[00:24:40] But everybody does the same thing and it takes them.

[00:24:42] They're allowed the same amount of time.

[00:24:44] But in an adaptive test, in brief, students start taking tests and the assessment engine adapts to their level of competency.

[00:24:57] So it serves up items to them.

[00:25:00] So if they succeed in some easier items, they're given some harder items.

[00:25:05] If they succeed in those, they're given some harder items until they get to a point where effectively they plateau in their ability to answer the questions.

[00:25:13] They may get some right, some wrong and so on.

[00:25:15] And at that point, the system says, OK, we've now assessed the level at which you are operating based on where you've got to.

[00:25:24] So that has advantages in the way that students take the test.

[00:25:31] It has advantages in the information and the speed that you can get information out.

[00:25:34] But it also means that not all students have to do 20 questions.

[00:25:39] I mean, dare I mention AI?

[00:25:42] I mean, it is the topic and there's just some astonishing work going on in how test items are being created using generative AI.

[00:25:55] But of course, I mean, what can be forgotten is that AI of one sort or another has been used in testing and assessment for a long time.

[00:26:06] And some examples, for example, scoring of extended essays has been using effectively AI for 20 odd years.

[00:26:17] But it's going to provide us with the means to create much larger item banks, more valid and reliable items as a result.

[00:26:35] And potentially items which are more innovative as well.

[00:26:40] So I think that's another area.

[00:26:45] And I mean, one has to say that the advent of the Internet, and I think we can't forget this because it seems like a relatively...

[00:26:57] We think it's been here forever, but of course it hasn't.

[00:27:01] And that, of course, is one of the things that has enabled us to do many of the things that we do do now.

[00:27:07] But in some guises were being done, but not on the scale that they are at the moment.

[00:27:22] I did some work in Pakistan when I first started doing consultancy for an organisation called the Punjab Exams Commission.

[00:27:30] And they set tests for primary school children.

[00:27:33] And basically the children had to take these tests.

[00:27:35] And if they passed, they moved on to the next grade.

[00:27:37] There's such a problem with malpractice and cheating that the whole system ran on the basis that the papers were printed the week before the exams took place.

[00:27:54] So if you imagine across the whole of the Punjab that this is taking place, you just imagine the number of question papers akin to what happens here.

[00:28:04] They were printed the week before and then they were trucked.

[00:28:08] And about 80,000 people trucked this stuff to the schools.

[00:28:12] They would deliver a half hour before the test took place.

[00:28:16] Oh, no, that's just too much stress on the day.

[00:28:19] Just in time.

[00:28:19] Yeah.

[00:28:21] And then the kids took the test.

[00:28:22] And it just makes your mind boggle how some organisations actually achieve that.

[00:28:29] And then you just reflect on what we take for granted.

[00:28:33] Yeah.

[00:28:33] And then the other thing, we're all in the business.

[00:28:37] It could be seen as a large machine.

[00:28:41] You know, you talk about scanning three million sheets of paper a day and processing the marks of five million students and so on.

[00:28:50] And somebody could say, what's your performance measure?

[00:28:53] Well, you know, we did this to this percentage of accuracy.

[00:28:56] But every 0.1 percent that you might have missed on a percentage actually accuracy represents a pupil, represents a student.

[00:29:07] Yeah.

[00:29:07] It's every individual.

[00:29:09] It's a life chance.

[00:29:10] And there were occasions when things perhaps didn't work quite as we expected.

[00:29:14] And we then had to go back and look and make sure that absolutely everything was right because it was that individual.

[00:29:21] It's not an environment that you can just measure with the standard measures of accuracy, efficiency.

[00:29:30] Yeah.

[00:29:30] And, you know, have you met your KPI?

[00:29:33] Yeah.

[00:29:33] Because it's a life chance.

[00:29:34] And that's always been, I think, my motivation.

[00:29:36] I know all my colleagues who've worked in this environment have that motivation too.

[00:29:41] Yeah.

[00:29:42] I can totally, totally see that.

[00:29:44] It matters, doesn't it?

[00:29:45] At the end of the day, it matters more than anything.

[00:29:48] It's interesting.

[00:29:48] When I think I wrote about this was when they did the, when we had the teacher assessment during COVID.

[00:29:55] And do you remember they introduced that algorithm to try and adjust grades to base it on past school performance?

[00:30:02] And, of course, kind of, it sort of made a kind of scientific sense.

[00:30:07] But at the same time, what it did to the individual was so kind of cold and so unfair in that sense that it, that's the thing with an assessment system, isn't it?

[00:30:20] It's like you've got these broad measures and these kind of global systems and things like that.

[00:30:26] But in the end, it comes down to an individual student and their life chances, doesn't it?

[00:30:31] Which makes it quite a complex system, really.

[00:30:35] I don't say that as a kind of criticism, really.

[00:30:38] But it's one of those things, isn't it?

[00:30:39] That it's just you're trying to make the best in a difficult situation, but the outcome can be kind of perverse, can't it?

[00:30:48] Yeah.

[00:30:48] And that's a trouble when you put models like that on because you lose the individual in the model.

[00:30:54] And that is the problem with it.

[00:30:56] And probably the, you know, later approaches that were based upon schools' work was a more defensible system.

[00:31:08] I mean, obviously had its flaws.

[00:31:10] And the whole issue about, you know, returning to a distribution that looks more like it was before COVID has also been a challenge to be fair and equitable.

[00:31:23] You know, so it's always tricky, isn't it?

[00:31:26] It really is.

[00:31:27] Yeah.

[00:31:28] Thank you so much.

[00:31:29] Thank you very much.

[00:31:30] It's so interesting.

[00:31:31] It's lovely to meet you as well.

[00:31:32] I hope you've enjoyed it as well.

[00:31:34] Thanks very much.

[00:31:35] No, I have.

[00:31:36] And thank you very much.

[00:31:36] And it's really nice.

[00:31:38] No problem at all.

[00:31:43] Thank you so much for listening to the Exam Man podcast.

[00:31:46] We really, really appreciate your support.

[00:31:48] Remember that you can access it on all the major podcast platforms.

[00:31:52] Give us a rating.

[00:31:53] Give us a follow.

[00:31:55] And we will catch you next time.

[00:31:57] Bye.

[00:31:57] Bye.