Highlights from The Exam Man podcast, Series 2 Episode 16
After a week in which there was a lot of discussion about the move from pen-and-paper to digital exams, we were joined by Graham Hudson, who is an international special advisor on assessment. He's worked in this area for 35 years, is the chair of the e-Assessment Association, and he's also on the board for Qualifications Wales. He supports business development for organisations enabling the transition from paper-based assessment to digital delivery for many national and professional organisations, and he shared with us some historical and current context for the changes.
Could we talk to you, Graham, a bit about e-assessment generally and the use of digital for examinations? One thing that I'm interested in is whether or not you think that our current system, how we run exams, can be adapted to digital or whether there have to be some changes in the system to accommodate digital. Do you see where I'm going with that?
Absolutely. Do you know, I think that's a real conundrum. I have thought about that lots, many, many times. I’ve felt for a long time that as students are using technology in schools, it seems lunacy for us to say, sit and write for two hours. Just at a base level, you think this is not right, because that's not what they do all the time. I'm not saying that they shouldn't learn to write. But in the environment that we are, where everybody sits down and you have cohorts of 400,000 students all sitting down to take English at one time, I find it difficult to see how that could be supported through online tests.
And then there is an element which I think again, emerged in previous discussions that you have on your show, which is, is that always fit for purpose? And sometimes it may not be. So if you look at other initiatives, the initiative that's been running in Wales for a number of years for primary school children, namely adaptive testing, which is more of a formative type assessment, those can be done at any point that the teacher wishes and provides ready feedback to the students.
So in that context, that works quite well. But I do wonder whether we'd have to move away from everybody having to do the test at the same time to make this practical across the board. That isn't to say that there aren't some areas where it is possible now, and AQA have taken a particular line with the way that they've done it. If you look at what we're doing in Wales, through Qualified for the Future and the new GCSEs, it's being done through some of the syllabuses there, but in a slightly different context. So they're taking things like performing arts and doing it in that context, which is a slightly different tack from what AQA did. So it is a bit of a conundrum. I'm not sure I really have an easy answer unless there is a change.
I do think we would have to change the way that we assess. And then the next question is, which I wanted to ask the Secretary of State when she was on the call yesterday that I sat in on - are you going to think of 11 to 19 in education, or are you going to continue to think of 11 to 16 and 16 to 19?
I think there are so many interesting questions in this, because I agree with you about the thing about 400 kids all sitting exams at the same time. I think schools look at that at the moment, and wonder how the hell would we deliver that? Some schools would be in a position to do it. But for example, ours would require large capital investment to be able to do that. And then I think the other question is about whether or not GCSEs are an appropriate kind of qualification, whether they're relevant nowadays. Of course, the whole school system is built 11 to 16, then 16 to 18. So for example, in Brighton, we don't really have any sixth forms left. We have secondary schools and then a couple of enormous sixth form colleges. So that idea of continuous 11 to 19 education, it doesn't really fit with the bricks and mortar. So there's some really massive questions actually, aren't there?
It shouldn't stop us from continuing down this route and looking to see how it can be done. What has happened in Wales is that the Welsh government has invested in infrastructure there. So all schools do have good connectivity and they also have invested in the hub, which is accessible to students and a wider audience. And in creating the whole Qualify for the Future we’ve had a specific team that's looking at what we call modernising assessment. It’s focused on what benefits can we actually bring in the most sensible manner into the new GCSEs as they start to emerge.
We’ve touched on a couple of the major challenges for thinking about exams and assessment, but also recognise that they're real opportunities as well. Like most of these things, it's not just a challenge, is it? It's an opportunity as well. What do you see are the other opportunities and challenges for assessment from all your experience and also from what you're seeing internationally at the moment as well?
What encourages me very much is what we see through the e-Assessment Association. And you'll probably be aware that we've now run for eight years a series of international awards, which are in a number of categories, and organisations put forward case studies and projects for an award. The range of what we see is amazing. And some of the innovation that we see is amazing. I've judged that myself for all of that time, and all those people who get involved in judging say what an amazing experience it is because of what they see. Sometimes people are awarded because they do a big thing, like TCS (Tata Consulting Services) introduced a system for recruiting students into universities. You can imagine in India, you're talking about hundreds of thousands, millions and millions of students.
So that was a big thing. But then there's others who do more niche things. We had some language assessments related to the European Union, for example. And also some work helping people transition from different types of jobs, supported through digital means. So the first thing that I'd say is that there's a huge amount of talent and innovation out there. And that's very encouraging. If there's opportunities, people are there, actually, starting to drive that. That would be the first thing. Then I think that the growing interest in things like adaptive assessment is very encouraging. It's maybe not fit for everything, but it certainly does have a place.
Graham, for our listeners, can you just explain the concept of adaptive assessment for us?
In straightforward terms, in a regular test, you might have 20 questions. All students complete the 20 questions, and those questions may be graded in terms of difficulty, and you add up what they get at the end, and you give them a grade based on what they achieve and how far they actually achieve on questions of different difficulty. But everybody does the same thing, and they are allowed the same amount of time.
But in an adaptive test, in brief, students start taking tests, and the assessment engine adapts to their level of competency. So it serves up items to them. So if they succeed in some easier items, they are given some harder items. If they succeed in those, they are given some harder items, until they get to a point where effectively they plateau in their ability to answer the questions. They may get some right, some wrong and so on. And at that point, the system says, okay, we've now assessed the level at which you are operating based on where you've got to. So that has advantages in the way that students take the test. It has advantages in the information, the speed that you can get information out, but it also means that not all students have to do 20 questions. Dare I mention AI?
I mean, it is the topic, and there's just some astonishing work going on in how test items are being created using generative AI. But of course, what can be forgotten is that AI of one sort or another has been used in testing and assessment for a long time. Some examples, for example, scoring of extended essays has effectively been using AI for 20 odd years.
But it's going to provide us with the means to create much larger item banks, more valid and reliable items as a result, and potentially items which are more innovative as well. So I think that's another area. And the advent of the internet, and I think we can forget this because it seems like it's been here forever - but of course it hasn't. And that, of course, is one of the things that has enabled us to do many other things that we do now. In some guises, they were being done, but not on the scale that they are at the moment.
I did some work in Pakistan when I first started doing consultancy for an organisation called the Punjab Exams Commission, and they set tests for primary school children. The children had to take these tests and, if they passed, they moved on to the next grade. There's such a problem with malpractice and cheating, that the whole system ran on the basis that the papers were printed the week before the exams took place. So if you imagine across the whole of the Punjab that this is taking place, you just imagine the number of question papers, akin to what happens here.
They were printed the week before, and then they were trucked. About 80,000 people trucked this stuff to the schools, so we delivered a half hour before the test took place. And then the kids took the test. It just makes your mind boggle how some organisations actually achieve that, and then you just reflect on what we take for granted. This business could be seen as a large machine. You know, you talk about scanning 3 million sheets of paper a day, and processing the marks of 5 million students, and so on. And somebody could say, what's your performance measure? Well, we did this to a percentage of accuracy, but every 0.1% that you might have missed represents a pupil, represents a student. It's every individual. It's a life chance.
And there were occasions when things perhaps didn't work quite as we expected. And we then had to go back and look and make sure that absolutely everything was right because it was about an individual. It's not an environment that you can measure with the standard measures of accuracy, efficiency, and have you met your KPI, because it's a life chance. And that's always been, I think, my motivation. I know all my colleagues who've worked in this environment have that motivation too.
I think I wrote about this when we had the teacher assessment during COVID. And do you remember they introduced that algorithm to try and adjust grades, to base it on past school performance. And of course, it sort of made a kind of scientific sense.
But at the same time, what it did to the individual was so cold and so unfair. That's the thing with an assessment system, isn't it? You've got these broad measures and these global systems but in the end, it comes down to an individual student and their life chances, doesn't it? This makes it quite a complex system, really. I don't say that as a kind of criticism, really, but it's one of those things, isn't it? You're trying to make the best in a difficult situation, but the outcome can be perverse, can't it?
You lose the individual in the model, and that is the problem with it. Probably the later approaches that were based upon schools’ work was a more defensible system. Obviously it had its flaws, and the whole issue about returning to a distribution that looks more like it was before COVID has also been a challenge to be fair and equitable, you know? So it's always tricky, isn't it?
It really is. Thank you so much. Seriously, so interesting. Lovely to meet you and I hope you've enjoyed it as well.
I have. Thanks very much.
To listen to and read more interviews like this, subscribe to The Exam Man - the #1 podcast about exams and assessment
To find out more about Examscreen please contact us: Examscreen
To get in touch with Graham, you can find him on LinkedIn: Graham Hudson | LinkedIn
More information on the e-assessment association can be found here: Welcome to The e-Assessment Association - The e-Assessment Association