Highlights from Season 3, Episode 6 of The Exam Man podcast
For this episode we had a good chat with Lisa Greenslade, who is the University Partnerships Manager for TeamCo- they help institutions to deliver digital exams at scale. We explored the work involved in getting some universities to switch over to digital exams, how that project has gone, how successful it has been, and the lessons for schools.
Could you start by telling us a bit about your experience of running the exams in universities?
My experience in exams operations started about 12 or 13 years ago. I used to work for what was called the Institute of Financial Services at the time, but it's now the London Institute of Banking and Finance. They're an awarding body for banking and finance qualifications. I started off in their schools ops team because they used to run what I thought were really good qualifications at roughly the same level as GCSE and A-levels, in financial capability. We used to do all the background ops of getting everything out to the various schools and getting it marked. That's how I got into exams, and from then I went on to work at various universities, and ended up at King's College as head of exams there. I absolutely loved it. It was the best job in the world, and it just challenged me every single day. I actually led the transformation from paper to pixel. So from paper-based exams through to digital, post-pandemic.
King's chose to do some assessment remotely online, particularly throughout the middle of COVID, when we obviously couldn't do an in-person setting. As we started to return, there were a lot of faculties who were very interested in bringing digital assessment back into an in-person environment. And so that's when we started exploring different options and we settled on partnering with the assessment management company that I work for now, to create real large scale computer suites at the Excel, and to sit thousands of students at once. So that was interesting at that time.
I'd love to get on to the mechanics of that and how you do that now. But I'm just wondering what you think the effect of the pandemic was? Do you think it would have happened so quickly if COVID hadn't happened? Do you think the university would be where it is now if that hadn't happened?
I don't think so. I think it would be wrong to say that it wouldn't have happened at all. It definitely would have, because they were exploring digital assessment prior to the pandemic, but the pandemic definitely did push that to the forefront. Everyone approached it in different ways during 2020 and 2021, when face-to-face stuff still wasn't really happening. It was a knee-jerk reaction for everyone at the time, apart from perhaps Brunel, because they'd been doing in-person assessment for a long time in a digital sense. But I think it did very much push it forward.
I think that if the pandemic hadn't happened, we might have seen an emergence around this time, rather than three or four years ago. I think it was definitely on people's radars, but no one really knew how to approach it, and one of the biggest barriers was that because it hadn't really been done before at any scale, no one really knew. The exams teams themselves might not have had the skills and the knowledge and the technical know-how to get it off the ground, because the bit I really struggled with at the time, was understanding what it actually entailed.
When you hadn't got a precedent to work on? And are you someone, Lisa, who is interested in technology, or has your interest in it formed through doing this?
I wouldn't necessarily say my interest is in technology. My interest is very much in efficiency. To me, digital assessment presents opportunity for massive improvements in efficiency in lots of different areas. I think all sorts of education providers are under a lot of pressure to squeeze as much into an academic year as they possibly can and to have more and more students because of funding problems etc. And very few of them have processes and procedures that benefit from economy of scale, whereas digital assessment really does. It takes just as long to churn through a thousand sets of data or a thousand students with marking and multiple choice questions that self mark or with AI marking, as it would to do a hundred thousand. But if you've got a physical paper script, every single student you add is more burden for all the downstream processes. So I've always been a real advocate for work smarter, not harder. I think long term, digital certainly is far more efficient than paper based, and so I'm all for it.
So obviously the school system at the moment is different. We are still pretty much entirely paper based for GCSEs and A levels. There are a few digital exams in other qualifications, but not that many. So from your experience of doing this transition, what are the biggest barriers? What are the biggest issues that have to be overcome to realise that transition?
I think the most obvious one is having the kit available at the scale that you need. It's one thing having enough desks and chairs and bits of paper, but it's something else entirely, having enough plug sockets, having a decent Wi-Fi coverage, and having enough devices for all your students. There's different ways of approaching that and different considerations, whether you would like to do a bring your own device model, whether you're going to invest in devices and tech, or whether you're going to enlist a third party to help you do that.
There's lots of options. It's choosing the right option for you as an institution. And then obviously with that, there are funding problems. You've got a lot of schools across the country who are underfunded, but I wouldn't say it's impossible. I think it's something that seems a much bigger beast than it is when you sit down and really think about it. I currently work for the Assessment Management Company, or TeamCo as we're known, and we specialise in coming into spaces that perhaps you wouldn't necessarily have first thought would be ideal exam spaces, particularly for digital exams. We specialise in transforming ordinary spaces into suitable exam venues for digital assessment.
We've done all sorts of things over the years. Some of our clients will go off site. It doesn't necessarily have to be a proper events venue. We will use hotels or conference suites, we'll use sports halls, we'll use stadiums etc. We can work with whatever's within the budget in terms of spaces. Equally, we go on to a campus space and we've used dance studios, we've used the flat floor spaces in lecture theatres, whatever it is that you're comfortable with. An exam hall doesn’t have to be a big flat floor, cold hall somewhere, with horrible walls and big echoey ceilings.
You may be thinking about a pop-up - use it for a couple of days as and when you need to flex up and down depending on your timetable, and then it's all packed away again and it's back to normal teaching space or whatever it's usually used for. I think that having the infrastructure in place is probably one of the main considerations.
And also for schools, I think, it's about having the time and space to come up with innovative solutions like that. So talking about pop-up examples I'm just trying to imagine that happening in a school on the ground, of getting to that point where we could do that. I definitely agree that the biggest thing that I hear about it, in terms of it being a challenge to achieve, are the logistical issues. I guess one of the ways in which schools and universities are different is that each university is also the exam board, as well as being the examining centre. So it's very clear that it's the university's job to sort all of this out. With the school system, it's more complicated, because it's a question of whose budgets? You talked about efficiency savings and most of the efficiency savings will come for the exam boards in terms of their systems. But most of the costs in terms of achieving the logistics will be borne by the schools. So I guess there's quite a big conversation to be had there in terms of who's paying!
I completely agree. The university funding model and the responsibilities are completely different to schools. But I don't think it's beyond the realms of being achievable. Even if something like a pop-up centre was made in a local place near to schools, so it became a shared resource that a number of schools sort of plugged together to create. There's loads of options available to explore how to tackle this.
One of the other things that gets talked about quite a lot is whether there needs to be a change in the way that exams are done. Obviously you're talking about big conference centres or big spaces, like King's use the Excel centre, where you do a lot of students all at once. Could there potentially be a change so that tests are more like on-demand rather than everybody sitting tests at the same time. Have you seen anything like that in your work at all? Or does it all tend to be terminal assessments, everyone doing the same thing at the same time?
We are starting to look at the on demand market. Team Co. already have a fixed test centre in London and we're in the process of acquiring two more in other locations. We're looking at the on-demand market because there certainly is demand for it. It's similar to your driving license when you go and do your theory tests or those sorts of things. There is a demand for that very much in the professional qualification market. Schools are not there yet, but I think that there is definitely room for that to be something that happens now we have digital assessment. If you're using an assessment platform, which will no doubt be rolled out as this becomes more commonplace for GCSEs, A-levels or other types of qualifications, there'll be other kinds of integrity measures that are built in that enable on-demand assessment without actually compromising the exam itself. So you might be able to have randomised question banks.
You might have all sorts of different things that essentially mean that no student sits the same exam, but are still equally tested against the same learning outcomes. There is definitely a market for it in the more professional side of stuff.
I guess the work that you're doing is testing out all of these things, at scale, in different ways, so that at the point that some schools might be able to move towards this, at least there are examples of how it's worked. When there's nothing to look at, it's just so overwhelming, because the cost, the logistics, all of it is just so huge. I can't see it in schools, unless they do change to everything basically being on-demand, and I'm not quite sure how that would work. Aside from that, I can't see it being anything other than gradual - some schools will be in a position to do it, and will move quicker than others. It's quite hard to envisage a situation where someone just hits a switch. But at the same time, it's hard to imagine being 10 years down the line and it's not happening.
I think it's universally accepted now that paper exams are on the decline.
It's just how long and in what way that happens. And I think schools as well have to be pretty conservative just because of the very young people that you're working with. With schools you can't really flick that switch and say, right, we're all trying this this year. I always tell this story, Lisa, about when we introduced Examscreen into schools, which was in 2018, so not that long ago. One of the biggest issues we had was that we designed it all for Chrome and Edge browsers and we found that, I'd say, getting on for 50% of users in schools were still on Internet Explorer, which we just couldn't believe in 2018. So all of a sudden, we had to do a really quick version of it that would work with Internet Explorer. That was a really good lesson for us in terms of being mindful of what schools are like compared to other places. They do seem to operate at different speed. Could you talk to us, Lisa a bit about some of the real benefits of digital assessments that you’ve seen in that transition? Obviously, there were great benefits for the institutions and their efficiency but what about the effects that they have for students and their experience?
I think one of the real benefits for students and for institutions as well in terms of efficiency, is how you can better support your reasonable adjustment candidates. If you're using an exam platform, it could be a cloud platform that's tailored already to your students' settings if they've got different needs, then that almost becomes very self-service with a student, rather than having to go through a whole application and approval process. I don't know what that's like in schools, but if it's anything like it is in universities, that process seems to be really long-winded, not just because of the paperwork and everything that needs to happen behind the scenes at the university, but also the waiting times for diagnosis and reports and things from professionals on the health side of it.
There just seems to be a real disconnect there that means that that process takes forever and then students are either missing out or becoming increasingly anxious as their exams approach and they still don't feel they're going to get the things they need. It becomes a bit more self-service.
At King's, we actually found that we were able to timetable very differently for a digital exam so that we could actually get more sittings in the day because the turnover between the exams was much shorter. You're not having to go pick up all these papers, put them in envelopes and then have to lay it all back out again, which takes forever and always longer than you think it's going to. It is literally just “your exams finished, please log off and leave.” And then the next person comes in, “please sit down, make sure you've got all the bits that you need, log in, is your exam correct? Yes, right, okay, we can go.” It's so much quicker. So you can either get additional sittings in so that you can decrease your exam periods in terms of the overall length of your exam period to give some more revision time or things like that. Or it means that your candidates that have got extra time aren't finishing or starting as late, or they're starting at the same time to their peers. That can be really challenging when you've got students that have got 100% extra time or have got rest breaks. How do you timetable that in and around to make sure they're not being disturbed by other candidates, but they're also not halting the operational aspects of your exams each day that they're timetabled for? We found that we could actually get away with using less rooms because we weren't needing to book the rooms out for longer periods. You could literally do the changeover between without having loads and loads of rooms. It meant we needed fewer invigilators and fewer single rooms.
With some of those students who might have additional needs, what sort of tools have you seen within software that can support those learners?
At King's we were using Moodle, which is like their VLE that they've been using for a long time. And within that they could change all the different look and feel of what Moodle looks like for individual students. If you're using other platforms, they can quite often work with other software that's been around for a long time, like Dragon for text-to-speech and all those sorts of things for students.
Because those softwares have been available for a long time, a lot of the platforms that we're seeing come through now are designed to work in tandem. So I wouldn't necessarily say that there's new technology in that sense. It's just being able to bring all the bits of technology to one place, which makes it so much easier for students, but also for people who are trying to administer these exams. And it becomes more self-serving for them.
I think a lot of the time when we talk to people in schools, there is quite a lot of reticence about digital exams. I think it's quite interesting thinking about what people's concerns are, and they're often what we were talking about earlier, about the logistical side of getting the equipment in place, getting the things working, getting everything set up. But actually, at the other end of the process, there's quite a lot of ways in which your job will become much easier, such as not having to package up papers and tick off attendance registers. You might have more stuff to do at the front end of it all but taking a lot of that stuff out of the back end of it is going to make a big difference to people. It's very interesting. It's really good to talk to somebody who's got the experience of having done this. At the moment, I think for people in schools, it feels like not an insurmountable task, but one that it's very hard to think through to the end.
I was in the same position because King’s was the first university to do it at this scale. To my knowledge Brunel only got to about 500 or 600 candidates in one go when they were doing it and I think they ran a bring your own device model, which I didn't want to explore with the number of candidates that we were looking at, because I thought it would be almost impossible to manage, as at Kings, we run about 1500 at one time. I also had about 1500 or 1600 paper based running at the same time as well. So I needed to be simple. I needed to be able to reduce the risk of things going wrong.
Another good point about the benefits is that things do happen in exams and sometimes there are paper mix ups, sometimes there are question queries, all sorts of things that for paper-based exams are almost like a showstopper. I think we've all been there where we've laid out a paper, all the candidates have looked at it and said, this is the wrong paper or this is the resit, and you would think that's what I've been given, I don't know what to do.
So you end up having to cancel an exam. With digital exams, if you have a problem, you can pause everything for everyone, go away and speak to the module lead or to the faculty or whoever it is and it can be resolved in 10 minutes. They can just take down one paper and put up another one and you can restart the exam again. And by the time they get to the end of the three hours they've forgotten all about the fact that there was a little hiccup at the start. With a paper exam you can't bring in another, you can't collect in 150 exam papers, then lay them back out again and pretend nothing happened. I think in terms of troubleshooting and problem resolution, it's really fantastic. The things that would be a showstopper previously just aren't when you've got digital assessment. To go back to the scale that we were looking at doing it at Kings, we were the first ones, so it was quite lonely. It came about because the medical school told me that the external regulators were really pushing for in-person assessment but digitalised, and that it had to be invigilated. Kings didn't want to go down the proctoring route because it didn't sit well with our student community.
So by proctoring, Lisa, do you mean remote invigilation?
Yes. Remote invigilation, either by way of AI or through actual real life people. We put out feelers and the students weren't too keen on that. So we were then looking at in-person options. The academics and the administrative colleagues in the med school just sort of left it with me. I had absolutely no idea how to approach it.
And as you say, you haven't got a network of people who've done it before in other places.
I had no one to reach out to.
And also, you know that everyone's watching, aren't they? Offering no support but watching.
Absolutely. Lots of eyes and no words of encouragement.
Do you get a lot of questions from people now who are exploring it? You need to charge consultancy rates to go out and support everyone!
This is how my role at TeamCo came about, really, because I'd done four and a half years at King's, and I actually live down in the South West. King's is a London-based university, and there were other things that I wanted to do in my life that meant being away from home for long periods wasn't going to work with families and things. So, I decided to leave King's, and I wanted to go into consultancy and help other universities do the same thing. Then TeamCo said why don't you come and do that with us? So that’s how it all came about. I'm helping other people who come to us and I'll give them advice and say you are in the same position that I was in four years ago.
And you've lived to tell the tale.
Yes. I've come out the other end and I've made all the mistakes, so you don't have to do it anymore, because I can tell you where they were etc. I always say to people that my role at TeamCo is to be the person at the other end of the phone that I didn't have.
I'm sure, as this becomes more commonplace in schools and colleges etc, there will be other types of people that get in touch with me and say, where do I even start? Maybe what we do at TeamCo isn't the answer to everyone's situations, but I think there are different ways to approach it and there's value in every avenue that you explore. It's always interesting to hear how people are thinking about approaching it.
So how long do you think it will take for basically every university in the country to be like King's?
I don't know if every university would actually want to be in the same position as King's, because everyone will approach it differently. Some will say we want to find a remote secure way of doing it. Some will have their own ways of approaching it in-person. Some are already there, and they're quite settled in and we only need to do it for these cohorts, because they're the ones that have got the external pressure, and everyone else sits at home, and that's fine.
I think during the pandemic, when everything was pushed online, academics certainly saw the benefits, because their marking became a lot easier. They could mark from anywhere in the world, not that we could go anywhere anyway, but there were all these different benefits that they might not have explored before.
It was more the burden on administrative staff and the pandemic brought about so many other things that put administrative burden on teams. But I think, as long as you approach it in the right way and you're very considered about how you do your rollout, I don't think I'd recommend full scale. That depends on the size of your institution and what you're doing, but for me, I think start small, proof of concept, and get people on board with it. You don't want to go full scale because there can be so many teething problems that you actually cause yourself more problems. At King's, by the time we got to the point where it was a fairly big pilot, about 450 seats - I think that was about a year in the making. I have recently written a white paper about this, which started off as a bit of a brain dump from my experience at King's. I felt the need to write all this down before I forgot the approach that I took and the things that I considered. It looks at the different things, different options. So bring your own device, leasing your own, all those sorts of things.
Thank you so much. Before you leave us, could you just let people know where they can contact you and where they can find out more about what you do?
If you go to the Teamco website, which is teamcouk.com, you can find my contact details on there, and copies of the white paper on there as well. And there's a bit about what we do as a business and how we can help various institutions at varying levels. We do what I call a pick and mix service. We do an all-in service from start to finish, but you can also pick and mix at various bits - it's all there on the website. You can contact me, drop me an email or find me on LinkedIn.
Thank you so much, Lisa. It's been really interesting to find out how it could actually happen. Someone has done it and survived, because we know, because we've spoken to her. It is possible.
And I was still smiling, too!
To listen to and read more interviews like this, subscribe to The Exam Man - the #1 podcast about exams and assessment
To find out more about Examscreen please contact us: Examscreen