Subjectivity of grading papers solved by new programme

Summary

High-tech meets common sense in a new assessment tool that works on the basis of comparisons. Local teachers are sold on D-Pac

Which is better?

A lot of research and some clever algorithms lie behind the Digital Platform for the Assessment of Competences, or D-Pac, a software tool developed by a group of Flemish researchers. Yet its approach to assessment is fundamentally driven by common sense.

The idea behind D-Pac is that the best way to assess the quality of something is to compare it with something similar. Take choosing a new pair of glasses, for example. Rather than consider each frame in isolation, most people will try on one, then try on another, and see which looks best. The comparison tells you which one you prefer, even if you might struggle to define why.

The researchers, from Ghent and Antwerp universities and digital research centre imec, set out to see if this approach could be applied to more complex tasks, such as grading students or choosing the most suitable candidate for a job. This is a hot topic at present.

“In education and in human resources people are more and more interested in skills and competences, not just knowledge,” explains Marije Lesterhuis, an education researcher at the University of Antwerp, “but that requires a change in how you assess students and employees.”

No two alike

Take writing skills, for example, such as how well a student can present an argument or construct a scientific paper. It is often tricky to devise a set of criteria to assess such a task, and even when you have a scheme all sorts of variables can creep in.

A teacher might mark a paper more harshly on a dark morning, before they’ve had their coffee, than after a good lunch on a sunny afternoon. They might also grade it differently if they see it early in a batch rather than later.

Getting more people involved can simply multiply the problems. “Then you see that judges differ in the way they interpret criteria, or in how strictly they apply them,” Lesterhuis explains. “And all these aspects have an impact on the reliability and the validity of the scores that come out.”

Every judge has his or her own expertise and things they are looking out for

- Marije Lesterhuis of D-Pac

D-Pac gets around this by presenting texts in pairs. “Instead of asking a teacher to grade a student’s paper, the question now is which of these two students wrote a better paper?”

This invites the teacher to use their expertise in a freer, more holistic way. As the papers are compared to others several times over, a rank order emerges.

“These comparative judgements are more stable within the judge, but also across judges,” Lesterhuis continues. “Every judge has his or her own expertise and things they are looking out for, so when multiple judges of this kind are involved in the assessment, the results are even more valid.”

Assessing each other

As for feedback, teachers can leave comments on the strengths and weaknesses of each piece of work. But more importantly, students can see where they lie in the ranking, as well as the feedback received by other students.

This can be taken further by using D-Pac for peer assessment, where students take the place of the teacher. “Students see a lot of examples of fellow students’ work,” Lesterhuis explains. “They have to compare and contrast in order to make the judgement, and they both provide and receive feedback.”

As well as written work, the tool has been used to assess presentation skills, as recorded on video, and the work of art students, such as portfolios, paintings, photographs and installations. These can be particularly hard to grade using fixed criteria.

You can use it in any situation where have to make complex judgements, where there is no easy answer

- Marije Lesterhuis

After four years of research, the notion of commercialising D-Pac came naturally. “In the beginning, most teachers were sceptical about the method, but at the end more and more said they wanted to keep using it,” Lesterhuis says. “That was the moment we decided to look into building a company around it.”

A commercial version of the tool has now been built and tested, and the company, also called D-Pac, was formally established this week. As well as education and human resources, a wide range of applications are foreseen, from helping committees select project proposals, to agenda-setting and even policy development.

“You can use it in any situation where have to make complex judgements, where there is no easy answer, and you want to involve the expertise or opinions of many people.”