Or the curse of the algorithm!
Despite recent issues surrounding exam results, why are educationalists unwilling to accept automated forms of assessment. Why is this? AV News looks at the increasing range of educational activities facilitated by the use of edtech and the implications for assessment.
Education and testing – the arguments continue to rage. Education Secretary Damian Hinds said recently: “Very few things matter as much as ensuring our children can read, write and add up. That is why all over the world, from France to Finland and America to Australia children’s learning is assessed. From Berlin to Bordeaux, Boston to Brisbane, children sit assessment tests. 28 out of 35 countries in the OECD assess primary school pupils through national, standardised tests. In Australia, tests take place in years 3, 5, 7 and 9. In most US States, they take place annually. There are very few things that are agreed the world over about education – the need to assess primary school attainment is one of them.”
“The tests themselves vary but the principle remains constant. These tests do not exist to check up on our children. Our national curriculum tests (often called SATs) exist to check up on the system – and those who oversee it on your behalf. There are few duties on me that are more serious than ensuring that children are literate and numerate by the time they leave primary school. It is absolutely right that you should know whether we are succeeding in this duty or not.”
“This is why it worries me deeply when I hear calls for primary school tests to be scrapped. Imagine if the government announced that it was going to ban dental checks or stop opticians checking our eyesight. People would be rightly horrified. Stopping testing means not checking whether something is ok or not. In the world of primary school education, that means stopping checking whether our children can read, write and add up.”
“This doesn’t mean that we should accept exam stress at primary school. The truth is that in many schools, there isn’t any. All over the world, schools guide children through tests without them feeling pressured. This is how it should be. For these tests are tests of our education system, not our children. They test whether we – the adults – are discharging our duty to the children of our country.”
The importance of assessment
“Those of you younger than 35 know this through experience, for our primary schools have been carrying out national curriculum tests for almost 30 years now. I refuse to countenance returning to a world where Government had no effective way of knowing how well our children were being taught, disproportionately to the detriment of those from the most disadvantaged backgrounds.”
“The importance of testing has been one of the main things Labour and Conservative governments have agreed on in education policy over the past quarter of a century. This consensus has allowed us to measure progress in discharging our duty to our children. As a result we can tell which areas of the country and which schools need more support and which should share their expertise with the rest. It allows us all to see the improvement over the last few years in children’s reading as well as the declining gap between disadvantaged students and their better off peers. Turning our back on testing would put this progress, and children’s future, at risk.”
There is a consensus among policy makers that assessment is a good thing – but that view is not universal. There is a body of opinion from students, teachers and administrators that points out the negative effects of over-testing. “Careless implementation of assessments may have negative consequences, especially when the needs of special education students are not considered. Using only a written formal assessment does not provide an overall picture of student achievement. Students that perform better with oral and visual skills or who display superior creativity are at a disadvantage.”
“The process of education is a cycle between teacher and student that is continually moving. The education process focuses on planning, delivery and assessment, and both teacher and student are continually assessing whether or not the content and delivery was successful. For the teacher, it is important to determine if the students received the information, retained it and were able to process it. For students, the process often involves making the determination as to whether or not the content was effectively presented and whether or not the information was retained. This determination is typically made through an assessment.
In the post pandemic work schools are being tasked to “do more with less,” which is often code for utilising digital technologies to curb “inefficiencies” and to “scale” services. The application of the language and practices of scientific management to education isn’t new.”
The digitisation of education was given new emphasis by the pandemic and further encouraged automation of the assessment process – but the use of such tools was not without its problems. In March UK authorities cancelled the traditional exams that students usually take at the end of primary and secondary school. These exams are critical in determining which secondary school or university students will attend. In fact, most students had offers from secondary schools and universities that were conditional upon them achieving a certain result in the exams so a new system had to be devised to grades hundreds of thousands of students.
The system devised required schools to give each student (1) a predicted grade (referred to as “centre-assessed grades”, or “CAGs”) and (2) a rank from best to worst (in their subject). That information was then sent to the government regulator (“Ofqual”) for a process of “moderation”. Ofqual had been tasked by the UK government to develop, in its words, “a system for awarding calculated grades, which maintained standards and ensured that grades were awarded broadly in line with previous years.”
Ofqual developed an algorithm to process all CAGs and school rankings. The algorithm compared CAGs with the performance of students at each school over the past three years. The algorithm then “moderated” (i.e., lowered, maintained or increased) each CAG to for consistency between individual students’ CAGs and historical results at their schools, which then resulted in the grade awarded to the student.
To a statistician perhaps this makes sense, but unfortunately some 40% of students had their grades lowered as compared with their CAGs. This led to nationwide complaints, largely because the methodology appeared to have exacerbated social inequality by generally favouring students at private schools at the expense of their state schools. The result was large numbers of high-achieving students being marked down by the algorithm, which had capped their schools’ previous high scores, and preventing them from meeting conditional offers from prestigious universities. Eventually, the government conceded and amended the algorithm such that students could rely on either their CAGs or their moderated grades, whichever was higher.
With a greater reliance on digital technologies for teaching and lesson preparation, further ingress of automated assessment is inevitable in the future. There is also no doubt that expanding the range of educational activities to take advantage of digital resources and activities is a genuinely exciting possibility – the challenge is to get the methods of assessment both reliable and acceptable to all stakeholders in education.
The failure of the UK government to anticipate the problems that its school algorithm would face resulted in enormous stress for hundreds of thousands of families, teachers, education administrators and politicians. It led to precisely the situation the algorithm was initially designed to prevent: significant grade inflation, with 38% of students taking English A-levels being awarded the top A or A* grades, compared to 25% last year, with adverse knock-on effects on universities as well as future school graduates.
Government actions have also undermined tolerance of algorithms more generally. The national press reported a trend of UK public bodies abandoning recently introduced algorithmic systems for visa applications, benefit claims and other social welfare issues. Let’s hope that a more ground-up approach by those with direct edtech experience will prevail.