Its “E-rater” engine is used to grade a number of statewide assessments, the GRE, and the Test of English as a Foreign Language (TOEFL), which foreign students must take before attending certain colleges in the U.S. The nonprofit Educational Testing Service is one of the few, if not the only, vendor to have published research on bias in machine scoring.

Only a handful of published studies have examined whether the engines treat students from different language backgrounds equally, but they back up some critics’ fears. Test scoring vendors closely guard their algorithms, and states are wary of drawing attention to the fact that algorithms, not humans, are grading students’ work. Research is scarce on the issue of machine scoring bias, partly due to the secrecy of the companies that create these systems. Traditionally, essays are scored jointly by two human examiners, but it is far cheaper to have a machine grade an essay, or serve as a back-up grader to a human. Nevertheless, test administrators and some state education officials have embraced the technology. The systems are also unable to judge more nuanced aspects of writing, like creativity. But the engines also focus heavily on metrics like sentence length, vocabulary, spelling, and subject-verb agreement-the parts of writing that English language learners and other groups are more likely to do differently. Training essay-scoring engines on datasets of human-scored answers can ingrain existing bias in the algorithms.

The education industry has long grappled with conscious and subconscious bias against students from certain language backgrounds, as demonstrated by efforts to ban the teaching of black English vernacular in several states.ĪI has the potential to exacerbate discrimination, experts say. states utilize automated essay scoring systems, according to a Motherboard investigation.

“And not only will these machine learning programs pick up bias in the training sets, they’ll amplify it.”Īn interactive map shows which U.S. Bender, a professor of computational linguistics at the University of Washington. “The problem is that bias is another kind of pattern, and so these machine learning systems are also going to pick it up,” said Emily M.

They then predict what score a human would assign an essay, based on those patterns. They’re trained on sets of hundreds of example essays to recognize patterns that correlate with higher or lower human-assigned grades. And as a Motherboard experiment demonstrated, some of the systems can be fooled by nonsense essays with sophisticated vocabulary.Įssay-scoring engines don’t actually analyze the quality of writing. But in the remaining 18 states, only a small percentage of students’ essays-it varies between 5 to 20 percent-will be randomly selected for a human grader to double check the machine’s work.īut research from psychometricians-professionals who study testing-and AI experts, as well as documents obtained by Motherboard, show that these tools are susceptible to a flaw that has repeatedly sprung up in the AI world: bias against certain demographic groups. Of those 21 states, three said every essay is also graded by a human. Three states didn’t respond to the questions. Natural language processing (NLP) artificial intelligence systems-often called automated essay scoring engines-are now either the primary or secondary grader on standardized tests in at least 21 states, according to a survey conducted by Motherboard.

Best essay grader