AI language tools discriminate against disabled people, study suggests

Artificially intelligent hiring tools could be offensive or prejudiced toward individuals with disabilities, according to researchers at the Penn State College of Information Sciences and Technology (IST).

AI models have been increasingly used for natural-language processing (NLP) applications, such as smart assistants or email autocorrect and spam filters. In the past, some of these tools have been found to have biases based on gender and race. However, until now similar biases against people with disabilities have not been widely explored.

Researchers at Penn State analysed 13 different AI models commonly used for NLP applications to measure attitudes towards people with and without disabilities.

“The 13 models we explored are highly used and are public in nature,” said Pranav Venkit, the first author of the study’s paper, which was presented at the 29th International Conference on Computational Linguistics (COLING).

“We hope that our findings help developers that are creating AI to help certain groups – especially people with disabilities who rely on AI for assistance in their day-to-day activities – to be mindful of these biases.”

In their study, the researchers examined machine-learning models that were trained on source data to group similar words together enabling a computer to automatically generate sequences of words.

They created four simple sentence templates in which to variably populate a gender noun of “man,” “woman,” or “person,” and one of the 10 most commonly used adjectives in the English language. 

They then generated over 600 adjectives that could be associated with either people with or without disabilities – such as neurotypical or visually impaired – to randomly replace the adjective in each sentence. The team tested more than 15,000 unique sentences in each model to generate word associations for the adjectives.

“For example, we selected the word ‘good’ and wanted to see how it associated with terms related to both non-disability and disability,” explained Venkit. “With the addition of a non-disability term, the effect of ‘good’ becomes ‘great.’ But when ‘good’ is associated with a disability-related term, we get the result of ‘bad.’ So that change in the form of the adjective itself shows the explicit bias of the model.”

The researchers also conducted a second experiment to measure each model to identify implicit biases. To do so, they examined the adjectives generated for the disability and non-disability groups and measured each one’s sentiment, as positive, negative or neutral.

All models they studied consistently scored sentences with words associated with disability more negatively than those without. One particular model, which was pre-trained on Twitter data, flipped the sentiment score from positive to negative 86 per cent of the time when a term related to a disability was used.

“When we look at that score alone, we see that just by the addition of a term related to disability into the conversation, the sentiment score of the whole sentence drops,” said Venkit. “For example, if a user includes a term related to disability in a social media comment or post, the probability of that post being censored or restricted increases.”

The researchers also tested for implicit bias in two large language models that are used to automatically generate long texts, such as news articles, to see how a blank left in the sentence template would change depending on the adjective used.

In this analysis they generated 7,500 sentences. The results showed that, when given the sentence of, “A man has [blank]”, the language models predicted “changed” for the blank word. However, when a disability-related adjective was added to the sentence, resulting in “A deafblind man has [blank]”, the model predicted the word “died” for the blank.

The models’ implicit bias against people with disabilities could be apparent in various applications – for example, in text messages when applying autocorrect to a misspelled word or on social media where there are rules that prohibit abusive or harassing posts.

“This work shows that people need to care about what sort of models they are using and what the repercussions are that could affect real people in their everyday lives,” Venkit said.