How that Machine Algorithm Ended up being Racist

Sometimes, people engage in backdoor discrimination on purpose. They won’t discriminate on race; they discriminate based on clothing or hairstyle or music that ends up being highly correlated with race. The federal government now considers using criminal records backdoor housing discrimination. People say the use of credit scores in job hiring are an attempt at backdoor discrimination. Voter IDs laws are regularly labeled backdoor discrimination. There is much debate as to whether such backdoors are legal or moral. Some people “see no racism” when it comes to using proxies for race:

Well, if you are hiring a secretary and there are two applicants with minimal experience and same resume, but one is called Jane Smith and other is Shereesha Taulava, name does have meaning since you probably can only pronounce one of these names right just by reading them. So you are more likely to interview them first. And if you have a quota to send X amount of letters, then these easily pronounceable names do have advantage. Still I see no racism, people just naturally go through path of least resistance with these sort of things.

Whether any specific case is intentional or not, intentional backdoor discrimination has a long history. It isn’t as if the day protected-class discrimination was made illegal everyone stopped wanted to do it anymore.

But backdoor discrimination can happen unintentionally. It can happen even if you go out of your way to try to not be racist. In fact, racism can be a consequence of excluding race as a factor. That’s likely what happened with the algorithm that produced these results:

• The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.
• White defendants were mislabeled as low risk more often than black defendants.

To get an intuitive sense of how this can happen, let’s consider a hypothetical example that isn’t quite so politically charged, but still yummily racist.

You want to determine which U.S. 8th-graders are likely to be gifted at picking up new languages. This will be used to identify children who will get special, desirable language instruction in high school targeted toward their talents.

You collect a lot of information about the kids. You have standardized test scores, teacher assessments, and their scores on that marshmallow test thing you read about in the Huffington Post. What you don’t bother to do is collect anything about the parents. Also, you don’t record anything about race because you are trying to not be racist today.

Whatever else you might add in to your model, one variable is likely to be highly predictive: does the kid already know any languages other than English and how well does the kid know them?

This way, you’re able to find the two gifted kids who can understand some Japanese because they were motivated to learn it so they could understand an anime series they were interested in. You’ll meet some other weirdly gifted kids who managed to pick up another language by overhearing some conversations in their neighborhoods.

But these won’t be the only people you identify. You would also eventually identify my daughter, who understands Chinese because she was Chinese. She’s not in the same league as the gifted kids of monolingual parents who nevertheless figured out how to comprehend any amount of a second language for themselves, but she will get credited by your algorithm that doesn’t consider information about the parents.

Your model does succeed in identifying linguistically gifted kids. Those kids really did understand that anime series. But your algorithm also picks up normal kids who happen to have parents who speak another language to them.

Your model is not perfect. But that’s not all. It’s going to be bad in a way that exhibits bias. And it will end up being racial bias despite your never including race in your model.

The gifted white kid who figured out how to watch her favorite anime show with 80% comprehension isn’t really going to be judged as that impressive by your model. After all, she’s not fluent. Her vocabulary is limited to what’s in the show. Despite her extraordinary accomplishment, your model will identify her as a prospect, but won’t give her all the credit she deserves.

Compare that to the kid born to Japanese parents. She is fluent in Japanese and English, but this is a direct consequence of her environment, not any particular giftedness on her part. Your model, will, nevertheless, be incredibly impressed with her.

Your model will systematically underrate the giftedness of kids of monolingual parents and systematically overrate the giftedness of kids of bilingual parents. This will almost certainly result in some net racism against non-Hispanic whites and towards races that have a higher percentage of recent migrants (Asians and Hispanics). It has this bias even though your model doesn’t even know the race of the kids it is classifying. You made a race-blind, racist computer.

Even though your heart is as pure as Seventh Generation chlorine-free bleach, your model ended up being racist anyway.

Here’s that happened. Reality racistly made some races in the United States be more likely to be bilingual and other races be somewhat more monolingual. You then unsuspectingly used this information that is highly-correlated with race, and it was through that that racial bias snuck in.

There are a couple ways to fix this. Ideally, one would be to include the number of languages known by the parents as a control variable. But you could also just include race as a control variable. By explicitly accounting for race, you could eliminate much of the racial bias that has snuck in through backdoor proxies for race. Asians would get a systematic penalty, and whites would get a systematic bonus, and the results would no longer be skewed against non-Hispanic whites. ¹

Now let’s consider the computer algorithm predicting recidivism. It “includes factors such as education levels, and whether a defendant has a job.” ProPublica further explains:

Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions. The survey asks defendants such things as: “Was one of your parents ever sent to jail or prison?” “How many of your friends/acquaintances are taking drugs illegally?” and “How often did you get in fights while at school?” The questionnaire also asks people to agree or disagree with statements such as “A hungry person has a right to steal” and “If people make me angry or lose my temper, I can be dangerous.”

This partial list seems loaded with potential backdoor proxies for race. The probability that one of your parents was ever sent to jail is surely inseparable from race. And it’s inclusion in a model is likely to create a bias in the direction that parallels our language-acquisition example. Thankfully, a white man with a parent who has done time in jail is an oddity. Unfortunately, a black man with the same is less so. Having a parent who has been in jail probably does do something to predict your own likelihood of recidivism, but it’s likely to sweep up ordinary black people along with the extraordinary whites just as our giftedness algorithm swept up first-generation immigrants along with extraordinary whites.

Image by Daniel E Lee

There’s a lesson here about how invidious racism is, but the more interesting lesson (to me) is how invidious correlations among your dependent variables are.  Race has affected everything. It is not present only in the categorical you check on a census form. It has left an imprint on much that has happened to you. To simply drop race as a variable and think you’ve become race-blind is farce. The net effect of doing so is likely to actually disadvantage those you seek to help by being blind to race.

1. Note that this revised model that includes the race of the children would create a new problem. Asians with monolingual parents would now be unfairly disadvantaged, and non-Hispanic whites with multilingual parents would be unfairly advantaged.