Racist risk assessments, algorithmic fairness, and the issue of harm

By now, you are likely to have heard of the fascinating report (and white paper) released by ProPublica describing the way that risk assessment algorithms in the criminal justice system appear to affect different races differently, and are not particularly accurate in their predictions. Even worse, they are even worse at predicting outcomes for black subjects than for white. Notice that this is a separate problem than ensuring equal outcomes pace disparate impact: it’s the problem of ensuring equal failure modes as well.


There is much to pick apart in this article, and you should read the whole thing yourself. But from the perspective of research in algorithmic fairness, and how this research is discussed in the media, there’s another very important consequence of this work.

It provides concrete examples of people who have possibly been harmed by algorithmic decision-making. 

We talk to reporters frequently about the larger set of questions surrounding algorithmic accountability and eventually they always ask some version of:

Can you point to anyone who’s actually been harmed by algorithms?

and we’ve never been able to point to specific instances so far. But now, after this article, we can.



White House Report on Algorithmic Fairness

The White House has put out a report on big data and algorithmic fairness (announcement, full report).  From the announcement:

Using case studies on credit lending, employment, higher education, and criminal justice, the report we are releasing today illustrates how big data techniques can be used to detect bias and prevent discrimination. It also demonstrates the risks involved, particularly how technologies can deliberately or inadvertently perpetuate, exacerbate, or mask discrimination.

The table of contents for the report gives a good overview of the issues addressed:

Big Data and Access to Credit
The Problem: Many Americans lack access to affordable credit due to thin or non-existent credit files.
The Big Data Opportunity: Use of big data in lending can increase access to credit for the financially underserved.
The Big Data Challenge: Expanding access to affordable credit while preserving consumer rights that protect against discrimination in credit eligibility decisions

Big Data and Employment
The Problem: Traditional hiring practices may unnecessarily filter out applicants whose skills match the job opening.
The Big Data Opportunity: Big data can be used to uncover or possibly reduce employment discrimination.
The Big Data Challenge: Promoting fairness, ethics, and mechanisms for mitigating discrimination in employment opportunity.

Big Data and Higher Education
The Problem: Students often face challenges accessing higher education, finding information to help choose the right college, and staying enrolled.
The Big Data Opportunity: Using big data can increase educational opportunities for the students who most need them.
The Big Data Challenge: Administrators must be careful to address the possibility of discrimination in higher education admissions decisions.

Big Data and Criminal Justice
The Problem: In a rapidly evolving world, law enforcement officials are looking for smart ways to use new technologies to increase community safety and trust.
The Big Data Opportunity: Data and algorithms can potentially help law enforcement become more transparent, effective, and efficient.
The Big Data Challenge: The law enforcement community can use new technologies to enhance trust and public safety in the community, especially through measures that promote transparency and accountability and mitigate risks of disparities in treatment and outcomes based on individual characteristics.

Friday links dump

What I’ve been reading (or meaning to read) this week:

Related links

A dump of what I’ve been reading lately:


NPR: Can Computers be Racist?


As will come as no surprise to readers of this blog, algorithms can make biased decisions.  NPR tackles this question in their latest All Tech Considered (which I was interviewed for!).

They start by talking to Jacky Alcine, the software engineer who discovered that Google Photos had tagged his friend as an animal:

As Jacky points out: “One could say, ‘Oh, it’s a computer,’ I’m like OK … a computer built by whom? A computer designed by whom? A computer trained by whom?” It’s a short segment, but we go on to talk a bit about how that bias could come about.

What I want to emphasize here is that, while hiring more Black software engineers would likely help and make it more likely that these issues would be caught quickly, it is not enough. As Jacky implies, the training data itself is biased. In this case, likely by including more photos of white people and animals than of Black people. In other cases, because the labels have been created by people whose past racist decisions are being purposefully used to guide future decisions.

Consider the automated hiring algorithms now touted by many startups (Jobaline, Hirevue, Gild, …). If an all-white company attempts to use their current employees as training data, i.e., attempts to find future employees who are like their current employees, then they’re likely to continue being an all-white company. That’s because the data about their current employees encodes systemic racial bias such as differences between white and Black SAT test-takers even when controlling for ability. Algorithmic decisions will find and replicate this bias.

We need to be proactive to keep such biases from influencing algorithmic decisions.

Disparate impact ruled discriminatory in housing

Symposium: Supreme Court’s victory for disparate impact includes a cautionary tale

The Supreme Court decided a huge case today in favor of civil rights… and in favor of statistics! More nuance on the legal opinion can be found above and at many other posts at SCOTUSblog. But for computational purposes, I find these sections interesting:

the high court cabined disparate impact liability to those policies that pose “artificial, arbitrary, and unnecessary barriers.” That important qualifier may ultimately determine the outcome of this case on remand.

Does this mean that if we develop a model that has high utility and decreases disparate impact that the previous decision will be ruled unnecessary?

Most importantly, the full majority cautions that disparate impact liability poses “special dangers” that must be limited to avoid serious constitutional questions that might arise under the FHA if, for example, such liability were imposed based solely on a showing of a statistical disparity. This requires giving housing authorities and private developers adequate leeway to explain the valid interests their policies serve, an analysis that is analogous to Title VII’s business necessity defense. The Court emphasizes that policies are not contrary to the disparate impact requirement unless they are “artificial, arbitrary, and unnecessary barriers.” And the Court confirms that a disparate impact claim relying on a statistical disparity must fail if the plaintiff cannot point to a defendant’s policy causing that disparity. The Court views this crucial causality requirement as necessary to ensure that defendants (and courts) do not resort to the use of racial quotas.

There’s a lot here, but I think the big point is the importance of interpretable models. It sounds like the plaintiff must be able to interpret the defendant’s model in order to bring a valid claim. So we also need interpretability of models even without the right to examine or build the model ourselves! (Though I would argue that this is an unreasonable standard for the law to expect and that the burden should be on the defendant to provide an interpretable version of their policy. Perhaps it is?)

As today’s decision presages, the next challenge to “disparate impact” theory the Court will undoubtedly be forced to consider may prove to be a far more difficult one. As Justice Scalia noted in his concurrence in Ricci v. DeStefano, whether any statute that affirmatively requires race-based actions to remedy “disparate impacts” can be harmonized with the Fourteenth Amendment’s guarantee of equal protection is not an easy question to answer.

This is the question of if it’s possible to repair disparate impact without seeing its effect first. The assumption seems to be “no”, but we’ve actually shown a way of doing this (see paper here).

Another inspiring interpretation of the ruling can be found here. It argues that the ruling acknowledges “unconscious prejudice.” I.e., the statistics might show that a decision was discriminatory even if you didn’t mean it to be!