Dominique Cardon on algorithmic fairness

Issues of fairness in algorithms are not limited to the US. There is a robust research effort in many parts of Europe on issues of fairness and discrimination. The regulatory regimes and legal frameworks are different in Europe, which makes the discussions quite different. But the concerns are universal.

The linked article (in French) is an interview with Dominique Cardon, a French sociologist who’s written a new book on algorithms and big data. Since I don’t speak French, my understanding of the interview is limited to the translation. However, he makes a number of interesting points about our new algorithmic world:

  • that the effect of algorithmic governance in our world cannot be thought of in terms of traditional controls like censorship. It’s more of a bottom up ‘nudge’-based system that constructs elaborate and invisible rewards. To use the somewhat trite but still-useful analogy, it’s not 1984, but a Brave New World, and is all the more insidious for it.
  • We can’t disengage from an algorithmic world: the whole “if you don’t like being watched, don’t go on Facebook” argument is rapidly losing credibility, because eventually we will have no choice but to participate and contribute to the floods of data being generated. This makes “opening the black box” even more crucial.

Not surprisingly, I was happy to hear his points about fairness in algorithms: specifically,

You can not ask an algorithm to be “neutral.” However, it must be “fair.” [….] And for that, it is useful for researchers and civil society [to] create instruments of verification and control.

Living in a bad neighborhood takes on a whole new meaning

The neighborhood you live in might control which schools you have access to and what a house costs. But it might also control what kind of sentencing guidelines are in effect if you have the misfortune of being sent to jail.

Pennsylvania will soon begin using factors other than a convict’s criminal history — such as the person’s age, gender, or county of residence — in its sentencing guidelines…

This is part of a long-running discussion on data-driven approaches to recidivism and criminal justice in general. The (honorable) rationale behind these efforts is to eliminate explicit bias/prejudice in sentencing, which (as the article points out) was governed by more lax guidelines than for investigating crimes in the first place.

And as always, I don’t think there’s a problem in using data-driven methods to inform and guide the underlying mechanisms that might cause recidivism. But it gets a lot trickier when these tools are used to guide sentencing. In the case of Pennsylvania, the tool will be used to “assist” judges in sentencing decisions. And as Goodhart’s law suggests, once you attempt to quantify something as nebulous as “likely to re-offend”, the quantification takes on a life of its own.

But even this isn’t the biggest problem. There’s now a long history of models for predicting recidivism, and one of the most studied ones is the LSI-R (and its generalization, the LSRNR). But these models are proprietary: you need to purchase the software to see what they do.

And if you think that’s not a problem, I’d like to point you to this article by Rebecca Weller in Slate. In particular

Today, closed, proprietary software can put you in prison or even on death row. And in most U.S. jurisdictions you still wouldn’t have the right to inspect it.

And as she goes on to point out,

Inspecting the software isn’t just good for defendants, though—disclosing code to defense experts helped the New Jersey Supreme Court confirm the scientific reliability of a breathalyzer.

At the heart of these arguments is a point that has nothing to do with algorithms.

Wanting transparency does not imply lack of trust. It reflects a concern about a potential violation of trust.