Today in horrible algorithms: COMPAS (Correctional Offender Management Profiling for Alternative Sanctions).

[A] ProPublica analysis has found the risk scores remarkably unreliable in forecasting violent crime. Only 20% of the people predicted to commit violent crimes actually went on to do so. When a full range of crimes were taken into account – including misdemeanors such as driving with an expired license — the algorithm used to determine the score was somewhat more accurate than a coin flip. Of those deemed likely to re-offend, 61% were arrested for any subsequent crimes within two years.

The analysis also found significant racial disparities. In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate, but in very different ways.

The formula wrongly labeled black defendants as future criminals at almost twice the rate as white defendants. White defendants were mislabeled as low risk more often than black defendants. The disparity could not be explained by other factors.

Horrible, horrible, horrible. There are no other words.