21 fairness definitions and their politics

link to tutorial

Introduction

Narayanan refers the audience to another work titled “Interventions Over Predictions”, which questions the assumption that algorithms should be trained to maximize predictive accuracy. Consider for example recidivism prediction algorithms. The same system could be used to try to understand why certain demographics appear prone to re-offend and identify opportunities for intervention.

Statistical Bias

The difference between an estimator’s expected value and the true value.

Here Narayanan gives the example of COMPAS and how its scores are not biased, as defined by statistical bias. But he is also quick to caution that the scores are not biased “w.r.t. rearrest”, which is distinct from recidivism.

Group Fairness

Do outcomes systematically differ between demographic groups (or other population groups)?

Impossibility Theorem

If an instrument satisfies predictive parity […] but the prevalence differs between groups, the instrument cannot achieve equal false positive and false negative rates across those groups.

Different stakeholders focus on different metrics - accuracy, false positive rate, false negative rate etc.

Narayanan claims that the three metrics of accuracy, false positive rate and false negative rate are not special in anyway and suggests that similar Impossibility Theorems can be induced with other metrics (found here).

Other Notes

  • The Impossibility Theorems also apply to human decision-making.
  • Ricci vs DeStefano is an interesting case
  • Narayanan notes three common reasons for different prevalences:
    • Measurement Bias - e.g. using rearrests instead of recidivism
    • Historical Prejudice - e.g. poverty cycles
    • Intrinsic Differences
  • Are randomized classifiers fair?