All experiments used in this paper were implemented in Python 3 with the following dependencies
numpy
matplotlib
scikitlearn
To re-run all experiments used in the paper, run the following from the command line
./run-all.sh
This will re-run all the experiments and output the data to plaintext files in the results/ subdirectory.
To generate all plots used in the paper, run the following from the command line:
python plot-all.py
The datasets are given the following names
adult
german
singles
For each dataset there is a data loader module and a baseline (see the
Baselines section below). We will use adult as the prototype, and unless
otherwise stated all datasets operate the same way with adult replaced by the
dataset name. The raw data files are adult.train and adult.test. If
preprocessing occurred to split a dataset into training and testing subsets,
then the unprocessed data files are in the preprocessing/ subdirectory along
with python scripts to perform the (randomized) preprocessing. Additional
preprocessing is performed to turn categorical features into (possibly many)
binary features.
To load a dataset, you can run the following commands from the base directory of the project.
$ python
Python 3.3.3 (default, Dec 30 2013, 23:51:18)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
>>> from data import adult
>>> trainingData, testData = adult.load()
>>> adult.protectedIndex
1
>>> len(trainingData)
32561
>>> trainingData[0]
((39, 1, 0, 0, 0, 0, 0, 1, 0, 0, 13, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 2174, 0, 40, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0), -1)
An example experiment, testing the linear regression learner on the German dataset.
Python 3.6.3 (default, Oct 4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
>>> from data import german
>>> train, test = german.load()
>>> from margin import *
>>> def lrLearner(train, protectedIndex, protectedValue):
... marginAnalyzer = lrSKLMarginAnalyzer(train, protectedIndex, protectedValue)
... shift = marginAnalyzer.optimalShift()
... print('best shift is: %r' % (shift,))
... return marginAnalyzer.conditionalShiftClassifier(shift)
...
>>> h = lrLearner(train, german.protectedIndex, german.protectedValue)
best shift is: -0.19250157835095894
>>> from errorfunctions import signedStatisticalParity, labelError, individualFairness
>>> labelError(test, h)
0.25825825825825827
Copy-pastable:
from data import german
from errorfunctions import signedStatisticalParity, labelError, individualFairness
from margin import *
train, test = german.load()
def lrLearner(train, protectedIndex, protectedValue):
marginAnalyzer = lrSKLMarginAnalyzer(train, protectedIndex, protectedValue)
shift = marginAnalyzer.optimalShift()
print('best shift is: %r' % (shift,))
return marginAnalyzer.conditionalShiftClassifier(shift)
h = lrLearner(train, german.protectedIndex, german.protectedValue)
labelError(test, h)
The experiments are organized by method, using the acronyms from the paper. So
random relabeling (RR) is in the experiment-RR.py file. Each experiment has a
runAll() function that runs all of the experiments for every dataset and
learner (SVM, logistic regression, and AdaBoost). Note that boosting and SVM
take ~5-30 minutes per run on large datasets, and each experiment averages over
10 runs.
The main plots in the paper are produced by the MarginAnalyzer class in
margin.py. See the MarginAnalyzer.plotMarginHistogram and
MarginAnalyzer.plotTradeoff functions for details.