# # We now know that our model is selecting individuals who are female far less # often than individuals who are male. There is a similar effect when # examining the results by race, with blacks being selected far less often than # whites (and those classified as 'other'). However, there are many cases where # presenting all these numbers at once will not be useful (for example, a high # level dashboard which is monitoring model performance). Fairlearn provides # several means of aggregating metrics across the subgroups, so that disparities # can be readily quantified. # # The simplest of these aggregations is ``group_min()``, which reports the # minimum value seen for a subgroup for each underlying metric (we also provide # ``group_max()``). This is # useful if there is a mandate that "no subgroup should have an ``fbeta_score()`` # of less than 0.6." We can evaluate the minimum values easily: grouped_on_race.group_min() # %% # As noted above, the selection rates varies greatly by race and by sex. # This can be quantified in terms of a difference between the subgroup with # the highest value of the metric, and the subgroup with the lowest value. # For this, we provide the method ``difference(method='between_groups)``: grouped_on_race.difference(method='between_groups') # %% # We can also evaluate the difference relative to the corresponding overall # value of the metric. In this case we take the absolute value, so that the # result is always positive: grouped_on_race.difference(method='to_overall') # %%
# %%
# Creating a derived metric
# =========================
#
# Suppose our key metric is the accuracy score, and we are most interested in
# ensuring that it exceeds some threshold for all subgroups
# We might use the :class:`~fairlearn.metrics.MetricFrame` as
# follows:
acc_frame = MetricFrame(
metrics=skm.accuracy_score,
y_true=y_test,
y_pred=y_pred,
sensitive_features=A_test["sex"]
)
print("Minimum accuracy_score: ", acc_frame.group_min())
# %%
# We can create a function to perform this in a single call
# using :func:`~fairlearn.metrics.make_derived_metric`.
# This takes the following arguments (which must always be
# supplied as keyword arguments):
#
# - :code:`metric=`, the base metric function
# - :code:`transform=`, the name of the aggregation
# transformation to perform. For this demonstration, we
# want this to be :code:`'group_min'`
# - :code:`sample_param_names=`, a list of parameter names
# which should be treated as sample
# parameters. This is optional, and defaults to
# :code:`['sample_weight']` which is appropriate for many