Embed Size (px)
Citation preview
A Moral Framework for Understanding Fair MLthrough Economic Models of Equality of Opportunity
Hoda HeidariETH Zürich
Michele LoiUniversity of Zü[email protected]
Krishna P. GummadiMPI-SWS
Andreas KrauseETH Zürich
ABSTRACTWe map the recently proposed notions of algorithmic fairness toeconomic models of Equality of opportunity (EOP)—an extensivelystudied ideal of fairness in political philosophy. We formally showthat through our conceptual mapping, many existing definition ofalgorithmic fairness, such as predictive value parity and equalityof odds, can be interpreted as special cases of EOP. In this respect,our work serves as a unifying moral framework for understand-ing existing notions of algorithmic fairness. Most importantly, thisframework allows us to explicitly spell out the moral assumptionsunderlying each notion of fairness, and interpret recent fairnessimpossibility results in a new light. Last but not least and inspiredby luck egalitarian models of EOP, we propose a new family of mea-sures for algorithmic fairness.We illustrate our proposal empiricallyand show that employing a measure of algorithmic (un)fairnesswhen its underlying moral assumptions are not satisfied, can havedevastating consequences for the disadvantaged group’s welfare.
CCS CONCEPTS• Computing methodologies → Supervised learning; Batchlearning; • Applied computing→ Economics; Sociology;
KEYWORDSEquality of Opportunity (EOP), Fairness for Machine Learning,Rawlsian and Luck Egalitarian EOP, Statistical Parity, Equality ofOdds, Predictive Value Parity
ACM Reference Format:Hoda Heidari, Michele Loi, Krishna P. Gummadi, and Andreas Krause. 2019.A Moral Framework for Understanding Fair ML, through Economic Modelsof Equality of Opportunity. In FAT* ’19: Conference on Fairness, Account-ability, and Transparency (FAT* ’19), January 29–31, 2019, Atlanta, GA, USA.ACM,NewYork, NY, USA, 10 pages. https://doi.org/10.1145/3287560.3287584
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected]* ’19, January 29–31, 2019, Atlanta, GA, USA© 2019 Association for Computing Machinery.ACM ISBN 978-1-4503-6125-5/19/01. . . $15.00https://doi.org/10.1145/3287560.3287584
1 INTRODUCTIONEquality of opportunity (EOP) is a widely supported ideal of fair-ness, and it has been extensively studied in political philosophyover the past 50 years [1, 6, 8, 9, 21, 26]. The concept assumes theexistence of a broad range of positions, some of which are moredesirable than others. In contrast to equality of outcomes (or posi-tions), an equal opportunity policy seeks to create a level playingfield among individuals, after which they are free to compete fordifferent positions. The positions that individuals earn under thecondition of equality of opportunity reflect their merit or deserv-ingness, and for that reason, inequality in outcomes is consideredethically acceptable [23].
Equality of opportunity emphasizes the importance of personal(or native) qualifications, and seeks to minimize the impact of cir-cumstances and arbitrary factors on individual outcomes [6, 8, 9, 21].For instance within the context of employment, one (narrow) in-terpretation of EOP requires that desirable jobs are given to thosepersons most likely to perform well in them—e.g. those with thenecessary education and experience—and not according to arbitraryfactors, such as race or family background. According to Rawls’s(broader) interpretation of EOP, native talent and ambition can jus-tify inequality in social positions, whereas circumstances of birthand upbringing such as sex, race, and social background can not.Many consider the distinction between morally acceptable andunacceptable inequality the most significant contribution of theegalitarian doctrine [25].
Prior work in economics has sought to formally characterizeconditions of equality of opportunity to allow for its precise mea-surement in practical domains (see e.g. [11, 24]). At a high level,in these models an individual’s outcome/position is assumed tobe affected by two main factors: his/her circumstance c and efforte . Circumstance c is meant to capture all factors that are deemedirrelevant, or for which the individual should not be held morallyaccountable; for instance c could specify the socio-economic statushe/she is born into. Effort e captures all accountability factors—those that can morally justify inequality. (Prior work in economicsrefers to e as effort for the sake of concreteness, but e summarizesall factors for which the individual can be held morally accountable;the term “effort" should not be interpreted in its ordinary sensehere.) For any circumstance c and any effort level e , a policy ϕ in-duces a distribution of utility among people of circumstance c andeffort e . Formally, an EOP policy will ensure that an individual’sfinal utility will be, to the extent possible, only a function of theireffort and not their circumstances.
FAT* ’19, January 29–31, 2019, Atlanta, GA, USA Heidari et al.
While EOP has been traditionally discussed in the context ofemployment practices, its scope has been expanded over time toother areas, including lending, housing, college admissions, andbeyond [28]. Decisions made in such domains are increasingly auto-mated andmade throughAlgorithmic Data Driven DecisionMakingsystems (A3DMs). We argue, therefore, that it is only natural tostudy fairness for A3DMs through the lens of EOP. In this work, wedraw a formal connection between the recently proposed notions offairness for supervised learning and economic models of EOP. Weobserve that in practice, predictive models inevitably make errors(e.g. the model may mistakenly predict that a credit-worthy appli-cant won’t pay back their loan in time). Sometimes these errorsare beneficial to the subject, and sometimes they cause harm. Weposit that in this context, EOP would require similar individuals(in terms of what they can be held accountable for) to have thesame prospect of receiving this benefit/harm, irrespective of theirirrelevant characteristics.
More precisely, we assume that a person’s features can be par-titioned into two sets: those for which we consider it morally ac-ceptable to hold him/her accountable, and those for which it isnot so. We will broadly refer to the former set of attributes as theindividual’s accountability features, and the latter, as their arbitraryor irrelevant features. Note that there is considerable disagreementon the criteria to determine what factors should belong to eachcategory. Roemer [22] for instance proposes that societies decidethis democratically. We take a neutral stance on this issue andleave it to domain experts and stake-holders to reach a resolution.Throughout, we assume this partition has been identified and isgiven.
We distinguish between an individual’s actual and effort-basedutility when subjected to algorithmic decision making. We assumean individual’s advantage or total utility as the result of beingsubject to A3DMs, is the difference between their actual and effort-based utility (Section 2). Our main conceptual contribution is tomap the supervised learning setting to economic models of EOP bytreating predictive models as policies, irrelevant features as individ-ual circumstance, and effort-based utilities as effort (Figure 1). Weshow that using this mapping many existing notions of fairness forclassification, such as predictive value parity [17] and equality ofodds [13], can be interpreted as special cases of EOP. In particular,equality of odds is equivalent to Rawlsian EOP, if we assume allindividuals with the same true label are equally accountable fortheir labels and have the same effort-based utility (Section 3.1).Similarly, predictive value parity is equivalent to luck egalitarianEOP if the predicted label/risk is assumed to reflect an individual’seffort-based utility (Section 4). In this respect, our work serves as aunifying framework for understanding existing notions of algorith-mic fairness as special cases of EOP. Importantly, this frameworkallows us to explicitly spell out the moral assumptions underlyingeach notion of fairness, and interpret recent fairness impossibilityresults [17] in a new light.
Last but not least, inspired by Roemer’s model of egalitarian EOPwe present a new family of measures for algorithmic (un)fairness,applicable to supervised learning tasks beyond binary classifica-tion. We illustrate our proposal on a real-world regression dataset,and compare it with existing notions of fairness for regression. We
Policy 𝝓 Predictive model h
Effort e Effort-based utility d
Circumstance c Irrelevant features z
Utility u Actual - effort-based utility (a - d)
Economic Models of EOP Fair Machine Learning
Figure 1: Our proposed conceptual mapping between FairML and economic literature on EOP.
empirically show that employing the wrong measure of algorith-mic fairness—when the moral assumptions underlying it are notsatisfied—can have devastating consequences on the welfare of thedisadvantaged group.
We emphasize that our work is not meant to advocate for any par-ticular notion of algorithmic fairness, rather our goal is to establish—both formally and via real-world examples—that implicit in eachnotion of fairness is a distinct set of moral assumptions about de-cision subjects; therefore, each notion of fairness is suitable onlyin certain application domains and not others. By making theseassumptions explicit, our framework presents practitioners with anormative guideline to choose the most suitable notion of fairnessspecifically for every real-world context in which A3DMs are to bedeployed.
1.1 Equality of Opportunity: An OverviewEquality of opportunity has been extensively debated among po-litical philosophers. Philosophers such as Rawls [21], Dworkin [8],Arneson [1], and Cohen [6] contributed to the egalitarian school ofthought by proposing different criteria for making the cut betweenarbitrary and accountability factors. The detailed discussion of theirinfluential ideas is outside the scope of this work, and the interestedreader is referred to excellent surveys by Arneson [2] and Roemerand Trannoy [25].
In this section, we briefly mention several prominent interpre-tations of EOP and discuss their relevance to A3DMs. FollowingArneson [3], we recount three main conceptions of equality ofopportunity:• Libertarian EOP: A person is morally at liberty to do whatshe pleases with what she legitimately owns (e.g. self, busi-ness, etc.) as long as it does not infringe upon other people’smoral rights (e.g. the use of force, fraud, theft, or damageon persons or property of another individual is considered aviolation of their rights). Other than these restrictions, anyoutcome that occurs as the result of people’s free choices ontheir legitimate possessions is considered just. In the contextof A3DMs and assuming no gross violations of individuals’data privacy rights, this interpretation of EOP leaves the en-terprise at total liberty to implement any algorithm it wishes
A Moral Framework for Understanding Fair MLthrough Economic Models of Equality of Opportunity FAT* ’19, January 29–31, 2019, Atlanta, GA, USA
for decision making. The algorithm can utilize all availableinformation, including individuals’ sensitive features such asrace or gender, to make (statistically) accurate predictions.• Formal EOP: Also known as “careers open to talents", for-mal EOP require desirable social positions to be open to allwho possess the attributes relevant for the performance ofthe duties of the position (e.g. anyone who meets the formalrequirements of the job) and wish to apply for them [24].The applications must be assessed only based on relevantattributes/qualifications that advances the morally innocentgoals of the enterprise. Direct discrimination based on factorsdeemed arbitrary (e.g. race or gender) is therefore prohibitedunder this interpretation of EOP. Formal EOP would permitdifferences in people’s circumstances—e.g. their gender—tohave indirect, but nonetheless deep impact on their prospects.For instance, if women are less likely to receive higher edu-cation due to prejudice against female students, as long asa hiring algorithm is blind to gender and applies the sameeducational requirement to male and female job applicants,formal equality of opportunity is maintained. In context ofA3DMs, Formal EOP is equivalent to the removal of the sen-sitive feature information from the learning pipeline. In thefair ML community, this is sometimes referred to as “fairnessthrough blindness".• Substantive EOP: Substantive EOPmoves the starting pointof the competition for desirable positions further back intime, and requires not only open competition for desirablepositions, but also fair access to the necessary qualificationsfor the position. This implies access to qualifications (e.g.formal requirements for a job) should not to be affected byarbitrary factors, such as race gender or social class. Theconcept is closely related to indirect discrimination: if theA3DM indirectly discriminates against people with a certainirrelevant feature (e.g. women or African Americans) thismay be an indication that the irrelevant/arbitrary feature hasplayed a role in the acquisition of the requirements. Whenthere are no alternative morally acceptable explanations forit, indirect discrimination is often considered in violation ofsubstantive EOP.
Our focus in this work is on substantive EOP, and in particular, ontwo of its refinements, called Rawlsian EOP and Luck EgalitarianEOP.
Rawlsian EOP. According to Rawls, those who have the samelevel of talent or ability and are equally willing to use them musthave the same prospect of obtaining desirable social positions, re-gardless of arbitrary factors such as socio-economic background [21].This Rawlsian conception of EOP has been translated into pre-cise mathematical terms as follows [18]: let c denote circumstance,capturing factors that are not considered legitimate sources of in-equality among individuals. Let scalar e summarize factors that areviewed as legitimate sources of inequality. For the sake of brevity,the economic literature refer to e as “effort", but e is meant to sum-marize all factors an individual can be held morally accountable
for.1 Let u specify individual utility, which is a consequence of ef-fort, circumstance, and policy. Formally, let Fϕ (.|c, e ) specify thecumulative distribution of utility under policy ϕ at a fixed effortlevel e and circumstance c . Rawlsian/Fair EOP requires that forindividuals with similar effort e , the distribution of utility shouldbe the same—regardless of their circumstances:
Definition 1 (Rawlsian Eqality of Opportunity (R-EOP)).A policy ϕ satisfies Rawlsian EOP if for all circumstances c, c ′ and alleffort levels e ,
Fϕ (.|c, e ) = Fϕ (.|c ′, e ).
Note that this conception of EOP takes an absolutist view of effort:it assumes e is a scalar whose absolute value is meaningful and canbe compared across individuals. This view requires effort e to beinherent to individuals and not itself impacted by the circumstancec or the policy ϕ.
Luck Egalitarian EOP. Unlike fair EOP, luck egalitarian EOPoffers a relative view of effort, and allows for the possibility of cir-cumstance c and implemented policy ϕ impacting the distributionof effort e . In this setting, Roemer [23] argues that “in comparingefforts of individuals in different types [circumstances], we shouldsomehow adjust for the fact that those efforts are drawn from distri-butions which are different". As the solution he goes on to propose“measuring a person’s effort by his rank in the effort distribution ofhis type/circumstance, rather than by the absolute level of effort heexpends".
Formally, let Fc,ϕE be the effort distribution of type c under pol-icy ϕ. Roemer argues that “this distribution is a characteristic ofthe type c , not of any individual belonging to the type. Therefore,an inter-type comparable measure of effort must factor out thegoodness or badness of this distribution". Roemer declares two in-dividuals as having exercised the same level of effort if they sit atthe same quantile or rank of the effort distribution for their corre-sponding types. More precisely, let the indirect utility distributionfunction Fϕ (.|c,π ) specify the distribution of utility for individualsof type c at the π th quantile (0 ≤ π ≤ 1) of Fc,ϕE . Equalizing opportu-nities means choosing the policy ϕ to equalize utility distributions,Fϕ (.|c,π ), across types at fixed levels of π :2
Definition 2 (Luck Egalitarian Eqality of Opportunity(e-EOP)). A policyϕ satisfies Luck Egalitarian EOP if for all π ∈ [0, 1]and any two circumstances c, c ′:
Fϕ (.|c,π ) = Fϕ (.|c ′,π ).
To better understand the subtle difference between RawlsianEOP and luck egalitarian EOP, consider the following example:suppose in the context of employment decisions, we consider yearsof education as effort, and gender as circumstance. Suppose Alice1Note that in Rawls’s formulation of EOP, talent and ambition are treated as a legiti-mate source of inequality, even when they are independent of a person’s effort andresponsibility. The mathematical formulation proposed here includes talent, ability andambition all in the scalar e . Whether natural talent should be treated as a legitimatesource of inequality is a subject of controversy. As stated earlier, throughout this workwe assume such questions have been already answered through a democratic processand/or deliberation among stakeholders and domain experts.2Note that in Roemer’s original work, utility is assumed to be a deterministic functionof c, e, ϕ . Here we changed the definition slightly to allow for the possibility ofnon-deterministic dependence.
FAT* ’19, January 29–31, 2019, Atlanta, GA, USA Heidari et al.
and Bob both have 5 years of education, whereas Anna and Benhave 3 and 7 years of education, respectively. Rawlsian EOP wouldrequire Alice and Bob to have the same employment prospects, soit would ensure that factors such as sexism wouldn’t affect Alice’semployment chances, negatively (compared to Bob). Luck egali-tarian EOP goes a step further and calculates everyone’s rank (interms of years of education) among all applicants of their gender. Inour example, Alice is ranked 1st and Anna is ranked 2nd. Similarly,Bob is ranked 2nd and Ben is ranked 1st. A luck egalitarian EOPpolicy would ensure that Alice and Ben have the same employmentprospects, and may indeed assign Bob to a less desirable positionthan Alice—even though they have similar years of education.
Next, we will discuss the above two refinements of substantiveEOP in the context of supervised learning.
2 SETTINGAs a running example in this section, we consider a business ownerwho uses A3DM to make salary decisions so as to improve businessproductivity/revenue. We assume a higher salary is considered tobe more desirable by all employees. An A3DM is designed to predictthe salary that would improve the employee’s performance at thejob, using historical data. This target variable, as we will shortlyformalize, does not always coincide with the salary the employeeis morally accountable/qualified for.
We consider the standard supervised learning setting. A learningalgorithm receives a training data set T = {(xi ,yi )}ni=1 consistingof n instances, where xi ∈ X specifies the feature vector for in-dividual i and yi ∈ Y , the true label for him/her (the salary thatwould improve his/her performance). Unless otherwise specified,we assume Y = {0, 1} and X = Rk . Individuals are assumed to besampled i.i.d. from a distribution F . The goal of a learning algorithmis to use the training data T to fit a model (or pick a hypothesis)h : X → Y that accurately predicts the label for new instances. LetH be the hypothesis class consisting of all the models the learningalgorithm can choose from. A learning algorithm receives T as theinput; then utilizes the data to select a model h ∈ H that minimizessome empirical loss, L (T ,h). We denote the predicted label for anindividual with feature vector x by y (i.e. y = h(x)).
Consider an individual who is subject to algorithmic decisionmaking in this context. To discuss EOP, we begin by assumingthat his/her observable attributes, x, can be partitioned into twodisjoint sets, x = ⟨z,w⟩, where z ∈ Z denotes the individual’sobservable characteristics for which he/she is considered morallynot accountable—this could include sensitive attributes such as raceor gender, as well as less obvious attributes, such as zip code. Werefer to z as morally arbitrary or irrelevant features. Let w ∈ Wdenote observable attributes that are deemed morally acceptable tohold the individual accountable for; in the running example, thiscould include the level of job-related education and experience.We refer to w as accountability or relevant features. We emphasizeonce again that determining what factors should belong to eachcategory is entirely outside the scope of this work. We assumethroughout that a resolution has been previously reached in thisregard—through the appropriate process—and is given to us.
Let d ∈ [0, 1] specify the individual’s effort-based utility—theutility he/she should receive solely based on their accountability
factors (e.g. the salary an employee should receive based on his/heryears of education and job-related experience. Note that this maybe different from their actual salary). Effort-based utility d is notdirectly observable, but we assume it is estimated via a functionд : X ×Y ×H → R+, such that
d = д(x,y,h).
Functionд links the observable information, x,y, andh, to the effort-based utility, d . Let a ∈ [0, 1] be the actual utility the individualreceives subsequent to receiving prediction y (e.g. the utility theyget as the result of their predicted salary). We assume there existsa function f : X ×Y ×H → R+ that estimates a:
a = f (x,y,h).
Throughout, for simplicity we assume higher values of a and dcorrespond to more desirable conditions.
Let u be the advantage or overall utility the individual earns asthe result of being subject to predictive model h. For simplicity andunless otherwise specified, we assume u has the following simpleform:
u = a − d . (1)
That is, u captures the discrepancy between an individual’s actualutility (a) and their effort-based utility d . With this formulation, anindividual’s utility is 0 when their actual and effort-based utilitiescoincide (i.e. u = 0 if a = d).
We consider the predictive advantage u to be the currency ofequality of opportunity for supervised learning. That is, u is whatwe hope to equalize across similar individuals (similar in terms ofwhat they can be held accountable for). Our moral argument for thischoice is as follows: the predictive model h inevitably makes errorsin assigning individuals to their effort-based utilities—this could bedue to the target variable not properly reflecting effort-based utility,the prediction being used improperly, or simply a consequence ofgeneralization. Sometimes these errors are beneficial to the subject,and sometimes they cause harm. Advantage u precisely capturesthis benefit/harm. EOP in this setting requires that all individuals,who do not differ in ways for which they can be held morallyaccountable, have the same prospect of earning the advantage u—regardless of their irrelevant attributes. As an example, let’s assumethe true labels in the training data reflects individuals’ effort-basedutilities (as we will shortly argue, this assumption is not alwaysmorally acceptable, but for now let’s ignore this issue). In thiscase, a perfect predictor—one that correctly predicts the true labelfor every individual—will distribute no predictive advantage, butsuch predictor almost never exists in real world applications. Thedeployed predictive model almost always distributes some utilityamong decision-subjects through the errors it makes. A fair model(with EOP rationale) would give all individuals with similar truelabels the same prospect of earning this advantage—regardless oftheir irrelevant attributes.
Our main conceptual contribution is to map the above setting tothat of economicmodels of EOP (Section 1.1).We treat the predictivemodel h as a policy, arbitrary features z as circumstance, and theeffort-based utilities d as effort (Figure 1). In the next Section, weshow that through our proposed mapping, most existing statisticalnotions of fairness can be interpreted as special cases of EOP.
A Moral Framework for Understanding Fair MLthrough Economic Models of Equality of Opportunity FAT* ’19, January 29–31, 2019, Atlanta, GA, USA
3 EOP FOR SUPERVISED LEARNINGIn this Section, we show that many existing notions of algorithmicfairness, such as statistical parity [10, 15, 16], equality of odds [13],equality of accuracy [4], and predictive value parity [17, 29, 30],can be cast as special cases of EOP. The summary of our results inthis Section can be found in Table 1. To avoid any confusion withthe notation, we define random variables X,Y to specify the featurevector and true label for an individual drawn i.i.d. from distributionF . Similarly given a predictive model h, random variables Y =h(X),Ah ,Dh ,U h specify the predicted label, actual utility, the effort-based utility, and advantage, respectively, for an individual drawni.i.d. from F . When the predictive model in reference is clear fromthe context, we drop the superscript h for brevity.
Before we formally establish a connection between algorithmicfairness and EOP, we shall briefly overview the Fair ML litera-ture and remind the reader of the precise definition of previously-proposed notions of fairness. Existing notions of algorithmic fair-ness can be divided into two distinct categories: individual- [7, 27]and group-level fairness. Much of the existing work on algorithmicfairness has been devoted to the study of group (un)fairness, alsocalled statistical unfairness or discrimination. Statistical notionsof fairness require that given a classifier, a certain fairness met-ric is equal across all (protected or socially salient) groups. Moreprecisely, assuming z ∈ Z specifies the group each individual be-longs to, statistical parity seeks to equalize the percentage of peoplereceiving a particular outcome across different groups:
Definition 3 (Statistical Parity). A predictive model h satis-fies statistical parity if ∀z, z′ ∈ Z,∀y ∈ Y :
P(X,Y )∼F [h(X) = y |Z = z] = P(X,Y )∼F [h(X) = y |Z = z′].
Equality of odds requires the equality of false positive and falsenegative rates across different groups:
Definition 4 (Eqality of Odds). A predictive model h satisfiesequality of odds if ∀z, z′ ∈ Z,∀y, y ∈ Y :
P(X,Y )∼F [Y = y |Z = z,Y = y] = P(X,Y )∼F [Y = y |Z = z′,Y = y].
Equality of accuracy requires the classifier to make equally ac-curate predictions across different groups:
Definition 5 (Eqality of Accuracy). A predictive model hsatisfies equality of accuracy if ∀z, z′ ∈ Z :
E(X,Y )∼F [(Y − Y )2 |Z = z] = E(X,Y )∼F [(Y − Y )2 |Z = z′].
Predictive value parity (which can be thought of as a weakerversion of calibration [17]) requires the equality of positive andnegative predictive values across different group:
Definition 6 (Predictive Value Parity). A predictive model hsatisfies predictive value parity if ∀z, z′ ∈ Z,∀y, y ∈ Y :
P(X,Y )∼F [Y = y |Z = z, Y = y] = P(X,Y )∼F [Y = y |Z = z′, Y = y].
3.1 Statistical Parity, Equality of Odds andAccuracy as Rawlsian EOP
We begin by translating Rawlsian EOP into the supervised learningsetting using the mapping proposed in Figure 1. Recall that weproposed replacing e with effort-based utility d , and circumstance c
with vector of irrelevant features z. In order for the definition ofRawlsian EOP to be morally acceptable, we need d to not be affectedby z and the model h. In other words, it can only be a function of wand y. Let Fh (.) specify the distribution of utility across individualsunder predictive model h. We define Rawlsian EOP for supervisedlearning as follows:
Definition 7 (R-EOP for supervised learning). Suppose d =д(w,y). Predictive model h satisfies Rawlsian EOP if for all z, z′ ∈ Zand all d ∈ [0, 1],
Fh (.|Z = z,D = d ) = Fh (.|Z = z′,D = d ).
In the binary classification setting, if we assume the true la-bel Y reflects an individual’s effort-based utility D, Rawlsian EOPtranslates into equality of odds across protected groups:3
Proposition 1 (Eqality of Odds as R-EOP). Consider thebinary classification task where Y = {0, 1}. Suppose U = A − D,A = h(X) = Y (i.e., the actual utility is equal to the predicted label)and D = д(W,Y ) where д(W,Y ) = Y (i.e., effort-based utility of anindividual is assumed to be the same as their true label). Then theconditions of R-EOP are equivalent to those of equality of odds.
Proof. Recall that R-EOP requires that ∀z, z′ ∈ Z,∀d ∈ D, andfor all possible utility levels u:
P(U ≤ u |Z = z,D = d ) = P(U ≤ u |Z = z′,D = d ).
ReplacingU with (A−D),D withY ,Awith Y , the above is equivalentto
∀z, z′ ∈ Z,∀y ∈ {0, 1},∀u ∈ {0,±1} :P[Y − Y ≤ u |Z = z,Y = y] = P[Y − Y ≤ u |Z = z′,Y = y]
⇔ ∀z, z′ ∈ Z,∀y ∈ {0, 1},∀u ∈ {0,±1} :P[Y ≤ u + y |Z = z,Y = y] = P[Y ≤ u + y |Z = z′,Y = y]
⇔ ∀z, z′ ∈ Z,∀y ∈ {0, 1},∀y ∈ {0, 1} :P[Y = y |Z = z,Y = y] = P[Y = y |Z = z′,Y = y]
where the last line is identical to the conditions of equality of oddsfor binary classification. □
The important role of the above proposition is to explicitly spellout the moral assumption underlying equality of odds as a measureof fairness: by measuring fairness through equality of odds, weimplicitly assert that all individuals with the same true label havethe same effort-based utility. This can clearly be problematic inpractice: true labels don’t always reflect/summarize accountabilityfactors. At best, they are only a reflection of the current state ofaffairs—which itself might be tainted by past injustices. For thesereasons, we argue that equality of odds can only be used as avalid measure of algorithmic fairness (with an EOP rationale) oncethe validity of the above moral equivalency assumption has beencarefully investigated and its implications are well understood inthe specific context it is utilized in.
Other statistical definitions of algorithmic fairness—namely sta-tistical parity and equality of accuracy—can similarly be thought ofas special instances of R-EOP. See Table 1. For example statistical
3Note that Hardt et al. [13] referred to a weaker measure of algorithmic fairness (i.e.equality of true positive rates) as equality of opportunity.
FAT* ’19, January 29–31, 2019, Atlanta, GA, USA Heidari et al.
Notion of fairness Effort-based utility D Actual utility A Notion of EOPAccuracy Parity constant (e.g. 0) (Y − Y )2 RawlsianStatistical Parity constant (e.g. 1) Y RawlsianEquality of Odds Y Y RawlsianPredictive Value Parity Y Y egalitarian
Table 1: Interpretation of existing notions of algorithmic fairness for binary classification as special instances of EOP.
parity can be interpreted as R-EOP if we assume all individualshave the same effort-based utility.4
Proposition 2 (Statistical Parity as R-EOP). Consider thebinary classification task where Y = {0, 1}. Suppose U = A − D,A = Y and D = д(W,Y ) where д(W,Y ) is a constant function (i.e.,effort-based utility of all individuals is assumed to be the same). Thenthe conditions of R-EOP is equivalent to statistical parity.
Proof. Without loss of generality, suppose д(X,Y ,h) ≡ 1, i.e.all individuals effort-based utility 1. Recall that R-EOP requires that∀z, z′ ∈ Z,∀d ∈ D,∀u ∈ R :
P(U ≤ u |Z = z,D = d ) = P(U ≤ u |Z = z′,D = d ).
Replacing U with (A − D), D with 1, and A with Y , the above isequivalent to
∀z, z′ ∈ Z,∀d ∈ {1},∀u ∈ {0,−1} :P[Y − D ≤ u |Z = z,D = 1] = P[Y − D ≤ u |Z = z′,D = 1]
⇔ ∀z, z′ ∈ Z,∀u ∈ {0,−1} :P[Y ≤ u + 1|Z = z] = P[Y ≤ u + 1|Z = z′]
⇔ ∀z, z′ ∈ Z,∀y ∈ {0, 1} :P[Y = y |Z = z] = P[Y = y |Z = z′]
where the last line is identical to the conditions of statistical parityfor binary classification. □
As a real-world example where statistical parity can be applied,consider the following: suppose the society considers all patients tohave the same effort-based utility—which can be enjoyed by accessto proper clinical examinations. Now suppose that undergoing aninvasive clinical examination has utility 1 if one has the suspecteddiseases and -1 otherwise, whereas avoiding the same clinical inves-tigation has utility 1 if one does not have the suspected disease, and-1 otherwise. For all subjects, the effort-based utility is the same(the maximum utility, let us suppose). In other words, all peoplewith a disease deserve the invasive clinical investigation and allpeople without the disease deserve to avoid it. Consider a policy ofgiving clinical investigation to all the people without the diseaseand to no people without the disease. This would achieve an equaldistribution of effort-based utility (D) and distribute no advantageU . Such policy, however, could only be achieved with a perfectaccuracy predictor. For an imperfect accuracy predictor, R-EOPwould require the distribution of (negative, in this case) utility (U)to give the same chance to African Americans and white patientswith (without) the disease to receive (avoid) an invasive clinicalexam.4Statistical parity can be understood as equality of outcomes as well, if we assume Yreflects the outcome.
Proposition 3 (Eqality of Accuracy as R-EOP). Considerthe binary classification task where Y = {0, 1}. SupposeU = A − D,A = (Y − Y )2 and D = д(W,Y ) where д(W,Y ) ≡ 0 (i.e., effort-basedutility of all individuals are assumed to be the same and equal to 0).Then the conditions of R-EOP is equivalent to equality of accuracy.
Proof. Recall that R-EOP requires that∀z, z′ ∈ Z,∀d ∈ D,∀u ∈R :
P(U ≤ u |Z = z,D = d ) = P(U ≤ u |Z = z′,D = d ).
ReplacingU with (A−D), D with 0, andAwith (Y −Y )2, the aboveis equivalent to ∀z, z′ ∈ Z,∀d ∈ {0},∀u ∈ {0, 1} :
P[(Y−Y )2−D ≤ u |Z = z,D = d] = P[(Y−Y )2−D ≤ u |Z = z′,D = d]
We can then write:
⇔ ∀z, z′,∀u : P[(Y − Y )2 = u |Z = z] = P[(Y − Y )2 = u |Z = z′]⇔ ∀z, z′ ∈ Z : E[(Y − Y )2 |Z = z] = E[(Y − Y )2 |Z = z′]
where the last line is identical to the conditions of equality ofaccuracy for binary classification. □
The critical moral assumption underlying equality of accuracyas a measure of fairness (with EOP rationale) is that errors reflectthe advantage distributed by the predictive model among decisionsubjects. This exposes the fundamental ethical problem with adopt-ing equality of accuracy as a measure of algorithmic fairness: itfails to distinguish between errors that are beneficial to the subjectand those that are harmful. For example, in the salary predictionexample, equality of accuracy would make no distinction betweenan individual who earns a salary higher than what they deserve,and someone who earns lower than their effort-based/deservedsalary.
Max-min distribution vs. strict equality. At a high level, R-EOPprescribes equalizing advantage distribution across persons withthe same effort-based utility. Some egalitarian philosophers haveargued that we can remain faithful to the spirit (though not theletter) of EOP by delivering a max-min distribution of advantage,instead of a strict egalitarian one [23]. The max-min distributiondeviates from equality only when this makes the worst off groupbetter off. Even though this distribution permits inequalities that donot reflect accountability factors, it is considered a morally superioralternative to equality, if it improves the utility of least fortunate.5The max-min distribution addresses the “leveling down" objectionto equality: the disadvantaged group may be more interested in
5The idea that inequalities are justifiable only when they result from a scheme arrangedto maximally benefit the worst off position is expressed through the Difference Principleby John Rawls in his theory of “justice as fairness" [20].
A Moral Framework for Understanding Fair MLthrough Economic Models of Equality of Opportunity FAT* ’19, January 29–31, 2019, Atlanta, GA, USA
maximizing their absolute level of utility, as opposed to their relativeutility compared to that of the advantaged group.
3.2 Predictive Value Parity as Egalitarian EOPNote that predictive value parity (equality of positive and negativepredictive values across different groups) can not be thought ofas an instance of R-EOP, as it requires the effort-based utility ofan individual to be a function of the predictive model h (as wewill shortly show, it assumes D = h(X)). This is in violation ofthe absolutist view of Rawlsian EOP. In this Section, we show thatpredictive value parity can be cast as an instance of luck egalitarianEOP.
We first specialize Roemer’s model of Egalitarian EOP to thesupervised learning setting. Recall that egalitarian EOP allows theeffort-based utility to be a function of the predictive model h, that isD = f (X,Y ,h). When this is the case, following the argument putforward by Roemer we posit that the distribution of effort-basedutility for a given type z (denoted by F z,hD ) is a characteristic of thetype z, not something for which any individual belonging to thetype can be held accountable. Therefore, an inter-type comparablemeasure of effort-based utility must factor out the goodness orbadness of this distribution. We consider two individuals as beingequally deserving if they sit at the same quantile or rank of thedistribution of D for their corresponding type.
More formally, let the indirect utility distribution function, de-noted by Fh (.|z,π ), specify the distribution of utility for individualsof type z at the π th quantile (0 ≤ π ≤ 1) of effort-based utility dis-tribution, F z,hD . Equalizing opportunities means choosing the pre-dictive model h to equalize the indirect utility distribution acrosstypes, at fixed levels of π :
Definition 8 (e-EOP for supervised learning). Suppose d =f (x,y,h). Predictive model h satisfies egalitarian EOP if for all π ∈[0, 1] and z, z′ ∈ Z,
Fh (.|Z = z,Π = π ) = Fh (.|Z = z′,Π = π ). (2)
Next, we show that predictive value parity can be thought ofas a special case of e-EOP, where the predicted label/risk h(X) isassumed to reflect the individual’s effort-based utility, and the truelabel Y reflects his/her actual utility.
Proposition 4 (predictive value parity as e-EOP). Considerthe binary classification task where Y = {0, 1}. SupposeU = A − D,A = Y and D = д(X,Y ,h) where д(X,Y ,h) = h(X) = Y (i.e., effort-based utility of an individual under h is assumed to be the same astheir predicted label). Then the conditions of e-EOP are equivalent tothose of predictive value parity.
Proof. Recall that e-EOP requires that ∀z, z′ ∈ Z,∀π ∈ [0, 1],and ∀u ∈ R :
P[U ≤ u |Z = z,Π = π ] = P[U ≤ u |Z = z′,Π = π ].
Note that since D = Y and in the binary classification, Y can onlytake on two values, there are only two ranks/quantiles possible interms of the effort-based utility—corresponding to Y = 0 and Y = 1.So the above condition is equivalent to ∀z, z′ ∈ Z,∀y ∈ {0, 1},∀u ∈{0,±1} :
P[U ≤ u |Z = z, Y = y] = P[U ≤ u |Z = z′, Y = y].
ReplacingU with (A−D),D with Y ,AwithY , the above is equivalentto
∀z, z′ ∈ Z,∀y ∈ {0, 1},∀u ∈ {0,±1} :P[Y − Y ≤ u |Z = z, Y = y] = P[Y − Y ≤ u |Z = z′, Y = y]
⇔ ∀z, z′ ∈ Z,∀y ∈ {0, 1},∀u ∈ {0,±1} :P[Y ≤ u + y |Z = z, Y = y] = P[Y ≤ u + y |Z = z′, Y = y]
⇔ ∀z, z′ ∈ Z,∀y ∈ {0, 1},∀y ∈ {0, 1} :P[Y = y |Z = z, Y = y] = P[Y = y |Z = z′, Y = y]
where the last line is identical to predictive value parity. □
Note that there are two assumptions needed to cast predictivevalue parity as an instance of e-EOP: 1) the predicted label/riskh(x) reflects the individual’s effort-based utility; and 2) the truelabel Y reflects his/her actual utility. The plausibility of such moralassumptions must be critically evaluated in a given context beforepredictive parity can e employed as a valid measure of fairness.Next, we discuss the plausibility of these assumptions throughseveral real-world examples.
Plausibility of assumption 1. The choice of predicted label, h(X),as the indicator of effort-based utility, may sound odd at first. How-ever, there are several real-world settings in which this assumptionis considered appropriate. Consider the case of driving under influ-ence (DUI): the law considers all drivers equally at risk of causingan accident—due to the consumption of alcohol or drugs—equallyaccountable for their risk and punishes them similarly, even thoughonly some of them will end up in an actual accident, and the restwon’t. In this context, the potential/risk of causing an accident—asopposed to the actual outcome—justifies unequal treatment, becausewe believe differences in actual outcomes among equally risky in-dividuals is mainly driven by arbitrary factors, such as brute luck.Arguably, such factors should never specify accountability.
Can assumptions 1 and 2 hold simultaneously? The following isan example in which ssumptions 1 and 2 hold simultaneously (inparticular, the true label Y specifies the actual utility an individualreceives subsequent to being subject to automated decision mak-ing). Consider the students of a course, offered online and open tostudents from all over the world. The final assessment of studentsenrolled in the course includes an essential oral exam. The oralexam is very challenging and extremely competitive. The instruc-tors hold an exam session every month. Every student is allowedto take the oral exam, but since resources for oral examinationsare limited, to discourage participation without preparation, therule is that, if a student fails the exam, he/she has to wait oneyear before taking the exam again. Suppose that students belongto one of the two groups: African Americans and Asians. AfricanAmerican and Asian students study in different ways, with differentcognitive strategies. As a result, an African American student with0.9 passing score may correspond to a very different feature vectorcompared to an Asian student with a 0.9 passing score. Suppose apredictive model is used to predict the outcome of the oral exam forindividual students, based on the student’s behavioral data. (Theonline learning platform records data on how students interact
FAT* ’19, January 29–31, 2019, Atlanta, GA, USA Heidari et al.
with the course materials.) Students are given a simple “pass/fail"prediction to help them make an informed choice about when totake the exam. In this example, we argue that both assumptionsunderlying predictive value parity are satisfied:
(1) A = Y : Passing the exam is a net utility, not passing the examis a net disutility (due to the one year delay). Also, beingpredicted to pass per se has no utility associated with it.
(2) D = Y . It is plausible to consider students morally respon-sible for their chances of success, because the predictionsare calculated based on how they have studied the coursematerial.
In this example, a fair predictor (with EOP rationale) should satisfiespredictive value parity. That means: students who are predicted topass, should be equally likely to pass the exam, irrespective of theirrace.
On Recent Fairness Impossibility Results. Several papers have re-cently shown that group-level notions of fairness, such as predictivevalue parity and equality of odds, are generally incompatible withone another and cannot hold simultaneously [12, 17]. Our approachconfers a moral meaning to these impossibility results: they can beinterpreted as contradictions between fairness desiderata reflect-ing different and irreconcilable moral assumptions. For examplepredictive value parity and equality of odds make very differentassumptions about the effort-based utility d : Equality of odds as-sumes all persons with similar true labels are equally accountablefor their labels, whereas predictive value parity assumes all personswith the same predicted label/risk are equally accountable for theirpredictions. Note that depending on the context, usually only one(if any) of these assumptions is morally acceptable. We argue, there-fore, that unless we are in the highly special case where Y = h(X),it is often unnecessary—from a moral standpoint—to ask for bothof these fairness criteria to be satisfied simultaneously.
4 EGALITARIAN MEASURES OF FAIRNESSIn this section, inspired by Roemer’s model of egalitarian EOP wepresent a new family of measures for algorithmic fairness. Ourproposal is applicable to supervised learning tasks beyond binaryclassification, and to utility functions beyond the simple linear formspecified in Equation 1. We illustrate our proposal empirically, andcompare it with existing notions of (un)fairness for regression. Ourempirical findings suggest that employing a measure of algorithmic(un)fairness when its underlying assumptions are not met, can havedevastating consequences on the welfare of decision subjects.
4.1 A New Family of MeasuresFor supervised learning tasks beyond binary classification (e.g. mul-ticlass classification or regression), the requirement of equation 2becomes too stringent, as there will be (infinitely) many quantilesto equalize utilities over. The problem persists even if we relax therequirement of equal utility distributions to maximizing the mini-mum expected utility at each quantile. More formally, let vz (π ,h)specify the expected utility of individuals of type z at the π th quan-tile of the effort-based utility distribution. For π ∈ [0, 1], we saythat a predictive model hπ satisfies egalitarian EOP at the π -slice
of the population, if:
hπ ∈ arg maxh∈H
minz∈Z
vz (π ,h).
Assuming we are concerned only with the π -slice, then hπ wouldbe the equal-opportunity predictive model. Unfortunately, whenwe move beyond binary classification, we generally cannot finda model that is simultaneously optimal for all ranks π ∈ [0, 1].Therefore, we need to find a compromise. Following Roemer, wedefine the e-EOP predictive model as follows:
h∗ ∈ arg maxh∈H
minz∈Z
∫ 1
0vz (π ,h)dπ . (3)
That is, we consider h∗ to be an e-EOP predictive model if it maxi-mizes the expected utility of theworst off group (i.e.
∫ 10 vz (π ,h)dπ ).6
Replacing the expectation with its in-sample analogue, our pro-posed family of e-EOP measures can be evaluated on the data set Tas follows:
F (h,T ) = minz∈Z
1nz
∑i ∈T :zi=z
u (xi ,yi ,h)
where u (xi ,yi ,h) is the utility an individual with feature vector xiand true label yi receives when predictive model h is deployed; andnz is the number of individuals in T whose arbitrary features valueis z ∈ Z. (The arbitrary features value z specifies the (intersectional)group each individual belongs to. We usem to denote the numberof such (intersectional) groups. For simplicity in our illustration,we refer to these groups as G1, · · · ,Gm .)
To guarantee fairness, we propose the following in-processingmethod: maximize the expected utility of the worst off group, sub-ject to error being upper bounded (by ϵ).
maxh∈H
F (h,T )
s.t. L (T ,h) ≤ ϵ (4)
Note that if the loss functionL is convex and F is concave in modelparameters, Optimization 4 is convex and can be solved efficiently.
We remark that our notion of fairness does not require us toexplicitly specify the effort-based utility D, since it only comparesthe overall expected utility of different groups with one another—without the need to explicitly compare the utility obtained by indi-viduals at a particular rank of D across different groups. Further-more, the utility function, u (x,y,h), does not have to be restrictedto take the simple linear form specified in Equation 1.
4.2 IllustrationNext, we illustrate our proposal on the Crime and Communities dataset [19]. The data consists of 1994 observations, each correspondingto a community/neighborhood in the United States. Each commu-nity is described by 101 features, specifying its socio-economic, lawenforcement, and crime statistics extracted from the 1995 FBI UCR.Community type (e.g. urban vs. rural), average family income, and
6Roemer in fact proposes two further alternatives: in the first solution, the objectivefunction for each π -slice of the population is assumed to beminz∈Z v z (π , h)—whichis then weighted by the size of the slice. In the second solution, he declares theequal opportunity policy to be the average of the policies hπ . Roemer expresses nostrong preference for any of these alternatives, other than the fact that computationalsimplicity sometimes suggests one over the others [23]. This is in fact the reasoningbehind our choice of Equation 3.
A Moral Framework for Understanding Fair MLthrough Economic Models of Equality of Opportunity FAT* ’19, January 29–31, 2019, Atlanta, GA, USA
(a) (b) (c)
Figure 2: NRD, PRD, and average utility of the disadvantaged group as a function of ϵ (the upperbound onmean squared error).The notion of fairness enforced on algorithmic decisions can have a devastating impact on the welfare of the disadvantagedgroup.
the per capita number of police officers in the community are afew examples of the explanatory variables included in the dataset.The target variable (Y ) is the “per capita number of violent crimes".We train a linear regression model, θ ∈ Rk , on this dataset to pre-dict the per capita number of violent crimes for a new community.We hypothesize that crime predictions can affect the law enforce-ment resources assigned to the community, the value of propertieslocated in the neighborhood, and business investments drawn to it.
We preprocess the original dataset as follows: we remove theinstances for which target value is unknown. Also, we remove fea-tures whose values are missing for more than 80% of instances. Westandardize the data so that each feature has mean 0 and variance1. We divide all target values by a constant so that labels rangefrom 0 to 1. Furthermore, we flip all labels (y → 1 − y), so thathigher y values correspond to more desirable outcomes. We assumea neighborhood belongs to the protected group (G1) if the majorityof its residents are non-Caucasian, that is, the percentage of AfricanAmerican, Hispanic, and Asian residents of the neighborhood com-bined, is above 50%. This divides the training instances into twogroups G0,G1. We include this group membership information asthe (sensitive) feature z in the training data (zi = 1[i ∈ G1]).
For simplicity, we assume the utility functionu has the followingfunctional dependence on x and θ :u (z,y, y); that is,u’s dependenceon x andθ are through z and y = θ .x, respectively. For communitiesbelonging toG0 andG1, we assumeu (z,y, y) = f (z,y, y)−д(z,y, y)is respectively defined as follows:• For a majority-Caucasian neighborhood,
u (0,y, y) = (1 + 0.5yy) − (0.5y).
• For a minority-Caucasian neighborhood,
u (1,y, y) = (1 + 3yy + 2y) − (y).
At a high level, neighborhoods in both groups enjoy a high utility iftheir predicted and actual crime rates are low, simultaneously (notethat the absolute value of utility derived from this case is higher forthe minority). The minority-Caucasian group further benefits fromlow crime predictions (regardless of actual crime rates). We assumethe effort-based utility for the minority group, is one minus theactual crime rate (y), and for the majority group, it is proportional
to one minus the predicted crime rate (0.5y). Note that these utilityfunctions are made up for illustration purposes only, and do notreflect any deep knowledge of how crime and law enforcementaffect the well-being of a neighborhood’s residents.
To illustrate our proposal, we solve the following convex opti-mization problem for different values of ϵ :
maxσ ,θ
σ
s.t.1n0
∑i ∈G0
−0.5θ .xi + 0.5(θ .xi )yi + 1 ≥ σ
1n1
∑i ∈G1
2θ .xi + 3(θ .xi )yi − yi + 1 ≥ σ
1n
n∑i=1
(θ .xi − yi )2 + λ∥θ ∥1 ≤ ϵ (5)
We choose the value of λ by running a 10-fold cross validation onthe data set. For each value of ϵ (Mean Squared Error), we measurethe following quantities via 5-fold cross validation:• Positive residual difference [5] is the equivalent of falsepositive rate in regression, and is computed by taking theabsolute difference of mean positive residuals across the twogroups:
�������
1n+1
∑i ∈G1
max{0, (yi − yi )} −1n+0
∑i ∈G0
max{0, (yi − yi )}�������.
In the above, n+д is the number of individuals in group д ∈{0, 1} who get a positive residual, i.e. yi − yi ≥ 0.• Negative residual difference [5] is the equivalent of falsenegative rate in regression, and is computed by taking theabsolute difference of mean negative residuals across thetwo groups.• Average utility of the disadvantaged group is computed bytaking the average utility of all individuals in the test dataset:
min
1n0
∑i ∈G0
u (xi ,yi ,h),1n1
∑i ∈G1
u (xi ,yi ,h).
FAT* ’19, January 29–31, 2019, Atlanta, GA, USA Heidari et al.
Figure 2 shows the results of our simulations. Blue curves corre-spond to our proposal (Optimization 5). As evident in Figures 2aand 2b, positive and negative residual difference increase with ϵ ,while the average utility increases (see Figure 2c).
To compare our proposal with existing measures of (un)fairnessfor regression, we utilize the in-processing method of Heidari et al.[14]. The method enforces an upperbound on
∑i (yi − yi ), and has
been shown to control the positive and negative residual differ-ence across the two groups. More precisely, we solve the followingoptimization problem for different values of ϵ :
maxθ
1n
∑i ∈T
θ .xi − yi s.t.1n
n∑i=1
(θ .xi − yi )2 + λ∥θ ∥1 ≤ ϵ (6)
Red curves in Figure 2 correspond to this baseline. As evident inFigures 2a and 2b, by enforcing a lower bound on
∑i (yi − yi ),
positive and negative residual difference go to 0 very quickly—asexpected. However, the trained model performs very poorly interms of average utility of the disadvantaged group.
5 CONCLUSIONOur work makes an important contribution to the rapidly growingline of research on algorithmic fairness—by providing a unifyingmoral framework for understanding existing notions of fairnessthrough philosophical interpretations and economic models of EOP.We showed that the choice between statistical parity, equality ofodds, and predictive value parity can be mapped systematically tospecific moral assumptions about what decision subjects morallydeserve. Determining accountability features and effort-based util-ity is arguably outside the expertise of computer scientists, and hasto be resolved through the appropriate process with input fromstakeholders and domain experts. In any given application domain,reasonable people may disagree on what constitutes factors thatpeople should be considered morally accountable for, and therewill rarely be a consensus on the most suitable notion of fairness.This, however, does not imply that in a given context all existingnotions of algorithmic fairness are equally acceptable from a moralstandpoint.
ACKNOWLEDGMENTH. Heidari and A. Krause acknowledge support from CTI grantno. 27248.1 PFES-ES. K. P. Gummadi is supported in part by theEuropean Research Council (ERC) Advanced Grant for the project“Foundations for Fair Social Computing", funded under the Euro-pean Union’s Horizon 2020 Framework Programme (grant agree-ment no. 789373). Michele Loi is supported by the CANVAS project,funded under the European Union’s Horizon 2020 Research andInnovation Programme (grant agreement no. 700540).
REFERENCES[1] Richard J. Arneson. 1989. Equality and equal opportunity for welfare. Philosophi-
cal Studies: An International Journal for Philosophy in the Analytic Tradition 56, 1(1989), 77–93.
[2] Richard J. Arneson. 2015. Equality of Opportunity. In the Stanford Encyclopediaof Philosophy (summer 2015 ed.), Edward N. Zalta (Ed.). Metaphysics ResearchLab, Stanford University.
[3] Richard J. Arneson. 2018. Four Conceptions of equal opportunity. (2018).
[4] Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracydisparities in commercial gender classification. In Proceedings of the Conferenceon Fairness, Accountability and Transparency. 77–91.
[5] Toon Calders, Asim Karim, Faisal Kamiran,Wasif Ali, and Xiangliang Zhang. 2013.Controlling attribute effect in linear regression. In Proceedings of the InternationalConference on Data Mining. IEEE, 71–80.
[6] Gerald A. Cohen. 1989. On the currency of egalitarian justice. Ethics 99, 4 (1989),906–944.
[7] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and RichardZemel. 2012. Fairness through awareness. In Proceedings of the Innovations inTheoretical Computer Science Conference. ACM, 214–226.
[8] Ronald Dworkin. 1981. What is equality? Part 1: Equality of welfare. Philosophy& Public Affairs 10, 3 (1981), 185–246.
[9] Ronald Dworkin. 1981. What is equality? Part 2: Equality of resources. Philosophy& Public Affairs 10, 4 (1981), 283–345.
[10] Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, andSuresh Venkatasubramanian. 2015. Certifying and removing disparate impact.In Proceedings of the International Conference on Knowledge Discovery and DataMining. ACM, 259–268.
[11] Marc Fleurbaey. 2008. Fairness, responsibility, and welfare. Oxford UniversityPress.
[12] Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016.On the (im)possibility of fairness. arXiv preprint arXiv:1609.07236 (2016).
[13] Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity insupervised learning. In Proceedings of the 30th Conference on Neural InformationProcessing Systems. 3315–3323.
[14] Hoda Heidari, Claudio Ferrari, Krishna P. Gummadi, and Andreas Krause. 2018.Fairness Behind a Veil of Ignorance: A Welfare Analysis for Automated DecisionMaking. In Proceedings of the 32nd Conference on Neural Information ProcessingSystems.
[15] Faisal Kamiran and Toon Calders. 2009. Classifying without discriminating. InProceedings of the 2nd International Conference on Computer, Control and Commu-nication. IEEE, 1–6.
[16] Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. 2011. Fairness-awarelearning through regularization approach. In Proceedings of the InternationalConference on Data Mining Workshops. IEEE, 643–650.
[17] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherenttrade-offs in the fair determination of risk scores. In In proceedings of the 8thInnovations in Theoretical Computer Science Conference.
[18] Arnaud Lefranc, Nicolas Pistolesi, and Alain Trannoy. 2009. Equality of opportu-nity and luck: Definitions and testable conditions, with an application to incomein France. Journal of Public Economics 93, 11-12 (2009), 1189–1207.
[19] M. Lichman. 2013. UCI Machine Learning Repository: Communities and CrimeData Set. http://archive.ics.uci.edu/ml/datasets/Communities+and+Crime.
[20] John Rawls. 1958. Justice as fairness. The philosophical review 67, 2 (1958),164–194.
[21] John Rawls. 1971. A theory of justice. Harvard university press.[22] John E. Roemer. 1993. A pragmatic theory of responsibility for the egalitarian
planner. Philosophy & Public Affairs (1993), 146–166.[23] John E. Roemer. 2002. Equality of opportunity: A progress report. Social Choice
and Welfare 19, 2 (2002), 455–471.[24] John E. Roemer. 2009. Equality of opportunity. Harvard University Press.[25] John E. Roemer and Alain Trannoy. 2015. Equality of opportunity. In Handbook
of income distribution. Vol. 2. Elsevier, 217–300.[26] Amartya Sen. 1979. Equality of What? The Tanner Lecture on Human Values
(1979).[27] Till Speicher, Hoda Heidari, Nina Grgic-Hlaca, Krishna P. Gummadi, Adish Singla,
Adrian Weller, and Muhammad Bilal Zafar. 2018. A Unified Approach to Quanti-fying Algorithmic Unfairness: Measuring Individual and Group Unfairness viaInequality Indices. In Proceedings of the International Conference on KnowledgeDiscovery and Data Mining.
[28] Wikipedia. 2018. Equal opportunity. https://en.wikipedia.org/wiki/Equal_opportunity.
[29] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna PGummadi. 2017. Fairness beyond disparate treatment & disparate impact: Learn-ing classification without disparate mistreatment. In Proceedings of the 26thInternational Conference on World Wide Web. 1171–1180.
[30] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P.Gummadi. 2017. Fairness Constraints: Mechanisms for Fair Classification. InProceedings of the 20th International Conference on Artificial Intelligence andStatistics.
