Ranking And Recommendation

Question 1

Q: What is the objective of ranking in information retrieval (IR)?

Answer 1

A: The objective of ranking in IR is to provide maximum relevance to the searcher. The items are ordered by decreasing probability of being relevant.

Question 2

Front:
Question: What are the main sources of this slide deck on defining fair rankings and recommendations?

Answer 2

Back:
Answer: The main source of this slide deck is Carlos Castillo’s paper “Fairness and Transparency in Ranking” published in SIGIR Forum, 2018.

Question 3

What is the importance of fairness in ranking?

Answer 3

Fairness in ranking is important to ensure that individuals or groups are not unfairly advantaged or disadvantaged in the ranking process.

Question 4

What are the different phases of processing methods of fairness in ranking?

Answer 4

Fairness in ranking can be achieved through pre-processing, in-processing, or post-processing methods.

Question 5

What is the significance of data in fairness in ranking?

Answer 5

Data plays a crucial role in the fairness of ranking as it provides the information needed to determine which groups or individuals should be protected and how the ranking should be conducted.

Question 6

What are some post-processing methods for achieving fairness in ranking?

Answer 6

One method is FA*IR, which

• ranks protected and non-protected groups separately and
• ensures that a minimum number of protected elements are included in every ranking position.

Question 7

How does FA*IR handle multiple protected attributes?

Answer 7

FA*IR handles multiple protected attributes by ranking candidates of all protected groups separately and ensuring that a minimum number of protected elements are included at every ranking position based on their individual probabilities.

Question 8

What is the ranking approach proposed by Singh and Joachims?

Answer 8

Singh and Joachims proposed a probabilistic ranking approach that maximizes utility and reduces disparate treatment and disparate impact.

Question 9

What is amortized fairness?

Answer 9

Ensuring that

Every element receives attention or exposure

proportional to its utility across multiple queries

while also taking into account past accumulated attention/utility deficits or surpluses.

Question 10

What is DELTR?

Answer 10

DELTR is a learning to rank approach that optimizes LTR with a combination of two losses:

• L, which is the difference between ranking predictions and training elements, and
• U, which is the expected difference in exposure.

Question 11

What is the significance of pre-processing methods in fairness in ranking?

Answer 11

Pre-processing methods are important as they ensure that rankings given as input satisfy a fair ranking condition, which can help to reduce disparate impact.

Question 12

What is the iFair approach?

Answer 12

The iFair (pre-processing) approach transforms input data to reduce the extent to which the distance between items is affected by protected attributes, resulting in individually fair data representations for algorithmic decision making.

Question 13

What is the importance of transparency in algorithmic rankings?

Answer 13

Transparency in algorithmic rankings is important to ensure that the ranking factors and composition of the list are clear and that fairness tests are conducted to avoid unfairness in the system.

Question 14

What are some possible pitfalls of lacking transparency in algorithmic rankings?

Answer 14

Lacking transparency in algorithmic rankings can lead to sneaking positive/affirmative action without a consensus or certifying an algorithm that is part of an unfair system or used in conditions of unfairness.

Question 15

What are the key take-home messages regarding fairness in ranking?

Answer 15

Remember the importance of pre-processing, in-processing, and post-processing methods in achieving fairness in ranking.
Be aware of the different problems that different solutions address.
Ensure transparency in algorithmic rankings to avoid unfairness and promote ethical compliance.

Question 16

Front:
Question: What is the objective of ranking in information retrieval (IR)?

Answer 16

Back:
Answer: The objective of ranking in IR is to provide maximum relevance to the searcher by ordering results by decreasing probability of being relevant.

Question 17

When do we need to consider utility when ranking?

Answer 17

e.g recruiting a candidate for a job

Question 18

Front:
Question: What are the three main criteria for fairness in rankings for those searched?

Answer 18

Back:
Answer: The three main criteria for fairness in rankings for those searched are a

sufficient presence of elements of the protected group (absence of statistical group discrimination, prevent allocative (distributional) harms),

a consistent treatment of elements of both groups (prevent individual discrimination), and a

proper representation of protected groups (prevent representational harms).

Question 19

Front:
Question: What is the fourth criterion for fairness in rankings for searchers?

Answer 19

Back:
Answer: The fourth criterion for fairness in rankings for searchers is an equal level of satisfaction across searcher groups, which is necessary to prevent allocative harms.

Question 20

Front:
Question: What are some possible biases in input data for ranking?

Answer 20

Back:
Answer: Possible biases in input data for ranking include biases in expert-provided training data (e.g., expert or editorially provided rankings), biases in user-provided training data (e.g., clicks and user feedback), and biases in document construction (e.g., completion of different CV sections by men/women).

Question 21

Front:
Question: What are representational harms in rankings?

Answer 21

Back:
Answer: Representational harms in rankings occur when systems reinforce the subordination of some groups along the lines of identity, such as sexualized search results or search suggestions reinforcing biases or stereotypes.

Question 22

Front:
Question: What is disparate exposure in rankings?

Answer 22

Back:
Answer: Disparate exposure in rankings refers to each position in a ranking having a certain probability of being examined. A ranking is fair if the exposure to elements of protected and non-protected groups is approximately equal.

Question 23

Front:
Question: What are the different measures for fairness in rankings?

Answer 23

Back:
Answer: The different measures for fairness in rankings include exposure-based methods, probability-based methods, pairwise ranking metrics, and the fair representation condition.

Question 24

Front:
Question: What is the ranked group fairness condition?

Answer 24

Back:
Answer: The ranked group fairness condition states that a ranking of k elements has ranked group fairness if for every i ≤ k, the prefix of size i of the list satisfies the fair representation condition.

Question 25

Front:
Question: What is the cross-AUC measure for fairness in rankings?

Answer 25

Back:
Answer: The cross-AUC measure for fairness in rankings is defined as the probability that a relevant item from one group is ranked higher than an irrelevant item from another group. It is used to measure differences in ranking between groups.

Question 26

What are exposure-based metrics in the context of ranking algorithms?

Answer 26

Back:
Measure fairness in rankings

based on the probability of each position being examined or “exposed” to users.

These metrics aim to ensure that each position has a fair chance of being seen and that certain groups are not systematically disadvantaged or advantaged in the rankings.

Examples of exposure-based metrics include utility-normalized exposure disparity and expected click-through rate disparity.

Question 27

Front:
What is the concept behind probability-based and pairwise metrics in the context of ranking algorithms?

Answer 27

Back:
Probability-based and pairwise metrics aim to measure the fairness of rankings based on the relative performance of protected and unprotected groups. Probability-based metrics evaluate the maximum probability that a ranking passes a fairness test for a given level of significance. Pairwise metrics evaluate the relative performance of protected and unprotected groups in pairwise comparisons. These metrics aim to ensure that protected groups are not systematically disadvantaged or advantaged in the rankings.

Question 28

Q: What are some examples of situations where searched utility matters in ranking?

Answer 28

Job search: When a user is searching for job openings, the ranking algorithm should take into account the relevance of the job to the user’s skills and experience, but also the user’s career goals, job preferences, and other relevant factors that affect the user’s likelihood to apply for the job.

A: Searched utility matters in ranking in situations such as finding a local business, purchasing a product or service, recruiting a candidate for a job, discovering events or groups to join, learning about a political candidate, dating/mating, and achieving success in business, marketing, career, social, political, or affective/reproductive domains.

Question 29

Q: What is “P-fairness” in the context of fairness for those searched?

Answer 29

A: “P-fairness” refers to the sufficient presence of elements of the protected group, absence of statistical group discrimination, and prevention of allocative (distributional) harms. It is a criterion for fairness for those searched.

Question 30

Q: What is “C-fairness” in the context of fairness for searchers?

Answer 30

A: “C-fairness” refers to an equal level of satisfaction across searcher groups due to different intents or different responses to relevance. It is a criterion for fairness for searchers.

Question 31

Q: What are some possible biases in input data that can affect ranking?

Answer 31

A: Some possible biases in input data that can affect ranking include biases in expert-provided training data, expert or editorially provided rankings, biases in user-provided training data such as clicks and user feedback, and biases in document construction such as completion of different CV sections by men/women.

Question 32

Q: What are representational harms?

Answer 32

A: Representational harms occur when systems reinforce the subordination of some groups along the lines of identity. Examples of representational harms include sexualized search results, search suggestions reinforcing biases or stereotypes, spreading misinformation, manipulative, pointing to adult material, and race stereotypes reinforced.

Question 33

Q: What are some goals of fair rankings?

Answer 33

A: Two different goals of fair rankings are to reduce discrimination when a protected group has higher utility but lower rankings, and to provide equal opportunity when a protected group has lower utility and lower rankings.

Question 34

Q: What are some take-home messages about fairness in ranking?

Answer 34

A: Fairness in IR/RecSys is less studied than in ML/DM. General criteria for fairness in ranking include sufficient presence, consistent treatment, proper representation, and (consumer-side) equal satisfaction. It is important to reduce biases in input data, avoid representational harms, and ensure that technology embodies certain values.