Bias¶
Elliot integrates the following bias metrics.
Summary¶
|
Average coverage of long tail items |
|
Average percentage of long tail items |
|
Average Recommendation Popularity |
|
Popularity-based Ranking-based Equal Opportunity |
Extended Popularity-based Ranking-based Equal Opportunity |
|
|
Popularity-based Ranking-based Statistical Parity |
Extended Popularity-based Ranking-based Statistical Parity |
ACLT¶
-
class
elliot.evaluation.metrics.bias.aclt.aclt.ACLT(recommendations, config, params, eval_objects)[source]¶ Bases:
elliot.evaluation.metrics.base_metric.BaseMetricAverage coverage of long tail items
This class represents the implementation of the Average coverage of long tail items recommendation metric.
For further details, please refer to the paper
\[\mathrm {ACLT}=\frac{1}{\left|U_{t}\right|} \sum_{u \in U_{f}} \sum_{i \in L_{u}} 1(i \in \Gamma)\]\(U_{t}\) is the number of users in the test set.
\(L_{u}\) is the recommended list of items for user u.
\(1(i \in \Gamma)\) is an indicator function and it equals to 1 when i is in Gamma.
To compute the metric, add it to the config file adopting the following pattern:
simple_metrics: [ACLT]
APLT¶
-
class
elliot.evaluation.metrics.bias.aplt.aplt.APLT(recommendations, config, params, eval_objects)[source]¶ Bases:
elliot.evaluation.metrics.base_metric.BaseMetricAverage percentage of long tail items
This class represents the implementation of the Average percentage of long tail items recommendation metric.
For further details, please refer to the paper
\[\mathrm {ACLT}=\frac{1}{\left|U_{t}\right|} \sum_{u \in U_{t}} \frac{|\{i, i \in(L(u) \cap \sim \Phi)\}|}{|L(u)|}\]\(U_{t}\) is the number of users in the test set.
\(L_{u}\) is the recommended list of items for user u.
\(\sim \Phi\) medium-tail items.
To compute the metric, add it to the config file adopting the following pattern:
simple_metrics: [APLT]
ARP¶
-
class
elliot.evaluation.metrics.bias.arp.arp.ARP(recommendations, config, params, eval_objects)[source]¶ Bases:
elliot.evaluation.metrics.base_metric.BaseMetricAverage Recommendation Popularity
This class represents the implementation of the Average Recommendation Popularity recommendation metric.
For further details, please refer to the paper
\[\mathrm {ARP}=\frac{1}{\left|U_{t}\right|} \sum_{u \in U_{t}} \frac{\sum_{i \in L_{u}} \phi(i)}{\left|L_{u}\right|}\]\(U_{t}\) is the number of users in the test set.
\(L_{u}\) is the recommended list of items for user u.
To compute the metric, add it to the config file adopting the following pattern:
simple_metrics: [ARP]
PopREO¶
-
class
elliot.evaluation.metrics.bias.pop_reo.pop_reo.PopREO(recommendations, config, params, eval_objects)[source]¶ Bases:
elliot.evaluation.metrics.base_metric.BaseMetricPopularity-based Ranking-based Equal Opportunity
This class represents the implementation of the Popularity-based Ranking-based Equal Opportunity (REO) recommendation metric.
For further details, please refer to the paper
\[\mathrm {REO}=\frac{{std}\left(P\left(R @ k \mid g=g_{1}, y=1\right) \ldots P\left(R(a) k=g_{A}, y=1\right)\right)} {{mean}\left(P\left(R @ k \mid g=g_{1}, y=1\right) \ldots P\left(R @ k \mid g=g_{A}, y=1\right)\right)}\]\(P\left(R @ k \mid g=g_{a}, y=1\right) = \frac{\sum_{u=1}^{N} \sum_{i=1}^{k} G_{g_{a}}\left(R_{u, i}\right) Y\left(u, R_{u, i}\right)} {\sum_{u=1}^{N} \sum_{i \in I \backslash I_{u}^{+}} G_{g_{a}}(i) Y(u, i)}\)
\(Y\left(u, R_{u, i}\right)\) identifies the ground-truth label of a user-item pair left(u, R_{u, i}right), if item R_{u, i} is liked by user 𝑢, returns 1, otherwise 0
\(\sum_{i=1}^{k} G_{g_{a}}\left(R_{u, i}\right) Y\left(u, R_{u, i}\right)\) counts how many items in test set from group {g_a} are ranked in top-𝑘 for user u
\(\sum_{i \in I \backslash I_{u}^{+}} G_{g_{a}}(i) Y(u, i)\) counts the total number of items from group {g_a} 𝑎 in test set for user u
To compute the metric, add it to the config file adopting the following pattern:
simple_metrics: [PopREO]
Extended PopREO¶
-
class
elliot.evaluation.metrics.bias.pop_reo.extended_pop_reo.ExtendedPopREO(recommendations, config, params, eval_objects, additional_data)[source]¶ Bases:
elliot.evaluation.metrics.base_metric.BaseMetricExtended Popularity-based Ranking-based Equal Opportunity
This class represents the implementation of the Extended Popularity-based Ranking-based Equal Opportunity (REO) recommendation metric.
For further details, please refer to the paper
To compute the metric, add it to the config file adopting the following pattern:
complex_metrics: - metric: ExtendedPopREO
PopRSP¶
-
class
elliot.evaluation.metrics.bias.pop_rsp.pop_rsp.PopRSP(recommendations, config, params, eval_objects)[source]¶ Bases:
elliot.evaluation.metrics.base_metric.BaseMetricPopularity-based Ranking-based Statistical Parity
This class represents the implementation of the Popularity-based Ranking-based Statistical Parity (RSP) recommendation metric.
For further details, please refer to the paper
\[\mathrm {RSP}=\frac{{std}\left(P\left(R @ k \mid g=g_{1}\right), \ldots, P\left(R @ k \mid g=g_{A}\right)\right)} {{mean}\left(P\left(R @ k \mid g=g_{1}\right), \ldots, P\left(R @ k \mid g=g_{A}\right)\right)}\]:math P(R @ k mid g=g_{A})) = frac{sum_{u=1}^{N} sum_{i=1}^{k} G_{g_{a}}(R_{u, i})} {sum_{u=1}^{N} sum_{i in I backslash I_{u}^{+}} G_{g_{a}}(i)}
\(\sum_{i=1}^{k} G_{g_{a}}\left(R_{u, i}\right)\) calculates how many un-interacted items from group {g_a} are ranked in top-𝑘 for user u.
\(\sum_{i \in I \backslash I_{u}^{+}} G_{g_{a}}(i)\) calculates how many un-interacted items belong to group {g_a} for u
To compute the metric, add it to the config file adopting the following pattern:
simple_metrics: [PopRSP]
Extended PopRSP¶
-
class
elliot.evaluation.metrics.bias.pop_rsp.extended_pop_rsp.ExtendedPopRSP(recommendations, config, params, eval_objects, additional_data)[source]¶ Bases:
elliot.evaluation.metrics.base_metric.BaseMetricExtended Popularity-based Ranking-based Statistical Parity
This class represents the implementation of the Extended Popularity-based Ranking-based Statistical Parity (RSP) recommendation metric.
For further details, please refer to the paper
To compute the metric, add it to the config file adopting the following pattern:
complex_metrics: - metric: ExtendedPopRSP