Accuracy¶

Elliot integrates the following accuracy metrics.

Summary¶

`AUC.auc.AUC`(recommendations, config, params, …)	Area Under the Curve
`AUC.gauc.GAUC`(recommendations, config, …)	Group Area Under the Curve
`AUC.lauc.LAUC`(recommendations, config, …)	Limited Area Under the Curve
`DSC.dsc.DSC`(recommendations, config, params, …)	Sørensen–Dice coefficient
`f1.f1.F1`(recommendations, config, params, …)	F-Measure
`f1.extended_f1.ExtendedF1`(recommendations, …)	Extended F-Measure
`hit_rate.hit_rate.HR`(recommendations, …)	Hit Rate
`map.map.MAP`(recommendations, config, params, …)	Mean Average Precision
`mar.mar.MAR`(recommendations, config, params, …)	Mean Average Recall
`mrr.mrr.MRR`(recommendations, config, params, …)	Mean Reciprocal Rank
`ndcg.ndcg.nDCG`(recommendations, config, …)	normalized Discounted Cumulative Gain
`ndcg.ndcg_rendle2020.nDCGRendle2020`(…)	normalized Discounted Cumulative Gain
`precision.precision.Precision`(…)	Precision-measure
`recall.recall.Recall`(recommendations, …)	Recall-measure

AUC¶

class elliot.evaluation.metrics.accuracy.AUC.auc.AUC(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Area Under the Curve

This class represents the implementation of the global AUC recommendation metric. Passing ‘AUC’ to the metrics list will enable the computation of the metric.

For further details, please refer to the AUC

Note

This metric does not calculate group-based AUC which considers the AUC scores averaged across users. It is also not limited to k. Instead, it calculates the scores on the entire prediction results regardless the users.

\[\mathrm {AUC} = \frac{\sum\limits_{i=1}^M rank_{i} - \frac {{M} \times {(M+1)}}{2}} {{{M} \times {N}}}\]

\(M\) is the number of positive samples.

\(N\) is the number of negative samples.

\(rank_i\) is the ascending rank of the ith positive sample.

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [AUC]

GAUC¶

class elliot.evaluation.metrics.accuracy.AUC.gauc.GAUC(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Group Area Under the Curve

This class represents the implementation of the GroupAUC recommendation metric. Passing ‘GAUC’ to the metrics list will enable the computation of the metric.

“Deep Interest Network for Click-Through Rate Prediction” KDD ‘18 by Zhou, et al.

For further details, please refer to the paper

Note

It calculates the AUC score of each user, and finally obtains GAUC by weighting the user AUC. It is also not limited to k. Due to our padding for scores_tensor in RankEvaluator with -np.inf, the padding value will influence the ranks of origin items. Therefore, we use descending sort here and make an identity transformation to the formula of AUC, which is shown in auc_ function. For readability, we didn’t do simplification in the code.

\[\mathrm {GAUC} = \frac {{{M} \times {(M+N+1)} - \frac{M \times (M+1)}{2}} - \sum\limits_{i=1}^M rank_{i}} {{M} \times {N}}\]

\(M\) is the number of positive samples.

\(N\) is the number of negative samples.

\(rank_i\) is the descending rank of the ith positive sample.

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [GAUC]

LAUC¶

class elliot.evaluation.metrics.accuracy.AUC.lauc.LAUC(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Limited Area Under the Curve

This class represents the implementation of the Limited AUC recommendation metric. Passing ‘LAUC’ to the metrics list will enable the computation of the metric.

“Setting Goals and Choosing Metrics for Recommender System Evaluations” by Gunnar Schröder, et al.

For further details, please refer to the paper

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [LAUC]

DSC¶

class elliot.evaluation.metrics.accuracy.DSC.dsc.DSC(recommendations, config, params, eval_objects, additional_data)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Sørensen–Dice coefficient

This class represents the implementation of the Sørensen–Dice coefficient recommendation metric. Passing ‘DSC’ to the metrics list will enable the computation of the metric.

For further details, please refer to the page

\[\mathrm {DSC@K} = \frac{1+\beta^{2}}{\frac{1}{\text { metric_0@k }}+\frac{\beta^{2}}{\text { metric_1@k }}}\]

Parameters

beta – the beta coefficient (default: 1)
metric_0 – First considered metric (default: Precision)
metric_1 – Second considered metric (default: Recall)

To compute the metric, add it to the config file adopting the following pattern:

complex_metrics:
- metric: DSC
  beta: 1
  metric_0: Precision
  metric_1: Recall

F1¶

class elliot.evaluation.metrics.accuracy.f1.f1.F1(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

F-Measure

This class represents the implementation of the F-score recommendation metric. Passing ‘F1’ to the metrics list will enable the computation of the metric.

For further details, please refer to the paper

\[\mathrm {F1@K} = \frac{1+\beta^{2}}{\frac{1}{\text { precision@k }}+\frac{\beta^{2}}{\text { recall@k }}}\]

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [F1]

Extended F1¶

class elliot.evaluation.metrics.accuracy.f1.extended_f1.ExtendedF1(recommendations, config, params, eval_objects, additional_data)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Extended F-Measure

This class represents the implementation of the F-score recommendation metric. Passing ‘ExtendedF1’ to the metrics list will enable the computation of the metric.

“Evaluating Recommender Systems” Gunawardana, Asela and Shani, Guy, In Recommender systems handbook pages 265–308, 2015

For further details, please refer to the paper

\[\mathrm {ExtendedF1@K} =\frac{2}{\frac{1}{\text { metric_0@k }}+\frac{1}{\text { metric_1@k }}}\]

Parameters

metric_0 – First considered metric (default: Precision)
metric_1 – Second considered metric (default: Recall)

To compute the metric, add it to the config file adopting the following pattern:

complex_metrics:
- metric: ExtendedF1
  metric_0: Precision
  metric_1: Recall

HR¶

class elliot.evaluation.metrics.accuracy.hit_rate.hit_rate.HR(recommendations: Dict[int, List[Tuple[int, float]]], config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Hit Rate

This class represents the implementation of the Hit Rate recommendation metric. Passing ‘HR’ to the metrics list will enable the computation of the metric.

For further details, please refer to the link

\[\mathrm {HR@K} =\frac{Number \space of \space Hits @K}{|GT|}\]

\(HR\) is the number of users with a positive sample in the recommendation list.

\(GT\) is the total number of samples in the test set.

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [HR]

MAP¶

class elliot.evaluation.metrics.accuracy.map.map.MAP(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Mean Average Precision

This class represents the implementation of the Mean Average Precision recommendation metric. Passing ‘MAP’ to the metrics list will enable the computation of the metric.

For further details, please refer to the link

Note

In this case the normalization factor used is \(\frac{1}{\min (m,N)}\), which prevents your AP score from being unfairly suppressed when your number of recommendations couldn’t possibly capture all the correct ones.

\[\begin{split}\begin{align*} \mathrm{AP@N} &= \frac{1}{\mathrm{min}(m,N)}\sum_{k=1}^N P(k) \cdot rel(k) \\ \mathrm{MAP@N}& = \frac{1}{|U|}\sum_{u=1}^{|U|}(\mathrm{AP@N})_u \end{align*}\end{split}\]

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [MAP]

MAR¶

class elliot.evaluation.metrics.accuracy.mar.mar.MAR(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Mean Average Recall

This class represents the implementation of the Mean Average Recall recommendation metric. Passing ‘MAR’ to the metrics list will enable the computation of the metric.

For further details, please refer to the link

\[\begin{split}\begin{align*} \mathrm{Recall@N} &= \frac{1}{\mathrm{min}(m,|rel(k)|)}\sum_{k=1}^N P(k) \cdot rel(k) \\ \mathrm{MAR@N}& = \frac{1}{|U|}\sum_{u=1}^{|U|}(\mathrm{Recall@N})_u \end{align*}\end{split}\]

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [MAR]

MRR¶

class elliot.evaluation.metrics.accuracy.mrr.mrr.MRR(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Mean Reciprocal Rank

This class represents the implementation of the Mean Reciprocal Rank recommendation metric. Passing ‘MRR’ to the metrics list will enable the computation of the metric.

For further details, please refer to the link

\[\mathrm {MRR} = \frac{1}{|{U}|} \sum_{i=1}^{|{U}|} \frac{1}{rank_i}\]

\(U\) is the number of users, \(rank_i\) is the rank of the first item in the recommendation list in the test set results for user \(i\).

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [MRR]

nDCG¶

class elliot.evaluation.metrics.accuracy.ndcg.ndcg.nDCG(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

normalized Discounted Cumulative Gain

This class represents the implementation of the nDCG recommendation metric.

For further details, please refer to the link

\[\begin{split}\begin{gather} \mathrm {DCG@K}=\sum_{i=1}^{K} \frac{2^{rel_i}-1}{\log_{2}{(i+1)}}\\ \mathrm {IDCG@K}=\sum_{i=1}^{K}\frac{1}{\log_{2}{(i+1)}}\\ \mathrm {NDCG_u@K}=\frac{DCG_u@K}{IDCG_u@K}\\ \mathrm {NDCG@K}=\frac{\sum \nolimits_{u \in u^{te}NDCG_u@K}}{|u^{te}|} \end{gather}\end{split}\]

\(K\) stands for recommending \(K\) items.

And the \(rel_i\) is the relevance of the item in position \(i\) in the recommendation list.

\(2^{rel_i}\) equals to 1 if the item hits otherwise 0.

\(U^{te}\) is for all users in the test set.

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [nDCG]

nDCGRendle2020¶

class elliot.evaluation.metrics.accuracy.ndcg.ndcg_rendle2020.nDCGRendle2020(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

normalized Discounted Cumulative Gain

This class represents the implementation of the nDCG recommendation metric.

For further details, please refer to the link

\[\begin{split}\begin{gather} \mathrm {DCG@K}=\sum_{i=1}^{K} \frac{2^{rel_i}-1}{\log_{2}{(i+1)}}\\ \mathrm {IDCG@K}=\sum_{i=1}^{K}\frac{1}{\log_{2}{(i+1)}}\\ \mathrm {NDCG_u@K}=\frac{DCG_u@K}{IDCG_u@K}\\ \mathrm {NDCG@K}=\frac{\sum \nolimits_{u \in u^{te}NDCG_u@K}}{|u^{te}|} \end{gather}\end{split}\]

\(K\) stands for recommending \(K\) items.

And the \(rel_i\) is the relevance of the item in position \(i\) in the recommendation list.

\(2^{rel_i}\) equals to 1 if the item hits otherwise 0.

\(U^{te}\) is for all users in the test set.

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [nDCG]

Precision¶

class elliot.evaluation.metrics.accuracy.precision.precision.Precision(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Precision-measure

This class represents the implementation of the Precision recommendation metric.

For further details, please refer to the link

\[\mathrm {Precision@K} = \frac{|Rel_u \cap Rec_u|}{Rec_u}\]

\(Rel_u\) is the set of items relevant to user \(U\),

\(Rec_u\) is the top K items recommended to users.

We obtain the result by calculating the average \(Precision@K\) of each user.

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [Precision]

Recall¶

class elliot.evaluation.metrics.accuracy.recall.recall.Recall(recommendations, config, params, eval_objects)[source]¶

Bases: elliot.evaluation.metrics.base_metric.BaseMetric

Recall-measure

This class represents the implementation of the Recall recommendation metric.

For further details, please refer to the link

\[\mathrm {Recall@K} = \frac{|Rel_u\cap Rec_u|}{Rel_u}\]

\(Rel_u\) is the set of items relevant to user \(U\),

\(Rec_u\) is the top K items recommended to users.

We obtain the result by calculating the average \(Recall@K\) of each user.

To compute the metric, add it to the config file adopting the following pattern:

simple_metrics: [Recall]