Metrics in Learn-To-Rank (LTR) problems
Suppose we have Q
queries, for each query q
, it brings \(N_q\) results and normally we care about the top K
docs retrieved.
All of the following metric is for 1 query, we need to average them to get an overall score for Q
queries.
-
Accuracy@K
: Whether top K results contain the relevant results.- Formula: \(A(K)=(\sum_{i=1}^K I(i)) > 0\). \(I(i)\) is an indicator function suggesting whether the \(i_{th}\) position is relevant or not.
-
Precision@K
: How many relevant results among the top K.- Formula: \(P(K) =\frac{1}{K}\sum_{i=1}^KI(i)\).
- Only for binary cases
-
Recall@K
: How many relevant results among the top K compared with all relevant results.- Formula: \(R(K) = \frac{1}{m}\sum_{i=1}^KI(i)\). \(m\) is the total number of relevant results overall.
- Only for binary cases
-
AP@K
(Average Precision): the weighted mean of precision at each threshold; the weight is the increase in recall from the prior threshold.- Formula: \(AP@K = \frac{1}{m}\sum_{k=1}^K P(k)I(k)\)
- A different way to represent \(AP@K\): We can also use
Recall@K
here by using its delta format \(\Delta R(k) = R(k) - R(k-1)\). It would be \(\frac{1}{m}\) if a relevant result is at position \(k\), otherwise it is 0. and it becomes \(AP@K = \sum_{k=1}^K P(k) * \Delta R(k)\). In other words, it is the area under Precision-Recall curve. - Only for binary cases
-
RR
(Reciprocal Rank): reciprocal of the rank of the first relevant item in a ranked list.- Formula: \(RR = \frac{1}{\text{rank of the first relevant item}}\)
- Only for binary cases.
-
NDCG@K
(Normalized Discounted Cumulative Gain)-
CG@K
: Sum of relevance of all results in a search result list \(CG@K = \sum_{k=1}^K rel(k)\), \(rel(k)\) is the relevance score of the result at \(k\). -
DCG@K
: Give earlier result higher weights: \(DCG@K = \sum_{k=1}^K\frac{2^{rel(k)} - 1}{ \log_2(k+1)}\) -
NDCG@K
: normalizeDCG
by its perfect ranking result (a.k.an IdealDCG
(IDCG
)) to reduce the effect that different search results can vary in length.
-
- Normalized Kendall tau distance at K:
- \(K_n@K(\tau_1, \tau_2) = \frac{K_d}{\frac{1}{2}n(n-1)} = \frac{2}{n(n-1)} \sum_{\{i,j\}\in \mathbb{K}, i < j} \bar K_{i,j}(\tau_1, \tau_2)\). \(\mathbb{K}\) is the set of unordered pairs of distinct elements among the top K in \(\tau_1\) and \(\tau_2\). \(\bar K_{i,j}(\tau_1, \tau_2) = 0\) if \(i\) and \(j\) are in the same order in \(\tau_1\) and \(\tau_2\), otherwise 1.
- Spearman Rho:
- Formula: \(\rho = \frac{cov(R(X), R(Y))}{\delta_{R(x)}\delta_{R(Y)}}\)
Reference:
- https://en.wikipedia.org/wiki/Discounted_cumulative_gain
- https://en.wikipedia.org/wiki/Kendall_tau_distance
- https://towardsdatascience.com/normalized-discounted-cumulative-gain-37e6f75090e9
Enjoy Reading This Article?
Here are some more articles you might like to read next: