Learning to rank
lastmod 2024-04-24created 2024-04-242 edits
Learning to rank or machine-learned ranking (MLR) is ranking using machine learning. It is the application of machine learning, typically supervised, semi-supervised, or reinforcement learning, to the construction of ranking models for information retrieval systems. For example, training data may consist of lists of items with some partial order specified between items in each list. This order is typically induced by giving a numerical or ordinal score or a binary judgment (e.g. "relevant" or "not relevant") for each item. The goal of constructing the ranking model is to rank new, unseen lists in a similar way to rankings in the training data.
What is Learning to Rank (LTR)?
Learning to Rank (LTR) refers to the methodology of using machine learning in a Ranking System.
Ranking optimization problem
Related topics:
- Top-K Correction
- Loss Function based on Ranking Metric - The LambdaLoss Framework for Ranking Metric Optimization
- NDCG optimization
- Optimization by EM Algorithm
Metrics used in Ranking
In Ranking, the following metrics (evaluation measures) are used to measure how well a model ranks Items.
- MRR(Mean Reciprocal Rank)
- A method that averages, for each Query, the score for getting the top Item right. When multiple Items are ranked for a Query, the reciprocal rank is calculated according to the position of the Item that is the correct answer (rank 1) in the test set. Then averaging the reciprocal ranks of all Queries yields the MRR score.
- This method has the limitation that it only looks at the position of the rank-1 item, ignoring the relevance of other Items.
- Precision at k
- Precision means the proportion of relevant items among the recommended top k Items. This metric only judges whether an item is relevant or not. That is, it has the limitation that it does not calculate a score for rank.
- nDCG(normalized Discounted Cumulative Gain)
- nDCG is a metric that compensates for the drawbacks of both MRR and Precision.
- DCG
- DCG is a method that calculates discounted relevance scores by gradually reducing the weight according to Ranking order. You can think of it as applying a penalty as the rank goes lower. If you want to weight relevance more than Ranking order, use the second of the formulas above.
- IDCG → nDCG
- Because DCG varies a lot with p, the length of the Ranking result, it needs to be normalized to have values on a consistent scale. This can be solved by computing IDCG.
- Dividing DCG by IDCG gives nDCG.
- DCG
Deep learning models
- GSF (Groupwise Scoring Function)
- seq2slate
- DLCM (Deep Listwise Context Model)
- Context-Aware Ranker - Shows the best performance (SOTA) on MSLR-WEB30K, an LTR benchmark dataset.
See also
- Recommender system (Korean: 추천 시스템)
- Content-based filtering (Korean: 컨텐츠 기반 필터링)
- Collaborative filtering (Korean: 협업 필터링)
- Hybrid recommender systems
- Machine Learning