Learning to Rank for In-Context Example Retrieval
Yuwen Ji, Luodan Zhang, Ambyer han, Yue Zhang, Haoran Que, Lei Shi, Wang Chao
Recent advances in retrieval-based in-context learning (ICL) train the retriever as a classification model, which categorizes in-context examples (ICEs) into the most useful and the rest based on absolute scores.However, during inference, ICEs are *retrieved by score ranking* rather than classification --- The classification training objective deviates from this test scenario.Hence, in this paper, we propose **SE-DPO** --- a novel algorithm that trains a retrieval model by ranking formulation, where the preference rankings between ICEs are given by comparing the likelihood of the LLM generating the correct answer conditioned on each exemplar.By learning to rank, we motivate the retriever to automatically learn diverse rationales why specific examples are more useful for ICL decisions.This addresses the issue that classification models poorly capture broader utility.Experimental results demonstrate the top-1 performance of our proposal across 9 NLP tasks, with ablation studies and case studies further validating the effectiveness of our design.