Evaluating D-MERIT of Partial-annotation on Information Retrieval
Retrieval models are often evaluated on partially-annotated datasets. Each
query is mapped to a few relevant texts and the remaining corpus is assumed to
be irrelevant. As a result, models that successfully retrieve false negatives
are punished in evaluation. Unfortunately, completely annotating all texts for
every query is not resource efficient. In this work, we show that using
partially-annotated datasets in evaluation can paint a distorted picture. We
curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to
contain all relevant passages for each query. Queries describe a group (e.g.,
``journals about linguistics'') and relevant passages are evidence that
entities belong to the group (e.g., a passage indicating that Language is a
journal about linguistics). We show that evaluating on a dataset containing
annotations for only a subset of the relevant passages might result in
misleading ranking of the retrieval systems and that as more relevant texts are
included in the evaluation set, the rankings converge. We propose our dataset
as a resource for evaluation and our study as a recommendation for balance
between resource-efficiency and reliable evaluation when annotating evaluation
sets for text retrieval.
667a6c233f20f47d49334f91
Royi Rassin
extracted_pending
2024-06-25T07:05:08.346Z
62a7581cf049be35252a2e7c
/avatars/91de4eb48f51bfd6e028c08ccfa98f8c.svg
Royi Rassin
user
Royir
667a6c233f20f47d49334f92
Yaron Fairstein
claimed_verified
2024-06-25T11:17:13.015Z
63a417660cf4daf616676972
/avatars/75cd5327713e078e5e70c8a8086f3488.svg
Yaron Fairstein
user
yyfairstein
667a6c233f20f47d49334f93
Oren Kalinsky
extracted_pending
2024-06-25T07:05:08.346Z
6306aed3cfbde33ef7dd6dac
/avatars/a59f8ffec3b87d71e428425d74c74795.svg
Oren Kalinsky
user
orenk
667a6c233f20f47d49334f94
Guy Kushilevitz
667a6c233f20f47d49334f95
Nachshon Cohen
extracted_confirmed
2024-10-27T08:03:33.794Z
63a417f9412fd71fb7ebd218
/avatars/0dcac15e5cc6c9cb89dd7af2818bacb9.svg
Nachshon Cohen
user
nachshonc
667a6c233f20f47d49334f96
Alexander Libov
claimed_verified
2024-06-25T13:34:22.404Z
63ad4bae9835dd4b124f32c5
/avatars/972929b6e560f328a305f99e5365b685.svg
Alexander Libov
user
alibov
667a6c233f20f47d49334f97
Yoav Goldberg
extracted_confirmed
2024-12-26T13:11:49.220Z
676d5323dd95830fd94d675f
/avatars/a8622e2a189a13c7b5a99177c3a87c37.svg
Yoav Goldberg
user
yoavgoldberg
Evaluating D-MERIT of Partial-annotation on Information Retrieval
Retrieval models are often evaluated on partially-annotated datasets. Each
query is mapped to a few relevant texts and the remaining corpus is assumed to
be irrelevant. As a result, models that successfully retrieve false negatives
are punished in evaluation. Unfortunately, completely annotating all texts for
every query is not resource efficient. In this work, we show that using
partially-annotated datasets in evaluation can paint a distorted picture. We
curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to
contain all relevant passages for each query. Queries describe a group (e.g.,
``journals about linguistics'') and relevant passages are evidence that
entities belong to the group (e.g., a passage indicating that Language is a
journal about linguistics). We show that evaluating on a dataset containing
annotations for only a subset of the relevant passages might result in
misleading ranking of the retrieval systems and that as more relevant texts are
included in the evaluation set, the rankings converge. We propose our dataset
as a resource for evaluation and our study as a recommendation for balance
between resource-efficiency and reliable evaluation when annotating evaluation
sets for text retrieval.
Evaluating retrieval models on partially-annotated datasets can lead to misleading rankings, and more comprehensive annotation improves evaluation reliability.
retrieval models partially-annotated datasets relevant texts false negatives D-MERIT passage retrieval Wikipedia evaluation set text retrieval resource-efficiency reliable evaluation