Randomised vs. Prioritised Pools for Relevance Assessments: Sample Size Considerations

Tetsuya Sakai*, Peng Xiao


研究成果: Conference contribution

4 被引用数 (Scopus)


The present study concerns depth-k pooling for building IR test collections. At TREC, pooled documents are traditionally presented in random order to the assessors to avoid judgement bias. In contrast, an approach that has been used widely at NTCIR is to prioritise the pooled documents based on “pseudorelevance,” in the hope of letting assessors quickly form an idea as to what constitutes a relevant document and thereby judge more efficiently and reliably. While the recent TREC 2017 Common Core Track went beyond depth-k pooling and adopted a method for selecting documents to judge dynamically, even this task let the assessors process the usual depth-10 pools first: the idea was to give the assessors a “burn-in” period, which actually appears to echo the view of the NTCIR approach. Our research questions are: (1) Which depth-k ordering strategy enables more efficient assessments? Randomisation, or prioritisation by pseudorelevance? (2) Similarly, which of the two strategies enables higher inter-assessor agreements? Our experiments based on two English web search test collections with multiple sets of graded relevance assessments suggest that randomisation outperforms prioritisation in both respects on average, although the results are statistically inconclusive. We then discuss a plan for a much larger experiment with sufficient statistical power to obtain the final verdict.

ホスト出版物のタイトルInformation Retrieval Technology - 15th Asia Information Retrieval Societies Conference, AIRS 2019, Proceedings
編集者Fu Lee Wang, Haoran Xie, Wai Lam, Aixin Sun, Lun-Wei Ku, Tianyong Hao, Wei Chen, Tak-Lam Wong, Xiaohui Tao
出版ステータスPublished - 2020
イベント15th Asia Information Retrieval Societies Conference, AIRS 2019 - Kowloon, Hong Kong
継続期間: 2019 11月 72019 11月 9


名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
12004 LNCS


Conference15th Asia Information Retrieval Societies Conference, AIRS 2019
国/地域Hong Kong

ASJC Scopus subject areas

  • 理論的コンピュータサイエンス
  • コンピュータ サイエンス(全般)


「Randomised vs. Prioritised Pools for Relevance Assessments: Sample Size Considerations」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。