Evaluating evaluation measures with worst-case confidence interval widths

Tetsuya Sakai*


    研究成果: Conference article査読


    IR evaluation measures are often compared in terms of rank correlation between two system rankings, agreement with the users' preferences, the swap method, and discriminative power. While we view the agreement with real users as the most important, this paper proposes to use the Worst-case Confidence interval Width (WCW) curves to supplement it in test-collection environments. WCW is the worst-case width of a confidence interval (CI) for the difference between any two systems, given a topic set size. We argue that WCW curves are more useful than the swap method and discriminative power, since they provide a statistically well-founded overview of the comparison of measures over various topic set sizes, and visualise what levels of differences across measures might be of practical importance. First, we prove that Sakai's ANOVA-based topic set size design tool can be used for discussing WCW instead of his CI-based tool that cannot handle large topic set sizes. We then provide some case studies of evaluating evaluation measures using WCW curves based on the ANOVA-based tool, using data from TREC and NTCIR.

    ジャーナルCEUR Workshop Proceedings
    出版ステータスPublished - 2017 1月 1
    イベント8th International Workshop on Evaluating Information Access, EVIA 2017 - Tokyo, Japan
    継続期間: 2017 12月 5 → …

    ASJC Scopus subject areas

    • コンピュータ サイエンス(全般)


    「Evaluating evaluation measures with worst-case confidence interval widths」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。