TY - GEN
T1 - The effect of score standardisation on topic set size design
AU - Sakai, Tetsuya
N1 - Publisher Copyright:
© Springer International Publishing AG 2016.
PY - 2016
Y1 - 2016
N2 - Given a topic-by-run score matrix from past data, topic set size design methods can help test collection builders determine the number of topics to create for a new test collection from a statistical viewpoint. In this study, we apply a recently-proposed score standardisation method called std-AB to score matrices before applying topic set size design, and demonstrate its advantages. For topic set size design, std-AB suppresses score variances and thereby enables test collection builders to consider realistic choices of topic set sizes, and to handle unnormalised measures in the same way as normalised measures. In addition, even discrete measures that clearly violate normality assumptions look more continuous after applying std-AB, which may make them more suitable for statistically motivated topic set size design. Our experiments cover a variety of tasks and evaluation measures from NTCIR-12.
AB - Given a topic-by-run score matrix from past data, topic set size design methods can help test collection builders determine the number of topics to create for a new test collection from a statistical viewpoint. In this study, we apply a recently-proposed score standardisation method called std-AB to score matrices before applying topic set size design, and demonstrate its advantages. For topic set size design, std-AB suppresses score variances and thereby enables test collection builders to consider realistic choices of topic set sizes, and to handle unnormalised measures in the same way as normalised measures. In addition, even discrete measures that clearly violate normality assumptions look more continuous after applying std-AB, which may make them more suitable for statistically motivated topic set size design. Our experiments cover a variety of tasks and evaluation measures from NTCIR-12.
UR - http://www.scopus.com/inward/record.url?scp=85007082699&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85007082699&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-48051-0_2
DO - 10.1007/978-3-319-48051-0_2
M3 - Conference contribution
AN - SCOPUS:85007082699
SN - 9783319480503
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 16
EP - 28
BT - Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings
A2 - Chang, Yi
A2 - Wen, Ji-Rong
A2 - Dou, Zhicheng
A2 - Zhao, Xin
A2 - Ma, Shaoping
A2 - Liu, Yiqun
A2 - Zhang, Min
PB - Springer Verlag
T2 - 12th Asia Information Retrieval Societies Conference, AIRS 2016
Y2 - 30 November 2016 through 2 December 2016
ER -