## Abstract

While previous work in comparing statistical significance tests for IR system evaluation have focused on paired data tests (e.g., for evaluating two systems using a common test collection), two-sample tests must be used when the reproducibility of IR experiments across different test collections must be examined. Using real runs and a test collection from the NTCIR-15 WWW-3 Task, the present study compares the properties of three two-sample significance tests for comparing two systems: Student's t-test (i.e., the classical parametric test), the Wilcoxon rank sum test (i.e., the classical nonparametric test), and the randomisation test (i.e., a population-free method that utilises modern computational power). In terms of the false positive rate (i.e., the chance of detecting a statistical significance even though the two samples of evaluation measure scores come from the same system), the three tests behave similarly, although the Wilcoxon rank sum test appears to be slightly more robust than the other two for very small topic set sizes (e.g., 10 topics each) with a large significance level (e.g., α=0.10). On the other hand, the t-test and the Wilcoxon rank sum test are very similar to each other from the following two viewpoints: "How often do they both detect a nonexistent difference?"and "How often do they both overlook a true difference?"Compared to the two classical significance tests, the randomisation test behaves markedly differently in terms of the above two viewpoints. Hence, we suggest that researchers should at least be aware of the above properties of the three two-sample tests when choosing from them.

Original language | English |
---|---|

Title of host publication | SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval |

Publisher | Association for Computing Machinery, Inc |

Pages | 1980-1984 |

Number of pages | 5 |

ISBN (Electronic) | 9781450380379 |

DOIs | |

Publication status | Published - 2021 Jul 11 |

Event | 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2021 - Virtual, Online, Canada Duration: 2021 Jul 11 → 2021 Jul 15 |

### Publication series

Name | SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval |
---|

### Conference

Conference | 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2021 |
---|---|

Country/Territory | Canada |

City | Virtual, Online |

Period | 21/7/11 → 21/7/15 |

## Keywords

- evaluation
- randomisation
- statistical significance
- two-sample tests

## ASJC Scopus subject areas

- Software
- Computer Graphics and Computer-Aided Design
- Information Systems