The reliability of metrics based on graded relevance

Tetsuya Sakai

The reliability of metrics based on graded relevance

Tetsuya Sakai^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Citations (Scopus)

Abstract

This paper compares 14 metrics designed for information retrieval evaluation with graded relevance, together with 10 traditional metrics based on binary relevance, in terms of reliability and resemblance of system rankings. More specifically, we use two test collections with submitted runs from the Chinese IR and English IR tasks in the NTCIR-3 CLIR track to examine the metrics using methods proposed by Buckley/Voorhees and Voorhees/Buckley as well as Kendall's rank correlation. Our results show that AnDCG_l and nDCG_l ((Average) Normalised Discounted Cumulative Gain at Document cut-off l) are good metrics, provided that l is large. However, if one wants to avoid the parameter l altogether, or if one requires a metric that closely resembles TREC Average Precision, then Q-measure appears to be the best choice.

Original language	English
Title of host publication	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages	1-16
Number of pages	16
Publication status	Published - 2005 Dec 1
Externally published	Yes
Event	2nd Asia Information Retrieval Symposium, AIRS 2005 - Jeju Island, Korea, Republic of Duration: 2005 Oct 13 → 2005 Oct 15

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	3689 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2nd Asia Information Retrieval Symposium, AIRS 2005
Country/Territory	Korea, Republic of
City	Jeju Island
Period	05/10/13 → 05/10/15

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Cite this

Sakai, T. (2005). The reliability of metrics based on graded relevance. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 1-16). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 3689 LNCS).

The reliability of metrics based on graded relevance. / Sakai, Tetsuya.
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2005. p. 1-16 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 3689 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Sakai, T 2005, The reliability of metrics based on graded relevance. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 3689 LNCS, pp. 1-16, 2nd Asia Information Retrieval Symposium, AIRS 2005, Jeju Island, Korea, Republic of, 05/10/13.

@inproceedings{dd3cfdff05d54189b5eb6d8c12534296,

title = "The reliability of metrics based on graded relevance",

abstract = "This paper compares 14 metrics designed for information retrieval evaluation with graded relevance, together with 10 traditional metrics based on binary relevance, in terms of reliability and resemblance of system rankings. More specifically, we use two test collections with submitted runs from the Chinese IR and English IR tasks in the NTCIR-3 CLIR track to examine the metrics using methods proposed by Buckley/Voorhees and Voorhees/Buckley as well as Kendall's rank correlation. Our results show that AnDCGl and nDCGl ((Average) Normalised Discounted Cumulative Gain at Document cut-off l) are good metrics, provided that l is large. However, if one wants to avoid the parameter l altogether, or if one requires a metric that closely resembles TREC Average Precision, then Q-measure appears to be the best choice.",

author = "Tetsuya Sakai",

year = "2005",

month = dec,

day = "1",

language = "English",

isbn = "3540291865",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "1--16",

booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

note = "2nd Asia Information Retrieval Symposium, AIRS 2005 ; Conference date: 13-10-2005 Through 15-10-2005",

}

TY - GEN

T1 - The reliability of metrics based on graded relevance

AU - Sakai, Tetsuya

PY - 2005/12/1

Y1 - 2005/12/1

N2 - This paper compares 14 metrics designed for information retrieval evaluation with graded relevance, together with 10 traditional metrics based on binary relevance, in terms of reliability and resemblance of system rankings. More specifically, we use two test collections with submitted runs from the Chinese IR and English IR tasks in the NTCIR-3 CLIR track to examine the metrics using methods proposed by Buckley/Voorhees and Voorhees/Buckley as well as Kendall's rank correlation. Our results show that AnDCGl and nDCGl ((Average) Normalised Discounted Cumulative Gain at Document cut-off l) are good metrics, provided that l is large. However, if one wants to avoid the parameter l altogether, or if one requires a metric that closely resembles TREC Average Precision, then Q-measure appears to be the best choice.

AB - This paper compares 14 metrics designed for information retrieval evaluation with graded relevance, together with 10 traditional metrics based on binary relevance, in terms of reliability and resemblance of system rankings. More specifically, we use two test collections with submitted runs from the Chinese IR and English IR tasks in the NTCIR-3 CLIR track to examine the metrics using methods proposed by Buckley/Voorhees and Voorhees/Buckley as well as Kendall's rank correlation. Our results show that AnDCGl and nDCGl ((Average) Normalised Discounted Cumulative Gain at Document cut-off l) are good metrics, provided that l is large. However, if one wants to avoid the parameter l altogether, or if one requires a metric that closely resembles TREC Average Precision, then Q-measure appears to be the best choice.

UR - http://www.scopus.com/inward/record.url?scp=33646126694&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33646126694&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:33646126694

SN - 3540291865

SN - 9783540291862

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 1

EP - 16

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

T2 - 2nd Asia Information Retrieval Symposium, AIRS 2005

Y2 - 13 October 2005 through 15 October 2005

ER -

The reliability of metrics based on graded relevance

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this