Journal Articles

Test theory for evaluating reliability of IR test collections

---

Classical test theory offers theoretically derived reliability measures such as Cronbach’s alpha,
which can be applied to measure the reliability of a set of Information Retrieval test results.
The theory also supports item analysis, which identifies queries that are hampering the test’s reliability,
and which may be candidates for refinement or removal. A generalization of Classical Test Theory, called Generalizability Theory, provides an even richer set of tools. It allows us to estimate the reliability of a test as a function of the number of queries, assessors (relevance judges), and other aspects of the test’s design. One novel aspect of Generalizability Theory is that it allows this estimation of reliability even before the test collection exists, based purely on the numbers of queries and assessors that it will contain. These calculations can help test designers in advance, by allowing them to compare the reliability of test designs with various numbers of queries and relevance assessors, and to spend their limited budgets on a design that maximizes reliability. Empirical analysis shows that in cases for which our data is representative, having more queries is more helpful for reliability than having more assessors. It also suggests that reliability may be improved with a per-document performance measure, as opposed to a document-set based performance measure, where appropriate. The theory also clarifies the implicit debate in IR literature regarding the nature of error in relevance judgments.

---

Availability

No copy data

Detail Information

Series Title

Information Processing & Management

Call Number

-

Publisher

s.l. : Elsevier., 2008

Collation

-

Language

English

ISBN/ISSN

0306-4573

Classification

-

Content Type

-

Media Type

-

Carrier Type

-

Edition

Vol. 44, No. 3, Page 1117-1145

Subject(s)

Evaluation
Test collections
Test theory
Generalizabilty theory

Specific Detail Info

-

Statement of Responsibility

-

Other version/related

File Attachment

Comments

---

You must be logged in to post a comment