This paper presents a simple approach to utilize past test collections as a material for user experiments. We have built a Web-based user interface for NTCIR-5 WEB run results, and conducted a user experiment with 29 subjects to investigate whether performance evaluation metrics of information retrieval systems used in test collections such as TREC and NTCIR comparable to user performance. In this experiment, we selected three types of systems from among systems that participated in NTCIR-5 WEB, and then selected three topics with roughly the same values from among several search topics. The results of the experiment showed no significant differences among these systems and topics in the time for search. While, in general, the user experiment itself have been successfully conducted and shown similar trends with prior study, the approach seems to have some limitations mainly on interactivity and cached page display.
雑誌名
Proceedings of the first international workshop on evaluating information access (EVIA)
ページ
78 - 81
発行年
2007-05-15
出版者
National Institute of Informatics
書誌情報
Proceedings of the first international workshop on evaluating information access (EVIA). Tokyo, Japan, 2007-05-15. National Institute of Informatics, 2007, p. 78-81.
An Application of the NTCIR-WEB Raw-data Archive Dataset for User Experiments
Saitoh002.pdf
(165.56KB)
AUE Repositoryへの登録について