Evaluation for (Cross-Language) Information Retrieval
Mark Sanderson
Dept. of Information Studies, University of Sheffield, United Kingdom
Abstract
Evaluation of search output is a critical component of any form of Information Access, however in MLIA, evaluation is additionally challenging. In this session, I will provide a broad introduction to evaluation of information access systems, where the main focus of will be on evaluation using test collections. Here the design and use of test collections will be described along with the measures employed to determine the effectiveness of a searching system. Alternative evaluation methods such as query log analysis and user studies will also be described. Next, the means by which evaluation methods have been adapted to work with multi-lingual systems are described. Finally, the additional assessments that need to be conducted to fully understand the success of MLIA are detailed.
Course Material
Evaluation in IR
- Learning User Interaction Models for Predicting Web Search Results Preferences
Eugene Agichtein, Eric Brill, Susan Dumais, Robert Ragno, Microsoft Research
- IR Evaluation Methods for Retrieving Highly Relevant Documents
Kalervo Järvelin and Jaana Kekäläinen, University of Tampere
- Accurately Interpreting Clickthrough Data as Implicit Feedback
Thorsten Joachims, Cornell University,
Laura Granka, Stanford University,
Bing Pan, Helene Hembrooke & Geri Gay, Cornell University
- Display Time as Implicit Feedback: Understanding Task Effects
Diane Kelly,
SILS University of North Carolina
Nicholas J. Belkin,
SCILS Rutgers University
- User Adaptation: Good results from Poor Systems
Catherine L. Smith & Paul B. Kantor,
Rutgers University, SCILS
