TrebleCLEF Logo
Evaluation, Best Practices and Collaboration for Multilingual Information Access
Latest News
CLEF 2010: Padua, Italy September 2010
Read more - Link

TrebleCLEF workshop at eChallenges : Best Practices for Multilingual Information Access Istanbul,...
Read more - Link

Read all news

TrebleCLEF Flyer

TrebleCLEF Deliverables & Reports

Del no.
Deliverable Title
Dissemination Plan - version 1
Operational Scientific Digital Library
Evaluation Resources for CLEF
CLEF2008 Evaluation Campaign Workshop
System Developers Workshop
MLIA User Communities Workshop
Best Practices in language Resources for MLIA
Analysis of Evaluation Campaign Results
Progress Reports
Financial Reports
Query Log Analysis Workshop
MLIA Summer School and Tutorials
CLEF 2008 Workshop Proceedings
System-oriented and User-oriented MLIA Best-practice Recommendations
CLEF Evaluation Resources
CLEF 2009 Evaluation Campaign
MLIA Technology Day
Best Practices for Test Collection Creation, Evaluation Methodologies and Language Processing Technologies
CLEF Evaluation Package
Analysis of Evaluation Campaign Results

Progress Report (Preliminary)

Financial Report
Exploitation Plan (Final)

Report by Mark Sanderson of the Workshop on Novel Technologies for Evaluation in Information Retrieval - held at ECIR 2008, Glasgow, UK, 30 March, 2008

TrebleCLEF Released Publications

  • Amigó E., Gonzalo J., Artiles J., Verdejo F., 2008. Comparison of extrinsic clustering evaluation metrics based on formal constraints. Information Retrieval Journal.
  • Braschler M., Di Nunzio G., Ferro N., Gonzalo J., Peters C., Sanderson M., 2007. From CLEF to TrebleCLEF: Promoting Technology Transfer for Multilingual Information Retrieval. In: Thanos C., Borri F., Launaro A. editors, Working Notes of the Second DELOS Conference on Digital Libraries, Pisa, Italy.
  • Braschler M., Ferro N., Gonzalo J., Sanderson M., 2008. Workshop on Novel Methodologies for Evaluation in Information Retrieval. In: Advances in Information Retrieval. 30th European Conference on IR Research, ECIR 2008, Glasgow, UK, March 30-April 3, 2008. Proceedings. Lecture Notes in Computer Science (LNCS) 4956, Springer Berlin/Heidelberg, p. 713.
    Link to the Springer page
  • Clough P., Eleta I., (to appear). Investigating Language Skills and Field of Knowledge on Multilingual Information Access in Digital Libraries. In International Journal of Digital Library Systems (IJDLS).
  • Clough P., Gonzalo J., Karlgren J., Barker E., Artile J., Peinado V., 2008. Large-Scale Interactive Evaluation of Multilingual Information Access Systems - the iCLEF Flickr Challenge. In Workshop on Novel Methodologies for Evaluation (ECIR 2008).
    google scholar
  • Dussin M., Ferro N., 2008. Direct: Applying the DIKW Hierarchy to Large-scale Evaluation Campaigns. In ACM Press, 2008, Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2008), NY, USA, p. 424.
    Link to the ACM portal
  • Dussin, M. and Ferro, N., 2008. The Design of the User Interface of a Scientific DLS in the context of the Data, Information, Knowledge and Wisdom. In Agosti, M. and Thanos, C., editors, Proceedings of the Fourth Italian Research Conference on Digital Library Systems (IRCDL 2008).
  • Dussin M., Ferro N. (2008). The Role of the DIKW Hierarchy in the Design of a Digital Library System for the Scientific Data of Large-scale Evaluation Campaigns. In ACM Press, 2008, Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2008), NY, USA, p. 450.
    Link to the ACM portal
  • Ferro, N. and Harman, D., 2008. Dealing with MultiLingual Information Access: Grid Experiments at TrebleCLEF. In Agosti, M. and Thanos, C., editors, 2008, Proceedings of the Fourth Italian Research Conference on Digital Library Systems (IRCDL 2008).
  • Herrera J., Peñas A., Verdejo F., LNAI, 2006. Textual Entailment Recognition Based on Dependency Analysis and WordNet. In: J. Quinonero-Candela, I. Dagan, B. Magnini, F. dAlché-Buc editors. MLCW 2005; . 231.
    google scholar
  • J. Herrera, A. Peñas, F. Verdejo, LNAI, 2006. Techniques for Recognizing Textual Entailment and Semantic Equivalence. In: Postproceedings of the XI Conference of the Spanish Association for Artificial Intelligence. Santiago de Compostela, Spain. 2006.
    google scholar
  • Herrera J., Peñas A., Verdejo M., 2007. Paraphrase Extraction from Validated Question Answering Corpora in Spanish. In Working Notes of the XXIII Conference of the Spanish Association for Natural Language Processing, (SEPLN 2007).
    google scholar
  • López-Ostenero F., Peinado V., Gonzalo J., Verdejo F., 2008. Interactive Question Answering: Is Cross-Language Harder than Monolingual Searching? In Information Processing and Management. Special topic issue on User-centered Evaluation of Information Retrieval Systems. 66-81.
    google scholar
  • Paramita M.L., Sanderson M., Clough P., (2009). Developing a test Collection to Support Diversity Analysis. In Redundancy, Diversity and Interdependent Document Relevance, July 23, 2009, Boston, Massachussets, US, Proceedings of the SIGIR 2009 Workshop.
  • Peñas A., Rodrigo Á., Verdejo F., 2006. SPARTE, a Test Suite for Recognising Textual Entailment in Spanish. In Alexander F. Gelbukh editor, 2006, Computational Linguistics and Intelligent Text Processing, 7th International Conference, CICLing 2006, Mexico City, Mexico, February 19-25, 2006, Proceedings, Lecture Notes in Computer Science. p. 275.
    google scholar
  • Peñas A., Sutcliffe R., 2006. Multilingual Question Answering - MLQA'06. EACL 2006 Workshop Proceedings.
    google scholar
  • Peñas A., Rodrigo Á., Sama V., Verdejo F., 2008. Testing the Reasoning for Question Answering Validation. Journal of Logic and Computation, Special Issue on Natural Language and Knowledge Representation,. To appear (draft version).
  • Rodrigo Á., Peñas A., Verdejo F., 2008. Evaluating Answer Validation in multi-stream Question Answering. In Proceedings of The Second International Workshop on Evaluating Information Access (EVIA-NTCIR 2008).
  • Téllez-Valero A., Montes-y-Gómez M., Villaseñor-Pineda L., Peñas A., 2008. Improving Question Answering by Combining Multiple Systems via Answer Validation. In Computational Linguistics and Intelligent Text Processing (CICLing 2008).

Related Literature

Multilingual Information Access / Information Retrieval System Evaluation
CLEF Publications

Pubications of Interest

  • De Groot & van Laer “The quality of legal dictionaries: an assesment” (full text on (pdf); abstract on ) [Maastricht Faculty of Law working paper & Social Science Research Network respectively].
    Not only criteria for reliable bilingual dictionaries have been developed (16 pages), but these criteria have been applied on more than 200 bilingual legal dictionaries as well (bibliography, 45 pages). The final, maybe provocative, conclusion is that most legal dictionaries are of dubious quality and therefore not useful to professional translators.