PatOlympics
Two PatSports (ChemAthlon and CrossLingual Retrieving) were the focus of the new interactive prototype evaluation campaign called PatOlympics. 5 referees (3 for ChemAthlon and 2 for CrossLingual Retrieving) worked with 3 participants and their systems. The objective: a new way to directly interact with 'the other side' and to concentrate discussions away from powerpoints and into real systems under development.
In the end, three medals were awarded:
- ChemAthlon: BiTeM Group (Univ. of Applied Sciences, Geneva, Switzerland)
- CrossLingual Retrieving: Spinque (Netherlands)
- Jury's favorite: BiTeM Group (Univ. of Applied Sciences, Geneva, Switzerland)
The event took place in the afternoon of second day of the IRFS, in the exhibition hall. This provided a very opened atmosphere, where participants, referees and observers mingled and enjoyed an interesting exchange of ideas. In rounds of just under 30 minutes each, the referees sat down and worked with the teams and their systems to solve a task they had prepared in advance. The systems' scores were then computed based on the number of relevant documents retrieved.
The referees were also asked to rate each system on a "user friendliness" scale between 1 and 5. In this sense, the winning system, from the BiTeM group, had a "lovely interface" according to Teresa Loughbrough, while Tony Trippe qualified it as an "interesting system, but not optimized for chemistry".
The logistics of the event were provided by the IRF and included a web interface for both participants and referees to submit relevant documents, as well as an API specification to allow participants to submit documents directly from their systems. The scoreboard, showing the final scores, is still available online at http://patolympics.ir-facility.org/
The organizers wish to thank the Referees: Teresa Loughbrough, Tony Trippe, Henk Tomas, Monika Hanelt and Pierre Buffet