CHIIR 2018 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2018)
Thursday, March 15 2018, 9:00-17:00
The goal of the BIIRRR 2018 workshop is to serve as a starting point for a community-driven effort to design and implement a platform for the collection, organization, maintenance, and sharing of resources for interactive information retrieval (IIR) experimentation. Examples include search tasks, experimental protocols, questionnaire designs, reporting standards, evaluation procedures, data collections, and the search interaction data produced in such experiments.
While the idea of collecting such resources in a central location is perhaps not a new one, the effort required in designing, implementing, and maintaining such a platform can only be borne by the community as a collective effort.
Please join us in a full-day workshop to brainstorm about the shape this iRepository should take. A pre-workshop survey will gather information on IIR experimental data and repository requirements.
One of the goals of the workshop is to produce a set of requirements and activities in order to develop an iRepository, which the organizers commit to publishing as a short report right after the workshop. After this immediate follow-up, we then plan further documented dissemination of the workshop outcomes but also envisage a more extensive activity, such as a Dagstuhl Seminar, which will provide the time to discuss the requirements and solutions in more detail.
Tentative program:
- Introduction of workshop purpose, goals and planned activities
- 2 keynotes:
- Nils Pharo (Oslo Metropolitan University): The importance and challenges for standardization in IIR Evaluation – Basis for an iRepository
- Luanne Freund (University of British Columbia): Previous experiences at IIR evaluation campaigns: INEX, TREC, CLEF, RepAST
- 2 break-out sessions:
- Viewpoints on standardization: which components of IIR experiments could be re-used?
- Requirements for IIR reuse: how can documentation and archiving be standardized?