CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019)
March 14, 2019 @ 9:00-17:00
What would be the equivalent of a re-usable TREC test collection for the interactive information retrieval (IIR) community? The aim of the BIIRRR 2019 workshop is to answer this question by continuing existing community-driven efforts to design and implement a platform for the collection, organization, maintenance, and sharing of resources for IIR experimentation.
To achieve this the BIIRRR 2019 workshop will provide an interactive forum to address relevant questions around re-use of IIR materials. Moving beyond the traditional re-use paradigm of shared tasks, this workshop will explore IIR re-use through sharing of methodologies, research designs, and resources.
In a survey we ran before the 2018 BIIRRR workshop, participants indicated that much of IIR is specific to an individual study and cannot be reused. One of the challenges in reuse is the different origins of terminology, design and methods. There are probably several aspects of IIR studies that researchers (re)use with the assumption that they are obvious aspects that everyone is aware of and understands in the same way. Yet, there are different perspectives and disciplinary backgrounds that gave shape to how individual researchers think about them, leading to a perception that reuse in IIR is difficult.
At the BIIRRR 2018 workshop a number of high-level focus areas were identified, that will be central in 2019 edition:
- Terminology: What terminology did you use to describe the different components of the study? Why did you choose this terminology? How did you develop this terminology?
- Research design: What overarching and specific methodologies did you employ in the study? How did you decide which methodologies to employ? Examples include qualitative/quantitative/mixed methods, theory/practice/design, distant/close reading, big data/small data, log studies, eye tracking, A/B testing, and simulated work tasks.
- Methodology: What research design(s) did you use? Which (aspects) of these have the potential to be re-used? To capture the variation within the IIR field, research designs should be interpreted broadly.
- Resources: What previously created materials did you re-use? This can cover all aspects, such as research designs, software, interfaces, data, scales, and specific survey questions. How did you decide what to re-use? How did you discover the materials that you re-used? Which problems did you encounter searching for them?
- Reporting: What aspects of your study could be re-used and how have you documented and represented them to enable re-use? What aspects were fully documented in your publication? What aspects do you feel should be documented outside the main publication?
Paper Contributions
We invite original contributions in the form of experience papers, in which authors analyze one of their previously published (or in-print) IIR studies in the context of the five topics listed above. Authors should not focus their contribution on the outcomes of their described studies, but on their thoughts and considerations in designing, setting up and adapting their studies as well as on efforts to connect their IIR study to previous work by others and aspects that make reuse of earlier work difficult.
These experiences are rarely made explicit and the focus of discussion, but are necessary to further the discussion on reuse and sharing in IIR. Any details you think might be relevant for others to understand the choices made are good to have in the paper.
The paper contributions will be presented in themed sessions in the morning. The afternoon will then pick up these ideas and discuss them in more depth in breakout groups based around the five themes. The workshop will conclude with a reporting and discussion session as well as planning future activities.
ll submissions should be in English and should not have been published or submitted for publication elsewhere. Papers should be between 4 and 8 pages formatted in the ACM Proceedings Style and submitted via EasyChair (https://easychair.org/conferences/?conf=biirrr2019). Submissions will be published in the workshop proceedings.
Timetable
- Submission deadline: January 18, 2019 January 25, 2019
- Notification of acceptance: February 4, 2019
- Camera-ready deadline: February 18, 2019
- Workshop: March 14, 2019
Registration for BIIRR 2019 Workshop goes via CHIIR 2019: http://sigir.org/chiir2019
Contact the organizers: biirrr@googlegroups.com