Call for Papers

Overview

The need for an assessment of the progress made in RE research is becoming increasingly felt across the RE community. A number of requirements and specification exemplars have appeared along the years, e.g. the meeting scheduler, the London ambulance computer aided dispatch system, the light control system. These exemplars have been useful for illustrating new RE tools, techniques and methods, and for identifying potential lines of research. Unfortunately, the commonly used exemplars in RE all lack well-defined evaluation criteria, so different approaches cannot be compared directly. However, with the development of performance measures for these exemplars, it becomes possible to benchmark different RE technologies.

Although RE processes are extremely rich and varied, it is possible to identify areas that are sufficiently understood to allow the definition of benchmarks. The utility of such benchmarks for both research and industry has been clearly demonstrated by analogous efforts in other fields (e.g. TREC in text retrieval or Robot Soccer in robotics). By their very nature, successful benchmarks need a community effort to be defined and established. In seeking to define an agreed benchmark, research communities often experience a great leap forward, both in terms of collaboration and consensus among researchers, and in terms of technical results. This workshop seeks to spark a community initiative in this direction.

So, how should we assess progress in Requirements Engineering research? As a young, multi-disciplinary field, we still lack any broad consensus on appropriate research methodology and evaluation criteria. And yet we need to do comparative evaluation of our research efforts, if we wish to develop and mature as a scientific discipline. Such comparisons are also a crucial component of technology transfer. This workshop will examine the research methods we currently use in RE, and will investigate how we might improve our ability to evaluate and compare our research results.

Themes

• Research method and research validation in RE:
  • How do we choose our research goals?
  • How do we evaluate success?
  • How do we measure the impact/importance of a research program?
  • Should we be more explicit about our research methods?
• The role of comparative evaluation in RE:
  • Establishing the necessary consensus on how to compare research results
  • Strengths and weaknesses of various comparative evaluation approaches
  • Experience of these evaluation approaches in other fields
• Determining which sub-areas in RE are ready for comparative evaluation.
  • Identifying task samples and evaluation criteria
  • Proposing potential benchmarks for specific RE activities.
• Reporting on the results of empirical studies and comparative evaluation of RE techniques, methods and tools.

Format

CERE will be a discussion-oriented workshop to promote interaction and exchange of ideas among participants. In the first half of the workshop, there will be presentations and discussion of submitted papers. During the afternoon, there will be a plenary session to synthesize the ideas and examine the way forward for comparative evaluation. An informal committee will be formed to continue the work after the workshop. Results will be used to set up a TREC-style competition in 2004.

Submissions

Paper of two types can be submitted to the workshop, short position papers and full technical papers. Both types of papers should tackle one of the topics or questions from the Themes section above.

• Position papers (max 4 pages)
  Short papers, stating the position of the author(s) on any of the topics within the scope of the workshop. For example, positions papers could describe experience with a particular research evaluation method, or could propose an area of RE that is ripe for benchmarking. Position papers will be evaluated based on their potential for generating discussion, and on the originality of the positions expressed.
• Technical papers (max 8 pages)
  Full papers either describing experience of comparative evaluation, or report on the results of such evaluation. For example, a full paper might describe how a comparative evaluation of RE techniques was performed in practice, either by research labs or in industrial settings; or it may present the results of the actual performance of RE tools, methods or processes, in lab-based experiments or in field trials.

Important dates

Organisers

Program Committee