The First International Workshop on
Comparative Evaluation in Requirements Engineering
Monterey Bay, California, USA
September 8, 2003
Held in conjunction with the
11th IEEE International Conference on Requirements Engineering (RE03).
Call for papers
Instructions for authors
Program & Schedule
Call for Papers
COMPARATIVE EVALUATION IN REQUIREMENTS ENGINEERING
September 8, 2003 Monterey Bay, California, USA
co-located with RE'03
IEEE International Requirements Engineering Conference
The need for an assessment of the progress made in RE research is becoming increasingly felt across the RE community. A number of requirements and specification exemplars have appeared along the years, e.g. the meeting scheduler, the London ambulance computer aided dispatch system, the light control system. These exemplars have been useful for illustrating new RE tools, techniques and methods, and for identifying potential lines of research. Unfortunately, the commonly used exemplars in RE all lack well-defined evaluation criteria, so different approaches cannot be compared directly. However, with the development of performance measures for these exemplars, it becomes possible to benchmark different RE technologies.
Although RE processes are extremely rich and varied, it is possible to identify areas that are sufficiently understood to allow the definition of benchmarks. The utility of such benchmarks for both research and industry has been clearly demonstrated by analogous efforts in other fields (e.g. TREC in text retrieval or Robot Soccer in robotics). By their very nature, successful benchmarks need a community effort to be defined and established. In seeking to define an agreed benchmark, research communities often experience a great leap forward, both in terms of collaboration and consensus among researchers, and in terms of technical results. This workshop seeks to spark a community initiative in this direction.
So, how should we assess progress in Requirements Engineering research? As a young, multi-disciplinary field, we still lack any broad consensus on appropriate research methodology and evaluation criteria. And yet we need to do comparative evaluation of our research efforts, if we wish to develop and mature as a scientific discipline. Such comparisons are also a crucial component of technology transfer. This workshop will examine the research methods we currently use in RE, and will investigate how we might improve our ability to evaluate and compare our research results.
CERE will be a discussion-oriented workshop to promote interaction and exchange of ideas among participants. In the first half of the workshop, there will be presentations and discussion of submitted papers. During the afternoon, there will be a plenary session to synthesize the ideas and examine the way forward for comparative evaluation. An informal committee will be formed to continue the work after the workshop. Results will be used to set up a TREC-style competition in 2004.
Paper of two types can be submitted to the workshop, short position papers and full technical papers. Both types of papers should tackle one of the topics or questions from the Themes section above.