KG-EmpiRE – A Knowledge Graph of Empirical Research in Requirements Engineering
“A Sustainable Literature Review on its State and Evolution”

GitHub Repository

Empirical research in Requirements Engineering (RE) is a constantly evolving topic, with an ever-growing number of publications applying it. In the last two decades, several researchers provided snapshots of the “current” state and evolution of empirical research in RE through literature reviews. However, these literature reviews were not sustainable, as none built on or updated previous works, resulting in overlap and redundancy. The underlying issue is the unavailability of the extracted and analyzed data from earlier works. We address this issue by developing, publishing, maintaining, and analyzing KG-EmpiRE: A Knowledge Graph (KG) of Empirical research in RE.

Our long-term goal is to constantly maintain KG-EmpiRE with the research community to synthesize a comprehensive, up-to-date, and long-term available overview of the state and evolution of empirical research in RE. In this way, we want to enable a replicable, (re-)usable, and thus sustainable literature review on the state and evolution of empirical research in RE to ensure the quality, reliability, and timeliness of the research results.

KG-EmpiRE is a manually curated KG that is maintained in the Open Research Knowledge Graph (ORKG) as part of the ORKG Observatory on Empirical Research in Software Engineering, making all its data openly accessible and long-term available according to the FAIR data principles. Currently, KG-EmpiRE contains data on the six themes research paradigm, research design, research method, data collection, data analysis, and bibliographic metadata extracted from (so far) 680 papers published in the IEEE International Requirements Engineering Conference (1994-2022) and described with roughly 45,000 triples, 64,000 resources, and 27,000 literals. Since its first release based on 199 papers (2014-2022), KG-EmpiRE and its analysis have been updated twice. We published the first update based on 570 papers (2000-2022) at the 17th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement 2023, where we received the best paper award. The second update is ongoing and covers 680 papers (1994-2022) so far. The goal for the second update is to cover all 766 research track papers from the IEEE International Requirements Engineering Conference (1993-2024).

Besides KG-EmpiRE, we provide its analysis with answers to 16 out of 77 competency questions regarding empirical research in RE that we derived from the vision of Sjøberg et al. on the role of empirical methods in software engineering, including RE, for 2020-2025. While the number of competency questions answered reflects the coverage of the curated topic in KG-EmpiRE, the answers to competency questions provide insights into the state and evolution of empirical research in RE. For each competency question answered, we provide all details of the analysis with its data, visualizations, and answers in our GitHub repository. This repository contains all (supplementary) materials with detailed instructions for replication and (re-)use locally and via executable environments.

KG-EmpiRE and its analysis demonstrate that data from a literature review can already be prepared during data extraction in such a way that it is FAIR, openly accessible, and long-term available for the research community. In this way, researchers can build on and update the data ideally collaboratively, enabling a sustainable literature review for comprehensive, up-to-date, and long-term available overview state and evolution of empirical research in RE.

An intermediate state of second update KG-EmpiRE and its analysis was reviewed on the Artifact Evaluation track of the 32nd IEEE International Requirements Engineering Conference 2024 and received the Badge – Available, the Badge – Reusable, and the best artifact award.

Number of papers that provide at least one URL to their data per year