Search papers, labs, and topics across Lattice.
This study investigates the challenges associated with Competency Question (CQ) verification in ontology evaluation, highlighting the time-consuming and error-prone nature of the process due to linguistic ambiguities and misalignments with formal constructs. By employing an LLM assistant to support 19 participants in verifying 20 tasks, the research identifies key factors that complicate CQ interpretation and proposes solutions to enhance user performance. The findings underscore the importance of refining CQs prior to publication to mitigate ambiguity and complexity, ultimately improving the ontology engineering workflow.
Ambiguous Competency Questions can derail ontology verification, but an LLM assistant can significantly enhance user performance in this critical process.
Competency Questions (CQs) are the central component of CQ-verification, an established process in which an ontology is evaluated against a set of natural language questions to determine whether the intended purpose of the ontology has been properly modelled. However, CQ-verification is often time-consuming and error-prone, as it requires careful interpretation of linguistic nuances and precise alignment with formal ontology constructs. Ambiguities and complexity in CQs can further complicate this process, leading to inconsistent modelling decisions and verification outcomes. In this paper, we investigate what makes a CQ challenging and possible solutions to enhance the users' performance in the CQ-verification process. We experimented with the data of 19 participants who performed CQ-verification on 20 tasks using an LLM assistant to support ontology evaluation. The results show the necessity of a tool to refine CQs before publishing them to avoid ambiguity or excessive complexity in later phases of the ontology engineering process.
