Human-in-the-Loop Evaluation of LLM-Assisted Requirements Engineering in Regulated Enterprises
The Business Systems Analyst (BSA) leads process and technology improvements, facilitates interaction and collaboration between business and technology teams, balances business and technology needs, and translates business problems and requirements into documented system requirements. Projects and businesses need quality to be successful. Common symptoms include unclear and vague requirements, stakeholder confusion, outdated requirement documents, scope creep, and missing requirements traceability. These problems can put a project on the fast track and cost time and money to rework and refactor, leading to project problems. This shift toward automation and AI within the BSA is reshaping the landscape and requires reaffirming and validating new procedures and processes. This research seeks to build and evaluate a method that, given raw notes from a JAD session, generates a structured Business Requirements Document (BRD) and corresponding user stories using prompt engineering and a large language model (LLM). Ambiguity, traceability, completeness, and the time needed to document the LLMs’ work and human evaluation will serve as metrics and criteria for evaluating the quality of the LLMs’ outputs against the human-generated baseline. Experimental results have not yet been acquired. The study will compare the time spent by the analyst, the LLM, and the LLM with the analyst to complete documentation, as well as the number of ambiguities, completeness, and traceability observed during the documentation process. This evaluation method is a proposed step-by-step approach for evaluating AI-supported Requirements Engineering in a regulated context. In a future empirical comparison and human review, the reflections on quality, efficiency, and performance improvements will be based on the proposed empirical comparison.