Frontiers in Emerging Technology

An Open Access Peer Reviewed International Journal.
Publication Frequency- Bimonthly
Publisher Name-APEC Publisher.

ISSN Online- 2945-3437
Country of Origin-South Africa
Language- English

Human-in-the-Loop Evaluation of LLM-Assisted Requirements Engineering in Regulated Enterprises

The Business Systems Analyst (BSA) leads process and technology improvements, facilitates interaction and collaboration between business and technology teams, balances business and technology needs, and translates business problems and requirements into documented system requirements. Projects and businesses need quality to be successful. Common symptoms include unclear and vague requirements, stakeholder confusion, outdated requirement documents, scope creep, and missing requirements traceability. These problems can put a project on the fast track and cost time and money to rework and refactor, leading to project problems. This shift toward automation and AI within the BSA is reshaping the landscape and requires reaffirming and validating new procedures and processes. This research seeks to build and evaluate a method that, given raw notes from a JAD session, generates a structured Business Requirements Document (BRD) and corresponding user stories using prompt engineering and a large language model (LLM). Ambiguity, traceability, completeness, and the time needed to document the LLMs’ work and human evaluation will serve as metrics and criteria for evaluating the quality of the LLMs’ outputs against the human-generated baseline. Experimental results have not yet been acquired. The study will compare the time spent by the analyst, the LLM, and the LLM with the analyst to complete documentation, as well as the number of ambiguities, completeness, and traceability observed during the documentation process. This evaluation method is a proposed step-by-step approach for evaluating AI-supported Requirements Engineering in a regulated context. In a future empirical comparison and human review, the reflections on quality, efficiency, and performance improvements will be based on the proposed empirical comparison.

Human-in-the-Loop Evaluation of LLM-Assisted Requirements Engineering in Regulated Enterprises

Keywords

Business Systems Analyst Requirements Engineering Large Language Models Prompt Engineering Human-in-the-Loop Agile Development Traceability

Authors

Akarambaig Saudagar Senior Business Systems Analyst MA USA

Abstract

The Business Systems Analyst (BSA) leads process and technology improvements, facilitates interaction and collaboration between business and technology teams, balances business and technology needs, and translates business problems and requirements into documented system requirements. Projects and businesses need quality to be successful. Common symptoms include unclear and vague requirements, stakeholder confusion, outdated requirement documents, scope creep, and missing requirements traceability. These problems can put a project on the fast track and cost time and money to rework and refactor, leading to project problems. This shift toward automation and AI within the BSA is reshaping the landscape and requires reaffirming and validating new procedures and processes. This research seeks to build and evaluate a method that, given raw notes from a JAD session, generates a structured Business Requirements Document (BRD) and corresponding user stories using prompt engineering and a large language model (LLM). Ambiguity, traceability, completeness, and the time needed to document the LLMs’ work and human evaluation will serve as metrics and criteria for evaluating the quality of the LLMs’ outputs against the human-generated baseline. Experimental results have not yet been acquired. The study will compare the time spent by the analyst, the LLM, and the LLM with the analyst to complete documentation, as well as the number of ambiguities, completeness, and traceability observed during the documentation process. This evaluation method is a proposed step-by-step approach for evaluating AI-supported Requirements Engineering in a regulated context. In a future empirical comparison and human review, the reflections on quality, efficiency, and performance improvements will be based on the proposed empirical comparison.

Scroll to Top