LLM-Based Agents for User Story Quality Enhancement in Agile Development

austrian_post_group · Government · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agile requirements engineering

Austrian Post Group IT wanted to improve agile user-story quality, where stories must be understandable, complete, and valuable enough for cross-team software delivery.

Constraints:

• The fetched source is only an arXiv abstract page, so this extraction relies on the long ZenML summary for detailed architecture.
• User-story quality depends on contextual relevance, acceptance criteria, clarity, business value, and team-specific agile practice.
• The evaluation was an early study using 25 synthetic user stories rather than a full production rollout.
• LLM-generated story improvements still needed professional judgment because GPT-4 could make stories longer and less practical.

Architecture

The Autonomous LLM-based Agent System (ALAS) models requirements enhancement as a collaboration between role-specific LLM agents. A task definition initializes context, a shared knowledge base carries task information and conversation history, Product Owner and Requirements Engineer agents analyze and rewrite the story from complementary perspectives, and responses are accumulated for subsequent agents. Prompt preparation uses persona, k-shot, AI planning, fact-check-list, and chain-of-thought patterns.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  User_story_task["User story task"] --> Prompt_preparation_phase["Prompt preparation phase"]
  Prompt_preparation_phase --> Product_Owner_agent["Product Owner agent"]
  Prompt_preparation_phase --> Requirements_Engineer_agent["Requirements Engineer agent"]
  User_story_task --> Shared_knowledge_base["Shared knowledge base"]
  Product_Owner_agent --> Shared_knowledge_base
  Requirements_Engineer_agent --> Shared_knowledge_base
  Shared_knowledge_base --> Response_set["Response set"]
  Response_set --> Practitioner_evaluation["Practitioner evaluation"]

Components

Data flow

A user-story improvement task is prepared with context, expected outcome, agent profiles, and planned steps. The Product Owner agent reviews business value and product alignment, the Requirements Engineer agent reviews clarity, ambiguity, and acceptance criteria, and each response is added to the shared knowledge base. The final improved stories are evaluated by practitioners across agile teams.

Why this architecture

The system maps requirements work to expert roles rather than a generic rewrite prompt. Product and requirements agents can inspect the same story from different responsibilities, while the shared knowledge base maintains context across their collaboration.

Models used: GPT models · GPT-4 · LLM Product Owner agent · LLM Requirements Engineer agent · CrewAI

Scale

• The study was assessed by 11 participants.
  “assessed by 11 participants across six agile teams.”article, stated
• The participants came from six agile teams.
  “assessed by 11 participants across six agile teams.”article, stated
• The evaluation used 25 synthetic user stories.
  “Evaluated by 11 professionals across six agile teams using 25 synthetic user stories”summary, stated

Alternatives considered (roads not taken)

Single generic LLM assistant for story rewriting

Rejected because: The reference model decomposes work into specialized agents with distinct responsibilities.

Prompting without shared conversation state

Rejected because: The model includes a shared knowledge base so subsequent agents can use task context and earlier responses.

Rely only on raw LLM output without verification prompts

Rejected because: The preparation phase included a fact-check-list pattern to check outputs for errors or inconsistencies.

Patterns

multi-agent-orchestration (orchestration)

ALAS coordinates Product Owner and Requirements Engineer agents over a shared task and knowledge base to improve user stories.

agent-memory (context)

The shared knowledge base stores task information and conversation history so later agent turns can incorporate prior responses.

eval-driven-development (evaluation)

The system was evaluated by professionals across agile teams to determine whether the agent-improved stories actually improved quality.

Tradeoffs

Use specialized agent roles for story enhancement.

Chose Product Owner and Requirements Engineer agents over A single general-purpose user-story rewriter.

Rationale: The source describes different quality dimensions tied to product value and requirements clarity, so role specialization makes those dimensions explicit.

Use GPT-4 despite longer outputs.

Chose More comprehensive GPT-4 improvements over Shorter but potentially less complete story rewrites.

Rationale: The study found stronger improvements but also practical concerns from longer stories, making human review important.

Gotchas (what bit them)

Role-specific output can become impractical even when more comprehensive.

Root cause: GPT-4 produced longer improved stories, which raised practical usability concerns.

Resolution: Keep human oversight and practitioner evaluation in the loop to judge contextual relevance and practicality.

LLM agents need careful role and task preparation to avoid generic rewrites.

Root cause: The quality task depends on product context, business value, requirements quality, and expected outcomes.

Resolution: Use persona profiles, k-shot prompts, AI planning, fact-check-list prompts, and chain-of-thought guidance during task preparation.

Evaluation

The researchers evaluated ALAS with 11 professionals from six agile teams using 25 synthetic user stories and assessed improvements in clarity, comprehensibility, and business-value articulation.

Metrics: 11 participants · six agile teams · 25 synthetic user stories · clarity · comprehensibility · business value articulation

Outcomes

• The study found that LLM agents can improve user-story quality in agile requirements engineering.
• GPT-4 produced more comprehensive improvements but also longer stories that raised practical concerns.
• The case supports LLM agents as assistance rather than autonomous replacement for requirements judgment.

This was an early report using 11 participants, six agile teams, and 25 synthetic user stories rather than a full production deployment.