Dynamic LLM Selection and Prompt Optimization Through Automated Evaluation and User Feedback

beekeeper · Tech · 2026 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Tech

Beekeeper needed a way to keep model and prompt choices current as LLM quality, prices, and requirements changed across frontline-worker use cases.

Constraints:

• model and prompt quality changes over time
• quality, latency, and cost must be balanced
• real user feedback should influence personalization
• personalized prompts must not drift below quality standards

Architecture

Beekeeper built a two-phase LLMOps control plane for chat summarization. A scheduled baseline phase tests model/prompt pairs against synthetic and annotated data, scores them on quality, latency, and cost, sends a statistical sample to human validation, mutates promising prompts, and stores winners on a leaderboard. A feedback phase takes production user feedback, fetches ranked model/prompt pairs, creates personalized prompt variants, gates them with drift detection against the baseline, and saves safe variants for specific users or tenants. Runtime requests are routed to the current top leaderboard choices rather than a single fixed model/prompt.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  eventbridge-scheduler["eventbridge-scheduler"] --> coordinator["coordinator"]
  coordinator --> model-prompt-evaluators["model-prompt-evaluators"]
  model-prompt-evaluators --> amazon-bedrock["amazon-bedrock"]
  model-prompt-evaluators --> mechanical-turk-validation["mechanical-turk-validation"]
  model-prompt-evaluators --> leaderboard-store["leaderboard-store"]
  leaderboard-store --> request-router["request-router"]
  coordinator --> prompt-mutator["prompt-mutator"]
  prompt-mutator --> amazon-bedrock
  prompt-mutator --> drift-detector["drift-detector"]
  drift-detector --> personalized-prompt-store["personalized-prompt-store"]
  personalized-prompt-store --> request-router

Components

Data flow

A scheduler starts the synthetic rank creator, which fetches test data and sends model/prompt pairs to evaluators. Evaluators generate summaries and run static and LLM-based checks, while a sample goes to Mechanical Turk for human validation. Scores update the leaderboard, the best prompt/model pairs are mutated and re-evaluated, and the winners are used in production traffic splits. When user feedback arrives, the coordinator sends the ranked pairs and feedback to a mutator; drift detection compares the personalized variant against the baseline before saving it for that user or tenant.

Why this architecture

The design treats model/prompt choice as a continuously changing optimization problem rather than a release-time decision. Beekeeper wanted to balance quality, latency, and cost while letting small teams test new models, personalize per tenant, and avoid prompt feedback corrupting the baseline. The leaderboard gives a routing surface, statistical human review checks whether automated evaluators remain trustworthy, and drift detection keeps personalization from becoming uncontrolled prompt injection or quality regression.

Models used: Amazon Bedrock · Amazon Nova · Anthropic Claude 4 Sonnet · Meta Llama 3 · Mistral 8x7B · Mistral Large · Anthropic Claude · Qwen3 embeddings

Scale

• The baseline example starts with eight model/prompt pairs across Amazon Nova, Claude 4 Sonnet, Llama 3, and Mistral 8x7B.
  “The baseline evaluation process starts with eight pairs of prompts and associated models (Amazon Nova, Anthropic Claude 4 Sonnet, Meta Llama 3, and Mistral 8x7B).”article, stated
• One baseline pass creates 480 LLM calls before mutation.
  “In total, this creates 480 LLM calls.”article, stated
• A mutation round adds 10 new prompts and about 600 more LLM calls.
  “These two prompts are mutated using user feedback, creating 10 new prompts, which are again evaluated, creating 600 calls to the LLM (10 x 20 + 10 x 20 x 2 = 600).”article, stated
• The worked evaluation exercise averages about 8.35M input tokens, 1.62M output tokens, and $48 of cost.
  “The whole process on average requires around 8,352,000 input tokens and around 1,620,000 output tokens, costing around $48.”article, stated
• Production traffic is split across the top three ranked model/prompt pairs at 50%, 30%, and 20%.
  “Newly selected model/prompt pairs are used in production with ratios 1st: 50%, 2nd: 30%, and 3rd: 20%.”article, stated

Alternatives considered (roads not taken)

Treating model/prompt choice as a one-time manual decision

Rejected because: Models, prices, requirements, and prompt complexity change over time, so a static decision would become stale.

Letting engineers manually re-benchmark and update prompts

Rejected because: The system is designed to automatically evaluate, rank, and evolve prompts so teams can focus on the feedback-driven setup instead of manual re-benchmarking.

Applying every user feedback item directly to future prompts

Rejected because: Beekeeper first checks whether feedback is still applicable and then uses drift detection so personalization does not undermine the baseline.

Manually reviewing every generated summary

Rejected because: They use a statistically determined Mechanical Turk sample instead, giving a human check without making evaluation prohibitively expensive or slow.

Patterns

eval-driven-development (evaluation)

Model/prompt candidates are continuously tested with synthetic data and production feedback before routing decisions are made.

model-routing (model-operations)

Requests are routed to the current best model/prompt pair for the use case, based on leaderboard results.

data-quality-loop (feedback)

User feedback drives prompt mutation, then drift detection gates personalized variants before they are saved.

Tradeoffs

Validate automated evaluators with a statistical human sample rather than reviewing every output.

Chose 7% Mechanical Turk validation based on statistical sampling over manual review of every evaluation output.

Rationale: The sample catches automated-evaluator drift and double hallucination while avoiding the cost and latency of full manual review.

Use a leaderboard traffic split instead of routing all requests to the single current winner.

Chose 50/30/20 traffic across the top three model/prompt pairs over 100% production traffic to one model/prompt pair.

Rationale: The split still favors the best-ranked option while keeping alternate high-performing choices in production as the model landscape changes.

Gotchas (what bit them)

Prompt and model optimization is not a one-time setup task.

Root cause: Models, prices, prompts, and business requirements change over time.

Resolution: Beekeeper continuously evaluates candidates and updates routing from a live leaderboard.

Evaluation

Synthetic test data is used for baseline scoring and user feedback validates production quality.

Metrics: quality · latency · cost · thumbs up/down

Outcomes

• The ZenML summary reports 13-24% better response ratings when aggregated per tenant.

The source describes a constantly optimizing setup rather than a one-off experiment.