Quantitative Framework for Production LLM Evaluation in Security Applications

elastic · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production LLM evaluation for security applications

Elastic needed a reproducible way to prove that changes to GenAI security features, prompts, LLM providers, and evaluators actually improved product quality for enterprise security users instead of relying on manual spot checks.

Constraints:

Architecture

Elastic built a quantitative evaluation framework around LangGraph traces and LangSmith datasets/evaluations. The framework defines test scenarios, curates gold-standard datasets with human-in-the-loop security expert review, captures AI-agent execution traces, applies LLM-as-judge rubrics, and computes weighted scores against thresholds such as an 85% accuracy cutoff. The same framework supports prompt tuning, LLM provider comparison, evaluator prompt evaluation, and product-quality regression checks.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  elastic-security-genai-features["Elastic Security GenAI features"] --> langgraph-agent-workflow["LangGraph agent workflow"]
  elastic-security-genai-features["Elastic Security GenAI features"] --> llm-connector["LLM connector"]
  elastic-security-genai-features["Elastic Security GenAI features"] --> elasticsearch-rag-context["Elasticsearch RAG context"]
  langgraph-agent-workflow["LangGraph agent workflow"] --> langsmith-traces["LangSmith traces"]
  langgraph-agent-workflow["LangGraph agent workflow"] --> elasticsearch-rag-context["Elasticsearch RAG context"]
  langgraph-agent-workflow["LangGraph agent workflow"] --> llm-connector["LLM connector"]
  test-scenarios["Test scenarios"] --> curated-test-dataset["Curated test dataset"]
  security-expert-review["Security expert review"] --> test-scenarios["Test scenarios"]
  security-expert-review["Security expert review"] --> curated-test-dataset["Curated test dataset"]
  langsmith-traces["LangSmith traces"] --> curated-test-dataset["Curated test dataset"]
  curated-test-dataset["Curated test dataset"] --> evaluation-runner["Evaluation runner"]
  langsmith-traces["LangSmith traces"] --> evaluation-runner["Evaluation runner"]
  prompt-and-provider-candidates["Prompt and provider candidates"] --> evaluation-runner["Evaluation runner"]
  evaluation-runner["Evaluation runner"] --> llm-as-judge-rubrics["LLM-as-judge rubrics"]
  llm-as-judge-rubrics["LLM-as-judge rubrics"] --> scoring-mechanism["Scoring mechanism"]
  evaluation-runner["Evaluation runner"] --> scoring-mechanism["Scoring mechanism"]
  scoring-mechanism["Scoring mechanism"] --> results-table["Results table"]
  results-table["Results table"] --> prompt-and-provider-candidates["Prompt and provider candidates"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Elastic Security GenAI features Production features such as AI Assistant, Attack Discovery, and Automatic Import that need quality evaluation. LangGraph agent workflow, LLM connector, Elasticsearch RAG context
LangGraph agent workflow Runs the AI agent workflow and structures the request-to-response execution path. Elastic Security GenAI features, LangSmith traces, Elasticsearch RAG context, LLM connector
LangSmith traces Captures LLM calls, execution graph metadata, and outputs that can be debugged or added to datasets. LangGraph agent workflow, Curated test dataset, Evaluation runner
Test scenarios Security scenarios such as malware samples, APTs, cloud attacks, and Log4j-style vulnerabilities. Curated test dataset, Security expert review
Curated test dataset Gold-standard examples used as baselines for automated evaluation. Test scenarios, Security expert review, LangSmith traces, Evaluation runner
Security expert review Human expertise used to create realistic alert sets and judge whether expected responses are good enough. Test scenarios, Curated test dataset
Evaluation runner Automated framework that runs candidate prompts, LLMs, and datasets through configurable evaluations. Curated test dataset, LangSmith traces, LLM-as-judge rubrics, Scoring mechanism, Prompt and provider candidates
LLM-as-judge rubrics Behavioral checks for hallucination, required fields, incident similarity, readability, and accuracy. Evaluation runner, Scoring mechanism
Scoring mechanism Weighted scoring and threshold logic that determines whether prompts or configurations pass. Evaluation runner, LLM-as-judge rubrics, Results table
Results table Human-readable comparison of prompts, providers, rubric items, and scores. Scoring mechanism, Prompt and provider candidates
Prompt and provider candidates Prompt versions, rubric prompts, and LLM provider choices under evaluation. Evaluation runner, Results table
Elasticsearch RAG context Retrieval context used by Elastic Security AI workflows. Elastic Security GenAI features, LangGraph agent workflow
LLM connector Provider integration required to enable AI Assistant and Attack Discovery. Elastic Security GenAI features, LangGraph agent workflow

Data flow

Security features such as AI Assistant and Attack Discovery generate responses from user requests, alert sets, RAG context, and LLM connectors. LangGraph runs the agent workflow and LangSmith captures traces of LLM calls and run metadata. Security experts and LLM outputs help create expected responses for curated test scenarios, which are stored as LangSmith datasets. Automated evaluation runs candidate prompts, model providers, and evaluator rubrics against those datasets; LLM-as-judge rubrics score desired behaviors, and the scoring mechanism produces results tables and threshold decisions.

3 supporting quotes
“we use LangGraph to design and run our AI Agent workflows behind the scenes, while LangSmith provides the tracing capabilities”article, stated
“Here are the components of the evaluation framework (which we will walk through in detail in following sections):”article, stated
“we wanted to have a threshold of accuracy, so we would drop a prompt if the accuracy was lower than 85%.”article, stated

Why this architecture

Elastic moved from manual spreadsheets to trace-backed automated evaluation because production security features needed repeatable evidence that changes improved quality. Curated datasets give each scenario a baseline, LangGraph/LangSmith provide traceability and evaluation automation, human security experts handle domain-specific judgment, and scoring thresholds turn rubric results into explicit release or prompt-selection decisions.

3 supporting quotes
“While this manual approach was effective for initial validation, it was time-intensive and did not scale well as the product evolved.”summary, stated
“This need for expert judgment meant that early evaluations relied heavily on manual review from Elastic’s security experts”article, stated
“However, this was a well-invested effort on an ongoing basis in order to know if our improvements are actually making the product better.”article, stated

Models used: LangGraph · LangSmith · LLM-as-judge evaluators · Elastic AI Assistant · Attack Discovery · LLM connectors

Scale

Alternatives considered (roads not taken)

Manual spreadsheet-based evaluation

Rejected because: It worked for early validation but was too time-intensive and hard to scale.

“they created spreadsheets filled with realistic queries that a security analyst might use in a SOC”summary, stated

Purely human-written expected outputs

Rejected because: For Attack Discovery, Elastic could run scenarios through LLMs and use human-in-the-loop review to decide whether outputs met quality criteria.

“we were able to run the scenarios through any LLM with a human-in-the-loop to decide if the result was good enough based on our criteria.”article, stated

Subjective prompt or provider selection

Rejected because: The framework compares prompts, providers, and evaluator rubrics quantitatively against datasets and score thresholds.

“Which LLM was best at a specific task?”article, stated

Evaluate only generator prompts

Rejected because: Elastic also evaluates evaluator LLMs and rubric prompts because judge quality affects the evaluation result.

“with this framework, we can evaluate an “evaluator” LLM as well as rubric prompts quantitatively as well.”article, stated

Patterns

eval-driven-development (quality)

Elastic uses curated datasets, automated evaluations, rubrics, and score thresholds to decide whether prompts and provider choices improve product quality.

“LangSmith's evaluation capabilities allowed them to build the first iteration of their internal evaluation framework”summary, stated

observability-tracing (operations)

LangGraph and LangSmith traces capture each LLM call, execution graph, and metadata for debugging and dataset construction.

“we started using LangSmith and LangGraph together, enabling us to capture traces of each LLM call.”article, stated

llm-as-judge (evaluation)

Elastic runs evaluator LLMs against rubric prompts to judge behaviors such as hallucination, incident similarity, and required-field coverage.

“This also enabled us to run automated LLM evaluations (“LLM-as-judge”)”article, stated

human-in-the-loop-gate (governance)

Security experts supply alert sets and judge whether generated expected responses are good enough before adding them to curated datasets.

“Second, determining the ideal output required genuine cybersecurity expertise, as the goal is to explain malicious attacks chronologically in a narrative style accessible to security analysts of varying experience levels.”summary, stated

model-routing (model-selection)

The framework compares LLM providers and recommends which model works best for a given security task.

“* Which LLM was best at a specific task?”article, stated

Tradeoffs

Invest in curated test datasets

Chose Time-consuming scenario and expected-response curation over Ad hoc manual checks after each change.

Rationale: Curated datasets made it possible to know whether changes improved the product across repeated evaluations.

“However, this was a well-invested effort on an ongoing basis in order to know if our improvements are actually making the product better.”article, stated

Use LLM outputs plus human review for expected responses

Chose Human-in-the-loop LLM-generated expected outputs over Only human-written reference responses.

Rationale: The team could create candidate outputs with LLMs, then rely on security expertise to decide if they met quality criteria.

“we were able to run the scenarios through any LLM with a human-in-the-loop to decide if the result was good enough based on our criteria.”article, stated

Set explicit scoring thresholds

Chose Weighted scoring with an 85% accuracy cutoff over Subjective prompt acceptance.

Rationale: Thresholds convert rubric results into an explicit drop/keep decision for prompt variants.

“In our case, we wanted to have a threshold of accuracy, so we would drop a prompt if the accuracy was lower than 85%. This is doable with your program of choice.”article, stated

Use trace-backed evaluation infrastructure

Chose LangGraph plus LangSmith traces and datasets over Disconnected logs and manual screenshots.

Rationale: Trace-backed workflows connect LLM calls, run metadata, dataset examples, and evaluation results.

“LangSmith provides the tracing capabilities as well as streamlined tools for us to create test datasets and run evaluations.”article, stated

Gotchas (what bit them)

Manual evaluation did not scale with product evolution.

Root cause: Spreadsheet tests required humans to manually input queries, record outputs, and compare responses.

Resolution: Move to LangSmith-backed automated evaluations with configurable LLM lists and input datasets.

“While this manual approach was effective for initial validation, it was time-intensive and did not scale well as the product evolved.”summary, stated

Attack Discovery evaluation needed security-domain judgment.

Root cause: The desired output is a chronological malicious-activity narrative, not a simple exact-match answer.

Resolution: Use security experts and human-in-the-loop expected-response selection for curated datasets.

“Second, determining the ideal output required genuine cybersecurity expertise, as the goal is to explain malicious attacks chronologically in a narrative style accessible to security analysts of varying experience levels.”summary, stated

Curating good examples is costly because LLM outputs vary.

Root cause: Initial scenario creation is time-consuming and output variance complicates selection of reference examples.

Resolution: Treat dataset curation as an ongoing investment needed to know whether improvements help.

“the initial effort of creating scenarios can be time-consuming, and since LLM outputs can have a high variance, some of the selection of curated examples can be difficult.”article, stated

Evaluator rubrics themselves need evaluation.

Root cause: LLM-as-judge prompts can be weak or ambiguous, so judge behavior can distort prompt comparisons.

Resolution: Quantitatively evaluate evaluator LLMs and rubric prompts, and tighten rubric wording when needed.

“Rubrics themselves can also be treated as prompts to improve on! For example, we tightened up the wording of the rubrics”article, stated

Evaluation

The project itself is an evaluation framework: scenario datasets, LangGraph/LangSmith traces, LLM-as-judge rubrics, provider comparisons, prompt comparisons, rubric-prompt evaluation, weighted scores, and an 85% accuracy threshold for prompt rejection.

Metrics: accuracy threshold · rubric item scores · prompt score · LLM provider score · hallucination rubric · required incident/user ID coverage · readability and summary accuracy

“Rubrics define the "desired behavior" of LLM outputs and contain multiple items, each responsible for checking a subset of desired behaviors. Examples include checking whether the response is written in plain language or whether it contains certain required elements.”summary, stated
“The evaluation criteria included whether the output was clear to read from a user standpoint and whether the LLM summary was accurate enough.”summary, stated

Outcomes

The case anchors scale with production enterprise users, three GenAI security features, five evaluation-framework components, datasets spanning attack scenarios such as APTs and Log4j 2021, and an 85% prompt accuracy threshold.

“With these tools, we successfully transitioned from manual to automated evaluations, significantly improving our workflow.”article, stated
“Putting it all together, you’d have an easily understandable results table — and you’d be able to see at a glance:”article, stated