Refining Input Guardrails for Safer LLM Applications Through Chain-of-Thought Fine-Tuning

capital_one · Finance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLM input moderation and conversational assistant safety

Capital One needed input guardrails that could detect malicious, jailbreak, and prompt-injection inputs before they reached LLM-powered conversational assistants, while keeping outputs parsable and false positives low enough for user-facing applications.

Constraints:

• Financial-services deployments carry reputational and regulatory risk from harmful, biased, misleading, or unsafe outputs.
• Post-training of the main assistant model is insufficient against sophisticated and evolving attacks.
• Guardrails must distinguish malicious/jailbreak inputs from safe user requests.
• Moderation responses need to be in a requested, parsable format.
• The research emphasized small training datasets, few epochs, and reduced computational requirements.

Architecture

Capital One implemented a proxy-defense input moderation guardrail using LLM-as-a-Judge classifiers. A user utterance goes first to a judge LLM prompted with policy definitions and chain-of-thought reasoning; the guardrail returns a safety verdict and explanation to the conversational agent. The team then fine-tuned medium-sized open-source Mistral/Llama judge models with LoRA using SFT, DPO, and KTO on accepted/rejected synthetic moderation responses, improving attack detection and parsability.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["User input"] --> B["Input moderation guardrail"]
  B --> C["Judge LLM"]
  C --> D["CoT prompt"]
  C --> E["Policy definitions"]
  C --> F["Fine-tuned LoRA adapters"]
  F --> G["Training dataset"]
  G --> H["Evaluation harness"]
  H --> C
  B --> I["Conversational agent"]

Components

Data flow

A user input enters the input moderation guardrail before the main assistant. The judge LLM receives policy definitions and CoT instructions, generates a concise explanation, and then emits a safe/unsafe verdict in a parsable format. Training data is built from public malicious benchmarks and synthetically generated safe/ideal/rejected responses. SFT updates the model on desired outputs; DPO and KTO align it further using rejected responses or binary signals. The conversational agent receives the guardrail verdict and explanation before deciding how to respond.

Why this architecture

Capital One used proxy defense because the main assistant model may still be vulnerable even after alignment. LLM-as-a-Judge was chosen over simpler classifiers for its reasoning versatility, and CoT plus fine-tuning improved both detection accuracy and parsable moderation formatting under industry constraints.

Models used: Mistral 7B Instruct v2 · Mixtral 8x7B Instruct v1 · Llama2 13B Chat · Llama3 8B Instruct · LlamaGuard-2 · ProtectAI DeBERTaV3 · Meta PromptGuard

Scale

• The work was published at AAAI 2025 and won an Outstanding Paper Award.
  “published in January 2025 to the workshop of Preventing and Detecting LLM Misinformation (PDLM) at the Association for the Advancement of Artificial Intelligence (AAAI) 2025 conference, and won the Outstanding Paper Award.”article, stated
• Four medium-sized open-source LLMs from Mistral and Llama families were evaluated.
  “We focused on four medium-sized open-source LLMs from Mistral and Llama families”article, stated
• The tuning data used 400 accepted responses with three rejected responses per query.
  “we narrowed down the training data to a total of 400 accepted responses (equally distributed between malicious and safe categories) and three rejected responses for each query.”article, stated
• Fine-tuning improved F1 and ADR by more than 50% while increasing FPR by at most 1.5%.
  “increase of 50+% for F1 and ADR, compared to a maximum hike of 1.5% in FPR”article, stated

Alternatives considered (roads not taken)

Rely on the main assistant model's post-training alignment alone

Rejected because: Aligned assistant models remain susceptible to evolving sophisticated attacks, so Capital One added a proxy guardrail.

BERT-based lightweight classifier backbone

Rejected because: The study focused on LLM-as-a-Judge because it was more versatile for policy-violation reasoning.

CoT prompting without fine-tuning

Rejected because: CoT helped, but base-model F1 remained well below 80%, validating additional fine-tuning.

Patterns

safety-policy-enforcement (security)

The proxy guardrail filters unsafe user utterances before the assistant LLM receives them.

llm-as-judge (evaluation)

An LLM judge is instructed to identify policy violations and classify inputs as safe or unsafe.

prompt-chaining (prompting)

The judge first produces a concise reasoning explanation and then uses it to arrive at the final safety verdict.

fine-tuning-adaptation (model-adaptation)

Capital One used SFT, DPO, KTO, and LoRA to adapt open-source judge models to adversarial-input detection.

structured-output-contract (interface)

One research question specifically targeted whether the judge could produce moderation responses in the requested parsable format.

synthetic-data-generation (data)

The team generated ideal and rejected moderation responses synthetically to support SFT, DPO, and KTO training.

Tradeoffs

Guardrail architecture

Chose proxy-defense input guardrail over relying only on the assistant model's own refusal behavior.

Rationale: A separate firewall reduces the chance that harmful interactions reach the main assistant.

Training method

Chose SFT first, with DPO/KTO as incremental alignment over RLHF-style reward-model fitting or alignment alone.

Rationale: SFT gave the largest lift; DPO/KTO added smaller gains and require fewer/preferential data structures than RLHF.

Model adaptation footprint

Chose LoRA and small-data/few-epoch tuning over large expensive full-model training.

Rationale: The team wanted practical industry guardrails with reduced computation and minimal hyperparameter effort.

Gotchas (what bit them)

Main-model alignment is not a complete defense.

Root cause: Sophisticated adversarial attacks continue to evolve even when the primary assistant has been post-trained for safety.

Resolution: Capital One added a separate proxy-defense guardrail before the conversational agent.

Base open-source judges struggled with jailbreak and prompt-injection detection.

Root cause: Open-source LLMs had a performance gap on dynamic jailbreak and prompt-injection vulnerabilities even with many-shot examples.

Resolution: The team fine-tuned and aligned judge models with CoT moderation responses.

Better attack detection can raise false positives.

Root cause: Fine-tuning improved F1/ADR but FPR rose marginally across most models and strategies.

Resolution: Capital One judged the FPR increase acceptable because it was small relative to attack-detection gains and did not create concerning customer impact.

Evaluation

Capital One evaluated four base and fine-tuned LLMs on malicious/safe datasets using F1, recall/ADR, false positive rate, and invalid response ratio, then compared the best Llama3-DPO model against public guardrails such as LlamaGuard-2, PromptGuard, and ProtectAI DeBERTaV3.

Metrics: F1 score · Recall / attack detection ratio (ADR) · False positive rate (FPR) · Invalid response ratio · 50+% increase in F1 and ADR · Maximum FPR hike of 1.5%

Outcomes

• CoT prompting improved or matched F1/recall on three of four base LLMs and reduced invalid response ratio for most models.
• SFT produced the largest F1 and ADR gains; DPO and KTO added marginal improvements on top.
• Fine-tuning and alignment improved F1 and ADR by more than 50% with a maximum FPR increase of 1.5%.
• Invalid response ratios dropped across base and fine-tuned LLMs.
• DPO-aligned Llama3 8B outperformed public guardrail baselines in the reported comparisons.

The case is a research-to-practice guardrail study using small datasets and four medium-sized open-source models, not a disclosed production traffic rollout.