Building and Scaling AI-Powered Password Detection in Production

github · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Application security

GitHub needed to detect generic passwords in code without burying security teams in false positives, a task that regex-based secret scanning handles poorly because user-created passwords lack strict provider-minted formats.

Constraints:

generic passwords have nuanced, varied structures that do not fit reliable regular expressions
security alerts must be precise enough to avoid alert fatigue while preserving recall
the model must handle unconventional file types and repository structures seen in customer code
secret scanning covers both incoming pushes and full Git history across all branches
LLM capacity has to be shared across workloads whose traffic patterns differ

Architecture

GitHub built Copilot secret scanning as an LLM-backed classifier inside GitHub Secret Protection. Candidate file context, source location, and vulnerability instructions are packed into a prompt with a strict JSON output contract. The system began with GPT-3.5-Turbo and few-shot prompting, then added diverse offline evaluation, private-preview feedback, GPT-4-generated synthetic cases, model experiments, GPT-4 confirmation, MetaReflection prompts, resource filters, tokenization improvements, workload-aware request management, and mirror testing against public-preview repositories before general availability.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  github-secret-protection[GitHub Secret Protection] --> secret-scanning-workload[Secret scanning workload]
  secret-scanning-workload --> content-filters[Content filters]
  content-filters --> tokenizer[Tokenizer]
  tokenizer --> prompt-builder[Prompt builder]
  prompt-builder --> llm-scanner[LLM scanner]
  llm-scanner --> confirming-scanner[Confirming scanner]
  llm-scanner --> structured-output-parser[Structured output parser]
  confirming-scanner --> structured-output-parser
  structured-output-parser --> secret-alert-pipeline[Secret alert pipeline]
  offline-evaluation-framework[Offline evaluation framework] --> prompt-builder
  synthetic-test-generator[Synthetic test generator] --> offline-evaluation-framework
  offline-evaluation-framework --> mirror-testing-framework[Mirror testing framework]
  secret-alert-pipeline --> mirror-testing-framework
  secret-scanning-workload --> workload-aware-request-manager[Workload-aware request manager]
  workload-aware-request-manager --> llm-capacity-pool[LLM capacity pool]
  llm-capacity-pool --> llm-scanner

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`GitHub Secret Protection`	Security product surface that owns secret scanning alerts across repositories.	`Secret scanning workload`
`Secret scanning workload`	Scans incoming Git pushes and entire Git history across all branches for possible generic passwords.	`GitHub Secret Protection`, `Content filters`, `Workload-aware request manager`
`Content filters`	Exclude file classes unlikely to contain credentials, such as media files and test/mock/spec paths.	`Secret scanning workload`, `Tokenizer`
`Tokenizer`	Prepares file portions for LLM scanning while retaining memory of previous tokenizations.	`Content filters`, `Prompt builder`
`Prompt builder`	Creates the LLM request with vulnerability description, source code location, file contents, examples, reasoning instructions, and JSON output contract.	`Tokenizer`, `LLM scanner`
`LLM scanner`	GPT model layer used to classify potential generic-password findings and return structured detection output.	`Prompt builder`, `Structured output parser`, `Confirming scanner`
`Confirming scanner`	Larger-model validation path used experimentally to improve precision on GPT-3.5-Turbo candidates.	`LLM scanner`, `Structured output parser`
`Structured output parser`	Automated processing layer that consumes the model's strict JSON response.	`LLM scanner`, `Secret alert pipeline`
`Secret alert pipeline`	Creates or suppresses actionable alerts for security teams and developers.	`Structured output parser`, `Mirror testing framework`
`Offline evaluation framework`	Diverse positive and negative test-bed measuring precision and recall across prompt and model changes.	`Prompt builder`, `Synthetic test generator`, `Mirror testing framework`
`Synthetic test generator`	GPT-4-based generator for additional test cases derived from existing open-source secret scanning alerts.	`Offline evaluation framework`
`Mirror testing framework`	Runs prompt and filtering changes on preview repositories to assess alert and false-positive changes without user impact.	`Offline evaluation framework`, `Secret alert pipeline`
`Workload-aware request manager`	Allocates LLM capacity across incoming-commit and full-history workloads with workload-specific ranges and fair-priority sharing.	`Secret scanning workload`, `LLM capacity pool`
`LLM capacity pool`	Shared model-serving capacity used by secret scanning and reused by other GitHub security products.	`Workload-aware request manager`, `LLM scanner`

Data flow

Secret scanning identifies code changes or history content that may contain generic passwords. Content filtering removes classes of files unlikely to contain credentials, tokenization prepares file context, and a prompt sends vulnerability guidance, source location, and file contents to an LLM with a strict JSON response specification. Evaluation and model experiments refine prompts and model choice offline. During preview, mirror testing reruns prompt and filtering changes against participating repositories to measure alert-volume and false-positive changes without affecting users. A workload-aware request manager allocates LLM capacity across incoming-commit and full-history workloads.

3 supporting quotes

“At the core of Copilot secret scanning lies a request to a large language model (LLM), expressed through an LLM prompt consisting of:”article, stated

“The source code location and contents of the file where we believe the vulnerability may exist. * A strict JSON format specification for the model output, to allow for automated processing. Our first iteration of the prompt used the few-shot prompting”article, stated

“We implemented a mirror testing framework that ran our prompt and filtering changes against a subset of repositories that participated in our public preview.”article, stated

Why this architecture

GitHub used an LLM because generic passwords are contextual: unlike provider-minted tokens, they cannot be reliably captured with strict regexes. The architecture keeps the model behind a structured JSON contract so downstream alerting remains automated, and it surrounds the model with evaluation, synthetic cases, preview feedback, mirror testing, and capacity controls because security detection needs precision, recall, and scale. Workload-aware capacity management was necessary because incoming pushes and full-history scans create different traffic patterns and static per-workload rate limits wasted capacity.

4 supporting quotes

“Historically, our detection approach has relied on regular expressions, which is an effective method for identifying secrets with strict, provider-minted formats. However, this method struggles with the nuanced and varied structures of generic passwords”article, stated

“A strict JSON format specification for the model output, to allow for automated processing. Our first iteration of the prompt used the few-shot prompting technique, which provides the LLM with example inputs and outputs to demonstrate how to perform the”article, stated

“This improved offline evaluation and gave us the breadth needed to measure both precision and recall. Precision is the ability to find secrets more accurately, with concerns to the false positive rate, while recall is the ability to find secrets more”article, stated

“assigning specific rate limits to individual workloads (such as scanning incoming Git commits or scanning the full history) was suboptimal.”article, stated

Models used: GPT-3.5-Turbo · GPT-4 · GPT-4-Turbo · GPT-4o-mini

Scale

GitHub Secret Protection protects millions of repositories.
“Copilot secret scanning is a feature of GitHub Secret Protection, which protects millions of repositories on GitHub”article, stated
Secret scanning detects hundreds of pattern types.
“protects millions of repositories on GitHub by detecting hundreds of pattern types through our partner program.”article, stated
Secret scanning scans incoming Git pushes and entire Git history on all branches.
“Secret scanning not only scans incoming Git pushes, but also your entire Git history on all branches.”article, stated
The system reduced false positives by 94% in some organizations during mirror testing.
“In some cases, we saw a 94% reduction in false positives across organizations!”article, stated
Copilot secret scanning detects passwords on nearly 35% of all GitHub Secret Protection repositories.
“Copilot secret scanning is now detecting passwords on nearly 35% of all GitHub Secret Protection repositories.”article, stated
GitHub Secret Protection became available to purchase starting April 1, 2025.
“GitHub Secret Protection is available to purchase starting April 1, 2025.”article, stated

Alternatives considered (roads not taken)

Regular expressions for generic-password detection

Rejected because: Regex works for strict provider-minted formats but creates excessive noise for varied user-created passwords.

“However, this method struggles with the nuanced and varied structures of generic passwords, often generating excessive noise for security teams and developers.”article, stated

Rely only on manually curated offline test cases

Rejected because: The private preview exposed unconventional customer file types and structures that the initial evaluation set did not cover.

“While it worked reasonably well at identifying credentials in our offline evaluation, it would fail spectacularly in some customer repositories.”article, stated

Voting by asking the same model many times

Rejected because: Voting made responses more deterministic but did not materially improve precision.

“We tried voting (asking the model the same question many times), which allowed for more deterministic responses but had no material impact on our precision.”article, stated

Always use GPT-4 as a confirming scanner

Rejected because: The confirming scan improved precision without reducing recall, but it cost more resources.

“This helped improve precision without reducing our recall, but was also more resource intensive. We also tried a few different prompting strategies, such as Fill-in-the-Middle, Zero-Shot, and Chain-of-Thought. We ended up collaborating with our colleagues at”article, stated

Static rate limits per scanning workload

Rejected because: Different workloads peak at different times, so static limits left capacity unused while one workload hit its limit.

“it was easy to land in a situation where an individual workload could run into rate limits within its operational context, leaving additional resources available elsewhere unused.”article, stated

Patterns

`structured-output-contract` (interface)

The LLM prompt requires strict JSON output so detections can flow into automated downstream processing.

“A strict JSON format specification for the model output, to allow for automated processing.”article, stated

`eval-driven-development` (quality)

Prompt, model, and filtering changes are guided by offline evaluation, diverse private-preview cases, precision/recall measurement, and mirror testing.

“we developed a basic offline evaluation framework, including manually curated test cases with both positive and negative findings”article, stated

“This improved offline evaluation and gave us the breadth needed to measure both precision and recall.”article, stated

`synthetic-data-generation` (evaluation)

GitHub used GPT-4 to create additional evaluation cases from existing open-source secret scanning alerts.

“used GPT-4 to create our own test cases based on learnings from existing secret scanning alerts in open source repositories.”article, stated

`model-routing` (model-selection)

The team experimented with GPT-3.5-Turbo for scalable detection, GPT-4 as a confirming scanner, and newer GPT-4 variants for cost/performance tradeoffs.

“We also tried using a larger model (GPT-4) trained on a larger set of parameters as a confirming scanner, to validate the accuracy of candidates found by GPT-3.5-Turbo.”article, stated

`cost-aware-routing` (operations)

A workload-aware request manager shares LLM capacity across scanning workloads instead of statically reserving capacity that may sit idle.

“the most impactful change came from creating a workload-aware request management system that allowed us to maximize and equitably share LLM capacity”article, stated

`canary-rollout` (release)

Mirror testing ran new prompts and filters on preview repositories to measure real alert impact before general availability without affecting users.

“Rescanning these repositories with our latest improvements allowed us to assess the change in real alert volumes and false positive resolutions, without impacting users.”article, stated

Tradeoffs

Detection method

Chose LLM contextual analysis over regex-only generic-password detection.

Rationale: Generic passwords need context about usage and location because their structures are too varied for precise regular expressions.

“using AI to analyze context—such as the usage and location of a potential secret—to limit noise and deliver relevant alerts”article, stated

Initial model choice

Chose GPT-3.5-Turbo with few-shot prompting over starting with the largest model everywhere.

Rationale: The team wanted a resource-effective model that could run detections at scale before adding more expensive confirmation paths.

“We wanted a resource-effective model to run the detections at scale and landed on GPT-3.5-Turbo.”article, stated

Evaluation data strategy

Chose real customer feedback and synthetic GPT-4 test cases over only manually curated pre-deployment cases.

Rationale: Preview failures showed the initial test bed did not cover the diversity of production repositories.

“First, we added reports from private preview participants to increase the diversity of our test cases.”article, stated

“used GPT-4 to create our own test cases based on learnings from existing secret scanning alerts in open source repositories.”article, stated

Capacity allocation

Chose workload-aware shared limits over specific static rate limits for each workload.

Rationale: Static limits can strand capacity; workload-aware ranges prevent overload while allowing unused capacity to be shared.

“an algorithm that allows us to set a range of limits for each workload, preventing the workload from completely overwhelming the LLM, while allowing it to tap into resources from other workloads going unused at the moment.”article, stated

Gotchas (what bit them)

Offline test cases missed unconventional customer repositories.

Root cause: The initial evaluation cases reflected conventional languages and structures rather than the diversity of real repositories.

Resolution: Add private-preview reports, visual deviation analysis, and synthetic GPT-4-generated cases based on real alert learnings.

“The model had difficulty interpreting file types and structures not typically seen in the conventional coding languages and patterns that LLMs train on.”article, stated

Precision improvements could trade off against recall or resource use.

Root cause: Larger models and confirmation scans improve precision but consume more LLM resources, while MetaReflection improved precision with a small recall penalty.

Resolution: Measure both precision and recall, experiment across model and prompt strategies, and combine techniques instead of relying on one tactic.

“This helped improve precision without reducing our recall, but was also more resource intensive.”article, stated

“experiential learnings from past trials to come up with a hybrid Chain of Thought (CoT) and few-shot prompt that improves precision with a small penalty in recall. We ultimately ended up using a combination of all these techniques and moved Copilot secret scanning into public preview, opening it”article, stated

LLM capacity grows linearly with customer adoption.

Root cause: Secret scanning processes incoming pushes plus full Git history across all branches, so each new customer adds scanning demand.

Resolution: Filter low-value content, tune context/tokenization, test more efficient models, and build workload-aware capacity management.

“With each new customer, the necessary resources increase linearly.”article, stated

Static workload rate limits wasted LLM capacity.

Root cause: Incoming commits and full-history scans follow different traffic patterns, so one workload can be limited while another has spare capacity.

Resolution: Use workload ranges and weighted fair-priority ideas so workloads cannot overwhelm the LLM but can borrow unused resources.

“different workloads have different traffic patterns—commits correlate with working hours while full history scans correlate with feature enablement events.”summary, stated

Evaluation

GitHub evaluated Copilot secret scanning through manually curated positive and negative test cases, private-preview feedback, visual deviation analysis, GPT-4-generated synthetic cases, precision/recall measurement, model and prompt experiments, and mirror testing against public-preview repositories.

Metrics: precision · recall · false positive rate · false negative rate · real alert volumes · false positive resolutions · 94% false-positive reduction in some organizations

“Precision is the ability to find secrets more accurately, with concerns to the false positive rate, while recall is the ability to find secrets more reliably, with concerns to the false negative rate.”article, stated