Accelerating LLM Inference with Speculative Decoding for AI Agent Applications

linkedin · HR · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Recruiting AI serving

LinkedIn needed to reduce Hiring Assistant latency for long structured recruiter-facing outputs without changing answer quality.

Constraints:

Thousands of input tokens
Long structured outputs
No quality degradation acceptable

Architecture

LinkedIn optimized Hiring Assistant's LLM serving path by adding n-gram speculative decoding to its vLLM stack. The assistant ingests job descriptions and candidate profiles, generates long rubric-style match analyses, and uses prompt-history n-grams to draft candidate token chunks. The target model verifies drafted tokens in parallel, accepting matching tokens and falling back to direct generation on mismatch, so the final output remains identical to the base model. This fits Hiring Assistant because its structured outputs reuse schema labels, skills, titles, tools, and evidence from the prompt.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  hiring-assistant[Hiring Assistant] --> structured-prompt[Structured prompt]
  hiring-assistant --> vllm-serving-stack[vLLM serving stack]
  structured-prompt --> n-gram-matcher[N-gram matcher]
  vllm-serving-stack --> n-gram-matcher
  n-gram-matcher --> target-model-verifier[Target model verifier]
  target-model-verifier --> fallback-generation[Fallback generation]
  speculation-parameters[Speculation parameters] --> vllm-serving-stack
  speculation-parameters --> n-gram-matcher
  hiring-assistant --> internal-evaluation-pipelines[Internal evaluation pipelines]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Hiring Assistant`	Recruiter-facing AI agent that analyzes jobs and candidate profiles.	`Structured prompt`, `vLLM serving stack`, `Internal evaluation pipelines`
`Structured prompt`	Long input containing job descriptions, candidate profiles, schema, and reusable phrases.	`Hiring Assistant`, `N-gram matcher`
`vLLM serving stack`	LLM serving layer where n-gram speculative decoding is enabled.	`Hiring Assistant`, `N-gram matcher`, `Target model verifier`
`N-gram matcher`	Drafts future tokens by looking up patterns in prompt history.	`Structured prompt`, `Target model verifier`
`Target model verifier`	Accepts or rejects drafted tokens using the base model probabilities.	`N-gram matcher`, `Fallback generation`
`Fallback generation`	Resumes ordinary generation from the last verified position after a mismatch.	`Target model verifier`
`Speculation parameters`	num_speculative_tokens, prompt_lookup_max, and prompt_lookup_min tuning knobs.	`vLLM serving stack`, `N-gram matcher`
`Internal evaluation pipelines`	Quality validation layer that checks no degradation from the serving optimization.	`Hiring Assistant`

Data flow

Hiring Assistant receives a job and candidate profile, builds a long structured prompt, and sends it through vLLM with n-gram speculation enabled. The n-gram matcher drafts tokens from prompt-history patterns, the target model verifies those drafts in parallel, accepted tokens advance generation, mismatches are discarded, and internal evaluation pipelines verify that latency/throughput gains do not change output quality.

3 supporting quotes

“Given a job and a profile, Hiring Assistant classifies the strength of the match (e.g. skills, seniority, domain experience) and explains its reasoning with grounded evidence.”article, implied

“Speculative decoding changes this dynamic in the prefill and generation stages by drafting multiple tokens ahead and verifying them in parallel.”article, stated

“If the guesses are wrong, the system falls back gracefully — meaning it discards the incorrect draft tokens and resumes generation directly from the last verified position”article, stated

Why this architecture

LinkedIn chose n-gram speculation because Hiring Assistant's outputs were long, structured, repetitive, and high-overlap with the prompt. That let LinkedIn improve throughput and latency without an auxiliary draft model, preserving traceability and policy compliance through lossless target-model verification.

“For our use case—long, structured outputs with recurring I/O schema—n‑gram speculation was the perfect fit.”article, stated

“N‑gram speculation offers operational simplicity at scale by delivering performance gains without introducing a second model”article, stated

Models used: LLMs served through vLLM

Scale

Hiring Assistant routinely processes thousands of input tokens and generates 1,000+ tokens per request.
“routinely ingests thousands of input tokens (such as job descriptions and candidate profiles) and generates a 1,000+ tokens of structured analysis.”article, stated
The serving change produced nearly 4x throughput and 66% lower P90 latency at the same QPS/SLA ceiling.
“We observed nearly 4× higher throughput at the same QPS and SLA ceiling, along with an average 66% reduction in P90 end-to-end latency”article, stated

Alternatives considered (roads not taken)

Draft-model speculation

Rejected because: A second model could help less repetitive text but added orchestration, cost, and latency-tail complexity.

“A smaller “draft” model proposes tokens, and the main model verifies them. This can accelerate less repetitive text but adds complexity because you’re serving two models.”article, stated

Use n-gram speculation for highly variable creative outputs

Rejected because: The article warns n-gram gains are smaller when text is less structured and less repetitive.

“On the other hand, if your text is highly variable, creative, or less structured, n‑gram speculation may deliver smaller gains. In those cases, a draft-model approach might be more suitable.”article, stated

Speed up serving without target-model verification

Rejected because: Recruiter-facing explanations needed trust, traceability, and identical base-model output distribution.

“lossless verification guarantees that the final text is identical to the base model’s output, aligning with Hiring Assistant standards for traceability and policy compliance.”article, stated

Patterns

`latency-optimization` (serving)

N-gram speculation speeds up repetitive structured generation while preserving the base model output.

“Speculative decoding changes this dynamic in the prefill and generation stages by drafting multiple tokens ahead and verifying them in parallel.”article, stated

`eval-driven-development` (evaluation)

LinkedIn verified that throughput and latency gains did not degrade output quality.

“We observed nearly 4× higher throughput at the same QPS and SLA ceiling, along with an average 66% reduction in P90 end-to-end latency—all without any quality degradation, as verified by our internal evaluation pipelines.”article, stated

Tradeoffs

Whether to use n-gram speculation or a draft model.

Chose n-gram speculative decoding over serving an auxiliary draft model.

Rationale: The source says n-gram speculation avoided extra latency tail risks, costs, and orchestration complexity.

“N‑gram speculation offers operational simplicity at scale by delivering performance gains without introducing a second model, avoiding additional latency tail risks, infrastructure costs, and orchestration complexity—critical for global deployment.”article, stated

How to accelerate long structured recruiter-assistant generations.

Chose n-gram speculative decoding over draft-model speculation.

Rationale: The workload's prompt/output repetition made n-gram lookup cheap and effective without the operational burden of a second served model.

“because the workload for the agent exhibits several unique characteristics. Hiring Assistant produces structured outputs in the form of rubric‑style summaries”article, stated

How much speculation to attempt per verification step.

Chose tunable num_speculative_tokens and prompt lookup bounds over fixed speculation with no mismatch-cost controls.

Rationale: Larger drafts can speed generation but waste compute after mismatches, so the serving path exposes knobs to trade speed against wasted attempts.

“Increasing this value can lead to significant speed-ups, as more tokens are accepted in a single step. However, the trade-off lies in the risk of mismatches.”article, stated

Gotchas (what bit them)

N-gram speculative decoding does not generalize equally to every generation workload.

Root cause: The optimization depends on repeated phrases and predictable structure in the prompt/output pair.

Resolution: LinkedIn positioned n-gram speculation for structured outputs and noted draft-model speculation for less repetitive text.

Evaluation

Internal evaluation pipelines checked quality while serving metrics measured latency and throughput.

Metrics: throughput · P90 latency · quality

Outcomes

Nearly four times higher throughput and lower P90 latency at the same QPS/SLA ceiling.

The serving change was aligned to a production recruiter assistant workload.