Standardized Bricks,
Custom Buildings

Compared as whole architectures, production agentic systems look wildly diverse: across 991 agentic systems, no single architecture shape describes more than ~13% of any problem domain. Compared as components, the industry has already converged hard. Two-thirds of everything orchestrates tools. Nearly half is governed by evaluation. Two in five keep a human gate in the loop.

This is the corpus's central lesson, and it took a measurement mistake to find it. Our first convergence metric compared full "solution shapes" and concluded everything was diverse — an unfalsifiable answer, because composition is combinatorial. Measured at two levels, the real structure appears: teams converge on which bricks to use, and stay creative about the building. Practically, the leverage sits at the brick level — with the ~10 blocks below, their variants, and their failure modes in hand, almost any production agent architecture reads as a composition.

The textbook definitions of these patterns are not what the corpus shows. Below is what each block actually means once it is carrying production traffic — the variant decisions inside it, and the wound it usually carries.

Tool-use orchestration 64.9% · the substrate

In production this is not "function calling." It is capability governance: a registry of what agents may do, machinery for choosing among capabilities at runtime, and interfaces redesigned so a model can operate them. LinkedIn built a central skill registry — gRPC auto-registration, embedding search over skills, service-principal authorization applied transitively to agents — because hand-wiring tools into every application "was not going to scale, so we inverted it." They built it before MCP existed; when MCP arrived, the layered design absorbed it cheaply. Hex went the other direction on the same problem: their agents saw too many tools, so they consolidated tool families and added tool search, retrieving the relevant few per task instead of exposing hundreds.

The recurring wound: naive one-API-endpoint-to-one-tool mapping floods the agent with choices and repeated instructions. The convergent fix is a smaller, curated tool surface with retrieval over it — the same shape as RAG, applied to capabilities.

Eval-driven development 47.9% · the governor

The mature form has a specific life cycle that appears again and again: define human rubrics per component → convert them to LLM-driven, reference-free automated evaluation as fast as possible (because human annotation is the iteration bottleneck) → attach evals to every release gate. LinkedIn evaluated each sub-agent in isolation before stitching; Dropbox treats model, retrieval, and prompt changes as gated code changes; one team's practice — every recurring failure spotted in logs becomes a tagged dataset in Braintrust — is the pattern in miniature: problems become datasets, datasets become regression tests.

Its wound is the largest gotcha cluster in the corpus (806 systems): evaluation coverage gaps, led by subjective-quality blind spots (334). Rovio's fine-tuned art generator passed the metrics; their artist looked at it and said the gestures were "a bit creepy and a bit awkward." And a coding-agent team discovered the sharper version: they optimized SWE-bench, got better SWE-bench scores, and a worse product — grading the wrong thing is worse than not grading.

Human-in-the-loop gates 40.0% · the accountability layer

The production version is precise about what the human owns. The strongest implementations gate on irreversibility and accountability, not on volume: agents "are not able to initiate any irreversible state changes without a human," while everything reversible flows freely. A voice-agent platform drew an even harder line — sensitive-data access is decided by deterministic permission systems, never by model judgment, because a prompt is not an access-control policy. The anti-form is equally well documented: approval on every low-risk step converts automation into a slow notification system and burns scarce expert attention.

Multi-agent orchestration 33.5% · the divisible workload

The dominant shape is a supervisor that treats sub-agents as tools — invoke, receive, act. What the corpus makes unusually clear is why teams accept its cost. Chained stages multiply error: two 90%-accurate stages yield 81%. LinkedIn wrote that math on a slide and chose modularity anyway, because the alternative — one monolithic prompt — cannot be quality-engineered by a team: any few-shot example added to fix one behavior silently changes another. Decomposition is a social architecture as much as a technical one: it buys parallel workstreams and isolated evaluation at the price of compounded error, then claws the error back by engineering each stage hard.

Observability & tracing 33.2% · the flashlight

The consistent testimony: classic APM-style observability fails on agents, because non-deterministic systems produce a different flow every time — a single trace explains nothing. Replit's agent traces ran to hundreds of steps and broke their ingestion and display tooling. The convergent design is aggregation before inspection: cluster traces into behavior families, feed OTel-compatible pipelines, and serve two distinct consumers — an engineer replaying one bad run, and a team watching aggregate failure modes shift in production.

RAG, in an agentic world 30.7% · demoted and promoted

The fixed retrieve-then-generate pipeline is receding; retrieval survives as a tool the agent invokes iteratively — deciding when to search, reformulating, and searching again. That's a promotion in capability and a demotion in architecture: retrieval is now a capability inside tool orchestration rather than the system's spine. Its wounds are the second-largest gotcha cluster (784): chunking that destroys document structure (table cells separated from their rows), citation fidelity, and ranking cutoffs that silently drop the evidence the model needed.

Agent memory 18.5% · scoped, layered, deliberate

Production memory is not a vector store bolted on. The mature design is scoped and layered — working memory (task context), episodic and procedural long-term memory, and collective memory at organizational granularities (what does this recruiter do; what do all recruiters at this company do) — with three retrieval mechanisms used together: most-recent-N, summarization, and semantic search. The corpus is also blunt that big context windows are not memory: long context is slower, more expensive, and hallucination-prone, which is why teams that tried "just stuff it in the window" migrated back to engineered retrieval.

Durable execution 9.4% · small share, sharpest lesson

Below the fundamental-block threshold in raw share but philosophically important: agent work outlives requests, waits on humans, and half-fails after side effects. Teams at Block, Instacart, and Hex converged on the same split — deterministic workflow code owns state and progress; every model call and external effect runs as a retryable activity. The universal gotcha: carrying rich agent context inside workflow history bloats it until replay breaks; store artifacts externally, persist references, use lifecycle controls. Its boundary question is the cleanest in the library: if the whole job is short and safely retryable end-to-end, it is not needed — the presence of an LLM alone justifies nothing.

Across 991 systems, the same handful of forks recur. These are the arguments that appear in the writeups — not our opinions.

We clustered every documented production issue. Two results stand out before the details: evaluation and retrieval failures lead, as expected — but human adoption and workflow fit is #3, nearly tied, ahead of every infrastructure category. Systems fail because people won't or can't work with them almost as often as because the AI is wrong.

The individual stories are where the transferable engineering judgment lives. A selection worth internalizing:

Adjacent patterns blur together until the one question that splits them is in hand. These came out of building the boundary matrix across all 23 patterns — each is a test that takes seconds to apply to a design.

Every pattern page carries an evidence-backed "when not to use" section. Collected, they form the corpus's anti-hype doctrine:

• No durable workflow engine for short, idempotent requests that can retry end-to-end. Hex adopted Temporal only after long-running iterative work outgrew queues — "lifecycle complexity, not the presence of an LLM," justifies the migration.
• No human gate on every step. Encode what can be policy as policy, automate stable quality criteria, and spend human attention only where ambiguity, accountability, or consequences demand it.
• No persistent memory for one-shot tasks — and no memory at all where sensitive data lacks a clear retention benefit. Manus deliberately ships task continuity without cross-conversation memory: an intentional midpoint, not a missing feature.
• No context platform for a small, stable prompt. And never use context selection to paper over missing authorization: choosing what the model sees is different from deciding what it may see.
• No fine-tuning while the problem statement is still moving.
• No LLM where procedural code does the job — reasoning is the only thing that pays for nondeterminism.

The corpus spans the whole modern arc, and several systems narrate their own evolution through it. The composite story: 2023 was prompt-in/string-out and the first RAG chatbots; trust defenses and conversational memory arrive as soon as users can type. 2024 is the copilot era — prompt chains with intent classification, planning, and synthesis stages; modularity emerges as a team-scaling necessity. 2025 is the agent turn: chains become graphs, tools become skills, human-in-the-loop becomes an explicit architectural concept, and MCP appears mid-year to standardize what registries like LinkedIn's had built privately. 2026, in the newest entries, is consolidation: durable execution under agents, message-bus orchestration between them, context engineering as a named discipline, agent identity/authorization as a first-class security problem, and forward-deployed engineering as the delivery model for enterprise agents.

Notably stable across all four years: evaluation discipline and human gates. They were in the earliest systems and the newest ones — the constants of the field, not its fashions.

Deep understanding includes knowing where this evidence is weak.

Where everything lives, for anything deeper than this page: the 23 pattern pages in patterns/ (each with variants, tradeoffs, checklists, and citations), the boundary matrix in pattern-boundary-matrix, per-system deep dives in projects/ (searchable index) (1,780 of them, with architecture diagrams), and the analysis layer in analysis/. To use the library the way it was designed: pick the pattern pages relevant to the system being built and place them in a coding agent's context — they were written to be read by humans and agents alike.