Corpus-Complete Gotcha Index

Snapshot: corpus-complete-20260711 (2026-07-11) · source commit 1737631198b184e05ba0a343d67df6fa9e71b938 Scope: 1780 approved schema-v2 extractions; 1754 deduped sources; 991 deduped agentic projects.

Corpus-complete analysis output. Counts are generated from extraction.json; citations link to deterministic report.md views. This is an input to, not the output of, Phase 4 synthesis.

Gotchas are clustered by an explicit rubric over issue, root cause, and resolution text. Counts are deduped by duplicate group. Broad clusters include one level of recurring sub-modes so large categories remain actionable.

Gotcha cluster Deduped projects Problem classes Top subclusters Common root causes Common resolutions Representative evidence
Evaluation coverage gaps
Automated tests, judges, or metrics miss important behavior.		 806 15 Subjective quality blind spots (334)
Other evaluation coverage gaps (248)
Judge and metric reliability (180)
Golden-data representativeness (173)		 Experiment evaluation required significant expert bandwidth and often took weeks. (2)
If internal documentation is unclear or inaccurate, the LLM cannot reliably produce high-quality answers no matter ho... (2)
Many improvements yielded only marginal gains without a clear path to better answer quality. (2)		 Add document evaluation that scores source documents and returns actionable suggestions for improving the knowledge b... (2)
Adopt the configurable agentic RAG architecture to test incremental improvements. (2)
Automate batch evaluation with LLM-as-Judge while using SME responses as reference context. (2)		 a-practical-blueprint-for-evaluating-conversational-ai-at-scale (dropbox, 2025)
"But they couldn’t enforce deployment-ready correctness." (article, stated)

abstractive-conversation-summarization-for-google-chat-spaces (google, 2022)
"Detecting low quality summaries: While the two methods above limited low quality and low value summaries, we still developed methods to detect and abstain from showing such ..." (article, stated)

accelerating-game-asset-creation-with-fine-tuned-diffusion-models (rovio, 2024)
"But when we showed it to our artist, they didn't agree with me, this was not good enough. You can see in some of the gestures of Red, it looks a bit creepy and a bit awkward." (transcript, stated)
Retrieval and grounding failures
Bad retrieval, ranking, grounding, chunking, or citation behavior harms answers.		 784 15 Other retrieval and grounding failures (346)
Chunking and structure loss (271)
Citation and grounding fidelity (213)
Ranking, recall, and cutoff mistakes (109)		 An LLM can generate misleading information if prompts do not force it to answer from relevant authoritative chunks. (2)
Answers are spread across Engwiki, internal Stack Overflow, engineering requirement documents, and other locations, m... (2)
Chunking still left Claude responsible for synthesizing too much transactional data. (2)		 Let users run custom evaluations and join context, responses, Slack metadata, and feedback into evaluation reports. (2)
Move to Google Docs plus a custom loader, table-aware chunking, and LLM-powered markdown formatting. (2)
Retrieve relevant knowledge, add sub-contexts and source URLs to prompts, and require answers to cite the source URL. (2)		 advanced-agent-monitoring-and-debugging-with-langsmith-integration (replit, 2024)
"As a result, Replit’s traces were very large - involving hundreds of steps. This posed significant challenges for ingesting data and displaying it in a visually meaningful way." (article, stated)

advanced-prompt-engineering-techniques-for-production-llm-applications (instacart, 2023)
"models knows that this emerging tech has limitless possibilities; Possibilities that are currently constrained by a number of challenges: from context size issues, to hallucinat..." (article, stated)

advanced-rag-implementation-for-ai-assistant-response-accuracy (nippon_india_mutual_fund, 2025)
"* Limited use of relevant data – As the dataset grows, only the configured set of top results are used for context, impacting accuracy" (article, stated)
Human adoption and workflow fit
Users need trust, review, UI fit, escalation, or habit change before the system works.		 738 15 Workflow and UI fit (593)
Trust calibration (99)
Review and escalation burden (96)
Change management and training (28)		 The model's tool-selection behavior was nondeterministic across conversations. (2)
The original on-call process involved several rounds of waiting for human responses and clarifications. (2)
A 1.3-million-line transformation could be functionally successful but still unacceptable if the result could not be ... (1)		 Add interaction buttons for follow-up questions, resolution marking, and contacting human support. (2)
Treat prompt engineering as only partial mitigation and defer high-risk write tools until additional safety controls ... (2)
1Password restricted agents to well-specified patterns and kept humans responsible for sequencing-sensitive work. (1)		 accelerating-game-asset-creation-with-fine-tuned-diffusion-models (rovio, 2024)
"The first one was about control. It was really hard when the artist used these tools the first time to get the resource they wanted, especially if they were focusing on generati..." (transcript, stated)

agent-based-ai-assistants-for-enterprise-and-e-commerce-applications (prosus, 2024)
"October 22 when we measured this it was about 10% it means one in 10 um it's just an answer that you cannot really trust but you don't know which one" (transcript, stated)

agent-identity-and-access-management-for-production-ai-systems (uber, 2026)
"Execution context (originating user, intermediate agents) is dropped across agent hops." (article, stated)
Context, memory, and state bloat
Context windows, memory stores, prompts, or workflow state become too large or stale.		 556 15 Stale or conflicting context (238)
Prompt and instruction sprawl (235)
Context-window overload (148)
Memory retrieval quality (3)		 The model was asked to reason over large JSON payloads and do math inside its context window. (2)
8B and 16B models strained the GPU-backed EC2 setup as prompt context increased. (1)
A caller may say something unexpected after the system has already started optimizing for a predicted response. (1)		 Ramp shifted computation into an in-memory SQL database and let Claude query structured tables. (2)
A separate text-to-SQL tool generates, lints, corrects, and executes SQL against the emissions database. (1)
Accept a random subset temporarily while planning to improve recipe selection. (1)		 accelerating-llm-inference-with-speculative-decoding-for-ai-agent-applications (linkedin, 2025)
"On the other hand, if your text is highly variable, creative, or less structured, n‑gram speculation may deliver smaller gains. In those cases, a draft-model approach might be m..." (article, stated)

advanced-context-aware-code-generation-with-custom-infrastructure-and-parallel-llm-processing (codeium, 2024)
"this comes at the cost of latency" (transcript, stated)

agent-driven-ui-framework-migration-at-enterprise-scale (block, 2026)
"The main problem was that model knowledge of Fluent UI was uneven and often stale." (article, stated)
Latency, cost, and scale pressure
Production load, latency targets, model cost, or infrastructure limits force design changes.		 328 15 Model, token, and compute cost (142)
Inference and serving latency (103)
Throughput and capacity bottlenecks (97)
Other latency, cost, and scale pressure (13)		 A 12-layer BERT model has large parameter count and CPU serving cost. (1)
A cross-encoder jointly evaluates every query-Pin pair, creating real-time latency and cost pressure. (1)
A development training run can produce the exact artifact teams want to promote, and rerunning may waste compute or c... (1)		 Abstract vector stores behind Bedrock Knowledge Bases while allowing tenant tiers to use different backends. (1)
Accept the small proxy overhead to avoid downstream changes while moving GPU-intensive inference to Bedrock. (1)
Account for rate limits, latency, cost, caching, and vendor lock-in in production design. (1)		 2x-engineering-throughput-through-ai-first-development-platform (intercom, 2026)
"After fixing bottlenecks and improving performance, CI is no longer the bottleneck, though code review has become one." (summary, stated)

accelerating-sap-s-4hana-migration-and-custom-code-documentation-with-generative-ai (axfood_/_harman, 2025)
"customizations in the system, and majority of those customizations have very poor documentation, and in some cases very" (transcript, stated)

agent-testing-and-evaluation-using-autonomous-vehicle-simulation-principles (coval, 2023)
"The problem addressed is that testing conversational AI agents is slow and manual, with engineers spending hours on repetitive "vibe checks" that become a game of whack-a-mole a..." (summary, stated)
Model reliability and behavior drift
Model nondeterminism, drift, upgrades, or weak reasoning make behavior unreliable.		 273 15 Other model reliability and behavior drift (107)
Model-version regressions and drift (73)
Reasoning, hallucination, and factual errors (61)
Model capability and domain mismatch (44)		 A 75-79% function-call success rate produces frequent visible failures for a voice assistant. (1)
A broad assistant will produce unreliable or fabricated answers, especially early in deployment. (1)
A central concern is that incorrect LLM labels might degrade human accuracy during assisted annotation. (1)		 A curated paraphrasing model summarizes the interpreted issue before presenting the article link. (1)
Add automated OCR/parsing, observability, governance, and validation before model consumption. (1)
Add Bosch brand logic for colors, tonality, composition, and authentic-looking people. (1)		 advanced-fine-tuning-techniques-for-multi-agent-orchestration-at-scale (amazon, 2026)
"Initial attempts using traditional RAG with foundation models yielded disappointing results, with accuracy hovering between 60 and 70%. The breakthrough came when they fine-tune..." (article, stated)

advanced-speaker-diarization-and-attributed-transcription-using-deep-learning-models (pyannoteai, 2026)
"most speech to text models are actually trained on single speaker data. As soon as you apply them on multi-speaker data with overlap, with speaker change, with all kind of mess,..." (transcript, stated)

ai-agent-development-and-evaluation-platform-for-insurance-underwriting (snorkel, 2025)
"* Tool use errors: Across models, including top performers, agents made at least one tool call error in 36% of the conversations despite access to the metadata required to u..." (article, stated)
Integration and legacy-system friction
Existing APIs, legacy systems, tools, or enterprise platforms do not fit agent workflows cleanly.		 243 14 API, connector, and contract mismatch (190)
Other integration and legacy-system friction (49)
Legacy-system and data-access friction (7)
Authentication and entitlement propagation (6)		 A competitor blocklist treated Microsoft as forbidden even when customers asked valid Salesforce Teams integration qu... (1)
A local crew is not automatically an autoscaling, protected, deployable API. (1)
A one-operation-to-one-tool mapping can expose hundreds of tools, especially for large APIs. (1)		 A spontaneous workstream optimized code paths and added new filesystem/database capacity. (1)
Add batching, distributed processing, logging, retries, and error handling before production. (1)
Add governance and guardrails around self-service tool publication. (1)		 accelerating-drug-development-with-ai-powered-clinical-trial-transformation (novartis, 2025)
"we realized that there are some challenges that we're going have to navigate through with the Novartis. Some of them you might find familiar." (transcript, stated)

adopting-model-context-protocol-mcp-in-financial-services-for-ai-system-integration (evergreen_wealth_/bloomberg/_saxo_bank, 2026)
"there are some interesting challenges uh in the just authentication authorization space in in terms of how you carry forward that identity, how you make sure entitlements are re..." (transcript, stated)

advancing-patient-experience-and-business-operations-analytics-with-generative-ai-in-healthcare (huron, 2025)
"it does take three to four months before the data is received back and by then, missed opportunities, early intervention to issues has been lost." (transcript, stated)
Orchestration, retry, and handoff failures
Multi-step workflows fail around retries, handoffs, ordering, idempotency, or coordination.		 240 15 Retry, idempotency, and recovery failures (109)
Other orchestration, retry, and handoff failures (77)
Timeout, ordering, and concurrency failures (31)
Handoff and state loss (23)		 Extracting large volumes through ordinary APIs was not optimized for reporting workloads. (2)
A bad answer can come from a tool problem, an inference problem, or another layer in the stack. (1)
A bad response may come from an intermediate inference inside a chain with up to 100 nested calls. (1)		 Ramp added an OLAP-powered reporting API path, while noting further optimizations like concurrency, async pulls, cach... (2)
Abridge consolidated around a single encounter workflow and moved cross-namespace calls to Nexus. (1)
Adapt the agent-visible error layer to each API's failure conventions. (1)		 actionable-ci-intelligent-analysis-and-auto-remediation-of-ci-pipeline-failures (block, 2026)
"For failures that are not eligible for autofix, the Results page gives developers the tools to move fast. Each issue has an Explain button that generates a plain-language brea..." (article, stated)

ai-enhanced-body-camera-and-digital-evidence-management-in-law-enforcement (an_garda_siochanna, 2023)
"At the moment, we're not allowed to do that. We need the AI legislation. The next one, looking to search for a car or a bicycle." (transcript, stated)

ai-orchestrated-code-review-system-at-scale (cloudflare, 2026)
"This is especially problematic if your application exits early before it has a chance to close everything out and write a valid JSON blob to disk — and this is often when you ne..." (article, stated)
Security, privacy, and compliance risk
Sensitive data, policy, permissions, privacy, or regulated requirements constrain the system.		 236 15 Sensitive-data and privacy leakage (97)
Regulatory, audit, and policy obligations (75)
Other security, privacy, and compliance risk (35)
Identity and access-control failures (33)		 Security and privacy documents differ by persona, geography, retention, classification, and sharing rules. (2)
Some internal sources are not safe to expose to broad Slack audiences or to the embedding/generation path. (2)
A malicious or mismatched email could ask for bank account or address details. (1)		 Pre-curate only widely accessible sources and scope embeddings to appropriate Slack channels. (2)
Use source identification, enriched metadata, BM25, and post-processing before generation. (2)
A planned phase has Wingman monitor Argentina tax law and propose policy updates for human validation. (1)		 ai-agent-automation-of-security-operations-center-analysis (doppel, 2025)
"These decisions aren’t limited to spotting basic patterns – they require judgment. Analysts must interpret unstructured data like screenshots, time-series activity, and custom..." (article, stated)

ai-agent-for-self-service-business-intelligence-with-text-to-sql (bgl, 2026)
"When engineering teams implement an AI agent for analytics use cases, a common anti-pattern is to have the agent handle everything including understanding database schemas, tran..." (article, stated)

ai-driven-student-services-and-prescriptive-pathways-at-ucla-anderson-school-of-management (ucla, 2024)
"we wanted to ensure the security and privacy of that student data. So we took extra care to architect that environment such that it mirrored the Information Security Policy" (transcript, stated)
Generic production risk
Production issue whose root cause is not specific enough for this analysis rubric.		 206 15 Other generic production risk (140)
Product scope and value fit (26)
Operational readiness gaps (25)
Upstream source availability (19)		 A basic assistant may be useful but not central to the learning product's core value. (1)
A celebrity name can also be a common local name. (1)
A cluster name or description could include specific website names or other identifiers even after summarization. (1)		 Abort the stream after 10 seconds of inactivity. (1)
Accelerate 10x backbone techniques into the 2024-2025 window. (1)
Accept the complexity when it substantially reduces IP router interfaces. (1)		 accelerating-drug-development-with-ai-powered-clinical-trial-transformation (novartis, 2025)
"However, these domains at the time were purposefully built for their own domains and specializations but not necessarily built for interoperability with each other." (transcript, stated)

advanced-embedding-based-retrieval-for-personalized-content-discovery (pinterest, 2024)
"We found that the model suffered from overfitting severely." (article, stated)

agentic-ai-manufacturing-reasoner-for-automated-root-cause-analysis (apollo_tyres, 2025)
"Initially, the generated code often contained inaccuracies or failed to handle large volumes of data correctly. To address this issue, they embarked on a process of continuous r..." (article, stated)
Data quality and schema drift
Bad data, changing schemas, missing metadata, or inconsistent source systems break behavior.		 76 14 Schema and contract drift (43)
Metadata and taxonomy quality (17)
Missing and inconsistent source data (17)
Label and training-data quality (1)		 A tool fetched database schemas from a private GitHub repository using credentials, but accepted attacker-controlled ... (1)
Accurate analytics requires parsing, ingesting, and normalizing metadata stored across many enterprise tools. (1)
Agents looped, hallucinated schema, ignored stop commands, and produced inconsistent answers under real user behavior. (1)		 Add deterministic gates, validation, expert curation, and hard/soft limits. (1)
Agents discover existing patterns and encode accessibility best practices. (1)
Attach Statsig exposure values as LangSmith tags so groups can be queried and exported. (1)		 ai-agent-for-customer-service-order-management-and-training (rhi_magnesita, 2023)
"Despite being an entry-level position, CSRs are responsible for generating significant revenue but lack formal training processes." (summary, stated)

ai-driven-multi-agent-system-for-dynamic-product-taxonomy-evolution (shopify, 2025)
"Every new product type, emerging technology category, and seasonal trend potentially requires taxonomy updates. Traditional manual curation simply can't keep pace." (article, stated)

ai-powered-fraud-detection-using-mixture-of-experts-and-federated-learning (feedzai, 2025)
"the format and the meaning of the data must be standardized, such that the patterns that the experts learned can be seamlessly integrated into new financial institutions." (article, stated)

Final Rubric Change Log

The first corpus-complete dry run exposed seven approved broad categories whose existing subcluster rubric placed every project in other. The final run adds one level of agent-authored, deterministic failure-mode rules for those categories. No extraction content changed.

Broad cluster Reason Before: fallback projects After: fallback projects Newly separated projects Known limitation
Latency, cost, and scale pressure 100% fallback in the first corpus-complete dry run 328 13 315 Signal rules are reproducible but still require reviewer boundary sampling.
Model reliability and behavior drift 100% fallback in the first corpus-complete dry run 273 107 166 Signal rules are reproducible but still require reviewer boundary sampling.
Integration and legacy-system friction 100% fallback in the first corpus-complete dry run 243 49 194 Signal rules are reproducible but still require reviewer boundary sampling.
Orchestration, retry, and handoff failures 100% fallback in the first corpus-complete dry run 240 77 163 Signal rules are reproducible but still require reviewer boundary sampling.
Security, privacy, and compliance risk 100% fallback in the first corpus-complete dry run 236 35 201 Signal rules are reproducible but still require reviewer boundary sampling.
Generic production risk 100% fallback in the first corpus-complete dry run 206 140 66 Signal rules are reproducible but still require reviewer boundary sampling.
Data quality and schema drift 100% fallback in the first corpus-complete dry run 76 0 76 Signal rules are reproducible but still require reviewer boundary sampling.

Boundary Flags

Fallback subclusters retain cases not separated by the current rubric. Large fallback shares should be split before synthesis if reviewer sampling finds a reusable boundary.

Broad cluster Fallback subcluster Deduped projects Share of broad cluster Review flag
Retrieval and grounding failures Other retrieval and grounding failures 346 44.1% review before synthesis
Evaluation coverage gaps Other evaluation coverage gaps 248 30.8% review before synthesis
Generic production risk Other generic production risk 140 68.0% review before synthesis
Model reliability and behavior drift Other model reliability and behavior drift 107 39.2% review before synthesis
Orchestration, retry, and handoff failures Other orchestration, retry, and handoff failures 77 32.1% sample
Integration and legacy-system friction Other integration and legacy-system friction 49 20.2% sample
Security, privacy, and compliance risk Other security, privacy, and compliance risk 35 14.8% sample
Human adoption and workflow fit Other human adoption and workflow fit 22 3.0% sample
Latency, cost, and scale pressure Other latency, cost, and scale pressure 13 4.0% sample
Context, memory, and state bloat Other context, memory, and state bloat 2 0.4% sample

Subcluster Breakdowns

Evaluation coverage gaps

Subcluster Deduped projects Description Representative evidence
Subjective quality blind spots 334 Automated metrics miss tone, usefulness, conversation quality, or other subjective behaviors. abstractive-conversation-summarization-for-google-chat-spaces (google, 2022)
"Detecting low quality summaries: While the two methods above limited low quality and low value summaries, we still developed methods to detect and abstain from showing such ..." (article, stated)

accelerating-game-asset-creation-with-fine-tuned-diffusion-models (rovio, 2024)
"But when we showed it to our artist, they didn't agree with me, this was not good enough. You can see in some of the gestures of Red, it looks a bit creepy and a bit awkward." (transcript, stated)
Other evaluation coverage gaps 248 Sub-mode not yet separated by the current signal rubric. agent-driven-development-for-ai-research-using-github-copilot-cli (github, 2026)
"We know that AI coding agents are really good at solving well-scoped problems but need handholding for the more complex problems you’d only entrust to your more senior engineers." (article, stated)

agentic-ai-architecture-for-investment-management-platform (blackrock, year n/a)
"we are very paranoid about telling the user the wrong things. So we toss um start by testing every single behavior that we uh are intending by writing that in the system prompt." (transcript, stated)
Judge and metric reliability 180 LLM judges, metrics, or grading rubrics are themselves noisy or poorly calibrated. a-practical-blueprint-for-evaluating-conversational-ai-at-scale (dropbox, 2025)
"But they couldn’t enforce deployment-ready correctness." (article, stated)

advanced-speaker-diarization-and-attributed-transcription-using-deep-learning-models (pyannoteai, 2026)
"we don't know in advance the number of classes that we're supposed to detect contrary to other let's say classical machine learning problems. And also we don't know the identity..." (transcript, stated)
Golden-data representativeness 173 Test sets, examples, or benchmarks do not cover production behavior well enough. agentic-ai-copilot-for-insurance-underwriting-with-multi-tool-integration (snorkel, 2025)
"to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI mode..." (summary, stated)

agentic-ai-search-with-custom-evaluation-framework-for-church-management (pushpay, 2026)
"## The challenges of initial solution without evaluation" (article, stated)
Production regression detection 131 Teams struggle to catch regressions across releases, model upgrades, or live traffic. agentic-ai-systems-for-drug-discovery-and-business-intelligence (loka, 2025)
"generative AI models have some advantages over traditional models in that they um can do a better job of course interpreting you know unstructured data but that doesn't mean tha..." (transcript, stated)

ai-agents-accelerating-gpu-kernel-engineering-for-llm-infrastructure (linkedin, 2026)
"Each of these tasks takes hours of expert time, and manual effort simply doesn't scale with the pace of model innovation." (article, stated)

Retrieval and grounding failures

Subcluster Deduped projects Description Representative evidence
Other retrieval and grounding failures 346 Sub-mode not yet separated by the current signal rubric. advanced-agent-monitoring-and-debugging-with-langsmith-integration (replit, 2024)
"As a result, Replit’s traces were very large - involving hundreds of steps. This posed significant challenges for ingesting data and displaying it in a visually meaningful way." (article, stated)

agentic-ai-for-legal-research-building-deep-research-in-westlaw-and-cocounsel (thomson_reuters, 2025)
"n New York where the company hired someone of the same race and gender as the rejected applicant. A traditional RAG system might suggest the claim can proceed. But that’s not en..." (article, stated)
Chunking and structure loss 271 Document structure, tables, layout, or chunks lose the information needed for grounded answers. advanced-rag-implementation-for-ai-assistant-response-accuracy (nippon_india_mutual_fund, 2025)
"* Limited use of relevant data – As the dataset grows, only the configured set of top results are used for context, impacting accuracy" (article, stated)

agentic-ai-system-for-document-summarization-and-analysis (moveworks, 2024)
"Current language models struggle to effectively utilize information from long input contexts. Performance is typically highest when relevant information appears at the beginning..." (article, stated)
Citation and grounding fidelity 213 Answers cite, ground, or attribute claims incorrectly even when retrieval returns plausible material. advanced-prompt-engineering-techniques-for-production-llm-applications (instacart, 2023)
"models knows that this emerging tech has limitless possibilities; Possibilities that are currently constrained by a number of challenges: from context size issues, to hallucinat..." (article, stated)

agentic-ai-system-for-document-summarization-and-analysis (moveworks, 2024)
"Most existing tools either don't provide citations or only offer citations at the document level. Brief Me, however, is capable of providing citations at the paragraph level" (article, stated)
Ranking, recall, and cutoff mistakes 109 Search retrieves the wrong items, ranks the right items too low, or cuts off useful context. agentic-ai-architecture-for-meeting-intelligence-and-productivity-automation (zoom, 2025)
"Meeting summaries are sometimes general, and as a result, may lack personal value. But with AI Companion Web 3.0, it knows your role and can tailor the summary" (transcript, stated)

ai-chatbots-for-customer-service-production-lessons-from-90-days (edsdev, 2026)
"The boring truth: most “the AI hallucinated” complaints we investigated were retrieval failures, not generation failures." (article, stated)
Stale or missing knowledge 47 Knowledge sources are incomplete, outdated, inaccessible, or not refreshed quickly enough. agentic-mapreduce-for-whole-codebase-security-scanning (devin, 2026)
"3. No explicit coverage boundary. A search-driven agent stops when it decides it’s done - not when a finite work queue has been exhausted." (article, stated)

ai-agent-system-for-automated-travel-itinerary-generation (aimpoint_digital, 2024)
"An LLM last trained in December 2023 is unlikely to be aware of a restaurant that closed in July 2024 and may incorrectly recommend it to a traveler." (article, stated)

Human adoption and workflow fit

Subcluster Deduped projects Description Representative evidence
Workflow and UI fit 593 The system has to fit existing user workflows, screens, channels, or habits. accelerating-game-asset-creation-with-fine-tuned-diffusion-models (rovio, 2024)
"The first one was about control. It was really hard when the artist used these tools the first time to get the resource they wanted, especially if they were focusing on generati..." (transcript, stated)

agent-identity-and-access-management-for-production-ai-systems (uber, 2026)
"Execution context (originating user, intermediate agents) is dropped across agent hops." (article, stated)
Trust calibration 99 Users need calibrated trust, confidence, or transparency before acting on model output. agent-based-ai-assistants-for-enterprise-and-e-commerce-applications (prosus, 2024)
"October 22 when we measured this it was about 10% it means one in 10 um it's just an answer that you cannot really trust but you don't know which one" (transcript, stated)

ai-augmented-code-review-system-for-large-scale-software-development (uber, 2025)
"A high false‑positive rate undermines engineers’ perception of the tool’s accuracy and usefulness—when they encounter many false-positive comments, they start to tune out and ig..." (article, stated)
Review and escalation burden 96 Human review, approval, or escalation becomes too heavy, late, or poorly routed. ai-enhanced-body-camera-and-digital-evidence-management-in-law-enforcement (an_garda_siochanna, 2023)
"there's 99 people we were trying to find after the Dublin riots. And the guards literally had 40,000 hours of video and they went through it frame by frame" (transcript, stated)

ai-native-multi-agent-system-for-customer-onboarding-and-kyc (brex, 2026)
"We watched them make decisions and asked them to narrate their thoughts. We documented everything that had never been written down: what signals they trusted, what they overvalu..." (article, implied)
Change management and training 28 Adoption depends on training, enablement, rollout, or organizational change. ai-agents-in-production-multi-enterprise-implementation-strategies (canva_/kpmg/autodesk/lightspeed, 2026)
"The companies faced challenges around scaling AI from proof-of-concept to production, managing agent quality and accuracy, and driving adoption across diverse teams." (summary, stated)

ai-powered-contact-center-copilot-from-research-to-enterprise-scale-production (cresta/_openai, 2025)
"we found that the people who are not using aren't actually veterans where they think they know actually how to do things and it's like a mindset shift" (transcript, stated)
Other human adoption and workflow fit 22 Sub-mode not yet separated by the current signal rubric. ai-assisted-product-attribute-extraction-for-e-commerce-content-creation (zalando, 2024)
"One of the biggest challenges was translating the Zalando-specific attribute codes provided by Master Data (e.g." (article, stated)

ai-powered-benefits-navigation-system-for-snap-recipients (propel, 2025)
"we recommend starting with small-scale pilots that allow for close monitoring, iteration, and human support for any necessary escalation." (article, stated)

Context, memory, and state bloat

Subcluster Deduped projects Description Representative evidence
Stale or conflicting context 238 Context or memory persists incorrectly, conflicts with fresh information, or becomes stale. agent-driven-ui-framework-migration-at-enterprise-scale (block, 2026)
"The main problem was that model knowledge of Fluent UI was uneven and often stale." (article, stated)

ai-agent-for-automated-feature-flag-removal (duolingo, 2025)
"But in Temporal, each activity may run on a separate worker, so cloning in a separate activity is not a viable option—we could find ourselves cloning the code and then trying to..." (article, stated)
Prompt and instruction sprawl 235 Prompt templates, policies, or instructions grow tangled and hard to maintain. accelerating-llm-inference-with-speculative-decoding-for-ai-agent-applications (linkedin, 2025)
"On the other hand, if your text is highly variable, creative, or less structured, n‑gram speculation may deliver smaller gains. In those cases, a draft-model approach might be m..." (article, stated)

agent-registry-and-dynamic-prompt-management-for-ai-feature-development (gitlab, year n/a)
"GitLab faced a challenge with their AI-powered issue description generation feature where prompts were hardcoded in Ruby classes, meaning any prompt changes required a full GitL..." (summary, stated)
Context-window overload 148 Prompts, histories, or retrieved context exceed useful token budgets. advanced-context-aware-code-generation-with-custom-infrastructure-and-parallel-llm-processing (codeium, 2024)
"this comes at the cost of latency" (transcript, stated)

agent-first-ai-development-platform-with-multi-surface-orchestration (google_deepmind, 2025)
"ultimately you're looking at a lot a lot of strings, a lot of tokens. This is like very hard to follow. And then this is actually like there's like 10 of these, right?" (transcript, stated)
Memory retrieval quality 3 Memory exists, but selecting or retrieving the right memory is unreliable. building-and-scaling-ai-powered-password-detection-in-production (github, 2025)
"This helped improve precision without reducing our recall, but was also more resource intensive." (article, stated)

building-fair-housing-guardrails-for-real-estate-llms-zillow-s-multi-strategy-approach-to-preventing-discrimination (zillow, 2024)
"Any semantic connection with the protected classes might result in a similar message, as indicated above." (article, stated)
Other context, memory, and state bloat 2 Sub-mode not yet separated by the current signal rubric. agentic-data-analyst-for-enterprise-self-service-analytics (ramp, 2025)
"Scale remained the challenge." (article, stated)

building-and-scaling-visual-intelligence-models-from-research-to-production (black_forest_labs, 2026)
"find perceptually equivalent but much lower-dimensional representations of pixel space" (summary, stated)

Latency, cost, and scale pressure

Subcluster Deduped projects Description Representative evidence
Model, token, and compute cost 142 Token use, model pricing, GPU demand, or repeated inference makes the system uneconomical. accelerating-sap-s-4hana-migration-and-custom-code-documentation-with-generative-ai (axfood_/_harman, 2025)
"customizations in the system, and majority of those customizations have very poor documentation, and in some cases very" (transcript, stated)

ai-agent-system-for-automated-b2b-research-and-sales-pipeline-generation (unify, 2025)
"In April, we pushed uh 36 billion tokens and that has increased" (transcript, stated)
Inference and serving latency 103 Model inference or end-to-end serving latency misses interactive or production targets. ai-powered-contact-center-transformation-for-pet-retail (petco, 2024)
"I will tell you, big lesson learned, we thought we had all of the utterances covered and we didn't. And so we're learning a lot and we're adding those in real time." (transcript, stated)

ai-powered-customer-feedback-analysis-system-for-container-shipping (hapag-lloyd, 2026)
"This generates insights that you can use to optimize the applications further like improving response latency or reducing costs." (article, stated)
Throughput and capacity bottlenecks 97 Traffic, concurrency, batch size, or infrastructure capacity cannot sustain production load. 2x-engineering-throughput-through-ai-first-development-platform (intercom, 2026)
"After fixing bottlenecks and improving performance, CI is no longer the bottleneck, though code review has become one." (summary, stated)

agent-testing-and-evaluation-using-autonomous-vehicle-simulation-principles (coval, 2023)
"The problem addressed is that testing conversational AI agents is slow and manual, with engineers spending hours on repetitive "vibe checks" that become a game of whack-a-mole a..." (summary, stated)
Other latency, cost, and scale pressure 13 Sub-mode not yet separated by the current signal rubric. ai-powered-code-review-system-at-scale (uber, 2025)
"Developers don’t like certain categories of comments from AI tools. Readability nits, minor logging tweaks, low-impact performance optimizations, and stylistic issues consistent..." (article, stated)

ai-powered-marketing-platform-for-small-and-medium-businesses (mowie, 2025)
"it's not an exact attribution link between the post and sales but can we at least come up with a statistically meaningful inference on how this marketing piece of marketing is i..." (transcript, stated)
Rate limits and resource pressure 7 Provider rate limits, memory limits, quotas, or scarce infrastructure constrain execution. building-a-unified-genai-platform-for-hundreds-of-production-use-cases (karrot, 2025)
"AI APIs typically enforce per-account rate limits. With teams operating independently, some accounts had plenty of headroom while others hit their limits" (article, stated)

building-customer-intelligence-mcp-server-for-ai-agent-integration (dovetail, 2025)
"The scalability claims are not substantiated with specific performance metrics or architectural details." (summary, stated)

Model reliability and behavior drift

Subcluster Deduped projects Description Representative evidence
Other model reliability and behavior drift 107 Sub-mode not yet separated by the current signal rubric. advanced-speaker-diarization-and-attributed-transcription-using-deep-learning-models (pyannoteai, 2026)
"most speech to text models are actually trained on single speaker data. As soon as you apply them on multi-speaker data with overlap, with speaker change, with all kind of mess,..." (transcript, stated)

ai-powered-clinical-documentation-with-multi-region-healthcare-compliance (heidi_health, 2025)
"healthcare isn't a single standard. Like we encountered like four distinct layers that I can think of, of complexity that we have to solve simultaneously." (transcript, stated)
Model-version regressions and drift 73 Model upgrades, provider changes, or temporal drift alter previously working behavior. ai-powered-clinical-documentation-and-data-infrastructure-for-point-of-care-transformation (veradigm, 2025)
"it's gen AI so it costs real money to run. It's not a SaaS application, right, where you're probably not thinking a whole lot about, well, how much compute am I going to use" (transcript, stated)

ai-powered-epilepsy-diagnosis-platform-reducing-diagnostic-time-through-multimodal-data-processing (australian_epilepsy_project, 2025)
"this data nexus is a data warehouse, an RDS instance um that we use for all the different model versioning, process orchestration and all these tasks that we need in order to ke..." (transcript, stated)
Reasoning, hallucination, and factual errors 61 The model invents facts, reasons incorrectly, or returns inaccurate task results. ai-agent-development-and-evaluation-platform-for-insurance-underwriting (snorkel, 2025)
"* Tool use errors: Across models, including top performers, agents made at least one tool call error in 36% of the conversations despite access to the metadata required to u..." (article, stated)

ai-powered-community-voice-intelligence-for-local-government (zencity, 2025)
"we found sometimes it was verbatim, sometimes it was just slightly rewarded, sometimes it just made something up that sounded what we had given it. So we realized no, we need to..." (transcript, stated)
Model capability and domain mismatch 44 The selected model lacks the domain knowledge, modality support, or capability required by the task. advanced-fine-tuning-techniques-for-multi-agent-orchestration-at-scale (amazon, 2026)
"Initial attempts using traditional RAG with foundation models yielded disappointing results, with accuracy hovering between 60 and 70%. The breakthrough came when they fine-tune..." (article, stated)

ai-agent-development-and-evaluation-platform-for-insurance-underwriting (snorkel, 2025)
"They occasionally hallucinated guidelines that might appear online but were not actually contained in the provided documentation." (article, stated)
Nondeterminism and inconsistent behavior 11 Equivalent requests produce unstable decisions, tool calls, or outputs. ai-powered-fraud-detection-using-mixture-of-experts-and-federated-learning (feedzai, 2025)
"As fraud patterns evolve and expert models are updated, the risk score must remain consistent and reliable over time." (article, stated)

ai-powered-on-call-assistant-for-airflow-pipeline-debugging (wix, 2026)
"LLMs are inherently non-deterministic, but SRE automation demands reliability." (article, stated)

Integration and legacy-system friction

Subcluster Deduped projects Description Representative evidence
API, connector, and contract mismatch 190 External APIs, connectors, schemas, or tool contracts do not align with agent execution needs. agentic-ai-for-automated-absence-reporting-and-shift-management-at-airport-operations (manchester_airports_group, 2024)
"So if an employee comes in and gives incomplete information, then our tool doesn't work already because it can't quite record the full absence. This is why we really need an Age..." (transcript, stated)

agentic-ai-for-automated-absence-reporting-and-shift-management-at-airport-operations (manchester_airports_group, 2024)
"what stops someone from social engineering the agent into doing something malicious? We're like, yeah, that's a great question. Well of course we would add guardrails." (transcript, stated)
Other integration and legacy-system friction 49 Sub-mode not yet separated by the current signal rubric. advancing-patient-experience-and-business-operations-analytics-with-generative-ai-in-healthcare (huron, 2025)
"it does take three to four months before the data is received back and by then, missed opportunities, early intervention to issues has been lost." (transcript, stated)

ai-managed-services-and-agent-operations-at-enterprise-scale (pricewaterhousecooper, 2026)
"the skill set of the people that we have in AI managed services is very different from your traditional managed services folks. We are hiring MLOps engineers. We are hiring data..." (transcript, stated)
Legacy-system and data-access friction 7 Legacy platforms, inaccessible data, or old interfaces block reliable automation. accelerating-drug-development-with-ai-powered-clinical-trial-transformation (novartis, 2025)
"we realized that there are some challenges that we're going have to navigate through with the Novartis. Some of them you might find familiar." (transcript, stated)

ai-powered-contact-center-transformation-for-student-support-services (anthology, 2024)
"every back to school we would have 12 unplanned outages in our busiest months of July, August and September because of the load and this back to school we had two." (transcript, stated)
Authentication and entitlement propagation 6 Identity, credentials, authorization, or user entitlements fail across system boundaries. adopting-model-context-protocol-mcp-in-financial-services-for-ai-system-integration (evergreen_wealth_/bloomberg/_saxo_bank, 2026)
"there are some interesting challenges uh in the just authentication authorization space in in terms of how you carry forward that identity, how you make sure entitlements are re..." (transcript, stated)

building-a-secure-kubernetes-platform-for-autonomous-ai-agents (grab, 2026)
"Containers help package the runtime, but they do not answer the harder platform questions:" (article, stated)

Orchestration, retry, and handoff failures

Subcluster Deduped projects Description Representative evidence
Retry, idempotency, and recovery failures 109 Retries duplicate work, lose progress, or cannot safely resume failed execution. actionable-ci-intelligent-analysis-and-auto-remediation-of-ci-pipeline-failures (block, 2026)
"For failures that are not eligible for autofix, the Results page gives developers the tools to move fast. Each issue has an Explain button that generates a plain-language brea..." (article, stated)

ai-orchestrated-code-review-system-at-scale (cloudflare, 2026)
"This is especially problematic if your application exits early before it has a chance to close everything out and write a valid JSON blob to disk — and this is often when you ne..." (article, stated)
Other orchestration, retry, and handoff failures 77 Sub-mode not yet separated by the current signal rubric. ai-enhanced-body-camera-and-digital-evidence-management-in-law-enforcement (an_garda_siochanna, 2023)
"At the moment, we're not allowed to do that. We need the AI legislation. The next one, looking to search for a car or a bicycle." (transcript, stated)

ai-powered-chief-of-staff-scaling-agent-architecture-from-monolith-to-distributed-system (outropy, 2024)
"Temporal’s Python SDK felt like a second-class citizen—even using standard libraries like Pydantic was a challenge, as the framework favors data classes." (article, stated)
Timeout, ordering, and concurrency failures 31 Concurrent or asynchronous work violates ordering, timing, or coordination requirements. ai-powered-compliance-investigation-agents-for-enhanced-due-diligence (stripe, 2025)
"Agents are kind of the opposite in all those aspects. So agents, particularly if you're using a vendor model, you're just waiting for it to come back. You're waiting for the LM ..." (transcript, stated)

ai-powered-contact-center-transformation-with-amazon-connect (traeger, 2024)
"by the time that you have identified the issue, that you have taken the time to put a message to your customers, they've missed it, they're in queue" (transcript, stated)
Handoff and state loss 23 Agents or workflow stages lose context, ownership, or execution state at handoff boundaries. ai-powered-accessibility-automation-for-e-commerce-platform (mercado_libre, 2025)
"The assistant’s prompt focuses on explaining, in simple and clear terms, how to annotate components like headers, links, buttons, groups, or dynamic areas." (article, stated)

ai-powered-conversational-food-ordering-with-ilo (ifood, 2026)
"they have a problem with I don't know, iFood because every company has a problem with a customer. And they report this problem by using ilo." (transcript, stated)
Routing, loops, and termination failures 12 Orchestrators choose the wrong path, loop indefinitely, or fail to stop safely. ai-powered-customer-support-automation-for-global-transportation-service (lime, 2024)
"Every support agent handled every type of ticket because there was nothing in place to prioritize and route specific tickets to specific agents." (article, stated)

ai-powered-marketing-compliance-monitoring-at-scale (performline, 2025)
"It was imperative that they not disrupt those processes or introduce cascading changes for this solution." (article, stated)

Security, privacy, and compliance risk

Subcluster Deduped projects Description Representative evidence
Sensitive-data and privacy leakage 97 PII, confidential content, or private context can leak into prompts, outputs, or logs. ai-driven-student-services-and-prescriptive-pathways-at-ucla-anderson-school-of-management (ucla, 2024)
"we wanted to ensure the security and privacy of that student data. So we took extra care to architect that environment such that it mirrored the Information Security Policy" (transcript, stated)

ai-powered-content-moderation-at-platform-scale (roblox, 2025)
"This text filter model was handling so many requests per second (RPS) that it was becoming difficult to support on our existing CPU-based serving stack." (article, stated)
Regulatory, audit, and policy obligations 75 Regulatory controls, auditability, retention, or policy requirements constrain the system. ai-agent-for-self-service-business-intelligence-with-text-to-sql (bgl, 2026)
"When engineering teams implement an AI agent for analytics use cases, a common anti-pattern is to have the agent handle everything including understanding database schemas, tran..." (article, stated)

ai-managed-services-and-agent-operations-at-enterprise-scale (pricewaterhousecooper, 2026)
"this cannot be done by an engineer alone. This has to you know you have to bring in people from and I will call this the business right to come in" (transcript, stated)
Other security, privacy, and compliance risk 35 Sub-mode not yet separated by the current signal rubric. ai-agent-automation-of-security-operations-center-analysis (doppel, 2025)
"These decisions aren’t limited to spotting basic patterns – they require judgment. Analysts must interpret unstructured data like screenshots, time-series activity, and custom..." (article, stated)

ai-powered-sales-intelligence-and-go-to-market-orchestration-platform (clay, 2024)
"if you know from the beginning that you can't trust it um in 100% of the cases or even 90 then we need to show you how we did it" (transcript, stated)
Identity and access-control failures 33 Authorization, permissions, or identity propagation expose actions or data incorrectly. ai-native-multi-agent-system-for-customer-onboarding-and-kyc (brex, 2026)
"Deferral is a feature, not a flaw. Teaching agents to say "I'm not sure" reduced errors on both sides." (article, stated)

ai-powered-trade-assistant-for-equities-trading-workflows (jefferies_equities, 2025)
"when the traders log in the credentials are verified to ensure the right entitlement so that they only see their data that they're authorized to see." (transcript, stated)
Prompt injection and tool abuse 7 Untrusted input manipulates model instructions or triggers unsafe tools and actions. building-and-deploying-background-coding-agents-at-scale (cognition, 2026)
"actual agent running not directly in the sandbox" (transcript, stated)

building-multi-agent-systems-with-mcp-and-pydantic-ai-for-document-processing (deepsense, 2025)
"security is super important. Uh GitHub users knows that already." (transcript, stated)

Generic production risk

Subcluster Deduped projects Description Representative evidence
Other generic production risk 140 Sub-mode not yet separated by the current signal rubric. advanced-embedding-based-retrieval-for-personalized-content-discovery (pinterest, 2024)
"We found that the model suffered from overfitting severely." (article, stated)

agentic-ai-manufacturing-reasoner-for-automated-root-cause-analysis (apollo_tyres, 2025)
"Initially, the generated code often contained inaccuracies or failed to handle large volumes of data correctly. To address this issue, they embarked on a process of continuous r..." (article, stated)
Product scope and value fit 26 The AI capability is technically possible but does not justify complexity or align with product value. ai-powered-fraud-detection-in-e-commerce-using-aws-fraud-detector (awaze, 2024)
"we will never completely know because the event never completed. We've interrupted it. Um and so that's always just something to consider based on like your risk score as a busi..." (transcript, stated)

automated-hcc-code-extraction-from-clinical-notes-using-healthcare-nlp (wvu_medicine, 2023)
"we filter out the codes that are not applicable to our patient population." (transcript, stated)
Operational readiness gaps 25 A prototype lacks the deployment, monitoring, documentation, or ownership needed for production. accelerating-drug-development-with-ai-powered-clinical-trial-transformation (novartis, 2025)
"However, these domains at the time were purposefully built for their own domains and specializations but not necessarily built for interoperability with each other." (transcript, stated)

ai-powered-marketing-content-generation-and-compliance-platform-at-scale (volkswagen, 2025)
"We often rely on pre-production cars, prototypes under camouflage to create launch assets before the car is officially revealed. And this is a huge confidentiality risk." (transcript, stated)
Upstream source availability 19 Deleted, blocked, stale, or inaccessible upstream material prevents reliable processing or evidence. ai-powered-contact-center-transformation-with-amazon-connect (traeger, 2024)
"when I was able to stand up Amazon Connect in over a weekend, immediately having access to our own data and was able to baseline for the first time" (transcript, stated)

ai-powered-hyper-personalized-email-marketing-system (hubspot, 2023)
"the first iteration that we had our hypothesis was around more of not the recommendation itself but the output" (transcript, stated)

Cluster Rubric

Reviewer Checks