Building Production-Grade Agentic AI Analytics: Lessons from Real-World Deployment

tellius · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agentic enterprise analytics

Tellius needed a production analytics agent architecture that could answer natural-language business questions over governed enterprise data without letting LLM randomness, schema sprawl, ambiguous language, or black-box execution corrupt SQL/MDX/GraphQL results.

Constraints:

• Business analytics requires governed metrics, row/column security, lineage, and repeatable answers rather than demo-style text-to-SQL.
• Enterprise schemas are wide enough that naive whole-warehouse prompting degrades accuracy and latency.
• Ambiguous terms, synonyms, and unsupported metric-dimension combinations must be detected instead of guessed.
• Compound analytics needs consistent plan/result replay across reruns.
• The product must stay interactive despite LLM generation, warehouse execution, and multi-hop workflows.

Architecture

Tellius advocates a deterministic analytics-agent stack: a pre-parser extracts intent/entities/time into a constrained schema; an LLM-assisted planner proposes a plan at temperature 0; a validator normalizes it into a typed AST checked against governed semantic constructs and policy; a compiler emits SQL/MDX/GraphQL; execution uses plan/result fingerprints, caches, latency budgets, fallbacks, and run telemetry. Clarification, preference memory, inline lineage, admin-governed feedback, and semantic dictionaries keep user-facing answers auditable and reusable.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  chat-or-api-app["Chat or API app"] --> agent-runtime["Agent runtime"]
  agent-runtime --> pre-parser["Pre-parser"]
  pre-parser --> ambiguity-detector["Ambiguity detector"]
  ambiguity-detector --> clarification-manager["Clarification manager"]
  clarification-manager --> preference-memory["Preference memory"]
  clarification-manager --> chat-or-api-app
  pre-parser --> planner["Planner"]
  planner --> plan-validator["Plan validator"]
  plan-validator --> typed-ast["Typed AST"]
  plan-validator --> semantic-model-repository["Semantic model repository"]
  plan-validator --> policy-engine["Policy engine"]
  typed-ast --> compiler["Compiler"]
  compiler --> policy-engine
  compiler --> warehouse-executor["Warehouse executor"]
  warehouse-executor --> plan-and-result-cache["Plan and result cache"]
  warehouse-executor --> inline-explanation-layer["Inline explanation layer"]
  inline-explanation-layer --> chat-or-api-app
  inline-explanation-layer --> feedback-governance["Feedback governance"]
  feedback-governance --> semantic-model-repository
  semantic-model-repository --> semantic-dictionary["Semantic dictionary"]
  semantic-dictionary --> chat-or-api-app
  plan-and-result-cache --> replay-harness["Replay harness"]
  replay-harness --> observability-pipeline["Observability pipeline"]
  agent-runtime --> observability-pipeline
  warehouse-executor --> observability-pipeline

Components

Data flow

A user enters a natural-language analytics request through a chat UI or API. The pre-parser extracts intent, entities, and time references, and an ambiguity detector checks whether terms or mappings require a clarification. The planner proposes a typed plan, which the validator checks against the semantic-model repository, allowed metric-dimension pairs, date rules, joins, row/column policies, and user permissions. The compiler emits SQL/MDX/GraphQL, execution uses deterministic plan/result cache keys and latency budgets, and the UI returns an answer with preamble, lineage, policy footprint, cache/fallback status, and feedback hooks. Observability carries run IDs and per-stage telemetry so teams can replay, diff, monitor drift, and update governed learnings.

Why this architecture

The system separates language from logic because analytics correctness depends on deterministic plans, governed semantics, explicit policy checks, and replayability. It rejects opaque chain frameworks and prompt-only execution in favor of a thin auditable runtime where the LLM proposes but validators, compilers, policies, caches, and telemetry control what executes.

5 supporting quotes

“from Building Agentic AI Analytics ‍ ## TL;DR Chat-based analytics requires way more than just an LLM turning text into SQL. Governed, trustworthy AI analytics requires: a deterministic plan boundary (LLMs can propose, but a validator normalizes”article, stated

“the same question yields different queries. What goes wrong if you don’t plan for it: Natural language is for understanding; execution must be deterministic. When a single prompt is allowed to parse, plan, enforce policy, and explain, tiny wording changes ripple into different”article, stated

“understanding and business semantics, randomness bleeds into logic. Why it happens in production: LLMs are good at proposing, not at enforcing rules. If you let stochastic generation decide dates, join paths, or metric–dimension pairs, the same question yields different”article, stated

“compiler then emits SQL/MDX/GraphQL from the plan. Keep language for explanations; keep logic deterministic. ## 4) Ambiguity is a state to manage ”article, stated

“continue or degrade gracefully without corrupting the whole run._ What to put in place: Keep the execution layer thin and auditable. Treat the planner output as a typed AST that must validate against schema and policy before any tool runs. Make retries”article, stated

Models used: LLM planner · GPT-4 · GPT-4o · Text-to-SQL models

Scale

• Tellius reports abstraction-layer overhead of 73% memory, 20% latency, and 15% cost in framework-based approaches.
  “Production measurements are sobering: 73% memory overhead, 20% higher latency, 15% higher costs from abstraction layers.”article, stated
• A cited Octomind example spent 12 months with LangChain before removing it.
  “Octomind spent 12 months with LangChain before ripping it out—they were spending more time debugging the framework than building features.”article, stated
• A 150-column table can consume about 25% of GPT-4's context window.
  “a single 150-column table can consume ~25% of GPT-4’s context—so accuracy and latency degrade fast.”article, stated
• Domain scoping is cited as exceeding 95% accuracy in AWS/Cisco reporting.
  “Scope by domain instead of whole-warehouse prompts (AWS/Cisco report >95% accuracy when narrowed).”article, stated
• AI pilot failure rates are cited near 95%.
  “industry surveys put AI pilot failure rates near 95%.”article, stated
• The latency budget target includes planning under 150 ms, compilation under 80 ms, cache hits under 100 ms, and warehouse execution under 700 ms.
  “Set latency budgets per hop—planning <150ms, compilation <80ms, cache hits <100ms, warehouse execution <700ms.”article, stated
• Tinybird is cited as finding 3.2-second query generation, 1.5-2x more data read, 6-7 second total processing, and costs up to $4 per answer.
  “best models needing 3.2 seconds for query generation alone, with AI-generated queries reading 1.5-2x more data than human-written ones.”article, stated

  “Total processing reaches 6-7 seconds, killing interactive use cases. Costs escalate to $0.40+ per completion, with worst cases hitting $4.00 per answer.”article, stated
• Sierra's tau-bench is cited as showing GPT-4o agents below 50% on realistic tasks and near 25% consistency across attempts.
  “Sierra’s τ-bench reports GPT-4o agents solving <50% of realistic tasks, dropping to ~25% when they must be consistent across attempts.”article, stated
• Enterprise schemas are cited as averaging 300+ columns versus Spider's 28.
  “Schema complexity kills naive approaches—enterprises average 300+ columns versus Spider's 28.”article, stated
• Cross-functional coordination is cited as a struggle for 74% of businesses.
  “74% of businesses struggle with the necessary collaboration across legal, IT, and compliance teams.”article, stated

Alternatives considered (roads not taken)

Use generic LLM chain frameworks as the runtime

Rejected because: The article says generic frameworks hide control flow, retries, timeouts, caching, and state, making production analytics hard to debug.

Let prompt text drive execution directly

Rejected because: The same question can yield different dates, joins, and metric rules when generative randomness selects logic paths.

Guess ambiguous mappings automatically

Rejected because: Ambiguity can produce wrong slices of data, so the system should stop and ask targeted clarification questions when confidence is low or mappings tie.

Scope prompts over the whole warehouse

Rejected because: Wide enterprise schemas hurt accuracy and latency; Tellius recommends domain scoping backed by a semantic model repository.

Hide explanations in logs

Rejected because: Trust and auditability require inline preambles, lineage, policy filters, cache status, and fallbacks.

Cache by raw natural language

Rejected because: Raw NL cache keys miss equivalent validated plans and amplify consistency problems, so plan/result caches should use normalized fingerprints.

Patterns

structured-output-contract (planning)

Tellius turns LLM proposals into typed AST plan artifacts that must validate against schema, semantic rules, and policies before execution.

guardrails-io-filtering (governance)

The validator and policy-aware compiler enforce whitelisted joins, metric-dimension pairs, RLS/CLS, column masking, and access checks before queries run.

confidence-thresholding (disambiguation)

The ambiguity detector uses confidence thresholds and separation tests to decide when to stop execution and ask a clarification question.

agent-memory (personalization)

Tellius stores clarified terminology as scoped preferences with TTL so future sessions can reuse user/team/org choices without re-asking.

semantic-caching (consistency)

The system normalizes validated plans and hashes them with semantic-model versions so plan/result caches and replay harnesses operate on stable fingerprints.

observability-tracing (operations)

Every analytics run carries IDs and per-stage records across parsing, planning, validation, compilation, execution, rendering, cache decisions, fallbacks, and policies.

latency-optimization (performance)

Tellius sets per-hop latency budgets, caches plans/results, parallelizes independent subqueries, caps concurrency, and falls back to aggregates when estimates exceed budgets.

eval-driven-development (quality)

Tellius recommends replay harnesses, drift checks, semantic correctness tests, factual consistency across rephrasings, and domain-specific evaluation with real policies.

human-in-the-loop-gate (governance)

High-risk decisions and organization-wide feedback changes remain gated through humans or admin approval until metrics are green.

Tradeoffs

Draw a deterministic plan boundary.

Chose LLM proposes, validator normalizes and gates execution over LLM-generated text directly deciding joins, dates, and metric logic.

Rationale: The boundary keeps language flexible while making execution repeatable, governed, and auditable.

Keep the runtime thin and auditable.

Chose Explicit plan artifacts and observable execution over Generic chain abstractions with hidden defaults.

Rationale: Analytics teams need to diagnose who chose a tool, which retries fired, and why a wrong join happened.

Clarify instead of guessing.

Chose Targeted clarification and scoped preference memory over Automatic low-confidence disambiguation.

Rationale: A short clarification prevents wrong slices of business data and can be reused through memory when scoped correctly.

Scope analytics by domain.

Chose Domain-scoped prompts backed by semantic models over Whole-warehouse prompts or raw-schema prompting.

Rationale: Domain scoping improves accuracy and latency by shrinking schema complexity and making business meanings explicit.

Use semantic fingerprints for caching and replay.

Chose Normalized plan/result fingerprints with semantic-model version over Raw natural-language cache keys.

Rationale: Validated plan fingerprints support consistency, replay, drift detection, and cache hits across equivalent phrasing.

Expose inline transparency.

Chose Preambles, lineage, policy footprints, cache status, and fallbacks in the answer over Keeping explanations in logs or aggregate feedback stores.

Rationale: Regulated analytics users and internal risk teams need visible audit evidence, not hidden implementation traces.

Fall back when exact execution exceeds budgets.

Chose Aggregated views/cubes with clear disclosure over Always running the full expensive query path.

Rationale: Interactive analytics must protect latency and cost while making approximations or fallbacks explicit.

Gotchas (what bit them)

Framework abstraction hides the facts engineers need during incidents.

Root cause: Generic chain runtimes hide retries, timeouts, caching, state, and control flow behind opaque defaults.

Resolution: Keep execution thin and auditable, with explicit typed plan artifacts and observable retries.

Teams blur agents, workflows, and apps.

Root cause: Stakeholders use the same words for runtime, versioned plan templates, and user-facing surfaces.

Resolution: Define a canonical vocabulary with separate meanings for agent, workflow, and app.

Prompt-only planning produces nondeterministic business logic.

Root cause: Generative randomness can pick different dates, joins, time windows, and rankings for the same business question.

Resolution: Split understanding from execution and validate typed plans against schema and policy before compiling queries.

Ambiguous business language returns the wrong slice of data.

Root cause: Synonyms collide, mappings tie, teams lack shared defaults, and raw schemas do not encode business meanings.

Resolution: Use ambiguity detection, targeted clarification, preference memory, and shared defaults in the semantic model.

Compound analyses drift across reruns.

Root cause: Relative windows, session defaults, missing tie-breakers, and raw-NL cache keys make equivalent questions produce different outputs.

Resolution: Use canonical plans, explicit time ranges, deterministic ORDER BY clauses, stable cache keys, and replay harnesses.

Correct answers lose trust when users cannot see why.

Root cause: Final-result UIs hide dataset choices, metric definitions, time windows, policy filters, cache status, and feedback effects.

Resolution: Show preambles, lineage, policy footprints, fallbacks, and governed feedback inline.

Cold start looks like model failure but is often catalog invisibility.

Root cause: Users do not know valid metrics, dimensions, pairings, periods, cohorts, or governed terminology.

Resolution: Publish a searchable semantic dictionary, role-specific starter workflows, and schema-aware autocomplete constrained by permissions.

Latency and cost make correct answers feel unusable.

Root cause: Broad prompts, naive compilation, no hop budgets, and expensive generated SQL create long-tail delays and high scan costs.

Resolution: Set per-hop latency budgets, use two-tier caching, push down predicates, cap concurrency, and disclose aggregate fallbacks.

Without end-to-end telemetry, teams cannot reproduce why a number changed.

Root cause: Operational signals are fragmented and lack stable request IDs, per-stage timing, mapping details, policy decisions, or replay artifacts.

Resolution: Thread run IDs through every stage, log policy/cache/fallback decisions, and build dashboards for drift, P95, accuracy, and failing mappings.

Evaluation

Tellius evaluates agentic analytics through domain-specific tests over the organization's own semantic layer and policies, replay harnesses for compound query drift, latency/cost dashboards, clarification metrics, and observability that treats missing telemetry as failure. The source also distinguishes academic NL2SQL benchmarks from real enterprise schemas and governed business definitions.

Metrics: Clarification rate greater than zero · Post-clarification accuracy compared with guessing · Repeat clarifications per user · Replay drift by plan fingerprint · P95 latency by dataset/workflow · Accuracy versus clarification rate · Policy and learning churn · Top failing mappings · Semantic correctness · Factual consistency across rephrasings · Bytes scanned · Cache hit rate · Hop latencies · Total cost per insight

Outcomes

• Tellius reports implementing typed plans that validate before execution.
• The platform emits metric definitions, time windows, lineage, cache/fallback notes, and policies with every answer.
• The product treats feedback as governed learnings instead of unreviewed aggregate counters.
• The article claims these patterns have been implemented and iterated in production.
• The case provides concrete SLO-style latency, cost, observability, and consistency targets for production analytics agents.

The source cites production measurements including 73% memory overhead, 20% higher latency, 15% higher costs from abstraction layers, 150-column tables consuming about 25% of GPT-4 context, >95% domain-scoped accuracy from AWS/Cisco reporting, 3.2s query-generation benchmarks, 6-7s total processing, and per-hop targets such as planning under 150 ms.