Building Production Analytics Agents with Semantic Layer Integration

wobby · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production analytics agents for business intelligence and text-to-SQL

Wobby needed analytics agents that business teams would actually use for ad-hoc questions over data warehouses. The challenge was not SQL syntax alone, but adoption in everyday channels, business-context grounding, semantic-layer governance, latency, testing, and avoiding raw-data access or brittle agent frameworks.

Constraints:

• Business users need answers in existing collaboration channels, not just a standalone chat UI.
• Text-to-SQL fails when agents see raw schemas without business logic, metrics, dimensions, and governance context.
• Academic text-to-SQL benchmarks do not prove production readiness when 1 in 5 queries can still fail.
• Golden SQL templates can create large review burdens and do not match day-to-day BI workflows.
• Analytics agents need low latency, but simple business questions can still require complex data work.
• Semantic layer changes must be tested and governed before reaching production.

Architecture

Wobby built three analytics agents around an agent-first semantic layer. Quick handles everyday questions, Deep handles open-ended research, and Steward helps build and maintain the semantic layer. The analytics agents operate over the logical semantic layer rather than raw data, while Steward can inspect physical-layer information and suggest dimensions, metrics, and KPIs. Slack and Teams integrations drive adoption and expose real failures to support staff. The codebase uses clean interfaces and API wrappers instead of heavyweight agent frameworks, tests obvious bugs in CI before evals, evaluates agent trajectories with composite quality metrics, and uses caching, compression, pruning, and progressive disclosure for latency.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  business-users["Business users"] --> slack-and-teams-channels["Slack and Teams channels"]
  slack-and-teams-channels --> quick-agent["Quick agent"]
  slack-and-teams-channels --> deep-agent["Deep agent"]
  quick-agent --> semantic-layer["Semantic layer"]
  deep-agent --> semantic-layer
  semantic-layer --> sql-compiler["SQL compiler"]
  sql-compiler --> data-warehouse["Data warehouse"]
  data-warehouse --> physical-data-layer["Physical data layer"]
  steward-agent["Steward agent"] --> physical-data-layer
  steward-agent --> semantic-layer
  steward-agent --> semantic-layer-environments["Semantic layer environments"]
  semantic-layer-environments --> semantic-layer
  customer-support-observers["Customer support observers"] --> slack-and-teams-channels
  customer-support-observers --> trace-and-eval-feedback["Trace and eval feedback"]
  quick-agent --> trace-and-eval-feedback
  deep-agent --> trace-and-eval-feedback
  trace-and-eval-feedback --> observability-wrappers["Observability wrappers"]

Components

Data flow

Business users ask analytics questions through Slack, Teams, or Wobby's interface. Quick or Deep receives the request, uses business-context prompts and the semantic layer to reason over metrics, dimensions, and logical entities, then compiles requests through Wobby's SQL compiler to the relevant dialect. Steward helps create and update semantic-layer entities from documentation or physical-layer inspection, with changes tested in environments before merge. Failures observed in customer channels become test or eval items. Traces flow through observability wrappers that call LangFuse and Logfire.

Why this architecture

Wobby found that business-context grounding mattered more than raw SQL generation. Raw schemas and golden-query RAG did not match how BI teams govern metrics, while a semantic layer matched existing BI concepts and kept analytics agents away from raw data. A separate Steward agent lets semantic-layer quality improve over time, while Slack/Teams channels make adoption and failure discovery social. Clean architecture and wrappers avoided early framework churn and let Wobby keep responsibilities in the app layer or agent layer as needed.

Models used: LLM analytics agents · LLM judges · OpenAI harmony message format · small language models

Scale

• Wobby's goal is business-team insight from a data warehouse in under one minute.
  “Wii helps business teams gets insight out of their data warehouse in under one minute and that's the goal”transcript, stated
• The system has connectors to semantic layers, catalogs, and databases.
  “we have connectors to various um semantic layers, cataloges and databases.”transcript, stated
• The presenter says Wobby had been building the system for about two years.
  “we'd had when we started building WBY about two years ago.”transcript, stated
• Text-to-SQL benchmark performance around 80% still means one in five queries can go wrong, which Wobby considers unshippable.
  “performance of their agents stalls at 80%. And so the real challenge is how can we um like up this 80% to 100%. because uh it's still one of five queries that goes wrong.”transcript, stated
• The first RAG/golden-query approach produced over 100 similar SQL queries that increased analyst review work.
  “what we ended up with was 100 plus similar looking SQL queries and that created even more work for the analysts to review.”transcript, stated

Alternatives considered (roads not taken)

Parameterized golden-query RAG over reusable SQL templates

Rejected because: It produced too many similar queries and review work, and did not align with BI teams' day-to-day process.

Use only existing semantic layers such as Cube or dbt Metric Flow

Rejected because: Wobby needed an agent-first integration layer across many semantic layers, APIs, and SQL dialects.

Adopt MCP/tool-server and multi-agent-network frameworks

Rejected because: Separate tool servers, ReAct prompting, and multi-agent networks added context, complexity, and debugging burden for this product.

Build a cross-data-source query engine first

Rejected because: Wobby stayed pragmatic by working with single data teams that already aggregate data in a warehouse, instead of solving cross-source joins initially.

Patterns

multi-agent-orchestration (agent-architecture)

Wobby separates everyday analytics, open-ended analysis, and semantic-layer stewardship into Quick, Deep, and Steward agents with distinct access boundaries.

data-quality-loop (semantic-grounding)

The semantic layer encodes business logic, metrics, dimensions, and governed definitions so agents improve as teams add missing context and eval items.

tool-use-orchestration (agent-tools)

Analytics agents call semantic-layer and SQL-compiler capabilities, while Wobby tests tool outputs as software bugs before relying on evals.

eval-driven-development (quality)

Real channel failures and team usage become eval items, while composite trajectory metrics compare agent configurations over representative datasets.

latency-optimization (operations)

Wobby optimizes agent response time with progressive disclosure, prompt-side intent handling, schema compression, message pruning, JSON compression, caching, and interleaved thinking.

observability-tracing (operations)

Function decorators and wrappers capture LLM inputs/outputs and forward traces into both LangFuse and Logfire without coupling the app directly to one vendor API.

prompt-versioning (agent-logic)

Wobby prefers portable prompt-contained logic for this domain because it is easier to compare, test, and deploy than workflow topology changes.

Tradeoffs

Build an agent-first semantic layer

Chose Own integration layer, APIs, and SQL compiler over logical entities over Relying only on existing semantic-layer APIs or raw warehouse schemas.

Rationale: Wobby needed cross-platform interoperability and agent-first semantics for accurate, governed analytics behavior.

Prioritize Slack/Teams integration early

Chose Native collaboration-channel adoption work over Treating channel UX as a later chat-box wrapper.

Rationale: The channel experience created group adoption, peer examples, onboarding, and live failure capture, but required more engineering than expected.

Use tests before evals for early production bugs

Chose Layered automated tests for prompt/tool/API failures over Putting all quality checks into evals.

Rationale: Tests catch deterministic bugs cheaply in CI, while evals measure comparative agent performance on representative tasks.

Put more domain logic in prompts

Chose Portable prompt-based agent logic over Rigid workflow branching for this analytics domain.

Rationale: As instruction-following improved, prompts became easier to compare, A/B test, and deploy than many workflow paths.

Target single aggregated data warehouses first

Chose Warehouse-bounded deployments over Cross-source joins across many non-interoperable systems.

Rationale: Single data-team/warehouse contexts kept the problem bounded while cross-source query engines remained a future possibility.

Gotchas (what bit them)

Text-to-SQL benchmark performance hides production context failures.

Root cause: SQL syntax is not the hard part; agents need business context, governed metrics, and logical concepts.

Resolution: Move agents from raw SQL/data toward a semantic layer that encodes business logic.

Golden-query RAG can increase analyst review workload.

Root cause: A library of parameterized SQL examples created many similar queries rather than reusable BI abstractions.

Resolution: Replace query-template reuse with a semantic layer of metrics, dimensions, and logical business entities.

Putting basic tool bugs into evals makes debugging harder.

Root cause: Access-management failures, bad tool formatting, and missing API data are deterministic bugs, not agent-performance questions.

Resolution: Catch failure modes and happy paths in automated tests and reserve evals for comparing agent behavior and configurations.

Agent frameworks and multi-agent networks can add unnecessary context and debugging complexity.

Root cause: Wobby needed direct control over app-layer versus agent-layer responsibilities, and extra agents increased context load.

Resolution: Use clean interfaces, wrappers, a factory, and selective abstractions instead of adopting unsuitable frameworks wholesale.

A well-governed semantic layer is a prerequisite, not a nice-to-have.

Root cause: Many companies lack trustworthy data and shared definitions, so agents receive wrong context and output wrong answers.

Resolution: Work with data engineering teams to build semantic layers, governed datasets, and logical abstractions that agents can use safely.

Evaluation

Wobby uses layered tests for deterministic bugs, eval items from real user/channel failures, a Streamlit job-container eval app, trajectory analysis, multiple LLM judges, reasoning-first structured judging, and an Overall Agent Effectiveness-style composite score.

Metrics: Fixed tests for tool output, prompt formatting, service downtime, and semantic-search/access bugs. · Eval items added from customer-channel failures. · Tool trajectory counts by step and next tool call. · Multiple judges to reduce evaluator variance. · Reasoning-first structured output for judge explanations. · Overall Agent Effectiveness composite weighted score across trajectories.

Outcomes

• Wobby has production analytics agents for everyday questions, open-ended research, and semantic-layer stewardship.
• Slack and Teams integrations helped business users adopt the agents through peer examples and channel-native usage.
• The first golden-query RAG approach was replaced by a semantic-layer architecture aligned with BI processes.
• The team built testing, eval, latency, and observability practices around production agent behavior rather than benchmarks alone.

The source describes two years of development, production use in logistics and finance analytics contexts, and a goal of under-one-minute insights, but does not provide total query volume.