Building Production-Ready Coding Agents with Skills and Observability

langfuse_/_clickhouse · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Coding-agent skills for LLM observability integration

Langfuse needed coding agents to add observability, prompt management, and evaluation correctly across diverse user codebases, despite hundreds of documentation pages, outdated model pretraining, many integration choices, and limited visibility into what agents struggled with during setup.

Constraints:

• Langfuse is intentionally unopinionated infrastructure, so the correct setup depends on the user's application architecture and evaluation goals.
• The documentation surface had grown to hundreds of pages across several feature areas.
• Coding agents relied on stale pretraining data and hallucinated outdated SDK methods or CLI parameters.
• A generic request like adding tracing did not tell the agent which evaluation strategy, data region, prompt-management workflow, or best-practice setup was appropriate.
• Embedding full documentation inside a skill would duplicate dynamic content and create another stale copy.
• Auto-research can optimize the wrong behavior if the target function omits important constraints such as fetching current docs or preserving approval gates.

Architecture

Langfuse built a coding-agent skill that acts like an expert integration guide. The skill defines the desired interaction style, references product modules instead of duplicating docs, asks follow-up questions before implementation, exposes a sitemap and markdown documentation paths, wraps existing APIs in an agent-friendly CLI, and adds a natural-language documentation search endpoint. Langfuse then instruments coding-agent runs with traces, reviews those traces to improve the skill, creates basic LLM-as-judge evaluations over file-system and diff state, and uses bounded auto-research plus human review to propose skill improvements.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  developer-user[Developer user] --> coding-agent-runtime[Coding agent runtime]
  coding-agent-runtime --> langfuse-skill[Langfuse skill]
  langfuse-skill --> skill-style-guide[Skill style guide]
  langfuse-skill --> product-module-references[Product module references]
  langfuse-skill --> documentation-access-layer[Documentation access layer]
  documentation-access-layer --> documentation-sitemap[Documentation sitemap]
  documentation-access-layer --> markdown-documentation-endpoint[Markdown documentation endpoint]
  documentation-access-layer --> documentation-search-endpoint[Documentation search endpoint]
  coding-agent-runtime --> langfuse-cli[Langfuse CLI]
  langfuse-cli --> langfuse-apis[Langfuse APIs]
  coding-agent-runtime --> repository[Repository]
  coding-agent-runtime --> trace-instrumentation[Trace instrumentation]
  trace-instrumentation --> production-signal-loop[Production signal loop]
  documentation-search-endpoint --> production-signal-loop
  repository --> evaluation-templates[Evaluation templates]
  evaluation-templates --> llm-as-judge-checks[LLM-as-judge checks]
  production-signal-loop --> auto-research-optimizer[Auto-research optimizer]
  auto-research-optimizer --> target-function[Target function]
  auto-research-optimizer --> human-reviewer[Human reviewer]

Components

Data flow

A developer asks a coding agent to add Langfuse to a repository. The coding agent reads the Langfuse skill, asks clarifying questions, inspects product-module references, uses the sitemap or search endpoint to retrieve current documentation, invokes the Langfuse CLI over APIs, and edits the repository. The run is traced in Langfuse; traces and search telemetry are reviewed to discover errors, documentation gaps, new use cases, and skill improvements. Evaluation templates and LLM-as-judge checks compare filesystem and diff state before and after skill execution.

Why this architecture

Langfuse chose a skill-based architecture because users wanted coding agents to perform integrations instead of reading documentation, while stale model pretraining and a large, flexible product surface made naive agent implementation slow and error-prone. The skill provides formalized shortcuts and current references without becoming another stale documentation cache, and observability closes the loop by showing what agents actually did.

Models used: Claude Code · Coding agents · Langfuse · LLM-as-a-judge · Docs Q&A RAG stack · Langfuse CLI

Scale

• Langfuse had been building for about three years.
  “We started LangFuse like 3 years ago”transcript, stated
• The docs surface included 478 pages.
  “478 pages of documentation.”transcript, stated
• The product surface spanned five feature areas.
  “across like five different feature areas, and a lot of implementation flexibility”transcript, stated
• Langfuse positions itself as infrastructure that can ingest billions of traces.
  “if you ingest like billions of traces, we'll still work.”transcript, stated
• The target user base included thousands of community teams and cloud customers.
  “there are thousand of teams in the community, thousand of customers on our cloud product”transcript, stated
• Trace inspection was described as getting the team about 80% of the detail needed.
  “looking at traces still like gets you to like 80% of the detail.”transcript, stated
• The basic evaluation setup used five initial evaluation templates.
  “we just created like five uh different ones and this was already helpful.”transcript, stated
• Auto-research produced six suggestions, three of which were accepted.
  “we accepted three out of the six improvements that were suggested”transcript, stated
• One roadmap example involved analyzing user feedback on 100 executions.
  “I got user feedback on 100 different executions, what do they have in common?”transcript, stated

Alternatives considered (roads not taken)

Let coding agents rely on pretraining context

Rejected because: Langfuse interfaces changed, so pretraining caused agents to use outdated methods and hallucinated APIs.

Embed full documentation in the skill

Rejected because: Duplicating dynamic docs in user-space skills would recreate the same staleness problem as pretraining data.

Fetch documentation pages one by one

Rejected because: Agents looped through pages inefficiently, so Langfuse added a sitemap and natural-language search over documentation chunks.

Wait for a perfect universal eval setup

Rejected because: Use cases were too broad, so five basic evaluation templates were better than having no guardrail at all.

Default the data region to Europe

Rejected because: Agents do not mind additional configuration prompts, and US enterprises also cared about data region.

Optimize auto-research for speed measured by number of turns

Rejected because: The optimizer removed documentation fetching, which defeated the purpose of current references.

Patterns

observability-tracing (production-operations)

Langfuse instruments coding-agent runs and reviews traces to understand runtime behavior, discover new use cases, and improve existing skills.

tool-use-orchestration (agent-control)

The Langfuse skill gives coding agents a CLI over the existing API surface so agents can create evaluations, manage prompts, configure regions, and perform UI-like workflows programmatically.

rag-retrieval (retrieval)

The skill exposes sitemap and natural-language documentation search so agents retrieve relevant, current documentation chunks instead of relying on stale pretraining or full-page browsing.

context-compaction (context-management)

Langfuse references canonical documentation and uses progressive disclosure, sitemap navigation, markdown rendering, and search to avoid stuffing a full documentation cache into the skill.

eval-driven-development (quality)

Langfuse creates basic skill-evaluation templates and checks repository changes with LLM-as-judge before accepting skill modifications.

llm-as-judge (evaluation)

The skill's basic evals use natural-language statements evaluated by an LLM judge over filesystem and git diff state.

human-in-the-loop-gate (workflow)

Auto-research generates skill-improvement ideas, but humans review and accept only the useful suggestions; the team also wants approval gates before pushing prompts remotely.

prompt-versioning (prompt-operations)

Langfuse's prompt-management use case moves prompts from local git into managed prompt infrastructure and links prompt versions to production traces.

Tradeoffs

Use skills as a middle ground

Chose Formalized shortcuts that preserve agent flexibility over Rigid workflows or fully unconstrained agents.

Rationale: Skills make agents more reliable for known domains while still allowing multi-domain requests that would be awkward across multiple workflows.

Reference dynamic content

Chose Canonical documentation references, sitemap, markdown pages, and search over Duplicating extensive docs inside the skill.

Rationale: Referencing keeps the skill current and avoids another stale copy of fast-changing product information.

Add explicit region prompts for agents

Chose Ask the agent to determine the user's data region over Defaulting to Europe to reduce human setup friction.

Rationale: Extra environment variables are cheap for agents, and enterprise region assumptions caused wrong setups.

Expose search server-side

Chose Natural-language docs search endpoint over Only local documentation fetching from user machines.

Rationale: Search returns relevant chunks faster and lets Langfuse observe which questions agents ask, revealing documentation gaps.

Start with basic evals

Chose Five practical evaluation setups with LLM-as-judge checks over Waiting for a perfect evaluation framework across every Langfuse use case.

Rationale: The product serves too many application types for one perfect eval, but basic checks were enough to catch regressions when modifying the skill.

Keep human review in auto-research

Chose Human-reviewed recommendations over Autonomously applying all agent-suggested skill changes.

Rationale: Auto-research expanded exploration capacity, but imperfect target functions meant many ideas did not make sense.

Gotchas (what bit them)

Pretraining context became a liability as Langfuse changed.

Root cause: Models remembered old SDK methods and interfaces that no longer existed.

Resolution: Force agents toward current documentation through skills, sitemap, markdown docs, and search endpoints.

Agents implemented first, failed, and only then fetched docs.

Root cause: Without the skill, the coding agent was not primed to discover the right setup or documentation path before editing.

Resolution: Use the skill to ask follow-up questions, fetch current docs, and guide the agent toward the best-fit setup first.

Agents hallucinated CLI parameters.

Root cause: They generalized from words such as trace and prior CLI patterns instead of checking the actual command surface.

Resolution: Advertise the help flag more aggressively, accepting one more turn to ground the agent in real CLI options.

Duplicated skill content goes stale like pretraining data.

Root cause: Embedding dynamic docs inside the skill creates another representation of Langfuse that can drift from canonical documentation.

Resolution: Reference canonical docs and expose discovery mechanisms rather than copying large content into the skill.

Auto-research optimized away the behavior the skill was meant to preserve.

Root cause: The target function measured speed in turns, so the optimizer removed documentation-fetching notes.

Resolution: Design target functions to include current-documentation usage, prompt-version trace links, approval gates, and other best-practice constraints.

Skill distribution and updates remain operationally immature.

Root cause: Skills are cached in user/agent environments, package-management conventions are weak, and auto-upgrading depends on the coding agent platform.

Resolution: Explore timestamp checks, well-known skill discovery, and user alerts when installed skills are old.

Evaluation

Langfuse evaluates the skill by manually inspecting coding-agent traces, creating five basic scenario templates, using LLM-as-judge checks over filesystem and git diff state, checking for instrumentation and retrieval spans, tracking documentation search telemetry, and human-reviewing auto-research suggestions before changing the skill.

Metrics: Trace review coverage · Documentation search queries · OpenAI instrumentation present · Retrieval spans present · Filesystem state before and after skill run · Git diff state before and after skill run · Accepted auto-research suggestions · Number of turns · Data-region selection correctness · CLI parameter hallucinations

Outcomes

• The Langfuse skill became the advertised and primary recommended way to add Langfuse through coding agents.
• Coding-agent traces after the skill showed more detailed evaluations and structured tool-execution steps.
• The skill reduced the initial wrong-implementation-then-doc-fetch loop caused by stale pretraining.
• Trace review revealed concrete skill improvements, including region prompting and help-flag usage.
• The search endpoint both improved agent navigation and gave Langfuse telemetry about user/agent problems.
• Five basic eval templates were enough to catch regressions while modifying the skill.
• Auto-research generated six suggestions, three of which were accepted after human review.
• The team identified unresolved platform problems around skill distribution, caching, timestamping, and upgrades.

The source gives concrete scale: three years of product growth, 478 documentation pages, five product areas, billions-trace positioning, thousands of community/cloud users, five eval setups, three of six auto-research suggestions accepted, and future examples involving 100 feedback executions.