Self-Improving Agent Through LLM-Based Session Analysis

factory · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI coding agent product analytics and self-improvement

Factory needed to understand where Droid users experienced friction or delight across thousands of sessions without requiring humans to read private user conversations.

Constraints:

Architecture

Factory built Signals, a daily batch pipeline that fetches Droid sessions from BigQuery, filters meaningful agentic interactions, analyzes them with LLM and embedding-based methods, extracts abstract facets plus friction/delight categories, clusters evolving patterns, stores aggregate results in BigQuery, posts Slack reports, correlates signals with logs and releases, and files Linear tickets when friction crosses thresholds so Droid can implement fixes subject to human PR approval.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Droid sessions"] --> B["BigQuery session store"]
  B --> C["Signals batch pipeline"]
  C --> D["LLM and embedding analysis"]
  D --> E["Facet and friction taxonomy"]
  E --> F["BigQuery analysis tables"]
  C --> G["Observability logs"]
  C --> H["Release notes"]
  C --> I["Slack reports"]
  C --> J["Linear ticket automation"]
  J --> K["Droid implementation loop"]
  K --> L["Human PR approval"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Droid sessions User-agent interaction records to be analyzed without exposing raw content to humans. BigQuery session store
BigQuery session store Stores sessions and receives historical Signals outputs for analysis. Droid sessions, Signals batch pipeline, BigQuery analysis tables
Signals batch pipeline Daily batch process that filters sessions and sends them for LLM/embedding analysis. BigQuery session store, LLM and embedding analysis, Observability logs, Release notes, Slack reports, Linear ticket automation
LLM and embedding analysis Extracts facets, friction, delight, abstract patterns, and clustering proposals. Signals batch pipeline, Facet and friction taxonomy
Facet and friction taxonomy Evolving schema of session facets, friction categories, and delight categories. LLM and embedding analysis, BigQuery analysis tables
Observability logs Backend error signals correlated with session friction windows. Signals batch pipeline
Release notes New release context used to detect regressions after CLI changes. Signals batch pipeline
Slack reports Daily team pulse on friction, delight, and changing patterns. Signals batch pipeline
Linear ticket automation Files improvement tickets when friction crosses thresholds. Signals batch pipeline, Droid implementation loop
Droid implementation loop Picks up tickets, implements fixes, reviews PRs, and awaits human merge approval. Linear ticket automation, Human PR approval
Human PR approval Final approval gate before Droid-generated fixes merge. Droid implementation loop

Data flow

Sessions from the past 24 hours are fetched from BigQuery, filtered to at least 30 agentic steps, sent to OpenAI's batch API for LLM/embedding analysis, decomposed into facets and friction/delight signals, clustered to propose new categories, written back to BigQuery and Slack, correlated with observability logs and release notes, and converted into Linear tickets when friction patterns cross thresholds.

3 supporting quotes
“Signals runs as a daily batch process designed for scale and cost efficiency.”article, stated
“Sessions from the past twenty-four hours get fetched from BigQuery, filtered to those with at least thirty agentic steps”article, stated
“Results flow to BigQuery for historical analysis and to Slack for daily reports.”article, stated

Why this architecture

Factory used batch LLM/embedding analysis because raw metrics missed qualitative session problems, while privacy constraints required abstracting sessions into aggregate facets and recurring patterns before humans saw results.

3 supporting quotes
“Traditional analytics would score this session as a success. A human reviewer would call it a near-disaster.”article, stated
“The model never surfaces raw conversation content to human analysts.”article, stated
“Batching lets us take advantage of lower API costs, and the twenty-four hour processing window works well”article, stated

Models used: LLM judges · embedding models · OpenAI batch API

Scale

Alternatives considered (roads not taken)

Rely on task-completion metrics alone

Rejected because: Metrics missed qualitative failure modes such as repeated rephrasing and frustration.

“The metrics look fine: forty-five tool calls, twelve-minute session, task completed.”article, stated
“Traditional analytics would score this session as a success. A human reviewer would call it a near-disaster.”article, stated

Have humans read raw sessions

Rejected because: Factory wanted human-like qualitative judgment without exposing user conversations to analysts.

“without actually requiring humans to read through thousands of daily sessions.”article, stated
“without anyone at Factory reading a single user conversation.”article, stated

Static facet and friction taxonomies

Rejected because: New patterns such as branch switches and context churn emerged from clustering and needed to become first-class categories.

“The facet schema itself evolves over time through semantic clustering.”article, stated
“The "context churn" category didn't exist in our original design.”article, stated

Patterns

eval-driven-development (evaluation)

Signals uses LLM judges to evaluate real sessions for friction and delight, then drives product fixes when patterns cross thresholds.

“Signals uses LLMs as judges to analyze Factory sessions at scale, identifying moments of friction and delight that metrics alone would miss.”article, stated
“Signals implements what AI researchers call recursive self-improvement—systems that autonomously enhance their own capabilities. When patterns cross a threshold, it files Linear tickets automatically. Droid picks up those tickets, implements fixes, and reviews its own PRs.”article, stated

observability-tracing (observability)

Signals correlates abstracted session friction with backend errors, release notes, BigQuery history, and Slack reporting.

“We pipe error logs from our observability system into Signals' analysis. When a session shows friction, Signals can cross-reference with backend errors that occurred during the same time window.”article, stated
“Daily Slack reports now include correlation with our release notes. When a new CLI version ships, Signals automatically tracks whether friction patterns change in subsequent sessions.”article, stated

context-engineering (agent-context)

Signals discovered context churn as a first-class category when users repeatedly changed context windows in ways correlated with abandonment.

“When the LLM examined the cluster, it identified the common thread: users repeatedly modifying their context window in ways that correlated with eventual abandonment. That pattern became a first-class friction category.”article, stated
“Context churn turned out to be the leading indicator of eventual frustration. When users repeatedly add and remove the same file from context, something fundamental is wrong.”article, stated

Tradeoffs

Run Signals as a daily batch.

Chose 24-hour batch processing through OpenAI's batch API over realtime alerting for every session.

Rationale: Factory was looking for patterns, not instant alerts, and batching reduced cost.

“We analyze thousands of sessions daily, dynamically adjusted based on a token budget. Batching lets us take advantage of lower API costs, and the twenty-four hour processing window works well since we're looking for patterns, not real-time alerts.”article, stated

Use abstraction rather than raw transcript review.

Chose facets, clustered signals, and aggregate statistics over human analysts reading user conversations.

Rationale: The abstraction layer preserves privacy while still surfacing product patterns.

“The LLM extracts patterns while omitting specific user content. Individual results flow into aggregate statistics that only become meaningful at scale.”article, stated

Keep human approval before merge.

Chose Droid-generated fixes with human PR approval over fully autonomous production changes.

Rationale: The improvement loop is automated through ticketing and PRs, but final merge still has human oversight.

“It's not fully automated yet. A human still approves the PR before merge. But the path from "users are frustrated by X" to "here's a fix for X" now happens without anyone manually triaging, assigning, or even noticing the pattern.”article, stated

Gotchas (what bit them)

Success metrics hid near-failure sessions.

Root cause: Task completion and tool-call counts did not capture repeated rephrasing or user frustration.

Resolution: Use LLM judges to identify friction/delight moments with citations and aggregate them across sessions.

“But buried in that session is a three-minute loop where the user rephrased the same request five times, growing increasingly frustrated, before the agent finally understood what they wanted.”article, stated

Raw user sessions could not be exposed for manual product analytics.

Root cause: Conversation content is sensitive and numerous enough that human review would be both invasive and unscalable.

Resolution: Extract abstract patterns, facets, and aggregate statistics without surfacing raw conversation content.

“Signals resolves this through multiple layers of abstraction. The LLM extracts patterns while omitting specific user content.”article, stated
“Individual results flow into aggregate statistics that only become meaningful at scale. And patterns only surface when they appear across enough distinct sessions to prevent identifying individual users.”article, stated

Release regressions could be hard to notice manually.

Root cause: Friction can emerge as an aggregate pattern after a product change rather than an obvious single failure.

Resolution: Correlate daily friction reports with release notes and backend errors.

“A release that changed how file context was assembled correlated with a spike in context churn friction the following day.”article, stated

Evaluation

Factory evaluates Droid experience by LLM-judged friction/delight, session facets, error-log correlation, release-note correlation, abandonment predictors, and before/after changes in friction rates.

Metrics: friction and delight moments · session facets · context churn · rephrasing count · completion/abandonment · release-correlated regression · friction-rate change after fixes

“Every session gets decomposed into structured metadata. We call these facets: the programming languages involved, the primary intent, how many tool calls were confirmed, whether the session ended in success or abandonment, what frameworks were referenced.”article, stated
“When we shipped a change to how Droid handles ambiguous requests, the "repeated rephrasing" friction rate dropped by thirty percent”article, stated

Outcomes

Signals processes thousands of daily sessions through a 24-hour batch pipeline and uses aggregation thresholds to protect individual privacy.

3 supporting quotes
“Context churn turned out to be the leading indicator of eventual frustration. When users repeatedly add and remove the same file from context, something fundamental is wrong.”article, stated
“We've caught regressions this way. A release that changed how file context was assembled correlated with a spike in context churn friction the following day.”article, stated
“When patterns cross a threshold, it files Linear tickets automatically. Droid picks up those tickets, implements fixes, and reviews its own PRs.”article, stated