Context Engine for Continual Learning in AI Coding Agents

applied_commute · Tech · 2026 · source Agentic: yes · promotional bias: high · summary-only (no fetched source) · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI coding-agent memory and continual learning

Applied Compute wanted coding agents to remember enterprise-specific development context, refine it from production traces, and retrieve it during future tasks instead of repeatedly losing institutional knowledge.

Constraints:

• The system had to collect traces across multiple coding-agent tools without disrupting developer workflows.
• Memory needed to capture both specific fixes and reusable procedures.
• Evaluation had to distinguish useful memories from distracting or stale memories.
• The case is vendor-authored and self-reported, so claims require provenance-aware interpretation.

Architecture

Applied Compute built Context Engine around a Remember/Refine/Retrieve loop. Production coding-agent traces from Cursor, Claude Code, and Codex flow into Applied Compute Logs, are refined daily into hierarchical Contextbase memories with deduplication, pruning, folders, and indexes, and are exposed back to coding agents through an MCP server. Human feedback and LLM-as-judge monitoring feed quality signals back into refinement.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  coding-agent-tools["coding agent tools"] --> applied-compute-logs["Applied Compute Logs"]
  applied-compute-logs --> remember-stage["Remember stage"]
  remember-stage --> refine-stage["Refine stage"]
  refine-stage --> contextbase["Contextbase"]
  contextbase --> mcp-retrieval-server["MCP retrieval server"]
  mcp-retrieval-server --> coding-agent-tools
  human-feedback-loop["human feedback loop"] --> refine-stage
  human-feedback-loop --> critical-memory-rate-monitor["Critical Memory Rate monitor"]
  contextbase --> aclbench-evaluation-harness["ACLBench evaluation harness"]
  critical-memory-rate-monitor --> aclbench-evaluation-harness

Components

Data flow

Coding-agent sessions are logged into ACL. The Remember stage ingests traces, Refine extracts low-level memories and higher-level procedural memories while deduplicating and pruning, and Retrieve exposes the Contextbase to agents through MCP. Engineer flags, Critical Memory Rate monitoring, and ACLBench results feed back into memory quality.

Why this architecture

The architecture treats production traces as the raw material for continual learning, then separates raw logging from refinement and runtime retrieval. MCP makes context injection portable across coding agents, while human feedback and benchmarks limit the risk that memory becomes stale or distracting.

Models used: Cursor coding agents · Claude Code · Codex · LLM-as-a-judge monitor · Claude Opus 4.6 benchmark agent

Scale

• Applied Compute tracked coding-agent interactions across Cursor, Claude Code, and Codex for several months.
  “The company tracked coding agent interactions across multiple platforms (Cursor, Claude Code, and Codex) for several months”summary, stated
• Refinement runs daily to incorporate new traces and feedback.
  “This refinement happens on a daily basis, incorporating new traces and feedback continuously.”summary, stated
• Critical Memory Rate roughly doubled over a two-week period.
  “The production results showed the Critical Memory Rate moving average climbing from under 10% to around 20% over a two-week period.”summary, stated
• Monitoring filters to buckets with more than 15 ACL-Wiki retrieve calls.
  “The company filters results to buckets with more than 15 ACL-Wiki retrieve calls”summary, stated
• ACLBench used approximately 25 hand-selected chunks.
  “Manual selection was used to identify approximately 25 chunks with clear memory needs”summary, stated

Alternatives considered (roads not taken)

coding agents without memory

Rejected because: The benchmark explicitly compared Contextbase against no-memory baselines and found gains on memory-relevant tasks.

larger-scale automated benchmark generation

Rejected because: LLM-as-a-judge filtration did not reliably identify tasks that distinguished good and bad memory systems.

flat key-value memory storage

Rejected because: The implementation organized memories into folders and indexes, implying a need for navigable hierarchical knowledge.

Patterns

agent-memory (context-management)

Context Engine turns production traces into a Contextbase that coding agents can query at runtime.

observability-tracing (operations)

The system logs coding-agent interactions across multiple tools as the substrate for memory creation.

data-quality-loop (continual-learning)

Engineer flags about missing, useful, or distracting memories feed back into the refinement stage.

llm-as-judge (evaluation)

Applied Compute uses an LLM judge with majority vote to label whether retrieved memories were critical.

eval-driven-development (evaluation)

ACLBench holds the model fixed and evaluates the memory system directly on production-derived tasks.

tool-use-orchestration (integration)

The Contextbase is exposed to coding agents through an MCP server for standardized retrieval.

Tradeoffs

Evaluate the memory system while holding the model fixed.

Chose memory/no-memory comparison in ACLBench over general model benchmarking where model and memory effects are entangled.

Rationale: This isolates whether the Contextbase itself improves task performance.

Use LLM-as-a-judge monitoring for nuanced memory utility.

Chose automated judge with trajectory context and majority vote over manual review of every retrieval.

Rationale: It scales production monitoring of a subtle quality signal, while introducing model-dependence and evaluation noise.

Refine memories daily rather than continuously on every trace.

Chose daily Refine cycle over more frequent updates.

Rationale: Daily processing balances freshness and computational cost, though faster updates could capture changes sooner.

Use human feedback despite trace-derived memory automation.

Chose engineer flagging for missing, useful, or distracting memories over memory creation driven only by traces.

Rationale: Human judgment supplies a utility signal that raw traces do not provide.

Gotchas (what bit them)

Critical Memory Rate alone could overstate the value of retrieval.

Root cause: A memory can be retrieved without helping, and the metric does not by itself distinguish helpful memory from harmful distraction.

Resolution: Build ACLBench with memory tasks and distractor tasks to measure both lift and regression.

Automated benchmark generation was not reliable enough for memory-system evaluation.

Root cause: Frontier models were poor judges of which tasks would distinguish strong and weak memory systems.

Resolution: Use manual selection of about 25 chunks with clear memory needs.

Only a minority of retrievals were critical even after improvement.

Root cause: Not every coding task requires institutional memory, so retrieval has a natural ceiling and cost-benefit questions remain.

Resolution: Track Critical Memory Rate while acknowledging the ceiling and evaluate distractor tasks for regression.

The case leaves several production failure modes unspecified.

Root cause: The vendor report does not detail latency, retrieval failures, refinement cost, staleness, or privacy/security controls.

Resolution: Treat the implementation as promising but not fully independently validated for operational risk.

Evaluation

Applied Compute used production Critical Memory Rate, LLM-as-judge majority voting, filtered production buckets, ACLBench memory/distractor tasks, and benchmark runs with Claude Opus 4.6 in a minimal coding-agent harness.

Metrics: Critical Memory Rate moving average under 10% to around 20% · buckets with more than 15 ACL-Wiki retrieve calls · approximately 25 curated ACLBench chunks · three samples per item · no significant regression on distractor tasks

Outcomes

• Critical Memory Rate roughly doubled over two weeks.
• Contextbase lifted benchmark performance in every memory-utility category.
• User preference tasks showed especially large gains.
• Distractor tasks showed no significant regression.

The deployment covered internal production use over several months of Cursor, Claude Code, and Codex traces, with a daily refinement loop and about 25 curated benchmark chunks.