Solving Tool Confusion and Design Slop in Open Model Coding Agents

commandcode · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Open-model coding agent reliability and preference learning

CommandCode needed open coding models such as DeepSeek to work reliably in production agent harnesses, where malformed tool calls, repeated schema errors, design slop, and missing developer preferences made otherwise capable models feel unusable.

Constraints:

• Open models could repeat the same invalid tool call dozens of times instead of learning from schema errors.
• The platform processes very high token volumes, so repeated tool failures waste time and tokens.
• Repair logic must not simply hide errors; it should return results while teaching the model the expected schema.
• Developer preferences and design taste need to persist across repositories without manual rule writing.

Architecture

CommandCode added deterministic repair layers around coding-agent behavior: malformed tool calls are intercepted, normalized into valid calls when the intent is clear, and returned with repair hints so the model learns the expected schema; the same repair-pattern approach is extended to UI design slop; and a Taste system automatically creates, maintains, and applies skills/rules that capture developer preferences across repositories.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Coding model"] --> B["Tool-call repair layer"]
  B --> C["Tool execution harness"]
  B --> D["Repair hints"]
  D --> A
  B --> E["Repair-pattern logs"]
  E --> F["Design repair layer"]
  F --> A
  A --> G["Taste memory"]
  G --> H["Skill files"]
  H --> A

Components

Data flow

A coding model emits a tool call. CommandCode detects known malformed patterns, repairs the call into a valid schema, executes it, returns the result plus a repair hint, and tracks failure patterns across logs. Taste separately observes coding preferences and writes portable skill/rule files that future coding agents can follow.

Why this architecture

CommandCode treats model failures as often deterministic harness-integration failures rather than pure model-quality problems: repairing clear malformed calls gives the model progress and feedback, reducing repeated errors without waiting for the base model to change.

Models used: DeepSeek V4 Pro · DeepSeek V4 Flash · Claude · GPT · Kimi · MiniMax

Scale

• LangBase previously reached 1.2 billion agent runs a month.
  “It was called LangBase. Uh it grew quite big. Uh 1.2 billion agent runs a month.”transcript, stated
• The team was doing a couple billion tokens per day at the time of the DeepSeek investigation, and later about 100 times more.
  “we are doing anywhere from like, you know, a couple billion tokens a day at that time. We are like 100 times more than that now.”transcript, stated
• DeepSeek repeated the same failed tool call 56 times on average in a billion-token sample.
  “It would repeat that same thing for like 56 times on average in a billion tokens.”transcript, stated
• CommandCode reports 16,000 repair variations across hundreds of billions of tokens.
  “now we have like 16,000 different repair you know, variations across hundreds of billions of tokens.”transcript, stated
• One user reportedly ran around 70 billion DeepSeek tokens and 12-plus-hour sessions.
  “one of the our users has actually done like I think 70 billion tokens on Deep Seek.”transcript, stated

  “I constantly run Deep Seek with Command Code for 12-plus hour-long sessions.”transcript, stated

Alternatives considered (roads not taken)

Return schema errors and ask the model to self-correct

Rejected because: Some open models repeated the same invalid call many times instead of using the error.

Assume the issue is DeepSeek-specific

Rejected because: The same patterns appeared in Kimi and MiniMax logs.

Let users manually maintain all preferences

Rejected because: Taste is designed to create and refresh skills automatically rather than relying only on manual memory.

Patterns

fallback-degradation (reliability)

Malformed tool calls are repaired when intent is clear rather than failing the interaction outright.

tool-use-orchestration (agent-architecture)

CommandCode wraps coding-agent tool execution with deterministic schema repair and hints.

agent-memory (agent-context)

Taste captures reusable coding preferences and maintains skills/rules across projects.

eval-driven-development (evaluation)

CommandCode mines large-scale tool failure logs to identify repairable model behavior.

Tradeoffs

Repair instead of reject malformed calls.

Chose deterministic repair plus hint over returning validation errors only.

Rationale: The repair keeps the agent moving and trains future calls toward the right schema.

Use repair files like migrations.

Chose many explicit repair variations over one broad prompt asking the model to behave.

Rationale: Specific deterministic repairs match recurring observed failures and can be generalized across models.

Automate preference memory.

Chose Taste-created skill files over manual preference steering in every session.

Rationale: Taste makes preferences portable and keeps skills from going stale.

Gotchas (what bit them)

Schema-error feedback can trap open models in loops.

Root cause: Some models treat their malformed output as correct and repeat it despite validation errors.

Resolution: Repair the call, return the result, and include a hint explaining the expected schema.

Tool errors can make capable models look slow or bad.

Root cause: Harnesses may hide repeated failures, leaving users to perceive model slowness.

Resolution: Expose and repair tool-call failures before they waste long sessions.

UI design slop repeats across models.

Root cause: LLMs share finite recurring design mistakes such as default gradient-heavy patterns.

Resolution: Use deterministic design repair guidance and skill files to constrain those patterns.

Evaluation

CommandCode evaluates repair logic by mining massive token-scale tool-call logs, comparing failure rates before and after deterministic repairs, observing model usability in long coding sessions, and applying the same pattern analysis to design-output failures.

Metrics: tool-call failures per session · repair variation count · token volume · session duration · model usability · design defect recurrence

Outcomes

• Repeated tool-call failures dropped from 50-plus-per-session behavior to repaired execution paths in the described examples.
• DeepSeek V4 Flash became practically useful enough to compete in CommandCode after repair logic.
• The same repair framework was applied to UI design slop.
• Taste provides portable preference memory through skill/rule files.

The source cites 1.2B agent runs per month at LangBase, hundreds of billions of current tokens, 16,000 repair variations, and user sessions exceeding 12 hours.