Enhancing Memory Retrieval Systems Using LangSmith Testing and Evaluation

new_computer · Tech · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agentic memory retrieval evaluation

New Computer needed to improve Dot's long-term memory retrieval so the personal AI assistant could retrieve relevant evolving memories accurately while preserving user privacy during evaluation.

Constraints:

• Dot's memory system dynamically creates and retrieves memories over time rather than searching a static document set.
• Memories need structure and optional meta-fields to support future retrieval.
• Multiple retrieval methods create a combinatorial experiment space.
• Evaluation must preserve user privacy.
• Prompt changes can improve one conversational query while degrading others.

Architecture

New Computer used LangSmith to create synthetic-user evaluation datasets for Dot's agentic memory system. Synthetic backstories and conversations seeded memory databases; queries and full memory sets were stored in LangSmith; relevant memories were labeled with an in-house tool; and retrieval methods such as semantic search, keyword search, BM25, and meta-field filtering were compared using precision, recall, and F1. The team also used LangSmith experiment comparison and prompt playground to inspect prompt changes and fix failures.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  synthetic-users["Synthetic users"] --> memory-database["Memory database"]
  memory-database --> langsmith-dataset["LangSmith dataset"]
  langsmith-dataset --> labeling-tool["Labeling tool"]
  labeling-tool --> retrieval-experiment-variants["Retrieval experiment variants"]
  retrieval-experiment-variants --> langsmith-experiment-comparison["LangSmith experiment comparison"]
  langsmith-experiment-comparison --> prompt-playground["Prompt playground"]
  prompt-playground --> dot-conversation-prompt["Dot conversation prompt"]
  dot-conversation-prompt --> memory-database

Components

Data flow

Synthetic users and conversations seed a memory database. Queries plus all available memories are stored in a LangSmith dataset, relevant memories are labeled, retrieval variants run experiments, precision/recall/F1 identify better retrieval approaches, and prompt changes are inspected in comparison views and adjusted in the playground.

Why this architecture

New Computer needed synthetic, measurable memory experiments because Dot's memory is dynamic, user-specific, and privacy-sensitive. LangSmith provided a way to run retrieval variants, inspect regressions, and iterate prompts without relying on real user memories.

Models used: Dot · LangSmith · LLM-generated synthetic users · semantic search · BM25

Scale

• LangSmith experiments produced 50% higher recall.
  “leading to 50% higher recall and 40% higher precision compared to a previous baseline implementation”article, stated
• LangSmith experiments produced 40% higher precision.
  “50% higher recall** and 40% higher precision compared to a previous baseline implementation of dynamic memory retrieval.”article, stated
• More than 45% of users converted to paid after hitting the free message limit.
  “more than 45% of which converted to the app’s paid tier after hitting the free message limit”article, stated

Alternatives considered (roads not taken)

Use only a simple semantic-search baseline.

Rejected because: Different query types benefited from other methods such as keyword search, BM25, and meta-field filtering.

Evaluate on real user memories.

Rejected because: Synthetic users preserved privacy while allowing labeled memory-retrieval experiments.

Patterns

agent-memory (memory)

Dot dynamically creates and retrieves memories to maintain an evolving picture of the user.

synthetic-data-generation (evaluation)

Synthetic users and LLM-generated backstories create privacy-preserving retrieval test data.

eval-driven-development (evaluation)

Precision, recall, and F1 experiments guide retrieval-method selection and prompt changes.

hybrid-search (retrieval)

The team compared semantic, keyword, BM25, and meta-field filtering retrieval methods across query types.

Tradeoffs

Evaluation data

Chose Synthetic user cohorts. over Real user memory data..

Rationale: Synthetic data preserved privacy while still exercising memory retrieval.

Retrieval approach

Chose Experiment across semantic, keyword, BM25, and meta-field filters. over One fixed semantic-search retriever..

Rationale: Different methods performed better for different query types.

Gotchas (what bit them)

Agentic memory is not static-document RAG.

Root cause: Memories must be created or structured now so they can be retrieved accurately later.

Resolution: Store structured memory meta-fields and evaluate retrieval over evolving memory sets.

Retrieval method performance varies by query type.

Root cause: Some queries work better with similarity search, others with keyword/BM25 or meta-field filters.

Resolution: Run multiple retrieval experiments quickly in LangSmith.

Prompt improvements can regress other queries.

Root cause: Dot's dynamic prompt combines memories, tool use, and behavioral instructions.

Resolution: Use experiment comparison to inspect global effects and prompt playground to adjust failures.

Evaluation

New Computer used labeled synthetic-user datasets in LangSmith, evaluated retrieval variants with precision, recall, and F1, and used experiment comparison plus prompt playground for prompt regressions.

Metrics: Recall · Precision · F1 · Regressed prompt runs · Paid-tier conversion after free message limit

Outcomes

• Achieved 50% higher recall over the previous dynamic memory baseline.
• Achieved 40% higher precision over the previous dynamic memory baseline.
• Enabled faster retrieval-method experimentation across synthetic user datasets.
• Improved prompt iteration by surfacing global regressions and letting the team edit prompts in LangSmith.

The source reports 50% higher recall, 40% higher precision, and over 45% paid conversion after users hit the free message limit.