Building Full-Text Search for Agent Traces with Custom Inverted Index on Object Storage

langchain · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agent observability storage, full-text search, and object-storage query engines

LangChain needed SmithDB to support full-text search and JSON filtering over very large LangSmith agent traces stored in object storage. Traditional log-search assumptions did not fit: agent trace inputs and outputs can be 1 MB+ or hundreds of megabytes uncompressed, source:index economics are worse than ordinary logs, term frequencies are highly skewed, users need path/value/full-text predicates, and object-store round trips make random I/O and large postings fetches expensive.

Constraints:

• Durable data lives in object storage so compute remains relatively stateless and can scale by adding nodes.
• Object-store requests have tens to hundreds of milliseconds of latency and modest per-request throughput.
• Agent traces contain enormous nested JSON payloads whose input/output columns dwarf metadata.
• The index must support path existence, keyed value search, and full-text search.
• Search must push down through the existing Apache DataFusion/Vortex scan pipeline rather than becoming a separate search stack.

Architecture

SmithDB implements a custom inverted index inside the Vortex columnar format, optimized for object storage. Vortex row groups, segment reads, statistics, masks, projections, and I/O coalescing prune and batch object-store reads. The v1 index used term_key/term_value columns, postings as List, positions as List<List>, and Vortex default encodings, but scale exposed per-column encoding inefficiency, term-skewed row groups, and expensive position reshaping during merge. The v2 layout sizes row groups by byte budgets, uses one FST per row group as the term dictionary, stores term_info offsets, encodes postings and positions as per-term block-bitpacked delta blobs, skips positions for path-only queries, and fetches only the byte ranges needed for the predicate.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  langsmith-agent-traces[LangSmith agent traces] --> smithdb[SmithDB]
  smithdb --> object-storage[Object storage]
  smithdb --> vortex-columnar-files[Vortex columnar files]
  smithdb --> apache-datafusion-scan-pipeline[Apache DataFusion scan pipeline]
  smithdb --> inverted-index-file[Inverted index file]
  object-storage --> vortex-io-scheduler[Vortex IO scheduler]
  vortex-columnar-files --> row-group-pruning[Row group pruning]
  vortex-columnar-files --> vortex-io-scheduler
  vortex-columnar-files --> inverted-index-file
  query-predicates[Query predicates] --> apache-datafusion-scan-pipeline
  apache-datafusion-scan-pipeline --> row-group-pruning
  apache-datafusion-scan-pipeline --> inverted-index-file
  row-group-pruning --> fst-term-dictionary[FST term dictionary]
  inverted-index-file --> fst-term-dictionary
  inverted-index-file --> term-info-table[Term info table]
  inverted-index-file --> postings-blob[Postings blob]
  inverted-index-file --> positions-blob[Positions blob]
  fst-term-dictionary --> term-info-table
  term-info-table --> postings-blob
  term-info-table --> positions-blob
  postings-blob --> apache-datafusion-scan-pipeline
  positions-blob --> apache-datafusion-scan-pipeline

Components

Data flow

LangSmith events arrive as large nested JSON traces in SmithDB/Vortex object-storage files. Query predicates enter the DataFusion/Vortex scan path as json_key, json_key_search, or search. Vortex prunes row groups, filters surviving rows, projects required columns, and coalesces segment reads. For a surviving index row group, the query walks the FST term dictionary, uses the ordinal to read term_info offsets, fetches the relevant postings byte range, and only fetches positions for phrase queries. Matching row IDs map directly to Vortex row positions, avoiding a translation table.

Why this architecture

LangChain chose a custom Vortex-native inverted index because SmithDB's workload combines huge agent trace payloads, object-store latency, columnar scan pushdown, and row-position identity requirements. Tantivy/Lucene-style ideas were useful, but local-disk mmap assumptions, separate segment/doc-ID semantics, and parallel query-stack integration would have fought SmithDB's storage and execution model. V2 then reworked the physical layout around byte budgets and per-term encodings because v1's textbook layout amplified skew and merge costs.

Models used: SmithDB · LangSmith · Apache DataFusion · Vortex · Finite State Transducer · FSST · zstd · Tantivy · Lucene-style inverted index

Scale

• SmithDB reports 400 ms P50 latency for full-text search and JSON filtering over agent traces.
  “SmithDB supports full-text search and JSON filtering over agent traces with a median (P50) latency of 400 ms”article, stated
• Agent trace inputs and outputs commonly exceed 1 MB and can stretch to hundreds of megabytes uncompressed.
  “with some of these stretching to hundreds of megabytes uncompressed.”article, stated
• LangSmith's source:index ratio for agent traces is about 1:1.9 versus about 1:1.25 for traditional logs.
  “for agent traces in LangSmith, we observed the average to be closer to 1:1.9.”article, stated
• Object-storage round trips are roughly 100 ms in the SmithDB/Vortex design discussion.
  “On object storage a round-trip costs roughly 100 ms”article, stated
• Vortex coalesces segment reads within a 1 MB gap into requests up to a 16 MB window.
  “merging reads within a 1 MB gap into one, up to a 16 MB window”article, stated
• V2 row groups use 32 MB posting-byte and 64 MB raw-term-string byte budgets.
  “bounds the worst-case object-store GET when a query reads postings for a row group.”article, stated

  “caps raw bytes per row group.”article, stated
• Positions were over 70% of index bytes in the v1 compaction bottleneck.
  “For an index where 70%+ of bytes are positions, this was the dominant compaction cost.”article, stated

Alternatives considered (roads not taken)

Adopt Tantivy directly.

Rejected because: Tantivy assumes mmap/local-disk random I/O, separate segment-local doc IDs, and a search stack that would not push down through SmithDB's Vortex/DataFusion scan path.

Scan content payloads without an index.

Rejected because: A content filter would scan many gigabytes just to return a few rows.

Use fixed term-count row groups.

Rejected because: Term skew let one high-frequency term push a row group past 100 MB or even 500 MB compressed, producing oversized GETs and merge memory spikes.

Rely on Vortex default per-column encoding for postings and positions.

Rejected because: A common token forced a larger bit width on every term in the column, hurting cache behavior and increasing reads.

Use zstd-compressed term dictionaries.

Rejected because: zstd can be smaller for high-cardinality term_value data, but it is opaque and requires decompression for lookup; the FST can be searched directly.

Patterns

observability-tracing (operations)

SmithDB indexes LangSmith agent traces so developers can search large agent inputs, outputs, tool data, and errors for debugging and observability.

latency-optimization (operations)

The index layout minimizes object-store requests and bytes read through row-group pruning, byte budgets, coalesced segment reads, FST lookup, and direct postings/positions byte-range fetches.

cost-aware-routing (operations)

SmithDB keeps durable data in object storage and designs index reads around object-store request and byte costs rather than local-disk assumptions.

batch-offline-processing (data)

Index construction and merge are decoupled from core run data by storing positions/postings in a separate sibling Vortex file.

Tradeoffs

Search library integration

Chose custom Vortex-native inverted index over direct Tantivy adoption.

Rationale: The custom design preserves object-storage-aware layout, DataFusion pushdown, and row-index identity with Vortex data files.

Storage model

Chose object storage with stateless compute over local-disk search engine assumptions.

Rationale: Object storage simplifies durable scaling by adding compute nodes, but requires careful request and byte minimization.

Row group sizing

Chose fixed byte budgets over fixed number of terms per row group.

Rationale: Byte budgets bound worst-case object-store reads and memory footprint under term skew.

Term dictionary encoding

Chose FST per row group over zstd/FSST-only compressed strings.

Rationale: FST can be searched directly for exact, prefix, range, LIKE, fuzzy, and regex-style walks without decompressing a block first.

Posting and position representation

Chose block-bitpacked deltas plus VInt tails over List and List<List> Vortex cells.

Rationale: Per-term blocks give better compression/control, allow byte-range fetches, and avoid full Arrow decode on reads.

Gotchas (what bit them)

Agent traces invert traditional log-search economics.

Root cause: LangSmith traces are enormous JSON documents where a single document can produce more index data than many log lines.

Resolution: Specialize the inverted index layout for huge documents, skewed term frequencies, and multiple JSON/text query modalities.

Object storage makes naive index reads expensive.

Root cause: Object-store round trips and per-request throughput make large postings or positions fetches dominate query time.

Resolution: Minimize requests and bytes read with coalesced segment reads, pruning, byte budgets, and range fetches only after lookup.

Per-column encoding punished all terms for a few frequent tokens.

Root cause: Vortex v1 chose encoding for the whole column, so common tokens forced larger bit widths across the chunk.

Resolution: Use per-term block-bitpacked deltas so each term chooses its own bit widths block by block.

Fixed-size row groups produced pathological object-store GETs.

Root cause: Term count was a poor proxy for I/O size under Zipfian term skew.

Resolution: Size row groups by independent posting-byte and raw-term-string byte budgets.

Position reshaping dominated compaction cost.

Root cause: Merging v1 segments required decoding full nested position lists, reshuffling inner lists, and recomputing outer offsets.

Resolution: Store postings and positions as opaque per-term encoded blobs that queries fetch and decode on demand.

Evaluation

LangChain evaluated the design through system-level search latency, payload and index-size ratios, object-store request/byte costs, row-group sizing bounds, encoding-size comparisons, and v1/v2 operational bottleneck analysis.

Metrics: Median P50 query latency of 400 ms. · 1 MB+ common payloads and hundreds-of-megabytes uncompressed outliers. · Source:index ratio of about 1:1.9 for LangSmith agent traces. · Object-store round-trip latency around 100 ms. · 32 MB posting-byte and 64 MB raw-term-string row-group budgets. · Positions making up 70%+ of index bytes in v1. · FST dictionary size comparisons against raw, FSST, and zstd.

Outcomes

• SmithDB supports full-text search and JSON filtering over agent traces with 400 ms median latency.
• The v2 design replaced fixed term-count row groups with byte-budgeted row groups.
• The index can answer path existence, keyed value, and full-text query predicates.
• FST dictionaries, term_info offsets, and block-bitpacked deltas let queries fetch only needed postings/positions ranges.
• The article is part 1; construction, merging, and read-path details are deferred to part 2.

The workload is large-scale observability infrastructure: billions of enormous documents, common 1 MB+ payloads, hundreds-of-megabytes outliers, object storage, 32/64 MB row-group budgets, and a reported 400 ms P50 search/filter latency.