Building Synthetic Filesystems for AI Agent Navigation Across Enterprise Data Sources

dust.tt · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise knowledge-agent navigation

Dust found that agents needed structural navigation over company data, not just semantic search, because many enterprise questions require finding known locations, adjacent content, or subtrees across tools like Notion, Slack, GitHub, and Google Drive.

Constraints:

• Enterprise data lives in multiple platforms with different native structures and APIs.
• Semantic search is insufficient when the agent needs a specific path, file, database entry, or section rather than meaning alone.
• LLM context windows prevent naive full-file reads.
• Some platforms, especially Notion, blur the file/folder boundary because pages can contain nested pages.
• The abstraction adds operational complexity around connectors, permissions, dynamic data, and API failures.

Architecture

Dust mapped enterprise data sources into synthetic filesystems backed by its content-nodes architecture, then exposed five Unix-inspired commands: list, find, cat, search, and locate_in_tree. Agents can navigate structure, inspect files in chunks with offset/limit/grep, search within subtrees, and combine path exploration with semantic search to solve multi-step knowledge tasks.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-data-connectors["enterprise data connectors"] --> content-nodes-architecture["content nodes architecture"]
  content-nodes-architecture --> synthetic-filesystem-layer["synthetic filesystem layer"]
  synthetic-filesystem-layer --> list-command["list command"]
  synthetic-filesystem-layer --> find-command["find command"]
  synthetic-filesystem-layer --> cat-command["cat command"]
  synthetic-filesystem-layer --> search-command["search command"]
  synthetic-filesystem-layer --> locate-in-tree-command["locate_in_tree command"]
  cat-command --> context-window-controls["context window controls"]
  agent-planner["agent planner"] --> list-command
  agent-planner --> find-command
  agent-planner --> cat-command
  agent-planner --> search-command
  agent-planner --> locate-in-tree-command

Components

Data flow

Connectors normalize Slack, Notion, GitHub, spreadsheets, and documents into content nodes; the synthetic filesystem layer presents those nodes as folders, files, or dual file-directories; agents call list/find/cat/search/locate_in_tree to narrow scope, read paginated content, or seed semantic search; the result is a navigation loop that alternates structure-aware browsing with meaning-based retrieval.

Why this architecture

Dust chose a filesystem abstraction because agents were already trying to express path-like intent, while human information work naturally mixes browsing, adjacent exploration, and semantic retrieval. The architecture gives agents both structural and meaning-based primitives instead of forcing every task through global semantic search.

Scale

• The behavior was observed in production logs in April 2025.
  “In April 2025, something kept showing up in our logs.”article, stated
• The implementation exposes five Unix-inspired commands.
  “We implemented five Unix-inspired commands:”article, stated
• The cat command uses offset and limit controls to read arbitrarily large documents within context-window limits.
  “We added limit and offset parameters: the agent chooses to see a certain number of lines starting from a certain line number.”article, stated

  “This lets agents handle arbitrarily large documents.”article, stated
• The tools are live as Advanced Search in Dust's Agent Builder.
  “The tools are live now as "Advanced Search" in the Agent Builder.”article, stated

Alternatives considered (roads not taken)

Treat path-like agent behavior as an instruction bug.

Rejected because: Dust interpreted it as a signal that agents needed structural navigation primitives.

Use semantic search alone.

Rejected because: Semantic search finds meaning, but it does not support known-path navigation such as finding a section in recent team notes.

Dump entire files with Unix-style cat.

Rejected because: LLM context windows make full-file reads impractical, so Dust added limit, offset, and grep.

Keep a strict file-versus-folder distinction.

Rejected because: Notion pages can contain sub-pages, so Dust allowed objects to be both readable and listable.

Patterns

tool-use-orchestration (agent architecture)

Dust gives agents a small set of explicit filesystem commands for navigation, content inspection, and scoped search.

hybrid-search (retrieval)

The system combines structural filesystem navigation with semantic search inside narrowed subtrees.

rag-retrieval (retrieval)

The agent retrieves content from enterprise knowledge sources, then reads specific files or sections to answer questions.

context-compaction (context management)

The cat command uses offset, limit, and grep to sample content without overflowing the LLM context window.

Tradeoffs

Add structural navigation alongside semantic search.

Chose Synthetic filesystem commands plus semantic search over Semantic search only.

Rationale: Some questions are about known structure and adjacency, while others require meaning-based lookup; combining both supports multi-step work.

Create synthetic structures independent of source-platform organization.

Chose A logical filesystem abstraction over Native platform structure only.

Rationale: Dust could reshape connectors into a hierarchy that made sense for agent navigation even when platforms lacked one.

Page large documents instead of reading them whole.

Chose cat with limit, offset, and grep over Full-document cat.

Rationale: Paged reads let agents inspect large documents within context windows while still jumping to sections and filters.

Gotchas (what bit them)

Naive file reads can immediately exceed the context window.

Root cause: Unix cat semantics dump full files, but LLMs have hard input-size limits.

Resolution: Add offset, limit, and grep parameters so agents can inspect documents in chunks.

Notion-like systems violate normal filesystem assumptions.

Root cause: Documents can recursively contain documents, so the object is both readable content and navigable container.

Resolution: Tell the model whether an item contains nested elements and allow it to be both cat-able and ls-able.

Synthetic navigation increases connector and reliability burden.

Root cause: Each command can translate into platform-specific API calls over dynamic third-party systems.

Resolution: The case implies this must be managed behind a consistent abstraction, but does not disclose detailed caching or reliability mechanisms.

Evaluation

The article evaluates behavior through production deployment examples and workflow demonstrations rather than quantitative accuracy metrics.

Metrics: Five Unix-inspired commands implemented · Observed in logs in April 2025 · Tools live as Advanced Search in Agent Builder · No formal latency or success-rate metric disclosed

Outcomes

• Agents can answer structure-dependent questions by finding databases, listing recent entries, and reading specific sections.
• The abstraction transforms agents from search-only systems into multi-step knowledge workers.
• The tools are live as Advanced Search in Dust's Agent Builder.

The article reports production deployment, five commands, multi-source coverage, context-window controls for arbitrarily large documents, and live availability in Agent Builder, but not usage volumes.