AI-Powered Email Search Assistant with Advanced Cognitive Architecture

superhuman · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Email and calendar search

Superhuman needed Ask AI to answer natural-language inbox and calendar questions reliably, quickly, and across more task types than a single-prompt RAG prototype could handle.

Constraints:

Users expect interactive email-search latency
Calendar and deadline questions require accurate date reasoning
Different query types need different tools, prompts, and retrieval parameters

Architecture

Superhuman's Ask AI evolved from a single-prompt RAG design into a parallel cognitive architecture. The old flow generated retrieval parameters in JSON mode, ran hybrid search and heuristic reranking, then answered with one LLM. The new flow runs tool classification and metadata extraction in parallel, invokes task-specific tools such as email search, calendar search, availability, scheduling, or direct LLM response, then uses task-specific response generation and post-processing to reduce hallucination.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ask-ai-query-interface["ask-ai-query-interface"]
  tool-classifier["tool-classifier"]
  metadata-extractor["metadata-extractor"]
  task-specific-tool-layer["task-specific-tool-layer"]
  hybrid-search-reranker["hybrid-search-reranker"]
  task-specific-response-prompts["task-specific-response-prompts"]
  post-processing-follow-up["post-processing-follow-up"]
  ask-ai-query-interface --> tool-classifier
  ask-ai-query-interface --> metadata-extractor
  tool-classifier --> task-specific-tool-layer
  metadata-extractor --> task-specific-tool-layer
  task-specific-tool-layer --> hybrid-search-reranker
  task-specific-tool-layer --> task-specific-response-prompts
  hybrid-search-reranker --> task-specific-response-prompts
  task-specific-response-prompts --> post-processing-follow-up

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`ask-ai-query-interface`	Natural-language entry point for inbox and calendar questions.	`tool-classifier`, `metadata-extractor`
`tool-classifier`	Intent classifier selecting email search, calendar search, availability, scheduling, or direct LLM response.	`task-specific-tool-layer`
`metadata-extractor`	Parallel process for time filters, sender names, attachments, and retrieval parameters.	`task-specific-tool-layer`
`task-specific-tool-layer`	Task execution layer for email search, calendar search, availability checking, and scheduling.	`hybrid-search-reranker`, `task-specific-response-prompts`
`hybrid-search-reranker`	Hybrid semantic and keyword retrieval plus reranking for email and calendar evidence.	`task-specific-response-prompts`
`task-specific-response-prompts`	Prompt layer specialized by query type to generate the final answer.	`post-processing-follow-up`
`post-processing-follow-up`	Response refinement layer used to reduce hallucinations and smooth the user experience.	—

Data flow

A user asks a natural-language inbox/calendar question. Tool classification decides whether the query needs email search, email plus calendar, availability, scheduling, or direct response, while metadata extraction pulls time filters, senders, and attachments. Search tasks go through hybrid semantic/keyword retrieval and reranking, and the selected task-specific prompt generates the final answer with post-processing or follow-up.

3 supporting quotes

“When a user submits a query, two parallel processes occur for the Ask AI agent:”article, stated

“The system classifies the query based on user intent to determine which tools or data sources to activate.”article, stated

“Simultaneously, the system extracts relevant tool parameters such as time filters, sender names, or relevant attachments.”article, stated

Why this architecture

Superhuman moved away from one large RAG prompt because the model did not reliably follow task-specific instructions, struggled with dates, and performed unevenly across query types. Parallel classification and metadata extraction let the system invoke only relevant tools and use task-specific prompts, while preserving low latency for a search product.

3 supporting quotes

“the single-prompt design had a few shortcomings. First, the LLM did not always follow task-specific instructions reliably.”article, stated

“They also found that the LLM struggled to reason about dates accurately”article, stated

“This tool classification ensures that only relevant tools are invoked, which improves response quality.”article, stated

Models used: LLM

Scale

Ask AI targets sub-2-second responses.
“Sub-2-second responses to maintain a smooth user experience”article, stated
The case summary reports a 14% reduction in search time.
“Through careful prompt engineering techniques like "double dipping" instructions and a rigorous staged evaluation process, they achieved sub-2-second response times with reduced hallucinations. The result was a 14% reduction in search time”summary, stated
The case summary reports 5 minutes saved per week for users who previously spent up to 35 minutes weekly on email searches.
“14% reduction in search time (5 minutes saved per week) for users who previously spent up to 35 minutes weekly on email searches.”summary, stated

Alternatives considered (roads not taken)

Single-prompt RAG

Rejected because: The original prompt produced retrieval parameters and final answers but failed task-specific instructions, date reasoning, and complex query coverage.

“the Superhuman team used a single prompt LLM that performed retrieval augmented generation (RAG).”article, stated

“These limitations pushed the Superhuman team to transition to a more complex cognitive architecture.”article, stated

Invoking all tools for every query

Rejected because: Tool classification invokes only the relevant tools or data sources for the user's intent.

“The classifier identifies whether the query requires:some text”article, implied

Search without extracted metadata filters

Rejected because: Metadata extraction narrows retrieval scope using time filters, sender names, and attachments.

“These will be used in retrieval to narrow the scope of search to improve accuracy.”article, stated

Patterns

`tool-use-orchestration` (query-routing)

Ask AI classifies each query to decide whether to use email search, calendar search, availability, scheduling, or a direct LLM answer.

“The system classifies the query based on user intent to determine which tools or data sources to activate. The classifier identifies whether the query requires:some text”article, stated

`structured-output-contract` (retrieval-parameter-generation)

The first version generated retrieval parameters in JSON mode before passing them to hybrid search and reranking.

“their first version, which generated retrieval parameters using JSON mode that were passed through hybrid search and heuristic reranking before the LLM produced an answer.”article, stated

`hybrid-search` (retrieval)

Search tasks use both semantic and keyword search, followed by reranking, to prioritize relevant inbox or calendar information.

“If the task required search, it would be passed into the search tool (with a hybrid semantic + keyword search) with reranking algorithms to prioritize the most relevant information.”article, stated

`prompt-chaining` (task-specific-response-generation)

The system separates classification, retrieval, and response generation, then selects prompts and preferences based on the earlier classification.

“Based on the classification in step 1, the system would select different prompts and preferences. Prompts would contain context-specific instructions with query-specific examples, and also encoded user preferences.”article, stated

`latency-optimization` (parallel-processing)

Tool classification and metadata extraction run in parallel to keep the agent responsive.

“When a user submits a query, two parallel processes occur for the Ask AI agent:”article, stated

Tradeoffs

Whether to keep one large prompt or split behavior across task-specific processing.

Chose parallel classification, metadata extraction, tool use, and task-specific response generation over one large all-encompassing prompt.

Rationale: Superhuman needed consistent quality across diverse inbox and calendar tasks.

“Instead of relying on one large, all-encompassing prompt, the Ask AI agent used task-specific guidelines during post-processing. This allowed the agent to maintain consistent quality across diverse tasks.”article, stated

Gotchas (what bit them)

Single-prompt RAG did not reliably follow task-specific instructions.

Root cause: One broad prompt had to handle too many query types and behavioral rules.

Resolution: Superhuman moved to task classification, task-specific prompts, and post-processing guidelines.

“the single-prompt design had a few shortcomings. First, the LLM did not always follow task-specific instructions reliably.”article, stated

The initial assistant struggled with date reasoning and complex searches.

Root cause: The first architecture handled some search types well but not calendar availability or complex multi-step tasks.

Resolution: The new architecture classified intent, extracted metadata such as time filters, and invoked the right tools for the query type.

“They also found that the LLM struggled to reason about dates accurately (e.g. identify upcoming deadlines). Their system also only handled certain search types well”article, stated

Evaluation

Superhuman evaluated the architecture against production expectations for latency and hallucination reduction, with the summary also describing staged user-feedback rollout.

Metrics: response latency · hallucination reduction · search time saved · retrieval accuracy · thumbs up/down feedback

“By transitioning to this parallel, multi-process architecture, Superhuman created a more reliable agent and also hit these RAG expectations:”article, implied

“Sub-2-second responses to maintain a smooth user experience”article, stated

Outcomes

The evolved Ask AI architecture met sub-2-second response expectations.
Post-processing layers and brief follow-up reduced hallucinations.
The summary reports a 14% reduction in user search time, or five minutes saved per week.

The case focuses on architecture and user-experience targets rather than infrastructure scale.

“Reduced hallucinations through post-processing layers and brief follow-up”article, stated

“The result was a 14% reduction in search time (5 minutes saved per week) for users who previously spent up to 35 minutes weekly on email searches.”summary, stated