Agentic AI System for Document Summarization and Analysis

moveworks · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise document analysis

Moveworks needed employees to upload documents and ask follow-up questions without manually reading or comparing long files.

Constraints:

Architecture

Moveworks Brief Me is a session-scoped document agent with online ingestion and content generation stages. Users upload PDFs, Word, or PowerPoint files; ingestion fetches, chunks, generates metadata, embeds, and indexes content with P90 latency under 10 seconds; the session locks retrieval to the processed sources; an operation planner chooses SEARCH or READ actions; SEARCH uses BM25, fine-tuned MPNet bi-encoder embeddings, OpenSearch ANN, and dynamic window expansion; READ uses map-reduce over long contexts; citation prediction and heuristics map claims back to paragraph-level chunks.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user-document-upload[User document upload] --> online-ingestion-pipeline[Online ingestion pipeline]
  online-ingestion-pipeline --> session-source-scope[Session source scope]
  session-source-scope --> operation-planner[Operation planner]
  operation-planner --> search-executor[Search executor]
  operation-planner --> read-executor[Read executor]
  search-executor --> citation-system[Citation system]
  read-executor --> citation-system
  citation-system --> generated-answer[Generated answer]
  generated-answer --> evaluation-framework[Evaluation framework]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
User document upload PDF, DOCX, PPT, or source input brought into Copilot. Online ingestion pipeline
Online ingestion pipeline Fetches, chunks, enriches metadata, embeds, and indexes sources. User document upload, Session source scope
Session source scope Focused mode where only processed sources are queryable until session close. Online ingestion pipeline, Operation planner
Operation planner Predicts SEARCH or READ actions and parameters with ICL/SFT. Session source scope, Search executor, Read executor
Search executor BM25, MPNet embeddings, OpenSearch ANN, and dynamic context lookup for targeted questions. Operation planner, Citation system
Read executor Map-reduce processing for whole-document understanding and summarization. Operation planner, Citation system
Citation system Model and heuristic attribution to sparse, complete paragraph-level citations. Search executor, Read executor, Generated answer
Generated answer Grounded summary, comparison, Q&A, or analysis returned to the user. Citation system
Evaluation framework Human and LLM-grader evaluation of turns and sessions. Generated answer

Data flow

A user uploads documents into chat. Brief Me ingests, chunks, embeds, indexes, and session-scopes the sources; subsequent queries enter the operation planner; SEARCH retrieves precise snippets with keyword and embedding search while READ processes full documents through split/map/reduce; generation returns grounded answers with sparse paragraph-level citations.

3 supporting quotes
“These steps—fetching, chunking, metadata generation, embedding, and indexing—form the core of the ingestion pipeline. We will be exploring this source content ingestion stage in more detail in our upcoming blog on this topic.”article, stated
“### Locking into Brief Me mode After the content ingestion pipeline runs successfully, the agent locks users into a focused mode where only the processed sources are available for querying.”article, stated
“The operation planner uses a combination of  in-context learning (ICL) and Supervised Fine Tuning (SFT) methods”article, stated

Why this architecture

Brief Me separates ingestion, planning, search, read, and citation because document questions need different retrieval modes. Snippet search works for targeted questions, full-document READ is needed for summaries, map-reduce mitigates long-context degradation, and paragraph citations make generated claims auditable.

3 supporting quotes
“Some tasks require a comprehensive understanding of the entire content. Hence, the context for a READ operation is the entire document.”article, stated
“### Map-reduce based parallel generation Current language models struggle to effectively utilize information from long input contexts.”article, stated
“Brief Me, however, is capable of providing citations at the paragraph level, enabling users to place greater trust in the generated responses.”article, stated

Models used: MPNet bi-encoder · LLMs

Scale

Alternatives considered (roads not taken)

Always solve document questions with search/RAG

Rejected because: Summaries may require reading the entire document rather than retrieving snippets.

“It is particularly crucial for addressing summarization-style problems that cannot be solved by a search or RAG approach. For example, when asked to summarize company A's instruction sheet, one would need to read the entire document and understand its complex nuances before writing a summary.”article, stated

Use document-level citations only

Rejected because: Brief Me needs citations tied to specific paragraphs that substantiate each claim.

“claim. Most existing tools either don't provide citations or only offer citations at the document level. Brief Me, however, is capable of providing citations at the paragraph level, enabling users to place greater trust in the generated responses. We employ a combination of LLM and heuristic-based”article, stated

Patterns

rag-retrieval (document-analysis)

Uploaded sources are ingested and made available as session-scoped context for user queries.

“After the content ingestion pipeline runs successfully, the agent locks users into a focused mode where only the processed sources are available for querying.”article, stated

hybrid-search (retrieval)

The summary reports hybrid search with custom-trained embeddings.

“two-stage agentic architecture with online content ingestion and generation capabilities, including hybrid search with custom-trained embeddings, multi-turn conversation support, operation planning, and a novel map-reduce approach for long context handling, the system achieves high accuracy metrics”summary, stated

prompt-chaining (reasoning)

The agentic pipeline plans operations before generating document-analysis responses.

“abilities, including hybrid search with custom-trained embeddings, multi-turn conversation support, operation planning, and a novel map-reduce approach for long context handling, the system achieves high accuracy metrics (97.24% correct actions, 89.21% groundedness, 97.98% completeness) with P90 lat”summary, stated

Tradeoffs

Route questions to SEARCH or READ instead of using one retrieval path for all document tasks.

Chose An operation planner with SEARCH for targeted snippets and READ for whole-document understanding over Always searching for snippets even when a summary requires comprehensive reading.

Rationale: Some summarization tasks are counterproductive to solve with search because skipping sections can make the summary incomplete.

“This action indicates that we need to read the entire document instead of attempting to search. It is particularly crucial for addressing summarization-style problems that cannot be solved by a search or RAG approach.”article, stated

How to handle long document summaries.

Chose map-reduce READ path over single long-context prompt or snippet search.

Rationale: Long contexts degrade and may exceed limits, while search can miss sections required for comprehensive summaries.

“longer inputs lead to higher latency.”article, stated

Gotchas (what bit them)

Long-context LLMs still struggled to use information buried in the middle of lengthy documents.

Root cause: Relevant information in the middle of long contexts is harder for current models to use, and long inputs add latency and context-limit risk.

Resolution: Moveworks used a map-reduce algorithm with split, map, and reduce stages for lengthy READ operations.

“Current language models struggle to effectively utilize information from long input contexts. Performance is typically highest when relevant information appears at the beginning or end of the input context”article, stated

Citations had to be sparse, complete, and paragraph-grounded rather than document-level labels.

Root cause: Existing citation approaches often omitted relevant attribution or linked claims to overly broad document-level sources.

Resolution: Brief Me combined model-predicted citation IDs with collators, dedupers, and referencers to link output claims to specific paragraphs.

“Most existing tools either don't provide citations or only offer citations at the document level. Brief Me, however, is capable of providing citations at the paragraph level”article, stated

Evaluation

Moveworks measured action correctness, groundedness, completeness, and ingestion latency.

Metrics: correct action rate · groundedness · completeness · P90 latency

“d a novel map-reduce approach for long context handling, the system achieves high accuracy metrics (97.24% correct actions, 89.21% groundedness, 97.98% completeness) with P90 latency under 10 seconds for ingestion, significantly reducing the hours typically required for document analysis tasks.”summary, stated

Outcomes

The feature reduces hours of document analysis work.

“In a hypothetical situation like this, this level of information synthesis could take hours, especially if you're not expertly versed with the content.”article, stated