Production LLM Systems: Document Processing and Real Estate Agent Co-pilot Case Studies

various · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production document-understanding RAG and real-estate agent copilot systems

Docugami and Rehat needed production LLM applications that could handle structurally complex enterprise documents or real-estate platform actions while remaining debuggable, reliable, cost-aware, and continuously improvable.

Constraints:

• Real-world documents contain tables, multi-column layouts, inline headings, and semantic relationships that naive text chunking loses.
• Complex LLM chains can fail through context overflow, invalid SQL, or broken structured outputs unless every step is traceable.
• A real-estate copilot with many tools is latency-sensitive because function calling may require iterative model turns.
• Evaluation must measure application behavior, not just standalone language-model quality.

Architecture

The webinar presents two production patterns. Docugami builds a document-engineering stack with multimodal structure extraction, XML knowledge graphs, hierarchical chunks, self-querying retrieval, custom models on Kubernetes/Triton, and LangSmith traces/datasets for chain debugging and improvement. Rehat's Lucy moves from a structured chat agent to OpenAI function calling for more reliable tool calls, then wraps it in synthetic CI tests, LangSmith trace links, human QA curation, and planned fine-tuning datasets.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-documents["Enterprise documents"] --> multimodal-structure-extractor["Multimodal structure extractor"]
  multimodal-structure-extractor["Multimodal structure extractor"] --> xml-knowledge-graph["XML knowledge graph"]
  xml-knowledge-graph["XML knowledge graph"] --> self-querying-retriever["Self-querying retriever"]
  self-querying-retriever["Self-querying retriever"] --> custom-document-models["Custom document models"]
  custom-document-models["Custom document models"] --> langsmith-traces["LangSmith traces"]
  lucy-copilot["Lucy copilot"] --> openai-function-calling["OpenAI function calling"]
  lucy-copilot["Lucy copilot"] --> synthetic-ci-tests["Synthetic CI tests"]
  openai-function-calling["OpenAI function calling"] --> langsmith-traces["LangSmith traces"]
  synthetic-ci-tests["Synthetic CI tests"] --> langsmith-traces["LangSmith traces"]
  synthetic-ci-tests["Synthetic CI tests"] --> human-qa-curation["Human QA curation"]
  human-qa-curation["Human QA curation"] --> fine-tuning-datasets["Fine-tuning datasets"]
  langsmith-traces["LangSmith traces"] --> fine-tuning-datasets["Fine-tuning datasets"]

Components

Data flow

For Docugami, documents are parsed into structural elements, stitched into reading order, labeled into an XML knowledge graph, retrieved by semantic and metadata filters, and passed through chains that can generate SQL, run it, fix failures, and explain results. For Rehat, a user request enters Lucy, the model selects platform functions through OpenAI function calling, test runs are tagged with commit and suite metadata, failures link to LangSmith traces, and curated examples feed future prompts or fine-tuning.

Why this architecture

Both systems reject generic one-shot prompting because the production problem lives in application structure. Docugami needs document topology and semantic labels before retrieval can work; Rehat needs reliable platform actions and application-level tests rather than model-only evals. LangSmith becomes the common debugging and dataset spine because failures occur inside chains, not just final answers.

Models used: custom LLM · smaller custom models · 70B parameter models · GPT-3.5 · GPT-4 · OpenAI function calling models · multimodal models

Scale

• Docugami had around five years of experience building document-understanding systems.
  “Docugami is a document engineering company founded by Jean Paoli (co-creator of the XML standard) and Mike Tuck, with around five years of experience building systems”summary, stated
• Rehat consolidates 10-20 real-estate agent tools into a single platform.
  “Rehat provides a SaaS platform for real estate agents, consolidating 10-20 different tools into a single "super app."”summary, stated
• OpenAI function calling improved Rehat latency by 3-4x.
  “Function calling embedded reasoning and structured output into the model itself, resulting in 3-4x latency improvements and much more reliable outputs.”summary, stated
• Rehat generated 50 synthetic user requests per use case for test coverage.
  “They generated test cases by manually asking ChatGPT to create 50 user requests per use case in JSON format, then imported them into their CI/CD system.”summary, stated
• Docugami samples 10% of every 500 fixed runs for few-shot learning updates.
  “Every 500 fixed runs, they sample 10% to add to their few-shot learning set, which is reloaded weekly.”summary, stated

Alternatives considered (roads not taken)

Naive text splitting for document RAG

Rejected because: Flat chunks lose document structure, reading order, and semantic relationships needed for precise retrieval.

Full agent architecture for Docugami's chain

Rejected because: Agents consumed many tokens and smaller custom models were not tuned for agent syntax.

LangChain structured chat agent for Lucy

Rejected because: As tools grew, prompts became too slow and structured outputs were unreliable.

Model-level evaluation frameworks for Rehat

Rejected because: The team needed to evaluate the behavior of the full application, not a language model in isolation.

Patterns

rag-retrieval (retrieval)

Docugami improves RAG by preserving document structure and semantic metadata so retrievers can filter by meaning rather than file names.

structured-output-contract (interface)

Rehat moved to OpenAI function calling because it embedded structured output and reasoning contracts into the model's tool interface.

observability-tracing (operations)

Both teams use LangSmith traces to inspect chain steps, reproduce failures, link CI failures, and create datasets for improvement.

eval-driven-development (quality)

Rehat built a synthetic CI test pipeline, while Docugami regularly reviews failed/disliked runs and converts them into datasets.

fine-tuning-adaptation (model-adaptation)

Docugami fine-tunes smaller production models from corrected examples; Rehat prepares fine-tuning infrastructure for future function-calling support.

human-in-the-loop-gate (governance)

Human annotators, reviewers, and QA teams decide which runs are bad, fix examples, and turn them into reliable datasets.

Tradeoffs

Preserve document structure before retrieval

Chose Hierarchical XML knowledge graph chunks over Flat text extraction and naive chunking.

Rationale: Structured chunks let retrieval disambiguate semantically related fields that flat retrievers miss.

Use LCEL-style chains rather than full agents for Docugami workflows

Chose Sophisticated chains with retry logic and parallel execution over Full agent architectures.

Rationale: The team wanted agent-like fix-up and branching without the token cost or syntax mismatch of agents for smaller custom models.

Move Lucy to OpenAI function calling

Chose Function calling with schema-constrained tool selection over LangChain structured chat agent.

Rationale: Function calling was faster and produced more reliable structured outputs as the tool count grew.

Use human QA before automated evaluators for Lucy

Chose Human good/bad determination and curated labels over Immediately scaling with LLM evaluators.

Rationale: The team valued human oversight while their product and failure modes were still being established.

Gotchas (what bit them)

Naive chunking breaks document RAG on complex layouts.

Root cause: Important semantic relationships can be visually separated across headings, tables, columns, and key-value structures.

Resolution: Use multimodal structure detection, reading-order stitching, hierarchical chunks, and XML semantic labels.

Complex chains are hard to debug unless traces are intentionally readable.

Root cause: Failures can occur across parsing, example selection, SQL generation, fix-up logic, and parallel explanation steps.

Resolution: Instrument LangSmith traces with proper names and config so each step can be inspected and reproduced.

Function-calling prompt descriptions may not reliably control model behavior.

Root cause: The model follows JSON schemas better than natural language instructions in tool descriptions, and examples in tool descriptions can be ignored or cause hallucinations.

Resolution: Rely on schema design, parsing-error handling, fine-tuning or software patches for behaviors that prompting cannot fix.

Application evaluation cannot be reduced to one deterministic test outcome.

Root cause: LLMs are nondeterministic and application behavior emerges from model, tools, prompts, and orchestration together.

Resolution: Run many synthetic cases concurrently and interpret statistical pass/fail trends by use case.

Evaluation

Docugami evaluates by reviewing failed and disliked runs, adding them to datasets, using larger offline models to propose corrections, and validating corrected syntax before fine-tuning. Rehat evaluates the full application with synthetic concurrent tests, CI tags, LangSmith trace links, playground reproduction, and human QA curation.

Metrics: statistical test pass/fail counts by use case · failed runs · user-disliked runs · valid SQL or XQuery syntax · commit hash and branch traceability · hundreds of daily test conversations

Outcomes

• Docugami retrieval succeeds on landlord-style metadata questions that fail without enriched document labels.
• Rehat saw 3-4x latency improvement and more reliable structured outputs after moving to function calling.
• Both teams built continuous improvement workflows that turn traced failures into datasets or future fine-tuning inputs.

The clearest numbers are Docugami's five years of document-engineering work, Rehat's 10-20 consolidated tools, 50 generated test requests per use case, hundreds of daily test conversations, and Docugami's 500-run/10% weekly few-shot refresh cadence.