Scaling Agentic AI Systems for Real Estate Due Diligence: Managing Prompt Tax at Production Scale

orbital · Legal · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real estate legal due diligence automation

Orbital needed to scale an agentic system that reads real estate legal documents, extracts due diligence information with citations, and keeps up with rapid frontier-model changes without drowning in prompt migration work.

Constraints:

• Real estate due diligence involves reading large collections of deeds, leases, and paperwork to find legal red flags.
• Documents can contain handwritten and typed text requiring OCR and structuring before LLM work.
• The system uses many domain-specific prompts written by real estate lawyers.
• New model releases create prompt tax because prompts need migration and retesting.
• The company scaled largely through human domain-expert testing rather than a comprehensive objective eval system.

Architecture

Orbital Co-Pilot ingests deeds and leases, OCRs and structures them, creates a plan, decomposes the due diligence task into subtasks, and runs each subtask through an agentic subsystem with multiple LLM calls. Outputs include clickable citations so licensed lawyers can verify against source documents before reports are stored and sent to clients. AI engineers own agentic/system prompts for tool choice, while embedded real-estate-lawyer domain experts write domain prompts and process user feedback into prompt changes.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Uploaded legal documents"] --> B["OCR and structuring"]
  B --> C["Agentic planner"]
  C --> D["Subtask agent systems"]
  D --> E["Model layer"]
  D --> F["Prompt library"]
  D --> G["Citation report"]
  G --> H["Lawyer review"]
  H --> I["Feedback loop"]
  I --> F
  F --> D

Components

Data flow

A user uploads legal documents for a report. Orbital OCRs and structures the documents, builds a plan, assigns subtasks such as lease date or current annual rent extraction to agentic subsystems, runs multiple LLM calls with agentic and domain prompts, emits answers with citations, and sends a final Word report for lawyer review. Product feedback flows to domain experts, who evaluate issues, modify prompts, and ship fixes into production.

Why this architecture

Orbital optimized for prompt-driven iteration because it was still finding product-market fit and needed domain experts to encode decades of real-estate-law practice quickly. The agentic decomposition allowed many focused LLM calls over long legal documents, while citations kept lawyers in the loop for verification.

Models used: GPT-3.5 · GPT-4 32K · GPT-4 Turbo · GPT-4o · GPT-4.1 · O1 Preview · O4 Mini

Scale

• Orbital had roughly 80 people, with half in product engineering.
  “We're roughly 80 people. Half of that is the product engineering team.”transcript, stated
• The demo processed a deed and lease totaling about 100 pages.
  “One I think has about 30 pages. The other has about 70 pages, so about 100 pages in turn.”transcript, stated
• Token consumption grew from under 1 billion monthly tokens to almost 20 billion monthly tokens in 18 months.
  “We're basically only burning less than a billion tokens. We're now within touching distance 18 months later almost consuming 20 billion tokens every month”transcript, stated
• Revenue grew from zero to multiple seven figures in annual recurring revenue.
  “We've also gone from 18 months ago doing exactly zero in revenue. We've scaled that aggressively too to now doing sort of multiple seven figures in revenue”transcript, stated
• Orbital's domain prompt library grew to well over 1,000 prompts.
  “we've gone from near zero in the early days to well over 1,000 prompts.”transcript, stated

Alternatives considered (roads not taken)

Fine-tune models first

Rejected because: Prompting let the team incorporate user feedback quickly during product-market-fit search.

Build a comprehensive eval system before scaling

Rejected because: The team wanted one but scaled mostly through domain-expert subjective testing and spreadsheets.

Keep old prompt instructions when moving to reasoning models

Rejected because: System-two models needed fewer how-to instructions, less repetition, and fewer constraints.

Patterns

tool-use-orchestration (orchestration)

Orbital's agentic system decomposes legal due diligence into subtasks and specialized agent subsystems that search documents for grounded answers.

prompt-versioning (prompting)

Prompt tax is the operational cost of maintaining and migrating a large prompt library across new frontier models.

human-in-the-loop-gate (governance)

Licensed lawyers manually review generated reports and click citations back to ground truth.

model-routing (model-selection)

Orbital moved across GPT-3.5, GPT-4 variants, and reasoning models, changing prompt strategy by model class.

data-quality-loop (feedback)

Product feedback is routed to domain experts, who decide which prompt changed and ship fixes back to production.

Tradeoffs

Model adaptation strategy

Chose prompting over fine-tuning.

Rationale: Prompting maximized development speed and let domain experts ship feedback-driven changes quickly.

Evaluation posture

Chose domain-expert vibes and manual testing over comprehensive automated evals.

Rationale: The team lacked a comprehensive eval system but still scaled via expert review, feedback, and subjective regression checks.

Model upgrade risk

Chose ship frontier upgrades after mitigation and risk ownership over waiting for certainty.

Rationale: New models bring uncertainty, but a team must decide when the positives outweigh prompt-tax and regression risks.

Gotchas (what bit them)

Prompt libraries create migration debt as models improve.

Root cause: Prompts tuned for earlier model behavior may be over-specified, repetitive, or constraining for newer reasoning models.

Resolution: Experiment with new models, estimate prompt tax, strip how-to instructions for system-two models, and use newer models to help migrate prompts.

Vibes-only evaluation does not provide objective regression coverage.

Root cause: Domain-expert subjective testing and spreadsheets are useful but not comprehensive.

Resolution: The speaker names this as a limitation and open question: an eval system may be needed as scale increases.

Legal users need explainability for complex model reasoning.

Root cause: Real estate lawyers need to understand how the model handled legal matters and verify answers.

Resolution: Expose citations and sometimes reasoning/thought-token explanations where useful.

Evaluation

Orbital's evaluation approach is explicitly informal: domain experts test releases, sometimes log spreadsheet regressions, user feedback flows into prompt changes, and citations let lawyers verify outputs. The speaker acknowledges the absence of a rigorous objective eval system.

Metrics: Less than 1B monthly tokens to almost 20B monthly tokens in 18 months · Zero revenue to multiple seven figures ARR · Over 1,000 domain-specific prompts · Domain-expert manual testing before release · User feedback routed into prompt fixes

Outcomes

• Orbital commercialized Co-Pilot in January 2024.
• The system scaled to almost 20 billion tokens per month within 18 months.
• ARR grew from zero to multiple seven figures.
• The prompt library grew to well over 1,000 domain-specific prompts.
• Lawyers can review extracted reports faster using clickable citations back to source documents.

Orbital reports roughly 80 employees, example due diligence over about 100 pages, nearly 20B monthly tokens, multiple seven figures ARR, and over 1,000 domain prompts.