Incremental LLM Adoption Strategy in Email Processing API Platform

nylas · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Incremental generative-AI adoption in an email-processing API platform

Nylas wanted to introduce generative AI into its Extract AI email-processing product family without betting the production system on immature, costly, and slower LLMs. The team needed to build confidence, justify roadmap capacity to leadership, automate low-risk work first, improve annotation speed/cost, then use LLMs only where they fit production latency and economics.

Constraints:

• team members had uneven generative-AI experience
• leadership approval required cost-benefit and long-term cost comparisons
• production Extract AI processed emails in under half a second
• Nylas processed 20 to 30 million emails per month
• LLM inference latency was seconds, which customers would not accept for every email
• LLM API costs would break pricing at fractions of a dollar per API call
• human engineers still needed to review generated pull requests for product/customer context

Architecture

Nylas used a three-month staged adoption strategy. First, it ran a two-week spike to compare OpenAI, Mistral, Meta, Google, and Anthropic on cost and fit. Then it built a GitHub Actions multi-agent workflow for toil tasks: a developer agent checks ground-truth data and proposes regex/config changes, a scripted QA tester runs tests, and an LLM reviewer approves or requests another iteration before a pull request is opened for human review. Next, GPT-4 replaced a blind dual-contractor annotation workflow, cutting annotation cost and cycle time. Finally, production Extract AI kept its fast strategy library as the primary path and used LLMs only as a low-confidence fallback, processing 1-3% of emails through LLMs. In parallel, Nylas built Llama.cpp-based self-hosted inference on GCP with prompts, model registry, feature storage, and model swapping to control costs at scale.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Nylas_platform["Nylas platform"] --> Extract_AI["Extract AI"]
  Email_data["Email data"] --> Extract_AI
  Email_data --> Annotation_tool["Annotation tool"]
  Email_data --> LLM_Anonymizer["LLM Anonymizer"]
  Adoption_strategy["Adoption strategy"] --> Model_spike_sprint["Model spike sprint"]
  Model_spike_sprint --> OpenAI_models["OpenAI models"]
  Model_spike_sprint --> Mistral_models["Mistral models"]
  Model_spike_sprint --> Meta_models["Meta models"]
  Model_spike_sprint --> Google_models["Google models"]
  Model_spike_sprint --> Anthropic_models["Anthropic models"]
  Adoption_strategy --> GitHub_Actions_workflow["GitHub Actions workflow"]
  GitHub_Actions_workflow --> Budget_limiter["Budget limiter"]
  Budget_limiter --> Developer_agent["Developer agent"]
  Ground_truth_data["Ground truth data"] --> Developer_agent
  Developer_agent --> QA_tester["QA tester"]
  QA_tester --> Reviewer_agent["Reviewer agent"]
  Reviewer_agent --> Developer_agent
  Reviewer_agent --> Engineer_reviewer["Engineer reviewer"]
  Adoption_strategy --> Annotation_tool
  Annotation_tool --> Blind_annotation_system["Blind annotation system"]
  Annotation_tool --> GPT_4_annotation_system["GPT-4 annotation system"]
  Adoption_strategy --> Extract_AI
  Extract_AI --> Strategy_library["Strategy library"]
  Strategy_library --> Confidence_estimator["Confidence estimator"]
  Confidence_estimator --> LLM_fallback["LLM fallback"]
  LLM_fallback --> Customers["Customers"]
  Extract_AI --> Inference_service["Inference service"]
  LLM_fallback --> Inference_service
  Inference_service --> Prompt_DB["Prompt DB"]
  Inference_service --> Model_registry["Model registry"]
  Inference_service --> Feature_storage["Feature storage"]
  Inference_service --> Llama_cpp_runtime["Llama.cpp runtime"]
  Llama_cpp_runtime --> GCP_L4_GPU_nodes["GCP L4 GPU nodes"]

Components

Data flow

For development automation, a cron job starts a GitHub Actions workflow with a budgeted multi-agent loop: the developer agent reads new ground-truth data and proposes changes, QA tests run as scripts, the reviewer agent evaluates the change and either opens a PR or asks for another iteration, and engineers review the PR. For production Extract AI, incoming emails are processed by existing template/statistics strategies; if confidence is low, the email is reprocessed with an LLM and that result is returned. For self-hosting, Extract AI calls an inference service, which selects prompts from a prompt DB, runs a model loaded from the model registry, and can support fine-tuned models through feature storage.

Why this architecture

Nylas deliberately avoided a big-bang LLM replacement. Low-risk internal milestones built skill and leadership confidence, annotation automation proved cost/time value, and production Extract AI kept the low-latency deterministic path while routing only low-confidence edge cases to LLMs. Self-hosting was added once API economics made break-even analysis favorable.

Models used: OpenAI models · Mistral models · Meta models · Google models · Anthropic models evaluated and dropped for cost · GPT-4 · GPT-4 Turbo · Llama.cpp-served open source models · fine-tuned models under exploration

Scale

• Nylas had over 1,000 businesses using its platform.
  “we have over a thousand businesses that are using our platform”transcript, stated
• The adoption strategy was scoped to three months.
  “in just three months we were able to uh do test and experimentation integrate generative AI”transcript, stated
• The spike sprint lasted two weeks.
  “this two week Sprint actually helped us to limit the number that we would have test”transcript, stated
• The toil tasks previously took one engineer two to three hours per week.
  “this task usually was done by one engineer once per week and it was taking two three hour”transcript, stated
• The multi-agent workflow opened more than 200 PRs and merged roughly 50.
  “since then we have opened more than 200 PR and of this roughly under 50 were actually merged into our code base”transcript, stated
• Roughly 10% of generated PRs involved LLM hallucination.
  “we also had roughly 10% cases in which yes the llms were hallucinating”transcript, stated
• The workflow moved updates from weekly to daily, a 7x latency reduction.
  “we went from a weekly update to actually daily update from data to the system so we actually lower by seven times the lency”transcript, stated
• The GPT-4 annotation prompt 99th percentile input was below 5,000 tokens and average output below 1,000 tokens.
  “the 99 percentile of the prompt input size for us it was below 5,000 token and instead for the output we took the aage”transcript, stated
• The LLM annotation system cost about $0.22 per annotation before GPT-4 Turbo.
  “this resulted below than th000 token this means that each annotation was costing us 22 cents”transcript, stated
• Annotating 5,000 emails cost about $1,100 with LLMs versus $10,000 with blind annotation.
  “if you have a 5,000 email data set translate in $1,100 TimeWise”transcript, stated

  “to annotate 5,000 uh data set email we had to actually do 10,000 annotation so $10,000 in costs”transcript, stated
• The annotation replacement saved 20x money and was 60x faster.
  “we ended up with this system saving uh 20 times money and having a system that actually was 60 times faster”transcript, stated
• Extract AI processed an email in less than half a second.
  “our system is able to process an email in less than half of a second”transcript, stated
• Nylas processed 20 to 30 million emails per month.
  “if you think about that we process 20 to 30 millions of email per uh per month”transcript, stated
• Only 1% to 3% of monthly emails were processed with LLMs under the fallback strategy.
  “thanks to this fullback strategy we actually monthly just process 1 to 3% of the email with llms”transcript, stated
• Bug tickets dropped by more than 90% from over 400.
  “last year we had over 400 tickets uh that we needed to handle”transcript, stated

  “however since we introduced this llm strategy those ticket number dropped by more than 90%”transcript, stated
• Self-hosted nodes cost roughly $500 per node per month.
  “this configuration will cost you roughly 500 bucks per node per month”transcript, stated
• The example self-hosting break-even point was 10,000 API calls.
  “you ended up finding that the break even point is 10,000 API call for you”transcript, stated

Alternatives considered (roads not taken)

Jump straight to production LLM replacement

Rejected because: The team needed confidence, experience, and clearer production fit before touching Extract AI.

Use Anthropic models

Rejected because: They liked model quality but cost was almost double for their use case.

Keep blind dual-contractor annotation

Rejected because: It doubled cost and required weeks for 5,000 emails, while GPT-4 annotation was much faster and cheaper.

Replace Extract AI entirely with LLMs

Rejected because: LLMs were too slow and too expensive for all monthly email traffic.

Use cloud-provider model garden only

Rejected because: It was quick but lacked autoscaling and fixed hardware/model flexibility.

Patterns

multi-agent-orchestration (agent-architecture)

Nylas uses LLM developer and reviewer agents plus a scripted QA tester to iterate on regex/config updates and open PRs.

human-in-the-loop-gate (governance)

Engineers review generated PRs and provide missing product/customer context before merge.

fallback-degradation (resilience)

Extract AI keeps the fast existing strategy path and sends only low-confidence emails to LLM fallback.

cost-aware-routing (operations)

Nylas compares provider costs, limits LLM fallback volume, and self-hosts when API call volume crosses break-even.

eval-driven-development (quality)

The adoption plan starts with spike testing models and uses tests, annotation quality, PR merge rates, latency, cost, and bug tickets to decide next steps.

privacy-preserving-processing (governance)

Nylas built an internal LLM Anonymizer to remove PII from emails before LLM processing.

Tradeoffs

Adoption sequence

Chose development automation and annotation milestones before production over start directly with a production-grade LLM replacement.

Rationale: Smaller, lower-risk milestones built team confidence, leadership support, and concrete evidence of value.

Production LLM role

Chose LLM fallback for low-confidence emails over LLM primary path for all emails.

Rationale: Fallback captures edge cases and reduces bug tickets while preserving sub-second latency and manageable cost.

Annotation workflow

Chose GPT-4 annotation replacement over two-contractor blind annotation.

Rationale: The LLM system was far cheaper and faster while avoiding weeks of delay when annotation definitions changed.

Model serving infrastructure

Chose Llama.cpp framework deployment over GCP Vertex AI Model Garden quick setup.

Rationale: The framework route required more build effort but supported autoscaling, model swapping, and flexibility.

Provider selection

Chose remove Anthropic from tested options for this use case over pay almost double for a liked model.

Rationale: Cost efficiency mattered more than raw model preference for the target use case.

Gotchas (what bit them)

LLM agents lacked full business context for many generated PRs.

Root cause: Customer exceptions and product requirements were not fully represented in the agent context.

Resolution: Keep human engineer review before merge and close unmergeable PRs.

LLM-generated PRs hallucinated.

Root cause: The developer/reviewer agents sometimes produced invalid changes despite the workflow.

Resolution: Use engineer review and close hallucinated PRs instead of auto-merging.

LLM replacement would violate production latency expectations.

Root cause: Extract AI runs in under half a second while LLM latency tends to be seconds.

Resolution: Use LLMs only as a fallback path for low-confidence emails.

LLM cost could make product pricing impossible.

Root cause: Processing tens of millions of monthly emails through APIs would not fit fractional per-call pricing.

Resolution: Limit LLM usage to 1-3% fallback and build self-hosting economics for higher-volume use.

Cloud-provider model garden was fast to launch but operationally rigid.

Root cause: The managed option lacked autoscaling and easy model swapping within the same infrastructure.

Resolution: Use Llama.cpp/Kubernetes-style framework deployment for flexibility.

Evaluation

Nylas evaluated adoption through a two-week model spike, operational metrics for the multi-agent PR workflow, annotation cost/time calculations, production latency and monthly volume constraints, fallback email share, bug-ticket reduction, and self-hosting break-even analysis.

Metrics: two-week spike sprint · 200+ PRs opened, roughly 50 merged · roughly 10% hallucinated PRs · weekly to daily updates: 7x latency reduction · annotation cost: $0.22 per annotation, then $0.10 with GPT-4 Turbo · 5,000-email annotation set: $1,100 LLM cost versus $10,000 blind annotation · 20x money saved and 60x faster annotation system · Extract AI latency: less than half a second · 20-30 million emails per month · 1-3% monthly emails processed by LLM fallback · bug tickets dropped more than 90% · self-hosting break-even example: 10,000 API calls

Outcomes

• The multi-agent development workflow opened more than 200 PRs and merged roughly 50.
• Ground-truth updates moved from weekly to daily, reducing integration latency by 7x.
• GPT-4 annotation replacement produced 20x cost savings and 60x speed improvement.
• Production LLM fallback processed only 1-3% of emails monthly while avoiding a product price increase.
• Bug tickets dropped by more than 90%, from over 400 to a manageable number.
• Self-hosted Llama.cpp infrastructure gave the team a break-even path for high-volume LLM usage.

The transcript is an internal engineering talk with ASR noise, but the operational numbers are specific enough to be useful. The main reusable lesson is not one model choice; it is the staged adoption sequence that uses low-risk internal workflows to learn before introducing LLMs into production paths.