Building Multilingual AI Agents with Translation Pipelines

boundary · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Multilingual agent pipeline architecture

Boundary's case addresses multilingual AI applications whose core prompts, tools, output schemas, and business logic are written in English. Even if a frontier model understands many languages, a bespoke agentic pipeline with mostly English steering can respond in English to non-English users, creating a reliability gap for customer-facing multilingual agents. The challenge is to support many languages without duplicating every prompt, tool definition, evaluation suite, and business-logic pipeline per language.

Constraints:

• Most of the steering in bespoke AI apps lives in the application pipeline, not in the user's message.
• If prompts, tool definitions, schemas, and user-message wrappers are mostly English, the model is likely to answer in English even for a non-English user.
• Parallel per-language pipelines multiply prompt maintenance and evaluation complexity across every supported language.
• Teams often do not speak all customer languages, making it hard to build trustworthy eval data without translators or domain experts.
• Translation layers add latency, especially for the majority-English path if every request passes through them.
• Output schema field names and tool definitions can accidentally steer the model toward English.
• Direct multilingual instruction following can work often, but the goal is higher-confidence reliability closer to production software expectations.

Architecture

The recommended architecture keeps the main business-logic agent and its evals in English, then wraps it with explicit language-normalization boundaries. The front layer detects or translates the user's message into English, optionally captures intent or metadata, and can preserve the original message to reduce translation loss. The English agent performs the core task, such as travel booking, using English prompts, tools, and output schemas. The final layer localizes the English response back into the user's language and tone. For latency, a heuristic classifier can bypass translation for mostly English messages. The demo live-coded this as a travel-booking pipeline with translation capture, optional intent capture, the main travel agent, and a localization postprocessor; small models were intended for translation parts and a heavier model for the main agent.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-message["User message"] --> language-heuristic["Language heuristic"]
  language-heuristic --> translation-preprocessor["Translation preprocessor"]
  language-heuristic --> english-agent-pipeline["English agent pipeline"]
  user-message --> intent-capture["Intent capture"]
  translation-preprocessor --> english-agent-pipeline
  intent-capture --> english-agent-pipeline
  english-agent-pipeline --> tool-layer["Tool layer"]
  english-agent-pipeline --> localization-postprocessor["Localization postprocessor"]
  translation-preprocessor --> localization-postprocessor
  localization-postprocessor --> user-facing-response["User-facing response"]
  small-translation-model["Small translation model"] --> translation-preprocessor
  small-translation-model --> intent-capture
  small-translation-model --> localization-postprocessor
  heavyweight-main-model["Heavyweight main model"] --> english-agent-pipeline
  evaluation-suite["Evaluation suite"] --> english-agent-pipeline
  evaluation-suite --> translation-preprocessor
  evaluation-suite --> localization-postprocessor

Components

Data flow

A user sends a message in French, Hindi plus English, Spanish, or another language mix. A lightweight preprocessor translates it to pure English and may capture intent, detected language, or metadata while preserving the original user text. The translated message and optional intent enter the main English agent, which contains the business logic and tool calls. The agent returns an English response. A localization pipeline receives the original message, the English response, and relevant language/tone context, then translates the response back to match the user's language and tone. For users who appear to be writing mostly English, a quick heuristic can skip the translation layer and send them directly to the English pipeline.

Why this architecture

The speakers recommend translation boundaries because duplicating pipelines per language makes evals, prompt updates, and tool-definition maintenance explode, while simply telling the model to respond in the user's preferred language relies on instruction following under a mostly English prompt. Normalizing to English keeps the expensive business logic and evals in one language, just as voice systems often use speech-to-text, an LLM text pipeline, and text-to-speech rather than direct audio-to-audio. The architecture is a little slower and more complex, but the speakers argue the accuracy, maintainability, and eval wins are worth it.

Models used: Claude · Anthropic Haiku · Small translation models · Heavyweight main model · BAML

Scale

• The speakers frame production reliability as the difference between 99% correctness and 99.99% correctness.
  “the difference between software that's right 99% of the time and 99.99% of the time”transcript, stated
• They cite 99.82% uptime as an example of a number that can look like a rounding error but still be unacceptable for GitHub.
  “it's not a rounding error when you have like 99.82 uptime is not acceptable for GitHub.”transcript, stated
• A prompt/context mix of roughly 20% French and 80% English is used to explain why the model may output English.
  “if I see a document that's 80% French, uh, sorry, that's like 20% French and 80% English, I'm going to go ahead and give you English as an output.”transcript, stated
• The speakers discuss supporting many languages, even infinite language-specific pipelines in the naive approach.
  “you could have infinite pipelines that you want for every language you want.”transcript, stated
• The demo asks for small models on translation parts and a more heavyweight model for the main agent.
  “we want to make sure that we use small models for any of the translation parts and obviously our main agent should actually be a little bit more u heavyweight”transcript, stated
• The speakers say the cost/engineering tradeoff changes once token spend reaches scales like thousands of dollars a day.
  “Now it's worth spending a week of engineering because the money we save by switching to Sonnet [snorts]”transcript, stated
• For medical companies, they warn that every language may need medical-terminology eval data.
  “if you're building a medical company you you almost definitely need to go and like get eval data for like every single medical term terminology”transcript, stated

Alternatives considered (roads not taken)

Build fully parallel pipelines per language.

Rejected because: This can produce higher confidence for each supported language, but it multiplies prompts, tool definitions, evals, translation files, and update coordination across every language.

Add a prompt instruction to reply in the user's preferred language.

Rejected because: The speakers say this relies on both language generation and instruction following, which reduces reliability compared with an explicit translation boundary.

Send non-English input directly to the main LLM and expect a non-English answer.

Rejected because: The live demo showed the same travel agent still responding in English when the translation layer was bypassed, because prompts and tool definitions remained English.

Run every request through translation and intent capture.

Rejected because: Most users may be English speakers, so the speakers propose a cheap heuristic that routes mostly English inputs straight to the English pipeline to avoid added latency.

Use direct audio-to-audio or direct non-English-to-non-English model calls.

Rejected because: The speakers argue production systems usually get better accuracy and maintainability by normalizing through text or English boundaries, even when that adds steps.

Build multilingual evals entirely with AI for languages the team does not understand.

Rejected because: The speakers say teams either accept the model's error tolerance for small translation tasks or hire people who speak each language to build proper eval data.

Patterns

prompt-chaining (pipeline)

The solution chains translation/normalization, optional intent capture, the main English agent, and localization back to the user's language/tone.

query-rewriting (input-normalization)

Non-English or mixed-language input is rewritten into pure English before reaching the core agent, while the original message can be retained for context and final tone matching.

model-routing (model-selection)

The demo prompt asks for small models on translation steps and a heavier model for the main agent, reflecting task-specific model selection.

latency-optimization (performance)

The design adds a cheap mostly-English heuristic so English users can bypass translation and classification steps rather than paying translation latency on every request.

eval-driven-development (quality)

The speakers emphasize building evals for the English pipeline and the smaller translation units, and using complaints to create new evals before prompt iteration.

structured-output-contract (interface-design)

The demo's output schema and field names affected model behavior; renaming a field from reply English was necessary because schema names act as prompt text.

Tradeoffs

Normalize through English instead of duplicating every pipeline per language.

Chose Translation boundaries around one English business-logic pipeline over Separate French, English, miscellaneous, and other language-specific pipelines.

Rationale: The translation-boundary design keeps the core pipeline evaluable and maintainable in one language, while avoiding the eval and prompt-maintenance explosion of fully parallel language pipelines.

Accept translation-boundary latency for higher correctness.

Chose Non-English to English to LLM to English to non-English over Direct non-English to LLM to non-English.

Rationale: The boundary architecture adds steps, but the speakers argue it improves accuracy, maintainability, and evalability enough to be worth the delay.

Use a fast English heuristic before translation.

Chose Common-word percentage threshold to bypass translation for mostly English messages over Always classifying/translating every message with an AI model.

Rationale: A cheap precheck can reduce latency for the majority English cohort while still routing non-English messages through the safer translation pipeline.

Use smaller models for translation only after the quality/cost tradeoff demands it.

Chose Small models for translation units and a heavier model for the main agent over Optimizing model cost too early or using the same heavyweight model everywhere forever.

Rationale: The speakers recommend throwing tokens at the problem until cost justifies engineering work, then routing simpler steps to cheaper models.

Keep original mixed-language text available to later stages.

Chose Preserve original user message alongside the translated English message over Discarding source text after translation.

Rationale: Keeping source text reduces translation loss and lets the localization step mimic mixed-language style and tone.

Gotchas (what bit them)

Multilingual capability in the base model does not guarantee multilingual product behavior.

Root cause: The application pipeline may be mostly English, so English prompt/tool/schema context dominates the user's non-English message.

Resolution: Add explicit translation boundaries instead of relying on the model to infer the desired output language from mixed context.

Parallel language-specific pipelines drift from the main product.

Root cause: A team may update the English pipeline faster than French or other translated versions, and may not know when an English change requires a localized update.

Resolution: Keep the core business logic in one evaluated pipeline and translate at the boundaries.

Teams cannot evaluate languages they do not understand without outside help.

Root cause: AI-generated tests for another language do not remove the need to understand whether the translation is correct when the domain requires high confidence.

Resolution: Either accept the model's residual error on small translation tasks or hire language/domain experts to build eval data manually.

A schema field name can accidentally force English output.

Root cause: Output schema names are part of the prompt, so a field named reply English steers the model to English even if the instruction says otherwise.

Resolution: Review schema names as prompt text and remove language-biased field names before testing multilingual behavior.

Bypassing translation can still fail even with a smart model.

Root cause: The main agent's prompts and tool definitions remain English, so the model answers in English when the translation/localization layer is disabled.

Resolution: Use the explicit non-English-to-English-to-LLM-to-English-to-non-English path for high-confidence multilingual behavior.

Translation layers add latency for English users unless routed carefully.

Root cause: A universal translation/intention preprocessor makes the common English path do unnecessary work.

Resolution: Add a cheap heuristic or classifier to route mostly English messages directly to the English pipeline.

Evaluation

The evaluation strategy is to build strong evals around the core English pipeline and smaller evals for the translation/localization units, then decide whether model-based translation quality is acceptable or whether humans who speak the language must create eval data. The live demo also treats test-running as prompt-editing back pressure: add a failing case, iterate prompts, and require tests to pass before trusting changes.

Metrics: Target correctness gap framed as 99% versus 99.99% · Language-output correctness when translation is bypassed versus enabled · Latency added by translation and intent-capture layers · English heuristic threshold for bypassing translation · Translation/localization quality for languages the team can or cannot read · Regression tests created from user complaints

Outcomes

• The demo generated a travel-booking agent with translation capture, intent capture, an English travel-agent response, and localization back to the user's tone.
• When the translation layer was bypassed for a non-English input, the same agent responded in English, proving the failure mode.
• Restoring the translation path returned the multilingual behavior the speakers wanted.
• The demo exposed an output-schema naming issue where reply English accidentally forced English output.
• The architecture supports mixed-language input by keeping the original message available through translation and localization.

The source is an architecture/live-coding discussion rather than a deployed KPI report. It quantifies reliability targets conceptually with 99% versus 99.99%, uses a 20% French/80% English prompt-mix example, and discusses operational scale in terms of many languages, English-majority traffic, and cost tradeoffs once token spend becomes large.