Building an AI-Native Browser with Integrated LLM Tools and Evaluation Systems

the_browser_company · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-native browser product development

The Browser Company needed to move from Arc to Dia, an AI-native browser, while giving the whole company enough tooling, evaluation, model-behavior discipline, and security patterns to ship LLM features safely and quickly.

Constraints:

AI product quality depends heavily on prompts, context, model choices, parameters, and evaluation data.
A small company needed fast hill-climbing without reinforcement learning or fine-tuning.
Browser assistants sit in a high-risk prompt-injection environment with private data, untrusted content, and external actions.
Model behavior had to become a cross-functional craft rather than only an engineering task.

Architecture

Dia is built as an AI-native browser with an assistant embedded into browsing. Internally, The Browser Company moved prompt, tool, context, model, and parameter editing into the product itself so employees can prototype with full context. The team uses evals and Jeba-style prompt optimization, created a model-behavior discipline to translate product requirements into prompts/evals, and adds confirmation UX around risky browser actions such as autofill, email, and calendar scheduling.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  employees[Employees] --> internal-prompt-tools[Internal Prompt Tools]
  internal-prompt-tools --> dia-browser[Dia Browser]
  dia-browser --> dia-assistant[Dia Assistant]
  internal-prompt-tools --> jeba-optimizer[Jeba Optimizer]
  jeba-optimizer --> eval-suite[Eval Suite]
  eval-suite --> model-behavior-team[Model Behavior Team]
  model-behavior-team --> dia-assistant[Dia Assistant]
  untrusted-web-content[Untrusted Web Content] --> dia-assistant[Dia Assistant]
  dia-assistant --> browser-action-tools[Browser Action Tools]
  browser-action-tools --> confirmation-ux[Confirmation UX]
  confirmation-ux --> dia-browser[Dia Browser]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Dia Browser`	AI-native browser with assistant, tabs, apps, personal context, and browser actions.	`Dia Assistant`, `Internal Prompt Tools`, `Confirmation UX`
`Dia Assistant`	LLM assistant that works alongside the user in the browser and personalizes work.	`Dia Browser`, `Model Behavior Team`, `Untrusted Web Content`, `Browser Action Tools`
`Internal Prompt Tools`	In-product tools for editing prompts, tools, context, models, and parameters.	`Dia Browser`, `Employees`, `Jeba Optimizer`
`Employees`	Engineers, CEO, PMs, designers, support, strategy, and ops people who prototype and refine AI features.	`Internal Prompt Tools`
`Jeba Optimizer`	Sample-efficient prompt optimization loop using tasks, scores, selection, reflection, and prompt mutation.	`Internal Prompt Tools`, `Eval Suite`
`Eval Suite`	Evaluation data and scoring used to clarify product requirements and refine model behavior.	`Jeba Optimizer`, `Model Behavior Team`
`Model Behavior Team`	Function defining, evaluating, and shipping desired model behaviors and assistant personality.	`Dia Assistant`, `Eval Suite`, `Internal Prompt Tools`
`Untrusted Web Content`	Web pages and HTML that may contain prompt injections.	`Dia Assistant`
`Browser Action Tools`	Autofill, scheduling, email writing, and other browser actions with external side effects.	`Dia Assistant`, `Confirmation UX`
`Confirmation UX`	User-facing control layer for reviewing data before writing forms, emails, or calendar actions.	`Browser Action Tools`, `Dia Browser`

Data flow

Employees prototype AI browser behaviors inside Dia with real prompts, tools, context, models, and parameters; useful ideas move into eval definition and product-requirement clarification; prompts are hill-climbed with code, manual refinement, or Jeba; features are dogfooded; and risky user-facing actions require confirmation before external side effects.

“So we evolved and since then we built all of our tools into our product, the product that we as a company internally use every day. And that includes the prompts, the tools, the context, the models, every parameter. Um, which has not only allowed us to 10x our speed of”transcript, stated

“In this case, before the form is written to, we actually let the user read and confirm that data in plain text. This doesn't prevent a prompt injection, but it gives the user control, awareness, and trust in what is happening. And this is a framing we carry throughout our”transcript, stated

Why this architecture

Dia required a company-wide AI development loop, not only browser engineering. By putting prompt and context tools inside the product, the team accelerated ideation and let non-engineers shape behavior. Security is handled as product design plus technical mitigation because prompt injection cannot be eliminated by prompting tricks alone.

“not only allowed us to 10x our speed of ideating, iterating and refining our products. But it has also widened the number of people who can access and iterate on our products themselves. from our CEO to our newest hire can ideate and create a new product in DIA and also”transcript, stated

“help, there are no guarantees and prompt injections will still happen. So what do we do? Well, it's on us to design a product with that in mind. We have to blend technology approaches and user experience and design into a”transcript, stated

Models used: GPT models · LLMs

Scale

The Browser Company started in 2019 and shipped Arc in 2022.
“this. We we started this company in 2019. Um, and so this is a screen cap of Josh, our CEO, sharing a little bit about our idea on the internet a few years ago, which we endearingly called the internet computer. So our mission”transcript, stated
“error, and some ups and downs, we shipped our first browser, Arc, in 2022. It was a browser we felt was an improvement over the browsers of that time. It made the internet more personal, more organized, and to us, a little more delightful with a little more craft.”transcript, stated
The team reports a 10x speedup in ideating, iterating, and refining AI products.
“not only allowed us to 10x our speed of ideating, iterating and refining our products. But it has also widened the number of people who can access and iterate on our products themselves. from our CEO to our newest hire can ideate and create a new product in DIA and also”transcript, stated
The company tried tens of computer-use strategies before building one into the product.
“computer use mechanism. We actually tried tens of different types of computer use strategies before landing on one before even building it into the product itself.”transcript, stated

Alternatives considered (roads not taken)

Keep AI tools in dev-only prompt editors.

Rejected because: Only engineers could access them, iteration was slow, and they lacked personal context.

“limited access as only engineers were able to access this. Two, slow iteration speeds. And three, none of your personal context.”transcript, stated

Use RL or fine-tuning for prompt refinement.

Rejected because: Jeba provided a sample-efficient way to improve complex systems by tuning text rather than weights.

“a sample efficient way to improve a complex LLM system without having to leverage RL or other fine-tuning techniques.”transcript, stated

Rely on prompt-injection tag wrapping or role separation alone.

Rejected because: These mitigations help but remain escapable, so risky actions need product-level confirmation.

“not the content. And while this can help, there are no guarantees and prompt injections will still happen. So what do we do? Well, it's on us to design a product with that in mind. We have to blend technology approaches and user experience and design into a”transcript, stated

Patterns

`eval-driven-development` (model-behavior)

The team collects eval data, clarifies product requirements through evals, and uses evals in the model behavior loop.

“collecting that data for measurement and training, clarifying those product requirements through eval.”transcript, stated

`prompt-versioning` (model-behavior)

Internal tools let employees create, rewrite, and optimize prompts inside Dia.

“engineers were writing the prompts at first and then we built these prompt tools to enable more people at the company”transcript, stated

`safety-policy-enforcement` (browser-security)

Risky browser actions use confirmation UX so users inspect data before forms, emails, or events are written.

“Scheduling events in DIA, we have a similar confirmation step. Writing emails India, we also have a similar confirmation step.”transcript, stated

`human-in-the-loop-gate` (browser-security)

Autofill and similar actions keep humans aware and in control before external side effects happen.

“it gives the user control, awareness, and trust in what is happening.”transcript, stated

Tradeoffs

Prompt optimization method

Chose Jeba text mutation with eval scoring over RL/fine-tuning for every refinement.

Rationale: The approach was sample-efficient and fit a small company improving prompts quickly.

“the ability to tune text and not weights.”transcript, stated

Security UX

Chose Plain-text user confirmation before writes over Assuming prompt-injection defenses prevent all attacks.

Rationale: Technical isolation reduces risk but cannot guarantee prompt injections will not happen.

“This doesn't prevent a prompt injection, but it gives the user control, awareness, and trust”transcript, stated

Behavior function

Chose Dedicated model behavior discipline over Ad hoc prompts owned only by engineers.

Rationale: The assistant's style, goals, data, context, and evals required product-design-like ownership.

“model behavior will continue to evolve and into a specialized and prevalent function of its own”transcript, stated

Gotchas (what bit them)

Dev-only prompt tools slow AI-native product development.

Root cause: They exclude non-engineers and lack the user's full product context.

Resolution: Move prompt/context/model tools into the product used internally every day.

“it has also widened the number of people who can access and iterate on our products themselves.”transcript, stated

Prompt injection is structurally dangerous in a browser.

Root cause: Browsers combine private data, untrusted content, and external communication/action capabilities.

Resolution: Blend technical mitigations with confirmation-first product design.

“It has access to your private data. It has exposure to untrusted content and it has the ability to externally communicate”transcript, stated

Model behavior ownership can be misassigned.

Root cause: Prompt and behavior quality may come from people outside traditional engineering roles.

Resolution: Treat model behavior as a specialized cross-functional craft.

“it might not be an engineer or it might be it could also be someone on the strategy and ops team”transcript, stated

Evaluation

The Browser Company evaluates AI features through in-product prototyping, dogfooding, eval collection, Jeba scoring loops, internal and external feedback, and confirmation UX for high-risk actions.

Metrics: Task scores for Jeba · Eval pass/fail against product requirements · Internal dogfood feedback · External feedback · Prompt-injection risk before side effects

“execute it across a set of tasks and score them.”transcript, stated

“we create evals, and then we ship, and then we collect more feedback”transcript, stated

Outcomes

The company shipped Dia, an AI-native browser with an assistant alongside users.
In-product tooling reportedly 10x'd ideation and refinement speed.
Prompt tools enabled non-engineers, including strategy/ops, to improve product behavior.
Security patterns such as confirmation steps became a cross-feature design frame.

The talk cites company founding in 2019, Arc in 2022, Dia in 2025, 10x iteration speed, tens of computer-use strategies tested, and a weekend prompt rewrite that formed the model behavior team.

“earlier this year we shipped DIA our AI native browser.”transcript, stated

“one weekend to rewrite all our prompts.”transcript, stated