Building Foundation Models for Computer Use Agents

tzafon · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Computer-use agents, multimodal foundation models, and agent training infrastructure

Tzafon wanted to train foundation models for computer-use agents: models that can observe a computer through screenshots and issue actions such as clicks and typing. The team found that existing state-of-the-art models performed poorly on simple computer-use benchmarks, that trajectory fine-tuning and reinforcement learning did not generalize well, and that the deeper bottleneck was visual grounding: models often did not know which on-screen element matched an instruction.

Constraints:

• Computer-use agents must interact through screenshots and low-level actions such as clicks, coordinates, and typing.
• There are few strong benchmarks for computer use, with OS World being one of the main references.
• Training requires large-scale data generation from browsers or operating systems.
• Automatic verification is difficult because models do not reliably understand whether a computer task was completed from screenshots.
• Domain-specific success on terminal or HTML tasks does not automatically generalize to broader computer use.

Architecture

Tzafon built Waypoint, an SDK/infrastructure layer for interacting with computers, first through browsers and later through OS-level environments, with warm browser pools for fast data collection. Early training generated trajectories for high-level tasks and used SFT or RL on successful paths, including a hierarchical Gemma-supervisor/Qwen-executor experiment. Those approaches improved in narrow settings but did not scale or generalize. Error analysis showed a grounding problem: coordinate errors were often either exactly correct or far off. Tzafon therefore moved toward pre-training a computer-use foundation model, filtering data, running ablations, mid-training, RL, and fine-tuning a Qwen vision encoder on screenshot/tutorial data to strengthen visual grounding. Training infrastructure shifted toward PyTorch Titan/Forge, with reported 40k tokens/sec/GPU throughput.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  computer-use-agent[Computer-use agent] --> screenshot-action-interface[Screenshot/action interface]
  screenshot-action-interface --> waypoint-sdk[Waypoint SDK]
  waypoint-sdk --> warm-browser-pool[Warm browser pool]
  waypoint-sdk --> os-environments[OS environments]
  warm-browser-pool --> trajectory-generator[Trajectory generator]
  os-environments --> trajectory-generator
  trajectory-generator --> trajectory-store[Trajectory store]
  trajectory-store --> sft-rl-trainer[SFT/RL trainer]
  trajectory-store --> verifiable-reward-domains[Verifiable reward domains]
  sft-rl-trainer --> supervisor-model[Supervisor model]
  supervisor-model --> executor-model[Executor model]
  executor-model --> screenshot-action-interface
  sft-rl-trainer --> benchmark-harness[Benchmark harness]
  verifiable-reward-domains --> benchmark-harness
  benchmark-harness --> grounding-analysis[Grounding analysis]
  grounding-analysis --> foundation-pretraining-pipeline[Foundation pretraining pipeline]
  foundation-pretraining-pipeline --> qwen-vision-encoder[Qwen vision encoder]
  foundation-pretraining-pipeline --> screenshot-tutorial-data[Screenshot tutorial data]
  screenshot-tutorial-data --> qwen-vision-encoder
  foundation-pretraining-pipeline --> pytorch-training-stack[PyTorch training stack]
  desktop-beta-app[Desktop beta app] --> computer-use-agent

Components

Data flow

Waypoint creates browser or OS environments, captures screenshots, and lets scripts or agents perform computer actions. The team generates trajectories for high-level tasks, filters successful trajectories, and tries SFT/RL or supervisor-executor training. Verifiable domains such as terminal and HTML tasks provide reward signals, while broader tasks expose sparse verification and grounding failures. Grounding analysis then feeds a foundation-model path: collect and filter data, train/pre-train, run ablations, mid-training, RL, and fine-tune the Qwen vision encoder with screenshot tutorial data where a human explains what is happening on screen.

Why this architecture

Tzafon first built Waypoint because every computer-use training strategy needed scalable interaction data. SFT/RL and supervisor-executor training were worth trying because successful trajectories and task decomposition are natural starting points, but they did not generalize. The observed grounding failures pushed the team toward domain-specific foundation-model pretraining and screenshot-specific vision encoder tuning rather than assuming a text-pretrained multimodal model would learn precise screen grounding from ordinary post-training.

Models used: Gemma · Qwen · Qwen vision encoder · Vision-language models · BERT-like transformers (conceptual comparison) · Mixture of experts · PyTorch Titan · PyTorch Forge · OS World

Scale

• Tzafon had worked on the computer-use effort for around a year.
  “we started around a year ago”transcript, stated
• Waypoint started with browser automation and expanded to OS support.
  “initially started with browsers now we support OS.”transcript, stated
• Waypoint maintains a warm pool of browsers for fast data collection.
  “It maintains a warm pool of browsers. So you don't have to like it's it's really fast if you want to want to want to uh collect data.”transcript, stated
• The team reported 40k tokens per second per GPU after PyTorch Titan modifications.
  “we we even uh achieved 40k tokens per second per GPU.”transcript, stated
• The infrastructure environment improved substantially within about half a year.
  “there's been a lot of infrastructure progress even half a year ago the infrastructure was really poor”transcript, stated
• Waypoint is useful for collecting a lot of data or running fleets of browsers/lightweight Linux environments.
  “as long as you as you need to collect a lot of data and uh or like you want a fleet of browsers or lightweight Linuxes”transcript, stated

Alternatives considered (roads not taken)

Use existing multimodal models as-is for screenshot/action computer use.

Rejected because: The speaker says state-of-the-art models perform poorly on simple OS World tasks, motivating custom work.

Fine-tune or RL-train on successful trajectories.

Rejected because: The approach somewhat worked but produced low-entropy trajectories, sparse rewards, and poor scaling.

Hierarchical supervisor/executor architecture.

Rejected because: Using Gemma to guide Qwen improved results in narrow settings, but the approach still failed to generalize across domains.

Attach a vision encoder to an already strong text model and rely on normal multimodal training.

Rejected because: The text-to-text component can explain much of the training signal, leaving weak image grounding.

Keep building custom training infrastructure.

Rejected because: As PyTorch Titan and Forge matured, the team no longer needed to own as much pre/post-training infrastructure itself.

Patterns

tool-use-orchestration (orchestration)

The agent operates the computer through an action interface: it sees screenshots and emits clicks, coordinates, or typing actions.

synthetic-data-generation (data)

Waypoint provides scalable computer environments for creating screenshots, actions, trajectories, and training examples for computer-use models.

fine-tuning-adaptation (model-operations)

The team fine-tuned existing components and trained computer-use-specific models, including Qwen vision encoder tuning on screenshot/tutorial data and SFT/RL on trajectories.

multi-agent-orchestration (orchestration)

Tzafon tried a hierarchical setup where a larger supervisor model decomposed tasks and instructed a smaller executor model.

eval-driven-development (evaluation)

The team used OS World and related benchmarks, verifiable reward domains, coordinate-error distributions, and grounding failure analysis to redirect the training strategy.

batch-offline-processing (model-operations)

The foundation-model path uses offline data filtering, ablations, mid-training, RL, and large-scale pre-training cycles before model release.

latency-optimization (operations)

Waypoint's warm browser pool and the training stack's throughput improvements reduce bottlenecks in data collection and model training.

Tradeoffs

Computer-control interface

Chose screenshot input plus click/type actions over more abstract or application-specific computer APIs.

Rationale: The speaker describes screenshot/action as the consensus among best-performing computer-use models, even if it seems unsophisticated.

First infrastructure investment

Chose Waypoint data-generation infrastructure over starting directly with model tuning.

Rationale: Every downstream training strategy required a way to interact with computers and generate data at scale.

Model development direction

Chose pre-training a computer-use foundation model over only SFT/RL on generated successful trajectories.

Rationale: Trajectory training somewhat worked but did not scale or generalize; grounding required a more fundamental solution.

Grounding fix

Chose fine-tune an existing Qwen vision encoder on screenshot/tutorial data over assuming generic multimodal pretraining is enough.

Rationale: Attaching vision remained hard, but Qwen was the best available open-source option and tutorial data helped tie text explanations to screens.

Training infrastructure ownership

Chose use maturing PyTorch Titan/Forge infrastructure over continuing to maintain all custom training infrastructure.

Rationale: The team initially had to build infra itself, but newer tools worked well enough to avoid that burden.

Gotchas (what bit them)

Successful-trajectory training produced low-entropy behavior.

Root cause: Generated trajectory trees often followed one path instead of exploring diverse solutions.

Resolution: Treat trajectory SFT/RL as insufficient and move toward broader grounding and pretraining work.

Automatic verification of computer-use tasks is sparse and unreliable.

Root cause: Models do not understand computers well enough to decide from screenshots whether a task is completed.

Resolution: Use domains with verifiable rewards, such as terminal tasks or HTML tasks with known answers, while recognizing the limitation.

Training on narrow domains did not generalize.

Root cause: Terminal-task and file-system browsing training improved local performance but failed to transfer to other computer-use domains.

Resolution: Use benchmark and failure analysis to identify grounding as the deeper cross-domain bottleneck.

Click errors were often wrong-object errors, not small coordinate errors.

Root cause: The model did not recognize which button matched the description, producing a bimodal error distribution.

Resolution: Focus on visual grounding rather than merely coordinate regression.

Text-pretrained multimodal models can underlearn screen grounding.

Root cause: The strong text-to-text part explains much of the variance, leaving weak signal from image data.

Resolution: Pre-train for the computer-use domain and fine-tune the vision encoder on screenshot/tutorial data.

Training infrastructure was immature early in the project.

Root cause: Pre-training/post-training tools and checkpointing did not work reliably.

Resolution: Adopt newer infrastructure such as PyTorch Titan and PyTorch Forge once they matured.

Evaluation

Tzafon evaluates computer-use models with OS World and related benchmarks, domain-specific verifiable reward tasks such as terminal and HTML tasks, trajectory success, coordinate-error distributions, and grounding analysis. The talk emphasizes that benchmark and verification quality remain weak across the field.

Metrics: OS World and related computer-use benchmark performance. · Success on simple tasks such as restoring a file from trash. · Verifiable rewards in terminal and HTML tasks. · Coordinate error distribution: exact hit versus large miss. · Generalization across terminal, file-system, browsing, and other domains. · Training throughput, including 40k tokens/sec/GPU.

Outcomes

• Tzafon released Waypoint as usable infrastructure for collecting computer-interaction data with browsers and OS environments.
• Early SFT/RL and supervisor-executor experiments improved in narrow settings but did not scale or generalize.
• Failure analysis identified grounding as a central bottleneck for computer-use agents.
• The team shifted toward pretraining a computer-use foundation model and fine-tuning a Qwen vision encoder on screenshot/tutorial data.
• At the time of the talk, the model and desktop app were upcoming/beta rather than proven general-purpose production agents.

The case is research-heavy rather than production deployment. Concrete scale signals include a one-year effort, Waypoint support for browser and OS environments, large trajectory/data generation, 40k tokens/sec/GPU training throughput, and a beta desktop app.