DragonCrawl: Uber's Journey to AI-Powered Mobile Testing Using Small Language Models

uber · Automotive · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Mobile application testing

Uber needed to scale mobile testing across constantly changing apps, cities, devices, and languages without spending massive engineering time maintaining brittle test scripts.

Constraints:

• Manual testing could not cover every minor code change at Uber's scale.
• Script-based tests broke on minor UI changes such as pop-ups and button changes.
• Testing had to work across many cities and 50+ languages.
• The model had to choose UI actions from screen text and test goals.
• A production CI system needed high stability without constant maintenance.

Architecture

DragonCrawl reframes mobile testing as a language/retrieval task. It converts the current mobile screen into text, combines that screen representation with a natural-language test goal, and asks a fine-tuned language model which UI element to interact with and how. Uber evaluated transformer models with precision@N and chose a tuned base MPNet model because it combined high retrieval quality, lower latency, 110M parameters, and 768-dimensional embeddings. The system then executes flows in CI across apps, cities, languages, devices, and OS versions.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  screen-text-representation[Screen text representation] --> dragoncrawl-model[DragonCrawl model]
  test-goal[Test goal] --> dragoncrawl-model
  dragoncrawl-model --> mobile-action-executor[Mobile action executor]
  mobile-action-executor --> ci-test-runner[CI test runner]
  evaluation-harness[Evaluation harness] --> dragoncrawl-model

Components

Data flow

A mobile test supplies DragonCrawl with the current screen text and a goal. The model embeds and interprets the screen/action options as a retrieval-style decision problem, chooses the next UI element and interaction, and repeats this process through the mobile flow. CI executes the selected actions and observes whether the target trip or app flow completes.

Why this architecture

Uber used a language-model decision loop because mobile tests are sequences of context-dependent steps that must adapt to UI changes, cities, and languages. A tuned base MPNet model gave nearly the same retrieval quality as larger models while reducing latency and embedding cost, making it better suited for frequent production CI use.

Models used: MPNet base · MPNet large · T5 11B · RoBERTa · OpenAI DALL-E

Scale

• Uber described mobile testing at the scale of thousands of developers and over 3,000 simultaneous experiments.
  “Mobile testing remains an unresolved challenge, especially at our scale, encompassing thousands of developers and over 3,000 simultaneous experiments.”article, stated
• Script maintenance consumed 30-40% of engineers' time.
  “Consequently, engineers working on this invest 30–40% of their time on maintenance.”article, stated
• Uber operates in 50+ languages.
  “imagine having to hire manual testers or mobile engineers for the 50+ languages that we operate in!”article, stated
• DragonCrawl achieved 99%+ stability in November and December 2023.
  “DragonCrawl executed flows with 99%+ stability in November and December 2023.”article, stated
• DragonCrawl completed trips in 85 of Uber's top 89 cities.
  “We evaluated DragonCrawl in 89 of our top cities, and DragonCrawl successfully requested and completed trips in 85 of them.”article, stated

Alternatives considered (roads not taken)

Manual testing for every code change.

Rejected because: Manual testing had too much overhead to cover every minor change.

Traditional script-based mobile tests.

Rejected because: Scripts broke under minor UI changes and required recurring manual updates.

Use a larger or untuned model for action selection.

Rejected because: Base MPNet had high precision, lower latency, smaller embeddings, and avoided the quality loss expected from an untuned model under frequent app changes.

Patterns

tool-use-orchestration (orchestration)

DragonCrawl chooses concrete UI elements and interactions as actions in a mobile testing loop.

fine-tuning-adaptation (model-operations)

Uber tuned MPNet for the mobile-testing action-selection task rather than using a generic untuned model.

eval-driven-development (evaluation)

DragonCrawl model selection was guided by precision@N retrieval metrics over candidate actions.

latency-optimization (operations)

Uber selected base MPNet because it was fastest and had smaller embeddings while preserving high precision.

Tradeoffs

Mobile test representation

Chose Natural-language screen text plus test goals. over Hardcoded test scripts tied to specific UI paths..

Rationale: Language representation lets the model adapt to UI changes, goals, and non-English contexts.

Model selection

Chose Fine-tuned base MPNet. over Larger models such as MPNet large or T5 11B..

Rationale: Base MPNet had high precision, lower latency, 110M parameters, and 768-dimensional embeddings.

Tune despite acceptable untuned precision.

Chose A customized model. over Using untuned T5-11B with decent precision@N..

Rationale: The expected usage frequency and app variability made small precision differences important.

Gotchas (what bit them)

Script-based tests created a large maintenance burden.

Root cause: Small UI changes like new pop-ups and button changes disrupted scripted tests.

Resolution: DragonCrawl adapts to screen text and goals instead of hardcoding brittle paths.

Global testing did not scale with manual or script maintenance.

Root cause: Uber needed testing across diverse cities and 50+ languages.

Resolution: DragonCrawl uses language-model resilience and natural-language inputs to test across cities and languages without code tweaks.

Untuned model precision looked acceptable but was not enough for frequent use.

Root cause: Uber's app changes constantly, making small precision gaps costly over repeated CI runs.

Resolution: Uber selected a customized base MPNet model instead of accepting untuned T5 precision.

Evaluation

Uber evaluated action selection as a retrieval problem with precision@N, compared several transformer models, and then measured production CI stability, maintenance, city coverage, device/OS resilience, and bug detection.

Metrics: MPNet base precision@1 0.9723, precision@2 0.9623, precision@3 0.9423 · 110M parameters and 768-dimensional embeddings for base MPNet · 99%+ stability in November and December 2023 · 85 of 89 top cities successfully completed trips · Ten high-priority bugs blocked in three months

Outcomes

• Blocked ten high-priority bugs in the first three months.
• Ran core-trip testing with 99%+ stability in late 2023.
• Completed trips in 85 of Uber's top 89 cities without code tweaks.
• Eliminated manual maintenance for the tested DragonCrawl flows.

Uber cites thousands of developers, 3,000+ simultaneous experiments, 50+ languages, 89 top-city evaluations, 85 successful city completions, 99%+ stability, and ten blocked high-priority bugs.