On-Device Grammar Correction with Sequence-to-Sequence Models

google · Tech · 2021 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Tech

Google needed real-time grammar correction in Gboard on mobile devices without sending user typing to a cloud service.

Constraints:

• mobile memory and compute limits
• low typing-latency requirements
• processing should remain on-device for privacy
• partial sentence prefixes must be corrected

Architecture

Google built Gboard grammar correction as a compact on-device sequence-to-sequence system. The runtime model combines a Transformer encoder with an LSTM decoder, shares/factorizes embeddings to keep a large token vocabulary without a large footprint, and uses post-training quantization to store 32-bit weights as 8-bit values. Training uses hard distillation: a cloud grammar model corrects hundreds of millions of public-web English sentences, then a prefix-capable cloud model generates sentence-prefix pairs so the smaller device model learns both complete-sentence and as-you-type correction. At inference time, Gboard calls the local model after more than three words, aligns original and corrected text with Levenshtein distance, and turns sentence edits into user-facing replacement suggestions.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  cloud-grammar-model["cloud-grammar-model"] --> distilled-training-data["distilled-training-data"]
  distilled-training-data --> on-device-grammar-model["on-device-grammar-model"]
  on-device-grammar-model --> transformer-encoder["transformer-encoder"]
  on-device-grammar-model --> lstm-decoder["lstm-decoder"]
  transformer-encoder --> shared-factorized-embeddings["shared-factorized-embeddings"]
  lstm-decoder --> shared-factorized-embeddings
  shared-factorized-embeddings --> quantized-weights["quantized-weights"]
  gboard-client["gboard-client"] --> on-device-grammar-model
  gboard-client --> edit-alignment["edit-alignment"]

Components

Data flow

Training starts with original/corrected text pairs produced by a cloud grammar model over hundreds of millions of web sentences. Google adds a second sentence-prefix dataset by sampling aligned prefixes, completing original prefixes with a neural language model, and distilling from a prefix-capable cloud model. The combined full-sentence and prefix data trains the compact on-device model. At runtime, Gboard sends typed text to the local grammar model once the user has typed more than three words; the model returns a corrected sentence, an alignment step minimizes Levenshtein distance between original and corrected text, and the app converts insertions/deletions into replacement suggestions.

Why this architecture

The architecture is shaped by phone constraints: grammar correction must run locally for privacy, fit mobile memory/compute limits, and remain fast enough to trigger while typing. The hybrid Transformer/LSTM model balances quality and latency; shared/factorized embeddings and quantization keep the model small; hard distillation transfers cloud-model behavior into a device-sized model; and prefix data handles the mobile reality that users type and send incomplete sentences.

Models used: cloud-based grammar model · on-device grammar model

Scale

• The compressed device model is 20MB and runs inference on 60 input characters in under 22ms on a Pixel 6 CPU.
  “the resulting model takes up only 20MB of storage and performs inference on 60 input characters under 22ms on the Google Pixel 6 CPU.”article, stated
• Distillation used hundreds of millions of English sentences from the public web.
  “We first collected hundreds of millions of English sentences from across the public web.”article, stated
• Runtime correction starts once the user has typed more than three words.
  “Gboard sends a request to the on-device grammar model whenever the user has typed more than three words, whether the sentence is completed or not.”article, stated
• The feature launched for English on Pixel 6 and is available on almost any app with Gboard, with more languages planned.
  “Currently, the feature is capable of correcting English sentences (we plan to expand to more languages in the near future) and available on almost any app with Gboard”article, stated

Alternatives considered (roads not taken)

Using the original cloud-model training data directly for the small on-device model

Rejected because: Hard distillation produced higher-quality suggestions for a similar-sized on-device model.

Triggering grammar correction only on complete sentences

Rejected because: Mobile messaging often lacks a final period, so complete-sentence-only correction would miss many typing-time errors.

Using a larger remote grammar system at inference time

Rejected because: The product goal was private, low-latency correction on the device without sending keystrokes to a remote server.

Patterns

model-distillation (model-operations)

A large cloud grammar model generated corrections used to train a smaller on-device model.

latency-optimization (serving)

The model was compressed enough to run 60-character inference under 22ms on a Pixel 6 CPU.

privacy-preserving-processing (governance)

The system keeps grammar correction local to the device instead of sending keystrokes to a remote model.

Tradeoffs

Compress the model for on-device latency while preserving acceptable quality.

Chose factorized embeddings and quantization over a larger cloud-scale model on the phone.

Rationale: The source states mobile constraints required controlling memory and latency, and that lower-fidelity weights did not materially affect model quality.

Use hard distillation for device-domain training data.

Chose cloud-generated correction pairs matched to the on-device domain over the original data used to train the large cloud grammar model.

Rationale: A similar-sized model trained from distilled data produced higher-quality suggestions.

Train on sentence prefixes as well as complete sentences.

Chose combined full-sentence and sentence-prefix training data over complete-sentence-only grammar correction.

Rationale: Typing-time mobile correction would miss many errors if it waited for completed sentences.

Gotchas (what bit them)

Mobile grammar correction must handle incomplete sentences, not just complete sentences.

Root cause: Messaging users often send text without a final period, and correction can happen while they type.

Resolution: Google generated sentence-prefix training pairs and combined them with full-sentence pairs.

Evaluation

The source describes quality comparisons between distilled on-device models and similar-sized models trained on original cloud-model data.

Metrics: model quality · model size · inference latency

Outcomes

• The model runs under 22ms on 60 characters with a 20MB footprint.

The feature is deployed through Gboard on Pixel 6.