Production-Scale NLP Suggestion System with Real-Time Text Processing

grammarly · Tech · 2022 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real-time writing-suggestion delivery and synchronization

Grammarly needed to serve ML-generated writing suggestions to a constantly changing editor while keeping suggestion positions, relevance, and batch application correct without freezing the UI or waiting for backend regeneration after every user edit.

Constraints:

Architecture

Grammarly represents document text, user edits, and ML suggestions as a unified Delta operational-transformation protocol. Backend services generate suggestions, while the client keeps them synchronized through a Suggestions Repository, Delta Manager, and Highlights Manager. Every user edit rebases active suggestion Deltas against the new document state, invalidates semantically stale suggestions, and composes multiple suggestion Deltas into a single batch operation when users accept many suggestions at once.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  backend-nlp-services["Backend NLP services"] --> suggestions-repository["Suggestions Repository"]
  suggestions-repository["Suggestions Repository"] --> delta-manager["Delta Manager"]
  suggestions-repository["Suggestions Repository"] --> highlights-manager["Highlights Manager"]
  delta-protocol["Delta protocol"] --> delta-manager["Delta Manager"]
  delta-protocol["Delta protocol"] --> editor-text-state["Editor text state"]
  editor-text-state["Editor text state"] --> delta-manager["Delta Manager"]
  editor-text-state["Editor text state"] --> highlights-manager["Highlights Manager"]
  user-edit-delta["User edit Delta"] --> delta-manager["Delta Manager"]
  suggestion-delta["Suggestion Delta"] --> delta-manager["Delta Manager"]
  suggestion-delta["Suggestion Delta"] --> batch-apply-composer["Batch apply composer"]
  delta-manager["Delta Manager"] --> relevance-invalidator["Relevance invalidator"]
  relevance-invalidator["Relevance invalidator"] --> suggestions-repository["Suggestions Repository"]
  delta-manager["Delta Manager"] --> batch-apply-composer["Batch apply composer"]
  batch-apply-composer["Batch apply composer"] --> editor-text-state["Editor text state"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Backend NLP services Scans text and generates writing suggestions. Suggestions Repository
Suggestions Repository Central client-side store for active suggestions and their states. Backend NLP services, Delta Manager, Highlights Manager
Delta protocol Unified operational transformation representation for text, edits, and suggestions. Delta Manager, Editor text state
Editor text state Current document content that changes as users type or apply suggestions. Delta protocol, Delta Manager, Highlights Manager
User edit Delta Representation of a user's live edit, used to shift or invalidate active suggestions. Delta Manager
Suggestion Delta Representation of an ML-generated change such as insertion, deletion, or retain operations. Delta Manager, Batch apply composer
Delta Manager Rebases suggestion positions and composes Deltas after edits or batch application. Suggestions Repository, Delta protocol, Editor text state, User edit Delta, Suggestion Delta, Relevance invalidator, Batch apply composer
Highlights Manager Renders mistakes and suggestion highlights in the user interface. Suggestions Repository, Editor text state
Relevance invalidator Hides suggestions that are no longer semantically valid after user edits. Delta Manager, Suggestions Repository
Batch apply composer Composes multiple accepted suggestion Deltas into one operation while rebasing later suggestions. Suggestion Delta, Delta Manager, Editor text state

Data flow

Backend NLP services send suggestions represented as Deltas into the Suggestions Repository. As the user edits, the client notifies the Delta Manager and Highlights Manager. The Delta Manager rebases each registered suggestion onto the edit Delta so retains and deletes still target the right text. Relevance logic hides suggestions invalidated by edits to semantically critical spans. When the user accepts a batch, the system rebases each suggestion onto the accumulating composed Delta and applies one combined edit to avoid repeated editor update cycles.

3 supporting quotes
“The Suggestions Repository serves as the central store for all active suggestions received from backend ML models.”summary, stated
“The Delta Manager is responsible for the critical task of keeping suggestion Deltas synchronized with the current text state through a continuous rebasing process.”summary, stated
“multiple Deltas can be composed together into a single Delta representing all changes at once.”summary, stated

Why this architecture

Grammarly's Delta abstraction lets the client adapt ML outputs to a changing document state without constantly asking backend models to regenerate suggestions. This preserves responsiveness at massive user scale and lets new suggestion types evolve without protocol changes. The rebase and compose logic moves hard correctness work into deterministic client-side state transformations rather than relying on model re-inference.

3 supporting quotes
“by using the same representation format for both user edits and ML suggestions, the system can apply the same transformation algorithms to both”summary, stated
“the client can maintain suggestion accuracy without backend involvement for every edit.”summary, stated
“Originally designed for single-word corrections, the system now handles suggestions that span sentences, paragraphs, or even entire documents”summary, stated

Models used: Grammarly backend NLP models · AI-generated writing suggestion models

Scale

Alternatives considered (roads not taken)

Wait for backend regeneration after every edit

Rejected because: Suggestions must remain instantly applicable and visually stable as users type.

“all registered suggestions must be updated to reflect the new document state—and this must happen instantly on the client side without waiting for the backend”summary, stated

Use separate representations for text, edits, and suggestions

Rejected because: A unified Delta representation lets the same transformation algorithms apply across user edits and ML suggestions.

“This protocol serves as the unified representation layer for three distinct types of data flows in the system: the document text itself, user-initiated edits, and AI-generated suggestions”summary, stated

Invalidate every sentence-level suggestion when any surrounding text changes

Rejected because: Some edits outside highlighted critical spans should preserve the suggestion.

“The system must distinguish between edits to highlighted portions (which should invalidate the suggestion) and edits to non-highlighted portions (which should preserve the suggestion).”summary, stated

Apply multiple accepted suggestions sequentially

Rejected because: Sequential application repeated the full update cycle and froze the UI.

“The naive implementation—iterating through suggestions and applying each one sequentially—technically works but creates serious UX problems when applying large batches.”summary, stated

Patterns

structured-output-contract (interface)

Deltas provide a compact, uniform contract for model suggestions, user edits, and document state transformations.

“A Delta consists of three operation types: "insert" for adding text, "delete" for removing text, and "retain" for specifying position.”summary, stated

model-output-rebasing (state-management)

The client rebases active model suggestions onto every user edit so positions and applicability remain correct without backend re-inference.

“The rebase procedure is the technical heart of how Grammarly maintains suggestion accuracy as documents evolve.”summary, stated

context-engineering (context)

The system manages a changing document context by invalidating stale suggestions and preserving suggestions whose critical spans remain valid.

“Beyond positional accuracy, suggestions must also remain semantically relevant. If a user independently fixes a mistake that a suggestion addresses, that suggestion should be immediately hidden as it's no longer useful.”summary, stated

Tradeoffs

Maintain suggestion accuracy on the client

Chose Client-side rebasing and invalidation over Backend regeneration for every edit.

Rationale: Client-side logic minimizes latency and backend load while preserving instant applicability.

“This dramatically reduces latency and backend load while ensuring responsive UX.”summary, stated

Use one protocol for all change types

Chose Delta operational transformation over Separate ad hoc formats for user edits and model suggestions.

Rationale: A single representation allows shared algorithms and future suggestion types without protocol changes.

“Notably, none of these advances required changes to the underlying protocol—a testament to the importance of building flexible abstractions when deploying ML systems in production.”summary, stated

Compose accepted suggestions into one operation

Chose Batch Delta composition with intermediate rebasing over Sequentially applying each accepted suggestion.

Rationale: Composition removes repeated expensive editor updates but requires careful offset correctness.

“By composing all suggestion Deltas before applying them to the text, the team transformed a repeated O(n) operation into a single operation”summary, stated

Gotchas (what bit them)

Suggestions become positionally wrong as users type.

Root cause: User edits shift text offsets after the backend has generated suggestion Deltas.

Resolution: Rebase suggestion Deltas onto each edit Delta on the client side.

“Every time a user makes an edit, all registered suggestions must be updated to reflect the new document state—and this must happen instantly on the client side without waiting for the backend to regenerate suggestions.”summary, stated

Suggestions can remain positionally correct but semantically stale.

Root cause: A user may independently fix or alter the text span that made a suggestion relevant.

Resolution: Track highlighted/critical spans and invalidate suggestions only when edits affect those spans.

“If a user independently fixes a mistake that a suggestion addresses, that suggestion should be immediately hidden as it's no longer useful.”summary, stated

Sequential batch application freezes the editor.

Root cause: The browser repeated the full update cycle for every accepted suggestion.

Resolution: Compose all suggestion Deltas into one operation while rebasing each against prior accepted suggestions.

“the most time-consuming operation was updating the text editor Delta, which was being repeated for every suggestion.”summary, stated

Batch composition can corrupt text if later suggestions are not rebased.

Root cause: Each subsequent suggestion's offsets assume the original document, not the intermediate state after earlier suggestions.

Resolution: Rebase each suggestion Delta onto the accumulating composed Delta before composing it.

“Without this rebasing step, the composed Delta would apply suggestions to incorrect positions, resulting in corrupted text with "characters all mixed up."”summary, stated

Evaluation

The article does not describe a formal model-evaluation harness; it focuses on deterministic correctness and performance of the suggestion orchestration layer, including rebasing correctness, invalidation behavior, and UI responsiveness during batch application.

Metrics: suggestion state · registered suggestions · applied suggestions · rebase operation correctness · UI freeze duration · batch application correctness · suggestion invalidation

“Each suggestion can exist in different states: "registered" (relevant and correct), "applied" (accepted by user), or removed (no longer relevant).”summary, stated
“suggestions must be instantly applicable when clicked, and cards must never flicker or point to incorrect text locations.”summary, stated

Outcomes

The system serves 30 million daily users and 30,000 professional teams; key operational scale concerns include instant client-side rebasing, several-second freezes in naive batch application, and protocol stability from single-word corrections to document-wide suggestions.

3 supporting quotes
“Originally designed for single-word corrections, the system now handles suggestions that span sentences, paragraphs, or even entire documents”summary, stated
“The architecture serves 30 million daily users and 30,000 professional teams, representing significant scale. This scale requirement drove many of the architectural decisions described in the article.”summary, stated
“The core concepts—using unified representations for model outputs and application state, building client-side intelligence to reduce backend dependencies”summary, stated