Production-Ready Question Generation System Using Fine-Tuned T5 Models

digits · Finance · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Accounting transaction clarification question generation

Digits needed to reduce repetitive accountant-client communication by suggesting transaction clarification questions while preserving financial-data privacy, professional tone, safety, and deployment reliability.

Constraints:

• Generated questions may be sent to clients, so toxicity, hallucinations, and awkward wording can harm professional trust.
• Financial transaction data is sensitive and should not be shared with external services without consent.
• Training and serving preprocessing must stay consistent to avoid training-serving skew.
• The model should generate semantically correct but not memorized questions.

Architecture

Digits fine-tunes a T5 encoder-decoder model on accounting-specific examples with transaction descriptions and persona/style inputs. A TFX pipeline on Vertex AI handles preprocessing with TensorFlow Transform, training, metadata, and evaluation. The exported model packages preprocessing, the fine-tuned T5 model, and detokenization so TensorFlow Serving can accept raw text. Generated questions pass toxicity screening, hallucination pattern matching, and mandatory human accountant review before reaching clients.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  transaction-description["Transaction description"] --> tfx-preprocessing-graph["TFX preprocessing graph"]
  persona-input["Persona input"] --> tfx-preprocessing-graph["TFX preprocessing graph"]
  tfx-preprocessing-graph["TFX preprocessing graph"] --> fine-tuned-t5-model["Fine-tuned T5 model"]
  tfx-preprocessing-graph["TFX preprocessing graph"] --> tfx-training-pipeline["TFX training pipeline"]
  tfx-training-pipeline["TFX training pipeline"] --> fine-tuned-t5-model["Fine-tuned T5 model"]
  tfx-training-pipeline["TFX training pipeline"] --> evaluation-component["Evaluation component"]
  tfx-training-pipeline["TFX training pipeline"] --> metadata-store["Metadata store"]
  fine-tuned-t5-model["Fine-tuned T5 model"] --> tensorflow-serving-signature["TensorFlow Serving signature"]
  tensorflow-serving-signature["TensorFlow Serving signature"] --> safety-filters["Safety filters"]
  safety-filters["Safety filters"] --> ml-team-alerts["ML team alerts"]
  safety-filters["Safety filters"] --> human-accountant-review["Human accountant review"]
  fine-tuned-t5-model["Fine-tuned T5 model"] --> evaluation-component["Evaluation component"]

Components

Data flow

A transaction description and persona enter the preprocessing graph, which tokenizes, pads, creates attention masks, and converts tensors. The fine-tuned T5 model generates a candidate question and detokenizes it into readable text inside the served model. Safety checks discard toxic, insulting, obscene, or known hallucination-pattern outputs and alert the ML team. A human accountant then reviews, edits, or sends the suggested question to the client.

Why this architecture

Digits uses a fine-tuned T5 model rather than generic prompting because the task is narrow, repetitive, domain-specific, and privacy-sensitive. Packaging preprocessing and detokenization with the served model reduces training-serving skew and removes an extra Python layer. Layered safety plus accountant review acknowledges that generated business communications cannot be trusted blindly.

Models used: T5 · fine-tuned T5 · TensorFlow Text FastSentencePieceTokenizer · TensorFlow Serving

Scale

• The article was published in March 2023.
  “This case study, published in March 2023, details how they implemented generative machine learning”summary, stated
• The article references GPT-3 3B training as requiring 50 petaflop/s-days of compute.
  “OpenAI's GPT-3 3B model required 50 petaflop/s-days of compute”summary, stated
• The evaluation component compares model versions against five metrics.
  “Checks model versions against five different metrics”summary, stated
• Every generated client-facing question must pass human accountant review.
  “A human accountant always reviews and confirms any suggested question before it is sent to a client.”summary, stated

Alternatives considered (roads not taken)

Training a generative model from scratch

Rejected because: Training from scratch would require massive compute compared with fine-tuning a pretrained T5 model.

Separate tokenization service at inference

Rejected because: Keeping preprocessing outside the served model increases deployment complexity and training-serving skew risk.

Trusting generated questions after automated checks only

Rejected because: Professional accounting communications still require human confirmation before client delivery.

Patterns

fine-tuning-adaptation (model-adaptation)

Digits adapts a pretrained T5 model to accounting transaction clarification using domain-specific training data.

human-in-the-loop-gate (governance)

A human accountant must approve every generated question before it reaches a client.

safety-policy-enforcement (safety)

The system discards outputs flagged for toxicity, insults, obscenity, or hallucination patterns before human review.

eval-driven-development (quality)

A custom TFX evaluation component compares versions using complementary metrics before deployment recommendations.

structured-output-contract (deployment)

The exported serving signature packages preprocessing, generation, and detokenization into one model contract that accepts raw text and returns readable questions.

Tradeoffs

Fine-tune T5 rather than train from scratch

Chose Domain adaptation of a pretrained encoder-decoder model over Training a generative language model from zero.

Rationale: Fine-tuning gives accounting-domain control while avoiding the compute cost of pretraining.

Embed preprocessing in the served model

Chose All-in-one TensorFlow Serving signature over Separate preprocessing/tokenization service.

Rationale: Packaging preprocessing and detokenization with the model reduces deployment complexity and skew risk.

Automate deployment recommendations with metrics

Chose Custom TFX evaluation component over Human-only release decisions.

Rationale: Quantitative gates make model version comparisons more consistent, while human review still gates client-facing content.

Gotchas (what bit them)

Autoregressive generation can loop into nonsense.

Root cause: Token-by-token generation can compound errors and fail to produce a stop token.

Resolution: Detect hallucination patterns and discard problematic outputs before review.

Training-serving skew can break production behavior.

Root cause: Preprocessing during training may differ from preprocessing used at inference.

Resolution: Export the TensorFlow Transform preprocessing graph alongside the model.

Human review can become a throughput bottleneck.

Root cause: Every suggested question requires accountant confirmation before reaching a client.

Resolution: Keep the human gate for safety while using automation to reduce drafting effort rather than fully automate sending.

Evaluation must reward semantic correctness without rewarding memorization.

Root cause: A model can match references lexically without producing diverse, useful wording.

Resolution: Use high Levenshtein distance together with high semantic similarity and ROUGE/token checks.

Evaluation

Digits evaluates every training pipeline with a custom TFX component comparing the candidate model against the last released version using five metrics, plus a curated human-written evaluation dataset.

Metrics: Levenshtein distance · semantic similarity · ROUGE · must-have token scoring · optional token scoring · toxicity screening · hallucination pattern matching

Outcomes

• Digits deployed a domain-specific fine-tuned T5 system that suggests accountant-client clarification questions.
• The production model accepts raw text inputs because preprocessing and detokenization are packaged into the served model.
• Safety checks and human review prevent generated content from going directly to clients.
• Evaluation compares model versions quantitatively before deployment recommendations.

The case gives limited production-volume metrics; concrete anchors include March 2023 publication, five evaluation metrics, GPT-3 3B's 50 petaflop/s-days training reference, and mandatory human review for every client-facing question.