Linguistic-Informed Approach to Production LLM Systems

mastercard · Finance · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: production LLM prompting and representation

Mastercard's talk argues that teams building LLM systems often optimize metrics without clear goals and miss linguistic failure modes in tokenization, pragmatics, phonetics, and model longevity.

Constraints:

• LLMs solve language tasks, so syntax, morphology, semantics, pragmatics, and phonetics shape model behavior.
• Statistical tokenization can hide arithmetic or borrowed-word structure from the model.
• Production systems need goal-setting and longevity planning, not only precision, recall, or F1 optimization.
• Financial-domain language may change more slowly than public internet language, affecting retraining and maintenance cadence.

Architecture

The proposed operating model is a linguistically informed LLM development framework. Teams set explicit goals first, then reason about five linguistic dimensions: syntax, morphology/tokenization, semantics, pragmatics/context, and phonetics/speech. The talk demonstrates that pragmatic instruction with Guidance and Falcon 7B Instruct can improve a biology-question task from 7/20 to 17/20 while reducing runtime from about one minute to about two seconds, and it frames multilingual tokenization and phonetic input as ways to preserve language information that plain text can lose.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Goal_definition["Goal definition"] --> Linguistic_analysis_framework["Linguistic analysis framework"]
  Linguistic_analysis_framework --> Tokenization_strategy["Tokenization strategy"]
  Linguistic_analysis_framework --> Pragmatic_instruction_layer["Pragmatic instruction layer"]
  Linguistic_analysis_framework --> Phonetic_representation_layer["Phonetic representation layer"]
  Tokenization_strategy --> Model_runtime["Model runtime"]
  Pragmatic_instruction_layer --> Model_runtime
  Model_runtime --> Retrieval_tools["Retrieval tools"]

Components

Data flow

A production team starts by defining the use-case goal and required lifetime. The team then chooses how much linguistic structure to preserve: tokenization for morphology, embeddings for meaning, prompt/context rules for pragmatics, retrieval or LangChain-style tools for external context, and phonetic/audio representations for speech tasks. For the biology demo, Guidance wraps Falcon 7B Instruct with pragmatic instructions and repeated prompting so the local model answers a 20-question set faster and more accurately than a vanilla ChatGPT-style query.

Why this architecture

The talk argues that language structure should drive LLM design decisions because raw metric optimization and statistical tokenization miss important meaning. Pragmatic instruction can improve both accuracy and speed, multilingual exposure can improve tokenization, and phonetic representations matter for speech meanings that text alone cannot express.

Models used: Falcon 7B Instruct · Guidance · ChatGPT / OpenAI framework · Goat 7B · GPT-4 · LangChain · vector databases · Tortoise · ElevenLabs · SVC · phonetic plus model

Scale

• The biology demo used 20 college-level biology questions.
  “it queries it for 20 college-level biology questions.”transcript, stated
• The vanilla ChatGPT-style query scored 7 out of 20.
  “using the OpenAI framework, it got seven out of 20”transcript, stated
• The Guidance/Falcon setup scored 17 out of 20.
  “it ended up getting 17 out of 20 instead of seven out of 20”transcript, stated
• The talk compares roughly one-minute warmed OpenAI runs with about two seconds for the local Guidance/Falcon run.
  “it took about 1 minute to do once the API had warmed up”transcript, stated

  “I'll just tell you it took about 2 seconds to play.”transcript, stated
• The speaker frames production longevity with an explicit 2030 horizon.
  “Is your LLM going to just be a period piece for 2023, or is it going to go further? Can you make it last to 2030”transcript, stated

Alternatives considered (roads not taken)

Optimize only for metrics

Rejected because: The speaker says KPIs are not goals unless the team is already late-stage or lacks direction.

Use purely frequency-based tokenization

Rejected because: The talk argues that morphemes are not frequency-defined and arithmetic can fail when numbers are grouped statistically.

Represent speech meaning as text only

Rejected because: The speaker says emphasis changes the meaning of the sentence and is lost when reduced to text.

Patterns

prompt-chaining (prompting)

Guidance is used to provide pragmatic instruction, chain-of-thought, repeated prompts, and system prompts around the model.

latency-optimization (serving)

The talk claims the local Guidance/Falcon setup completed the 20-question biology task in about two seconds rather than about a minute.

rag-retrieval (retrieval)

Retrieval tools such as LangChain and vector databases are named as additional ways to add context and document retrieval to pragmatic instruction.

model-routing (architecture)

The talk compares when model maintenance needs differ by domain, such as slow-changing financial language at Mastercard.

Tradeoffs

Production maintenance strategy

Chose Domain-specific longevity planning over Treating all LLMs as equally perishable public-language snapshots.

Rationale: The speaker notes that financial language at Mastercard may change more slowly, so model-longevity choices depend on domain language drift.

Instruction style for the biology task

Chose Pragmatic Guidance around Falcon 7B Instruct over Vanilla ChatGPT query through the OpenAI framework.

Rationale: The speaker reports much better accuracy and speed for the same 20-question task using local pragmatic instruction.

Gotchas (what bit them)

Metric-only optimization can send an LLM project in the wrong direction.

Root cause: Precision, recall, and F1 are treated as goals before the team defines the desired system behavior.

Resolution: Define the production goal and desired model lifetime before optimizing KPIs.

Frequency-based tokenization can make arithmetic harder.

Root cause: Commonly grouped numbers become tokens that prevent the model from seeing the mathematical structure.

Resolution: Consider morphology and tokenization strategy explicitly instead of assuming frequent token groups are semantically right.

Borrowed words can be split into misleading morphemes.

Root cause: A monolingual tokenizer may identify familiar English pieces inside a borrowed word with different morphological structure.

Resolution: Use multilingual exposure or linguistic knowledge to improve tokenization and word-boundary handling.

Text-only speech representation loses meaning carried by emphasis.

Root cause: The same words can mean different things depending on which word is emphasized.

Resolution: For speech tasks, consider phonetic and audio-reference representations in addition to text.

Evaluation

The talk uses a 20-question biology demo as its concrete evaluation: vanilla ChatGPT via OpenAI framework versus Falcon 7B Instruct with Guidance-style pragmatic instruction.

Metrics: questions answered correctly out of 20 · runtime before and after API warm-up · local runtime with Guidance · reported accuracy and speed improvement

Outcomes

• The demo improved biology-question accuracy from 7/20 to 17/20.
• The demo reduced reported runtime from about one warmed minute to about two seconds.
• The broader contribution is a linguistic checklist for LLM production design rather than a deployed Mastercard product metric.

The source is a conference talk with one 20-question demo and qualitative Mastercard financial-language context; it is not a production case study with deployment volume.