LLM Production Case Studies: Consulting Database Search, Automotive Showroom Assistant, and Banking Development Tools

globant · Consulting · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production LLM consulting delivery

Globant's production LLM projects repeatedly hit failures caused by ill-defined retrieval semantics, non-canonical source data, hallucinating platform services, vague success criteria, and premature technology commitments.

Constraints:

• The consulting database project could not create supplementary databases or modify the client's core SQL database.
• Business users expected deterministic, fast BI-like responses despite LLM non-determinism.
• The automotive showroom assistant had non-canonical source documents across countries and strict restrictions on price, competitor, and off-brand responses.
• A required cloud file-search service introduced hallucinated metadata summaries.
• The banking copilot POC had unclear success criteria and buzzword-driven fine-tuning expectations.
• The talk aggregates five cases rather than describing one unified production architecture.

Architecture

Globant describes a set of practical workarounds across five LLM projects: text-to-SQL plus LLM-generated keyword filters and row-level LLM reranking for a financial database; a multi-layer automotive RAG workaround that ignores hallucinated provider summaries and regenerates safer metadata; hybrid vector and SQL retrieval for internal knowledge access; LLM middleware for an underdefined banking copilot POC; and a GenAI accelerator platform where technology adoption had to be disciplined by use-case fit rather than benchmarks.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  SQL["Financial text-to-SQL layer"] --> Keywords["LLM keyword mapper"]
  Keywords --> Rerank["LLM row filter and reranker"]
  FileSearch["Automotive file search"] --> Clean["Clean-summary LLM"]
  Clean --> Eval["Stage-by-stage evaluator"]
  Hybrid["Knowledge assistant hybrid retrieval"] --> Eval
  Bank["Bank copilot middleware"] --> Eval
  Accelerator["GenAI accelerator platform"] --> Eval
  Rerank --> Eval

Components

Data flow

Across the cases, raw user questions first meet use-case-specific retrieval or structuring: SQL queries and LLM keyword generation for financial data, provider file search followed by LLM-created clean summaries for automotive documents, embeddings for structured reports and SQL recovery for success cases, and LLM middleware for code-assistant gaps. Each flow then needs independent measurement of retrieval, ranking, generation, factuality, tone, and instruction following rather than a single end-to-end score.

Why this architecture

The recurring architecture lesson is to avoid defaulting to embeddings-plus-vector search. Globant matched retrieval and data-shaping techniques to each case: SQL remained necessary where structured data and determinism mattered, graph-like relationships would have better captured belonging/hierarchy, LLM metadata generation compensated for unmodifiable sources, and stage-by-stage evaluation isolated where failures actually occurred.

Models used: LLMs · GPT-3.5 · legacy pre-GPT-4 model

Scale

• Globant frames its experience as more than 10 years and more than 400 AI projects.
  “we have delivered this for more than 10 years for more than 400 projects”transcript, stated
• The financial case involved a huge proprietary database of market caps, valuations, M&A, and financial information.
  “they have this huge database of um market caps valuations mergers Acquisitions operations uh a lot of financial data”transcript, stated
• The banking copilot was constrained to GPT-3.5 because GPT-4 was considered too expensive.
  “let's use TBT 3.5 because four was too expensive at the time and they didn't want to commit a lot of uh budget”transcript, stated
• The accelerator-platform theme targeted reducing experimentation from a day to an hour.
  “enabling the platforms that help companies accelerate their Innovation cycle and go from a day to an hour in experimentation”transcript, stated

Alternatives considered (roads not taken)

Pure vector similarity for financial search

Rejected because: Similarity did not capture business notions like belonging, dependency, or hierarchy, and spurious retrieval had very low tolerance.

Trust cloud-provider file-search summaries

Rejected because: The provider summaries introduced hallucinated content, including removed price information.

Embeddings plus vector database for every document type

Rejected because: Structured technical reports worked with embeddings, but success cases performed better after pre-structuring into SQL-style recovery.

Accept client buzzword requirements without pushback

Rejected because: The bank POC lacked outcome definitions and fine-tuning understanding, creating reputational risk.

Patterns

hybrid-search (retrieval)

Globant combines SQL/text-to-SQL, keyword LIKE filters, embeddings, and vector retrieval depending on source type and business semantics.

retrieval-ranking (retrieval)

LLMs filter and rerank retrieved rows/documents when the first-stage retrieval mechanism cannot fully express relevance.

eval-driven-development (evaluation)

The talk argues for measuring retrieval, ranking, generation factuality, tone, instruction following, and input filtering separately so failures are assignable.

data-quality-loop (data)

The cases repeatedly show that canonical documents, metadata, structured sources, and data access determine whether the LLM system works.

human-in-the-loop-gate (governance)

Globant frames client challenge, validation-point setting, and evidence-based pushback as necessary gates before accepting flawed technical commitments.

Tradeoffs

Use SQL and LLM keyword generation where vector similarity was too loose

Chose text-to-SQL plus LIKE keywords and LLM reranking over pure vector database similarity.

Rationale: The client's data constraints blocked metadata stores, and business relevance required deterministic structured filtering plus lean post-processing.

Layer LLMs around flawed platform services

Chose LLM-generated clean summaries and metadata on the fly over trusting the provider's file-search summaries.

Rationale: The required provider component was injecting hallucinated summaries, so Globant used LLMs to sanitize and rank source content while still satisfying platform constraints.

Risk relationship friction to avoid unrecoverable delivery risk

Chose challenge assumptions and say no when needed over accepting vague POC commitments for relationship comfort.

Rationale: The talk argues that agreeing to flawed requirements creates worse reputational risk than early conflict.

Gotchas (what bit them)

Vector similarity can fail when business users mean belonging or hierarchy, not semantic closeness.

Root cause: The financial-search project treated similarity as the organizing concept even though user expectations depended on industry/domain relationships.

Resolution: Use text-to-SQL, LLM-generated keywords, and row-level filtering/reranking; consider graph structures when belonging/hierarchy matter.

Retrieval infrastructure can hallucinate before generation even starts.

Root cause: The cloud file-search service generated summaries with an internal LLM and introduced removed price information.

Resolution: Ignore provider summaries, regenerate clean summaries from source documents, and measure each RAG stage independently.

Non-canonical source documents turn correct answers into stakeholder disputes.

Root cause: Different country documents described the same automotive products inconsistently, and stakeholders sometimes disagreed with their own source docs.

Resolution: Use measured evidence showing the model matched the provided document and require the client to fix canonical data.

Buzzword-driven fine-tuning POCs can lack measurable success criteria.

Root cause: The bank knew it wanted fine-tuning and productivity improvement but had not defined outcomes or measurement.

Resolution: Push back early rather than accepting commitments that cannot be validated.

Evaluation

Globant's core evaluation recommendation is stage-by-stage measurement: retrieval relevance, ranking quality, generation factuality, tone, instruction following, input filtering, and use-case-specific effectiveness should be evaluated separately.

Metrics: document relevance · ranking quality · generation factuality · usefulness · tone compliance · instruction following · input filtering · response time

Outcomes

• The financial case was salvaged with text-to-SQL, LLM keyword mapping, and row-level filtering/reranking under strict data-modification constraints.
• The automotive case identified retrieval-stage hallucinations from required cloud services and worked around them with LLM-generated clean summaries.
• The internal knowledge-assistant case showed that structured SQL recovery can outperform vector retrieval for image-heavy success cases.
• The bank copilot case became a project-governance lesson about refusing vague fine-tuning/productivity commitments.
• The talk distilled a cross-case lesson: data sources, evaluation, and use-case fit matter more than defaulting to embeddings, benchmarks, or buzzwords.

The presentation draws on Globant's claimed 10+ years and 400+ AI projects, but the individual cases are anonymized and do not provide production volume metrics.