Panel Discussion: Real-World LLM Production Use Cases

various · Other · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production LLM use cases across customer service, extraction, and sales intelligence

Teams deploying LLMs need to turn generic capabilities such as retrieval, summarization, extraction, and evaluation into measurable customer value without over-engineering prompts or optimizing before production feedback exists.

Constraints:

• Property-management knowledge is scattered across PDFs, posters, email replies, handwritten notes, and staff memory.
• Resident support needs high question-resolution rates while reducing call-center and associate workload.
• A simple LLM capability can be low-value in one market and high-value in a specific workflow with expensive human labor.
• LLM development behaves more like experimentation than deterministic engineering, so long planning cycles can waste time.
• Evaluation needs to run quickly enough to build intuition and support continuous improvement.
• Customers may value human-feeling interactions more than obviously magical AI experiences.

Architecture

The panel presents a production playbook anchored by Resides' RAG-based property knowledge system: ingest unstructured property materials into a vector database, retrieve relevant context for resident questions, generate answers for customer-service workflows, and continuously evaluate conversations in production. Supporting examples extend the same playbook to structured transcript extraction for executive coaches and sales-call argument analysis, with project prioritization driven by customer value, revenue, fast experiments, and LLM-based evaluation loops.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Resident_question_intake["Resident question intake"] --> Context_retrieval_layer["Context retrieval layer"]
  Resident_question_intake --> Associate_workflow["Associate workflow"]
  Property_knowledge_ingestion["Property knowledge ingestion"] --> Vector_database["Vector database"]
  Property_knowledge_ingestion --> Context_retrieval_layer
  Vector_database --> Context_retrieval_layer
  Context_retrieval_layer --> LLM_answer_generator["LLM answer generator"]
  LLM_answer_generator --> Associate_workflow
  LLM_answer_generator --> Conversation_scoring_loop["Conversation scoring loop"]
  Associate_workflow --> Conversation_scoring_loop
  Conversation_scoring_loop --> Evaluation_example_store["Evaluation example store"]
  Conversation_scoring_loop --> LLM_answer_generator
  Evaluation_example_store --> Conversation_scoring_loop
  Transcript_extraction_workflow["Transcript extraction workflow"] --> Human_training_or_presentation_workflow["Human training or presentation workflow"]

Components

Data flow

Property manuals, posters, replies, and staff knowledge are collected into a retrieval store; resident questions retrieve relevant context and generate answers; unresolved or noteworthy conversations feed workflow scoring; examples are collected in batches, turned into scoring buckets, and used to refine future evaluation; separate transcript-analysis workflows extract high-value quotes or sales arguments and feed coaching, presentations, or sales training.

Why this architecture

The architecture emphasizes retrieval, fast production iteration, and value-specific extraction because the panelists repeatedly distinguish technical capability from business value: useful systems come from shipping, measuring, and improving workflows that customers will pay for, not from perfect prompts or generic demos.

Models used: LLMs · GPT-5 · Gemini · Instructor · LangSmith

Scale

• Resides says its resident-question resolution moved from about half of questions to roughly 95-99%.
  “we were answering maybe half of our resident's question in any in any given month”transcript, stated

  “today we're answering about 95 to 99% of Resident questions”transcript, stated
• Resides reports cutting expected associate headcount by about half.
  “it's cut down our expected headcount on the number of Associates we need by like half”transcript, stated
• Associate coverage reportedly increased from about 7,000-8,000 apartments to about 20,000 apartments per person.
  “one person supporting about seven to 8,000 Pro apartments at a time before this now they support maybe closer to 20,000”transcript, stated
• Executive coaches in Jason Liu's example charge about $800-$1,000 per hour and spend about two hours extracting quotes.
  “executive coach where these are people that will charge you know $800 to $1,000 an hour”transcript, stated

  “they will take about two hours to just extract quotes to prepare the presentation”transcript, stated
• The transcript extraction example contrasts a 50-cent API call with downstream value in the thousands of dollars.
  “the API call for gb5 ends up being 50 cents the actual upside for the downstream user becomes thousands of dollars”transcript, stated
• The sales-call analysis example used thousands of hours of transcripts and reportedly increased sales by 30%.
  “transcripts like thousands of hours of transcripts which we used to detect arguments and objections”transcript, stated

  “our customer used that argument and increased the sales by 30%”transcript, stated
• The evaluation workflow uses batches of 1,000 examples to develop and refine scoring categories.
  “collect a thousand examples use that to develop your your buckets use that to score your next thousand examples”transcript, stated
• The panel says LLMs can run about 100 tests per hour for workflow evaluation.
  “they can do 100 eye tests for you every every hour so you can continue improve your workflows”transcript, stated

Alternatives considered (roads not taken)

Over-optimize prompts and latency before shipping

Rejected because: Resides found this created week-scale lead times and delayed production feedback.

Define all conversation scoring categories upfront

Rejected because: The team spent months trying to define exact segments before discovering that simple scoring plus example collection was faster.

Plan deterministic AI work as a set of engineering tickets

Rejected because: Jason argues LLM optimization is closer to science than deterministic engineering, so teams need metrics, hypotheses, and experiments.

Prioritize AI projects by impact/effort alone

Rejected because: Resides found effort became so small that many projects looked attractive, so willingness to pay became a better prioritization signal.

Build for an obviously magical AI experience

Rejected because: Resides found its customers prefer AI interactions that feel human and fit the service workflow.

Patterns

rag-retrieval (retrieval)

Resides ingests unstructured property documents, replies, and questions into a vector database so resident support can retrieve building-specific answers.

eval-driven-development (evaluation)

The panel recommends shipping quickly, scoring workflows continuously, collecting examples, and using metrics and experiments rather than predicting all categories upfront.

tool-use-orchestration (workflow)

The production use cases connect LLM outputs to customer-service workflows, executive-coaching presentations, and sales-training programs rather than treating generation as the final product.

human-in-the-loop-gate (operations)

Resides uses AI to augment human associates, and the sales-analysis example feeds LLM-discovered arguments into human training rather than replacing all human judgment.

prompt-versioning (prompt-operations)

The panel's iterative scoring advice turns an initial broad prompt into categorized evaluation buckets and then uses those buckets on the next example batch.

observability-tracing (operations)

The panel frames production LLM work as continuous monitoring of workflow quality, tests, and data rather than a one-time launch.

Tradeoffs

Prioritize shipping and production evaluation over prompt/latency perfection.

Chose Fast MVP plus continuous LLM evaluation over Weeks of pre-shipment prompt and latency optimization.

Rationale: The team learned more by shipping and monitoring than by optimizing before real production examples existed.

Build a RAG repository from messy property knowledge rather than keep manual staff lookup.

Chose Vector database over unstructured property artifacts over Manual question routing back to property staff.

Rationale: The RAG repository turned bespoke offline knowledge into reusable answers and sharply improved resident question resolution.

Sell workflow value rather than raw extraction capability.

Chose Niche executive-coaching extraction tied to expensive human workflow over Generic transcript extraction or summarization.

Rationale: A 50-cent API capability can justify $100-$200 pricing when it saves expensive expert time and drives a valuable presentation workflow.

Use LLMs to evaluate LLM workflows.

Chose Automated LLM-based workflow evaluation over Manual-only validation of every workflow iteration.

Rationale: LLM judges can run many tests per hour and support continuous improvement, though the panel still anchors success in revenue and customer value.

Make AI feel human in customer-service contexts rather than obviously magical.

Chose Human-feeling service experience over Flashy AI-first product behavior.

Rationale: Resides' customers preferred interactions that fit existing human service expectations.

Gotchas (what bit them)

Over-engineering prompts before shipment slowed production learning.

Root cause: The team treated LLM development as an engineering optimization problem instead of a product-feedback loop.

Resolution: Ship a simple version, evaluate continuously in production, and improve from real examples.

Trying to design perfect scoring categories upfront wasted months.

Root cause: The team attempted to predict exact segments before seeing enough real scored examples.

Resolution: Start with a broad scoring prompt, collect 1,000 examples, derive buckets, then score the next 1,000 examples.

Capability demos can obscure whether the workflow creates economic value.

Root cause: The same summarization or extraction capability can be nearly free in one context and worth thousands in another.

Resolution: Tie the LLM capability to a specific user, workflow, willingness to pay, and downstream outcome.

Traditional impact/effort prioritization can break when LLM experiment effort collapses.

Root cause: If many experiments become cheap, effort no longer discriminates between priorities.

Resolution: Prioritize by customer willingness to pay and business value rather than effort alone.

Research excitement can distract from user value.

Root cause: Builders can chase the latest AI technique rather than the customer outcome.

Resolution: Start from user value, a clear headline, or a press-release-style product promise before building.

Evaluation

The panel recommends practical production evaluation: resolution rates, productivity, revenue, willingness to pay, fast tests, example-driven scoring buckets, LLM-as-judge loops, and reviewing high- and low-performing examples to build intuition.

Metrics: Resident question resolution rate · Associate apartments supported · Expected headcount reduction · Training time · Sales lift · Revenue · Customer willingness to pay · Experiment run time · High- and low-performing example review · Workflow value · Human productivity

Outcomes

• Resides reports increasing resident question resolution from about 50% to 95-99%.
• Resides reports cutting expected associate headcount roughly in half while increasing apartment coverage per associate.
• The executive-coaching extraction example turns a low-cost API call into high-value expert workflow output.
• The sales-call analysis example reportedly increased sales by 30% by identifying better arguments for objections.
• The panel converges on fast experimentation, continuous LLM evaluation, and customer-value prioritization as production lessons.

The panel includes quantified outcomes across several production contexts: 95-99% question resolution, 7,000-8,000 to about 20,000 apartments per associate, $800-$1,000/hour executive coaching, 50-cent API calls priced at $100-$200 by value, thousands of transcript hours, a 30% sales increase, 1,000-example scoring loops, and 100 LLM tests per hour.