Practical Lessons Learned from Building and Deploying GenAI Applications

bolbeck · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production GenAI application architecture and operations

Bolbeck needed to explain the practical operational burden that appears when teams add LLMs and agents to ordinary applications: model choice, hosting, hallucination control, RAG data quality, fine-tuning risk, evaluation, prompt lifecycle, agent latency, cost, and observability all become production concerns.

Constraints:

• AI code generation does not build complete applications automatically despite marketing claims.
• Adding AI introduces model selection, fine-tuning, prompt engineering, RAG, hallucination prevention, model replacement, and GPU hosting concerns.
• Benchmarks do not guarantee that a new model works for a particular application.
• Guardrails add latency and cost, and their classifier is still probabilistic.
• RAG answers are only as fresh and accurate as the vector-database contents.
• Fine-tuning is expensive, slow, and can degrade quality if done poorly.
• Agent tool calls multiply latency and cost across sequential LLM calls.
• Agent behavior is probabilistic and hard to debug without traces.

Architecture

The talk proposes a production GenAI stack with explicit layers: choose and host models locally or in the cloud; evaluate candidate models continuously with tools such as Ollama, Hugging Face, OpenRouter, and LangSmith; improve quality through prompt engineering, guardrail classifiers, RAG, and selective fine-tuning; externalize prompts in repositories such as LangChain Hub so experts can test and commit prompt changes; build agents that route user tasks through LLM assistants and tool/function calls; optimize agent latency with parallel calls and branching; model token/provider costs before deployment; and instrument agent traces so probabilistic failures can be debugged.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  AI_application_frontend["AI application frontend"] --> Model_selection_layer["Model selection layer"]
  Model_selection_layer --> Hosting_layer["Hosting layer"]
  Hosting_layer --> Cost_model["Cost model"]
  AI_application_frontend --> Prompt_repository["Prompt repository"]
  Prompt_repository --> Domain_expert_workflow["Domain expert workflow"]
  AI_application_frontend --> Main_LLM["Main LLM"]
  Guardrail_classifier["Guardrail classifier"] --> Main_LLM
  RAG_vector_database["RAG vector database"] --> Main_LLM
  Fine_tuning_workflow["Fine-tuning workflow"] --> Model_selection_layer
  Evaluation_toolkit["Evaluation toolkit"] --> Model_selection_layer
  Main_LLM --> Agent_assistant["Agent assistant"]
  Agent_assistant --> Tool_function_layer["Tool/function layer"]
  Parallel_execution_layer["Parallel execution layer"] --> Agent_assistant
  Agent_assistant --> Observability_platform["Observability platform"]
  Observability_platform --> Evaluation_toolkit
  Cost_model --> Agent_assistant

Components

Data flow

A user request enters an AI application whose model, hosting provider, and quality controls have been selected through evaluation. For direct chat or RAG flows, prompts and retrieved company data go to an LLM, while guardrail classifiers may run before or after the main call. For agentic flows, the assistant LLM decides which tools or functions to call, may run some calls in parallel, and returns the final answer. Prompt updates are externalized through a hub so domain experts can test and commit changes. LangSmith-style traces record node order, metadata, inputs, outputs, errors, and tool calls for debugging.

Why this architecture

The stack is layered because each production risk requires a different control: local and cloud hosting for exploration versus deployment, evaluation tools for constant model churn, guardrails and RAG for answer quality, prompt repositories for collaboration and future-proofing, agents and tools for real-world action, parallel branches for latency, cost modeling for economics, and observability for debugging probabilistic multi-node behavior.

Models used: GPT-3.5 · GPT-4V · OpenAI o1 · DeepSeek V3 · Claude · Llama 3 · Llama 3.1 70B · Llama 3.3 70B · Mistral · Llama Guard

Scale

• The lessons are drawn from a year and a half of building GenAI applications.
  “lessons I've learned from building gen application over the last year and a half”transcript, stated
• The speaker has more than 15 years of IT experience.
  “I have over 15 years experience uh in the IT industry”transcript, stated
• A chatbot prototype can now be built in an hour or two, but correctness remains hard.
  “building a chat B power di is very simple you can build it in an hour or two”transcript, stated
• The call-center example assumes 3,000 calls per day.
  “this call center takes 3,000 calls a day and each one of those calls requires 15 uh function calls or tool calls”transcript, stated
• The cost model assumes 15 tool/function calls per call.
  “each one of those calls requires 15 uh function calls or tool calls”transcript, stated
• The o1 example uses 1,500 input tokens and 3,000 output tokens.
  “inputting 1500 input tokens and 3,000 output tokens”transcript, stated
• The o1 call-center estimate reaches almost $300,000 per month.
  “which turn is almost $300,000 per month”transcript, stated
• Switching the example to Llama 3.3 70B reduces the monthly estimate to almost $50,000.
  “we go all from almost $300,000 per month to almost $50,000 per month”transcript, stated
• A100 GPU hosting is described as four or five dollars per hour versus cents for CPU instances.
  “having an a100 GPU for an hour will cost you like potentially four or five dollars an hour”transcript, stated

Alternatives considered (roads not taken)

Assume AI coding tools will build the whole application

Rejected because: The speaker says coding assistants can help and produce code, but they do not build the full application for you.

Pick models based on benchmarks alone

Rejected because: The speaker warns that better benchmark scores do not mean the model works for the application.

Hardcode prompts in application code

Rejected because: Externalized prompts let domain experts edit, test, and commit prompts without coding and make model upgrades easier.

Use a single expensive frontier model for every call

Rejected because: The cost example shows a dramatic monthly difference when replacing OpenAI o1 with Llama 3.3 70B if the cheaper model can do the task.

Run agent tool calls serially when they are independent

Rejected because: The speaker recommends platforms that support concurrent calls and branching to reduce wall-clock latency.

Debug agents with only print statements or a debugger

Rejected because: Probabilistic multi-node behavior made the lower-case John failure hard to diagnose without traces.

Patterns

eval-driven-development (quality)

The speaker recommends evaluating models and applications at each step of development and continuously comparing new models.

prompt-versioning (operations)

Prompts are externalized so experts can edit and commit them, and teams can adjust quickly when model versions change.

tool-use-orchestration (architecture)

Agents are described as assistants that decide which functions/tools to call until they can answer the user.

observability-tracing (operations)

Agent traces record the sequence of LLM/tool calls, metadata, inputs, outputs, and errors so non-deterministic failures can be debugged.

cost-aware-routing (operations)

The talk models per-call token economics and recommends choosing the cheapest model/provider that can satisfy the task.

model-routing (architecture)

The speaker compares local, open-source, provider-routed, and expensive frontier models during exploration and deployment.

safety-policy-enforcement (safety)

Guardrails run a secondary LLM classifier before or after the main model to reject unsafe or unwanted questions and answers.

rag-retrieval (architecture)

The RAG pattern stores company information in a vector database and retrieves relevant context for the LLM.

latency-optimization (operations)

Agent graphs use concurrent calls and branching to reduce end-user wait time when multiple tools are involved.

fine-tuning-adaptation (modeling)

Fine-tuning is presented as an option, but one with cost, time, and quality-regression risks.

Tradeoffs

Use local tools for exploration and cloud/GPU platforms for deployment

Chose Ollama/local exploration plus cloud providers, Modal, or SkyPilot for hosting over one hosting mode for every stage.

Rationale: Local tools are cheap for exploration, while production deployment requires thinking about GPUs, cloud availability, containers, and cost optimization.

Externalize prompts

Chose LangChain Hub-style prompt repositories over hardcoded prompts in application code.

Rationale: Externalized prompts let experts collaborate and make it easier to retest prompts against changing model versions.

Use guardrails despite their overhead when safety matters

Chose secondary LLM classifiers over single unchecked model call.

Rationale: The guardrail can classify unsafe questions or bad answers, but the team must account for added latency, cost, and errors.

Use RAG when company context matters

Chose chunked company data in a vector database over asking the base model without proprietary context.

Rationale: RAG lets the model answer with company information, but it creates a data-quality dependency.

Parallelize independent agent tool calls

Chose concurrent calls and branching over purely serial agent execution.

Rationale: Parallel execution can reduce elapsed time when independent tasks such as translation and audio generation do not depend on each other.

Choose cheaper models when they meet the task

Chose Llama 3.3 70B-style cheaper model in the example over OpenAI o1 for every call.

Rationale: The modeled call-center workload drops from nearly $300,000 per month to nearly $50,000 per month if the cheaper model is sufficient.

Instrument agent traces from the start

Chose LangSmith-like observability over debugger/print-only investigation.

Rationale: Multi-node probabilistic failures are hard to reconstruct after the fact, while traces show node order, inputs, outputs, and errors.

Gotchas (what bit them)

AI increases the application architecture surface area immediately.

Root cause: Teams must handle models, fine-tuning, prompts, RAG, hallucination prevention, model churn, GPU hosting, and cost in addition to ordinary front-end/back-end/infrastructure work.

Resolution: Plan AI systems as full production architectures rather than adding a single LLM call to a traditional app.

Correct chatbot answers remain hard even when building the chatbot is easy.

Root cause: LLMs can produce fluent but wrong answers and may ignore prompt instructions.

Resolution: Layer prompt engineering, guardrails, RAG, fine-tuning when justified, and continuous evaluation instead of relying on raw model output.

Guardrails are not free or deterministic.

Root cause: A secondary LLM classifier adds another API call, latency, cost, and probabilistic errors.

Resolution: Use guardrails where needed, but account for extra latency/cost and evaluate their mistakes.

RAG quality collapses if the vector database contains stale information.

Root cause: The LLM answers from whatever context retrieval supplies, even if that content is outdated.

Resolution: Treat data curation and vector-store freshness as operational requirements.

Fine-tuning can make model quality worse.

Root cause: New training data can cause forgetting or confusion with existing knowledge.

Resolution: Use fine-tuning selectively and evaluate whether it actually improves the target behavior.

Agent latency multiplies across sequential LLM/tool calls.

Root cause: Each LLM call can take seconds, and agents may call four or five LLMs/tools in one process.

Resolution: Use concurrent calls, branching, and careful graph design to keep user-facing latency within expectations.

Agent failures can appear only after many successful runs.

Root cause: Probabilistic models add unknowns to an otherwise deterministic application flow.

Resolution: Record traces with node order, inputs, outputs, metadata, and errors so edge-case failures can be diagnosed.

Evaluation

The speaker frames evaluation as continuous and lifecycle-wide. Teams should use local tools such as Ollama for early model exploration, Hugging Face for open-source model and code examples, OpenRouter to compare models/providers cheaply during testing, and LangSmith to record runs and compare model performance over time. The same discipline applies after deployment because new models appear frequently and may not work for the existing application.

Metrics: Use-case-specific model performance · Prompt behavior across model versions · Run histories and model performance over time · Latency per LLM/tool call · Per-call, daily, and monthly token cost · Provider/model output speed in tokens per second · Error traces, inputs, outputs, and node order for agent runs

Outcomes

• The talk identifies AI integration as a new production layer on top of traditional application development.
• It recommends externalized prompts as a collaboration and future-proofing mechanism.
• It shows agent cost can become hundreds of thousands of dollars per month if model choice and tool-call counts are not modeled.
• It demonstrates that switching to a cheaper capable model can reduce the example workload from almost $300,000 to almost $50,000 per month.
• It recommends observability for agents because traces made a case-sensitivity failure diagnosable.
• It argues that no single quality technique guarantees correctness, but combining evaluation, prompt engineering, guardrails, RAG, fine-tuning where justified, and tracing can get teams much closer.

The talk gives several concrete scale anchors: 18 months of GenAI work, over 15 years of IT experience, chatbots buildable in an hour or two, agent calls taking seconds each, a call-center example with 3,000 calls/day and 15 tool calls/call, o1 token prices of $15 and $60 per million tokens, 1,500 input and 3,000 output tokens, nearly $300,000/month versus nearly $50,000/month, A100 GPU costs of four to five dollars/hour, and Llama 3.1 70B served at over 1,000 or even 2,000 tokens/second by some providers.