Improving LLM Food Tracking Accuracy through Systematic Evaluation and Few-Shot Learning

taralli · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM-powered nutrition extraction and evaluation-driven prompt optimization

Taralli's initial calorie-tracking endpoint returned schema-valid nutrition JSON but made severe semantic mistakes, especially around quantities such as treating 100g of peanut butter as 100 servings. The developer needed a practical way to turn real user failures into a golden dataset, evaluate output quality, optimize the prompt, and deploy the improved system without losing production logging.

Constraints:

• initial product validation favored a simple implementation over complex upfront optimization
• structured-output compliance was high but semantic accuracy was poor
• quantity interpretation errors could produce wildly incorrect calorie totals
• evaluation had to reflect user-relevant accuracy rather than perfect nutrition science
• the validation set was small, so the result still needed ongoing monitoring
• few-shot examples increased token count and could affect response time

Architecture

Taralli began with a FastAPI-style endpoint that sent a food description to GPT-4o-mini using structured outputs and a Pydantic nutrition schema. After observing failures, the developer used W&B Weave to log production inputs and outputs, curated a golden dataset from real mistakes with assistance from stronger models and manual editing, and defined a metric requiring calories within 10% plus food-group overlap. Baselines showed GPT-4o-mini at about 17% and Gemini 2.5 Flash zero-shot at about 25%. DSPy's BootstrapFewShotWithRandomSearch selected few-shot examples from training data and improved validation accuracy to about 76%. The optimized DSPy program was saved, loaded once at application startup, asyncified, deployed as /calories-v2, and kept under Weave logging for continued improvement.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  User_food_description["User food description"] --> Calories_API_endpoint["Calories API endpoint"]
  Calories_API_endpoint --> GPT_4o_mini_structured_output["GPT-4o-mini structured output"]
  GPT_4o_mini_structured_output --> Pydantic_nutrition_schema["Pydantic nutrition schema"]
  GPT_4o_mini_structured_output --> W_B_Weave_logging["W&B Weave logging"]
  W_B_Weave_logging --> Golden_dataset_curation["Golden dataset curation"]
  Golden_dataset_curation --> o3_and_Gemini_2_5_Pro_assistants["o3 and Gemini 2.5 Pro assistants"]
  Golden_dataset_curation --> Dataset_visualizer["Dataset visualizer"]
  Golden_dataset_curation --> Train_validation_split["Train/validation split"]
  Train_validation_split --> Evaluation_metric["Evaluation metric"]
  Evaluation_metric --> DSPy_evaluator["DSPy evaluator"]
  DSPy_evaluator --> Gemini_2_5_Flash["Gemini 2.5 Flash"]
  DSPy_evaluator --> DSPy_optimizer["DSPy optimizer"]
  DSPy_optimizer --> DSPy_optimized_program["DSPy optimized program"]
  Gemini_2_5_Flash --> DSPy_optimized_program
  DSPy_optimized_program --> Calories_API_endpoint
  DSPy_optimized_program --> W_B_Weave_logging
  Calories_API_endpoint --> API_key_authentication["API key authentication"]
  DSPy_optimized_program --> Type_validation["Type validation"]
  Type_validation --> Pydantic_nutrition_schema

Components

Data flow

A user submits a food description. The original endpoint sends it to GPT-4o-mini with a Pydantic structured-output schema and logs input/output pairs through W&B Weave. Logged failures are reviewed, corrected, and added to a golden dataset. The dataset is split into train/validation sets; a metric checks calorie tolerance and food-group overlap. DSPy evaluates baselines, runs BootstrapFewShotWithRandomSearch to select demonstrations, saves the optimized program, and the FastAPI /calories-v2 endpoint loads that program to return improved NutritionAnalysis outputs while continuing to log new examples.

Why this architecture

The architecture follows a pragmatic evaluation flywheel: ship a simple structured-output model, log real use, turn observed failures into a golden dataset, define a metric aligned with the failures, optimize few-shot examples, and keep logging after deployment. This avoided guessing at prompt quality and converted real production mistakes into measurable improvement.

Models used: GPT-4o-mini · OpenAI o3 · Gemini 2.5 Pro · Gemini 2.5 Flash · DSPy Predict program · BootstrapFewShotWithRandomSearch

Scale

• The initial failures included 58,000 kcal and 60,000 kcal-style outputs for ordinary quantities.
  “100g of peanut butter came back with 58 000 kcal - 350g of pasta as 60 000 kcal - quantities were all over the place.”article, stated
• The author reported improving tracking accuracy from 17% to 76%.
  “After some work, I managed to improve tracking accuracy from 17% to 76%.”article, stated
• Format compliance was nearly 100%.
  “the model respected the format almost 100% of the time, cracks started showing pretty quickly”article, stated
• The metric allowed calorie predictions within 10% of the golden example.
  “Is the total calories of the prediction within 10% of the total calories of the golden example?”article, stated
• The production GPT-4o-mini baseline scored 17.24%.
  “Score: 17.24%”article, stated
• The Gemini 2.5 Flash zero-shot baseline scored about 25%.
  “Using this zero-shot approach, the score was about 25% - better than my GPT-4o mini implementation”article, stated
• DSPy optimization used 16 maximum labeled demos and 16 candidate programs.
  “metric=eval_metric, max_labeled_demos=16, num_candidate_programs=16”article, stated
• The validation run used 29 examples and reached 75.9%.
  “Average Metric: 22.00 / 29 (75.9%): 100%|██████████| 29/29”article, stated
• The deployed async DSPy configuration used async_max_workers=4.
  “dspy.settings.configure(lm=lm, async_max_workers=4)”article, stated

Alternatives considered (roads not taken)

Keep the naive GPT-4o-mini structured-output prompt

Rejected because: It was schema-compliant but only about 17% accurate on the validation set and made severe quantity mistakes.

Switch to Gemini 2.5 Flash zero-shot

Rejected because: It improved the baseline but remained far below acceptable accuracy.

Design a highly refined metric upfront

Rejected because: A simple metric based on the observed failure modes was sufficient to guide meaningful improvement.

Optimize before shipping

Rejected because: The author's stated process was to ship early, learn from real failures, and then improve with monitoring and evals.

Patterns

structured-output-contract (quality)

The system constrains model responses to a Pydantic NutritionAnalysis schema with food items, quantities, calories, macros, and food groups.

observability-tracing (operations)

W&B Weave logs every food-analysis input/output pair so real failures can be inspected and turned into evaluation data.

data-quality-loop (data)

Production failures are reviewed, corrected, added to a golden dataset, and used to improve future prompts.

eval-driven-development (quality)

A concrete metric and validation split guide baseline measurement, DSPy optimization, and production integration.

prompt-chaining (workflow-orchestration)

DSPy compiles a few-shot prompt/program from examples, then the saved program is loaded into the production API.

latency-optimization (operations)

The production integration loads the DSPy classifier once, asyncifies it, and uses Gemini 2.5 Flash to mitigate prompt-length latency.

Tradeoffs

Initial product architecture

Chose ship a naive structured-output endpoint first over build complex LLMOps infrastructure before proving usage.

Rationale: The author prioritized validating that people would use the app, then improved quality based on real failures.

Metric simplicity

Chose 10% calorie tolerance plus food-group overlap over a more refined nutrition metric.

Rationale: The simple metric captured the dominant observed failures and was sufficient to drive large improvement.

Few-shot optimized prompt

Chose DSPy BootstrapFewShotWithRandomSearch over zero-shot Gemini 2.5 Flash.

Rationale: The optimizer raised validation accuracy from roughly 25% to 75.9% by selecting demonstrations.

Few-shot examples versus response time

Chose accept a larger few-shot prompt with Gemini 2.5 Flash over keep a shorter lower-accuracy prompt.

Rationale: The fast model made the latency hit acceptable for this use case.

Gotchas (what bit them)

Schema-valid outputs can still be wildly wrong.

Root cause: The model followed the JSON format but misinterpreted quantities as multipliers.

Resolution: Create a semantic evaluation metric and optimize examples against real quantity failures.

Synthetic assumptions would miss real user edge cases.

Root cause: Failures emerged from actual user inputs after launch.

Resolution: Log production input/output pairs and curate the golden dataset from observed failures.

A model swap alone did not solve accuracy.

Root cause: Gemini 2.5 Flash zero-shot improved over GPT-4o-mini but remained far from acceptable.

Resolution: Use DSPy few-shot optimization rather than relying on zero-shot model selection.

Few-shot optimization adds latency pressure.

Root cause: The optimized prompt includes several examples, increasing token count.

Resolution: Use a relatively fast model and load/asyncify the optimized DSPy program in the API.

The improved system is not finished despite a large lift.

Root cause: A 76% validation score still leaves a meaningful error rate and the validation set is small.

Resolution: Keep Weave logging and add new reviewed data to the golden dataset as more users interact.

Evaluation

The author built a golden dataset from real logged failures, split it into training and validation sets, and used a binary metric requiring both calorie totals within 10% and food-group overlap. Baselines were measured for GPT-4o-mini and zero-shot Gemini 2.5 Flash before DSPy's BootstrapFewShotWithRandomSearch optimized few-shot demonstrations.

Metrics: calorie totals within 10% of the golden example · predicted food groups overlap expected food groups · GPT-4o-mini production baseline: 17.24% · Gemini 2.5 Flash zero-shot baseline: about 25% · DSPy optimized validation score: 22/29, 75.9% · format compliance: almost 100%

Outcomes

• Tracking accuracy improved from 17% to 76%.
• The optimized prompt correctly handled the 100g peanut-butter quantity example as one 100g item.
• The improved DSPy program was deployed behind /calories-v2 with API-key authentication.
• The endpoint preserved W&B Weave logging for continued monitoring and dataset expansion.
• The case demonstrates an evaluation flywheel: real users create data, reviewed data improves the golden set, and the prompt can be continuously re-optimized.

This is a small independent application, not a large-enterprise system. Its value is the clarity of the LLMOps loop: production traces, golden data, metric, baseline, optimization, deployment, and continued logging.