Iterative Prompt Optimization and Model Selection for Nutritional Calorie Estimation

taralli · Healthcare · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Nutritional calorie estimation from free-text meal descriptions

Taralli needed to improve LLM-based calorie and nutrition estimation without relying on subjective vibe tests. The task required predicting total calories and macronutrients from messy food descriptions, measuring whether predictions were close enough to ground truth, comparing prompt optimization methods and model choices, and supporting both cloud fallback and offline iOS inference.

Constraints:

• calorie estimates need quantitative accuracy rather than subjective inspection
• food descriptions vary in detail, portion sizes, and food groups
• the local evaluation set had only 107 examples to keep experiments fast
• the target metric was Accuracy@10%, not exact calorie prediction
• prompt optimizers can overfit to one model and fail on another
• few-shot prompts need to preserve output format across models
• production inference needed deterministic behavior with temperature 0.0
• offline iOS inference required converting the DSPy program into OpenAI-compatible messages

Architecture

Taralli's improvement loop starts from a measured evaluation dataset and DSPy metric rather than vibe testing. The developer mixed 107 examples from NutriBench and a prior golden dataset, defined an Accuracy@10% calorie metric, and tested vanilla DSPy prompting, bootstrap few-shot variants, MIPROv2, and GEPA across Gemini 2.5 Flash, Gemini 3 Flash, and DeepSeek v3.2. The winning production update used Gemini 3 Flash with a 16-example few-shot prompt, improving accuracy by about 15% over Gemini 2.5 Flash. Production inference uses OpenRouter with Gemini 3 Flash as primary, DeepSeek v3.2 as fallback, temperature 0.0, and an iOS path that can fetch or bundle an OpenAI-compatible prompt template for on-device LLM inference.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  NutriBench_dataset["NutriBench dataset"] --> Taralli_evaluation_dataset["Taralli evaluation dataset"]
  Previous_golden_dataset["Previous golden dataset"] --> Taralli_evaluation_dataset
  Taralli_evaluation_dataset --> DSPy_Accuracy10_metric["DSPy Accuracy10 metric"]
  DSPy_Accuracy10_metric --> Prompt_optimizer_experiments["Prompt optimizer experiments"]
  Model_candidates["Model candidates"] --> Prompt_optimizer_experiments
  Prompt_optimizer_experiments --> Selected_few_shot_program["Selected few-shot program"]
  Selected_few_shot_program --> OpenRouter_model_gateway["OpenRouter model gateway"]
  OpenRouter_model_gateway --> DeepSeek_fallback_model["DeepSeek fallback model"]
  Selected_few_shot_program --> Template_converter["Template converter"]
  Template_converter --> Template_endpoint["Template endpoint"]
  Template_converter --> Bundled_prompt_template["Bundled prompt template"]
  Template_endpoint --> iOS_on_device_LLM["iOS on-device LLM"]
  Bundled_prompt_template --> iOS_on_device_LLM

Components

Data flow

The developer builds a 107-example dataset from NutriBench v2 and a previous golden dataset, each with food description, calories, food groups, and source. A DSPy evaluation function computes whether the predicted calories are within 10% of ground truth and returns score plus feedback. Multiple prompt optimizers and model candidates are evaluated against that metric. The selected DSPy program is served through an OpenRouter-backed API with a fallback model, and a cached template converter turns the optimized DSPy program into OpenAI-compatible messages that the iOS app can use with an on-device model or bundle directly as a backup.

• Vanilla: Zero shot prompt using the DSPy format."

Why this architecture

The architecture separates measurement, optimization, production fallback, and edge inference. DSPy forces an explicit Accuracy@10% metric, optimizer experiments compare prompts and models reproducibly, OpenRouter gives model-switching and fallback flexibility, and the iOS template path moves inference offline while still reusing the server-optimized few-shot program.

Models used: Gemini 2.5 Flash · Gemini 3 Flash · DeepSeek v3.2 · GPT-4o · DSPy · MIPROv2 · GEPA · BootstrapFewShotWithRandomSearch · on-device LLM

Scale

• NutriBench contains roughly 12,000 meal descriptions.
  “The team created a dataset for ~12K meal descriptions”article, stated
• NutriBench tested 4 prompting techniques and 12 LLMs.
  “To answer the question, they tested 4 different prompting techniques and 12 different LLMs.”article, stated
• GPT-4o with chain-of-thought responded to about 99% of NutriBench prompts and achieved 66.82% Acc@7.5g.
  “The best performer is GPT-4o with a Chain-of-Thought prompt. It responds to ~99% of prompts (models can refuse to answer) and achieves an [email protected] of 66.82%”article, stated
• Taralli's local evaluation dataset has 107 examples.
  “I created a new dataset of 107 examples of food descriptions and their corresponding total calories”article, stated
• The winning configuration used 16 few-shot examples and reached about 60% Accuracy@10%.
  “The best performing model is Gemini 3 Flash with a set of 16 examples in the prompt. It achieves a score of around 60%.”article, stated
• Switching from Gemini 2.5 Flash to Gemini 3 Flash improved accuracy by about 15%.
  “This approach is ~15% more accurate when compared to the old version, which was running on Gemini 2.5 Flash, with the exact same optimizer.”article, stated
• The iOS app is described as 5 SwiftUI files and around 4.5 MB.
  “It's a 5 file SwitfUI app with around 4.5 MB.”article, stated

Alternatives considered (roads not taken)

Continue vibe testing

Rejected because: The author argues that improvement requires numerical measurement before and after changes.

Use GEPA-optimized prompts for production across models

Rejected because: GEPA worked respectably on Gemini 2.5 Flash but failed output formatting on other models, indicating model-specific overfitting.

Use the incumbent Gemini 2.5 Flash production model

Rejected because: Gemini 3 Flash with the same optimizer and few-shot approach was about 15% more accurate.

Rely only on cloud inference

Rejected because: Taralli added offline on-device inference, with a bundled template backup to avoid the API call entirely.

Patterns

eval-driven-development (evaluation)

Taralli defines an explicit Accuracy@10% metric, evaluates multiple prompt optimizers and models against a 107-example dataset, and chooses updates based on measured accuracy.

model-routing (architecture)

The production configuration uses OpenRouter with Gemini 3 Flash as the primary model and DeepSeek v3.2 as a fallback model.

cost-aware-routing (optimization)

The experiment explicitly compares cheap and fast closed/open models and later moves inference on-device to remove per-call API dependence for offline use.

structured-output-contract (interface)

The calorie estimator expects structured nutritional outputs and the on-device path converts the optimized DSPy program into OpenAI-compatible message templates.

rag-retrieval (retrieval)

RAG is not the chosen production update, but the source treats NutriBench RAG and possible future web/RAG augmentation as a candidate path for better nutritional estimation.

Tradeoffs

Use a 107-example eval set.

Chose a small mixed NutriBench/golden dataset for fast experiments over a larger but slower evaluation set.

Rationale: The developer wanted enough measurement to compare approaches while keeping iteration fast.

Choose few-shot Gemini 3 Flash for production.

Chose Gemini 3 Flash with 16 few-shot examples over GEPA, vanilla, MIPROv2, and the incumbent Gemini 2.5 Flash setup.

Rationale: This configuration had the best measured score while remaining reliable and model-agnostic in output formatting.

Use OpenRouter for model fallback.

Chose Gemini primary with DeepSeek backup through OpenRouter over a direct single-model API integration.

Rationale: The abstraction makes it easy to configure a second-best fallback model without changing the application path.

Run inference on-device for iOS.

Chose offline on-device LLM using an OpenAI-compatible prompt template over requiring cloud API inference for every nutrition estimate.

Rationale: The app can work in airplane mode, avoid a network dependency, and optionally skip even the template endpoint by bundling the prompt.

Gotchas (what bit them)

Vibe testing cannot tell whether an LLM nutrition change helped or hurt.

Root cause: Without a numerical metric and baseline, prompt/model changes are judged subjectively.

Resolution: Define Accuracy@10%, create a test dataset, measure current performance, change one thing, and measure again.

Small eval sets make results fast but less statistically settled.

Root cause: The developer used 107 examples to keep experiments fast, not because 107 was necessarily optimal.

Resolution: Treat the 60% result as a pragmatic benchmark and acknowledge room for larger or more rigorous evaluation.

Prompt optimizers can overfit to model-specific behavior.

Root cause: GEPA optimized a prompt that worked with Gemini 2.5 Flash but failed output formatting on other models.

Resolution: Prefer the more model-agnostic few-shot approach for production portability.

Model upgrades can produce large gains with tiny code changes, but that also reveals model dependency.

Root cause: Changing only the model string from Gemini 2.5 Flash to Gemini 3 Flash improved accuracy about 15%.

Resolution: Evaluate model updates explicitly and keep a fallback model configured through OpenRouter.

Cloud-only nutrition inference can fail privacy, cost, or connectivity goals.

Root cause: A mobile calorie tracker may need to work without internet and avoid calling an API for every prediction.

Resolution: Convert the optimized DSPy program to OpenAI-compatible messages for an on-device LLM, with the prompt template bundled as backup.

Evaluation

Taralli evaluates nutrition estimation with a DSPy Accuracy@10% function over a 107-example dataset. The experiments compare prompt optimizers and models, track whether calorie predictions fall within 10% of ground truth, use feedback text for incorrect and correct predictions, and report the winning configuration's approximate 60% score plus a 15% improvement over the previous Gemini 2.5 Flash setup.

Metrics: Accuracy@10% for total calorie prediction · score 1 for within threshold and 0 for incorrect · feedback text for incorrect predictions · 107 examples · 16 few-shot examples in the winning prompt · around 60% score for Gemini 3 Flash few-shot · ~15% improvement over Gemini 2.5 Flash · NutriBench Acc@7.5g 66.82% carb baseline · NutriBench ~99% response rate for GPT-4o CoT

Outcomes

• Taralli updated production inference to Gemini 3 Flash with a 16-example few-shot prompt.
• The selected configuration reached around 60% Accuracy@10%.
• The model-string change improved accuracy by about 15% versus Gemini 2.5 Flash with the same optimizer.
• The GEPA prompt was rejected for poor cross-model output-format portability.
• The iOS app can run nutritional analysis completely offline with an on-device LLM.
• The prompt template can be bundled into the app to skip the API call entirely.

The experiments used 107 local evaluation examples from NutriBench v2 plus a prior golden dataset. NutriBench itself has roughly 12,000 meal descriptions and tested 4 prompting techniques across 12 LLMs. The selected Taralli configuration uses 16 few-shot examples, reaches around 60% Accuracy@10%, and improves about 15% over the previous Gemini 2.5 Flash setup.