AI-Powered Performance Optimization System for Go Code

uber · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Go service performance optimization

Uber needed to make Go performance tuning repeatable across high-cost services without requiring scarce compiler/runtime experts to manually profile, benchmark, and inspect code for every optimization opportunity.

Constraints:

• top Go services created multi-million-dollar compute pressure
• manual profiling and code analysis could take days or weeks
• LLM recommendations must be precise enough to earn developer trust
• single-shot LLM analysis produced hallucinations and non-runnable code
• analysis must focus on service-owned hotpaths instead of noisy dependencies or runtime internals

Architecture

PerfInsights combines Uber's production profiling with constrained GenAI code analysis. A daily fleet-wide profiler captures CPU and memory profiles during peak traffic. Static filtering selects the top 30 expensive service-owned functions by flat CPU, triggers memory analysis when runtime.mallocgc exceeds 15% of CPU time, and excludes open-source dependencies and internal runtime functions. The GenAI stage passes full hotpath source code plus a curated Go antipattern catalog to LLMs. LLM juries and LLMCheck rule validators reduce hallucinations and false positives, while dashboards track detection accuracy, failure reasons, antipattern frequency, and model drift. The system integrates optimization opportunities into CI/CD and developer workflows.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  fleet-wide-profiler[fleet-wide-profiler] --> hotpath-filter[hotpath-filter]
  hotpath-filter --> antipattern-catalog[antipattern-catalog]
  antipattern-catalog --> llm-antipattern-detector[llm-antipattern-detector]
  llm-antipattern-detector --> llm-jury-validation[llm-jury-validation]
  llm-jury-validation --> llmcheck-validators[llmcheck-validators]
  llmcheck-validators --> llmcheck-dashboards[llmcheck-dashboards]
  llmcheck-dashboards --> developer-workflows[developer-workflows]

Components

Data flow

Production services emit CPU and memory profiles through Uber's fleet-wide profiler. PerfInsights selects hotpath functions, filters candidates to service-owned code, and attaches a curated catalog of Go performance antipatterns. The LLM analyzes the full source code for those hotpaths, flags likely inefficient constructs, and proposes more performant alternatives. LLM juries and LLMCheck validators review the finding, dashboards record accuracy and drift metrics, and accepted opportunities become diffs or workflow items in Uber's Go monorepo and CI/CD process.

Why this architecture

PerfInsights narrows LLM work to expensive, service-owned hotpaths because unconstrained code analysis is noisy and costly. The antipattern catalog injects institutional Go performance expertise, while juries and LLMCheck make the suggestions trustworthy enough for developers. This hybrid of profiling, static filtering, prompt engineering, and validation turns performance work from expert-led manual analysis into a repeatable workflow.

Models used: large language models · LLM juries · LLMCheck

Scale

• In March 2024, Uber's top 10 Go services accounted for multi-million-dollar compute spend.
  “In March 2024 alone, the top 10 Go services accounted for more than multi-million dollars in compute spend alone”article, stated
• PerfInsights selects the top 30 most expensive functions by flat CPU usage for each service.
  “For each service, it identifies the top 30 most expensive functions based on flat CPU usage.”article, stated
• Memory-profile analysis is triggered when runtime.mallocgc exceeds 15% of CPU time.
  “Additionally, if runtime.mallocgc—the Go runtime’s memory allocation function—accounts for more than 15% of CPU time, which means the runtime is spending a lot of time allocating memory, so PerfInsights also analyzes memory profiles to uncover potential allocation inefficiencies.”article, stated
• PerfInsights reduced false positives from over 80% to the low teens.
  “reducing false positives from over 80% to the low teens.”article, stated
• Detection and fix time fell from 14.5 hours to almost 1 hour of tool runtime, a 93.10% time savings.
  “By using our streamlined tool, we reduced the engineering time required to detect and fix an issue from 14.5 hours to almost 1 hour of tool runtime—a 93.10% time savings.The impact goes beyond dollars saved.”article, stated
• The expert team would have consumed approximately 3,800 hours without the streamlined tool.
  “Addressing this volume manually, as the Go expert team would have consumed approximately 3,800 hours.”article, stated

Alternatives considered (roads not taken)

manual performance tuning by Go experts

Rejected because: Profiling, benchmarking, and code analysis could take days or weeks and require specialized expertise.

single-shot LLM antipattern detection

Rejected because: Early single-shot attempts were inconsistent, hallucinated, and sometimes generated non-runnable code.

unfiltered whole-codebase LLM analysis

Rejected because: Static filtering keeps the analysis focused on service-owned hotpaths rather than dependencies and runtime internals.

Patterns

batch-offline-processing (operations)

PerfInsights runs a profiling and analysis pipeline over production services rather than waiting for developers to manually inspect every function.

llm-as-judge (evaluation)

LLM juries validate optimization findings to reduce hallucinations and improve confidence.

guardrails-io-filtering (safety)

Static filtering and LLMCheck validators restrict analysis to relevant code and catch false positives.

prompt-versioning (model-operations)

Prompt strategies were iterated with few-shot examples, contextual cues, and explicit antipattern definitions to improve detection precision.

observability-tracing (operations)

LLMCheck dashboards track detection accuracy, false positives, failure reasons, and model regressions.

Tradeoffs

Constrain analysis to profiled hotpaths.

Chose top CPU and allocation candidates from production profiles over broad LLM codebase analysis.

Rationale: Profiling identifies expensive functions and limits the model's search space to high-impact code.

Validate LLM findings with both juries and rule-based checks.

Chose LLM juries plus LLMCheck over trusting a single model's recommendation.

Rationale: The dual validation strategy reduces hallucinations and false positives enough to earn developer trust.

Encode institutional Go performance knowledge into prompts.

Chose curated antipattern catalog and Go-specific prompt cues over relying only on pretrained model knowledge.

Rationale: The source says explicit definitions, examples, and idioms improved detection precision.

Gotchas (what bit them)

Unconstrained GenAI code analysis creates too much noise.

Root cause: Early single-shot detection varied between runs, hallucinated, and produced non-runnable code.

Resolution: Score only detected antipatterns, add targeted prompt strategies, and validate with juries plus LLMCheck.

Large Go functions can be truncated if the model context is too small.

Root cause: Hotpath source code needs to be passed into the prompt for useful analysis.

Resolution: Use high input token limits and prompt phrasing that includes Go-specific idioms and antipattern definitions.

Developer adoption depends on trust, not novelty.

Root cause: Performance recommendations can waste engineering time if false positives and hallucinations are not visible and controlled.

Resolution: Use validation dashboards, track failure reasons, and surface clear wins before broad adoption.

Candidate selection matters before any model call.

Root cause: Sending non-service code, dependencies, and irrelevant runtime functions into the model increases noise.

Resolution: Filter on production CPU and allocation profiles, top functions, and service boundaries before invoking GenAI.

Evaluation

Uber evaluates PerfInsights with false-positive reduction, hallucination reduction, engineering time saved, merged diffs, LLMCheck dashboards, detection accuracy, failure patterns, and model-regression signals.

Metrics: false positives from over 80% to low teens · 93.10% time savings · 14.5 hours to almost 1 hour tool runtime · approximately 3,800 expert hours avoided · hallucination rates cut by more than 80% · hundreds of generated and merged diffs

Outcomes

• PerfInsights generated and merged hundreds of diffs into Uber's Go monorepo.
• False positives dropped from over 80% to the low teens.
• Detection and fixing time dropped from 14.5 hours to about one hour of tool runtime.
• The process turned performance tuning into a continuous, data-driven developer workflow.

The source gives strong production context: top 10 Go services with multi-million-dollar compute spend, top-30 hotpath selection per service, a 15% memory threshold, hundreds of diffs, 3,800 avoided expert hours, and 93.10% time savings.