Building a Comprehensive LLM Evaluation Framework with BrainTrust Integration

hostinger · Tech · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer-support chatbot evaluation

Hostinger needed a repeatable LLM evaluation framework for customer-support chatbots that could test offline changes, continuously assess live behavior, and identify issues for iteration across teams.

Constraints:

changes must be compared against predefined datasets before release
live chatbot performance needs continuous assessment
automated metrics must be complemented by human scores and user feedback
evaluation should run programmatically in development workflows
multiple teams need to reuse the same framework

Architecture

Hostinger uses Braintrust to orchestrate experiments over golden examples, chatbot outputs, automated metrics such as factuality and general direction, human review scores, and user feedback. The evaluation function can run programmatically through GitHub Actions on changes, compare against prior experiments, and let teams drill into failed cases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  golden-examples[Golden examples] --> braintrust-experiment[Braintrust experiment]
  chatbot-engine[Chatbot engine] --> braintrust-experiment
  braintrust-experiment --> automated-metrics[Automated metrics]
  braintrust-experiment --> human-scores[Human scores]
  human-scores --> user-feedback[User feedback]
  github-actions[GitHub Actions] --> braintrust-experiment

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Golden examples`	Dataset of inputs and expected answers used for offline evaluation.	`Braintrust experiment`
`Chatbot engine`	Customer-support chatbot under evaluation.	`Braintrust experiment`
`Braintrust experiment`	Evaluation orchestration and experiment-comparison tool.	`Golden examples`, `Chatbot engine`, `Automated metrics`, `Human scores`, `GitHub Actions`
`Automated metrics`	Checks factuality, general direction, component correctness, and URLs.	`Braintrust experiment`
`Human scores`	Manual or domain-expert review scores for cases automation cannot fully judge.	`Braintrust experiment`, `User feedback`
`User feedback`	Front-end feedback collected into the evaluation framework.	`Human scores`
`GitHub Actions`	Programmatic trigger for evaluation runs on changes.	`Braintrust experiment`

Data flow

Examples, inputs, expected outputs, and live chatbot calls enter the Braintrust experiment flow. Automated metrics score correctness, URLs, factuality, and direction against golden answers; human/domain-expert scores and user feedback are added to the same place; GitHub Actions or manual triggers run comparisons against previous experiments.

3 supporting quotes

“URLs are correct then the key meat uh of the evaluation is kind of using these metrics like uh factuality and general direction to compare This Kind of”transcript, stated

“giving and then uh into the same framework we want to add some human answers because sometimes it's still good to have someone uh like a person”transcript, stated

“here is that we can run this programmatically anytime we push a change uh a part of GitHub actions or anytime we want to trigger this and”transcript, stated

Why this architecture

A unified evaluation workflow lets Hostinger compare changes before release while still capturing human judgment and real user feedback. Running the same tests from GitHub Actions makes evaluation part of the development loop rather than a separate manual ceremony.

“if you're developing offline and you made a change and you want to know well what you've done did really improve on your data set”transcript, stated

“user feedback goes in the same place and so that's the kind of experiment View and another thing we're going to be”transcript, stated

Models used: LLMs · Braintrust evaluators

Scale

The framework is intended for multiple teams and different services.
“we're trying to make this so that multiple uh teams can use it too uh for different sorts of services and on the UI side uh it looks”transcript, stated

Alternatives considered (roads not taken)

Automated metrics only

Rejected because: The team still wants humans or domain experts to review cases where automation is insufficient.

“answers because sometimes it's still good to have someone uh like a person actually reviewing uh how we're doing uh”transcript, stated

Ad hoc local evaluation

Rejected because: The framework runs programmatically and compares experiments so changes can be judged consistently.

“is that it Compares against your previous experiment so if you're developing offline and you made a change and you want to know well what you've”transcript, stated

Patterns

`eval-driven-development` (evaluation)

GitHub Actions and experiment comparisons evaluate changes before release.

“anytime we push a change uh a part of GitHub actions”transcript, stated

`human-in-the-loop-gate` (quality)

Human and domain-expert scores are added beside automated evaluation.

“a domain expert can to and the user feedback goes in the same place”transcript, stated

`llm-as-judge` (evaluation)

The framework can ask an LLM to evaluate arbitrary criteria.

“there's a lot of flexibility to kind of ask nlm to evaluate whatever way we want”transcript, stated

Tradeoffs

Evaluation tooling

Chose Braintrust orchestration over building every evaluation UI and experiment comparison in-house.

Rationale: Braintrust automated the flow and added experiment-comparison functionality.

“Brain Trust here because that's a third party tool that we started using to help us automate all of this flow”transcript, stated

Quality inputs

Chose automated metrics, human scores, and user feedback over one metric source.

Rationale: Support quality needs both scalable metrics and human/user judgment.

“the user feedback goes in the same place”transcript, stated

Gotchas (what bit them)

Automated chatbot evaluation is incomplete on its own.

Root cause: Some support-answer quality judgments require human/domain expertise.

Resolution: Add human scores and user feedback into the same evaluation framework.

“some automated CH valuation and adding user feedback from that uh front end in there as well”transcript, stated

Teams need to know whether a change improved a fixed dataset.

Root cause: LLM chatbot changes can alter many outputs at once.

Resolution: Run experiments and compare against previous experiment scores.

“you automatically get comparisons and then of course you can uh drill in a little bit more”transcript, stated

Evaluation

Hostinger evaluates chatbot changes with golden examples, factuality, general direction, URL/component checks, human scores, user feedback, experiment comparisons, and GitHub Actions triggers.

Metrics: factuality · general direction · URL correctness · component correctness · human scores · user feedback

“whether uh the outputs are correct whether the URLs are correct”transcript, stated

Outcomes

The framework lets teams run programmatic evaluations on changes.
Experiment comparison helps identify what improved or regressed.
Human and user feedback are consolidated with automated metrics.

The source does not provide deployment volume, but says the framework is being designed for multiple teams and services.

“multiple uh teams can use it too uh for different sorts of services”transcript, stated