Building an AI Agent for Real Estate with Systematic Evaluation Framework

rechat · Other · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real estate AI assistant evaluation

Rechat needed to move Lucy, an AI assistant for real estate professionals, from prototype demos to production without relying on subjective vibe checks.

Constraints:

• Prompt changes could improve one use case while breaking another.
• The team lacked reliable success and failure rates for production launch decisions.
• The assistant had to use real estate-specific tools such as contacts, email, listings, and website creation.
• Human review was necessary but could become too expensive without automation and better data curation.

Architecture

Rechat wrapped Lucy with an evaluation system rather than treating prompting alone as the production path. The system combines unit tests and assertions over observed failure modes, trace logging, a Rechat-specific human review interface, synthetic real-estate-agent inputs, Metabase dashboards, and later LLM-as-judge/fine-tuning workflows that are calibrated against human labels.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  real-estate-user[Real Estate User] --> lucy-assistant[Lucy Assistant]
  lucy-assistant --> rechat-tools[Rechat Tools]
  lucy-assistant --> trace-logging[Trace Logging]
  trace-logging --> human-review-app[Human Review App]
  human-review-app --> evaluation-dataset[Evaluation Dataset]
  evaluation-dataset --> assertions[Assertions]
  evaluation-dataset --> llm-judge[LLM Judge]
  trace-logging --> metabase-dashboard[Metabase Dashboard]
  assertions --> metabase-dashboard[Metabase Dashboard]

Components

Data flow

A synthetic or real real-estate task is sent to Lucy, Lucy calls Rechat tools, assertions and unit tests check known failure modes, traces are logged and reviewed in a custom annotation interface, results are visualized in Metabase, and high-quality reviewed cases feed later fine-tuning or judge alignment.

Why this architecture

The architecture moves Rechat from ad hoc prompt iteration to measurable development: cheap assertions catch known regressions, traces expose failures for humans, synthetic inputs bootstrap coverage before many users exist, and only then does the team add LLM judging or fine-tuning.

Models used: GPT-3.5 · LLM-as-judge

Scale

• The team needed to know whether changes worked 50% or 80% of the time before production launch.
  “is it going to work 50% of times or 80% of times and it's very difficult to launch a production app when you don't really know”transcript, stated
• The assistant covered multiple real-estate tool families, including contacts, email, listings, and website creation.
  “create a contact for me with this information or send an email to somebody with some instructions um find me some listings”transcript, stated

Alternatives considered (roads not taken)

Continue with vibe checks and a few manual invocations.

Rejected because: The team could not quantify success rates or know whether prompt changes caused regressions.

Jump straight to generic LLM judges.

Rejected because: Known failure modes are better captured first with cheap unit tests and assertions.

Buy or force generic trace tools for all review.

Rejected because: The team found too much friction and needed Rechat-specific trace metadata and filters.

Patterns

eval-driven-development (operations)

Rechat made unit tests, assertions, trace review, and judge alignment the mechanism for deciding whether Lucy was improving.

observability-tracing (operations)

The team logged assistant traces and built low-friction viewing/annotation so humans could inspect failures.

synthetic-data-generation (evaluation)

LLMs generated real-estate-agent inputs to bootstrap test cases and improve coverage before production usage supplied enough examples.

llm-as-judge (evaluation)

After deterministic assertions, Rechat could add an LLM judge calibrated against domain-expert labels.

fine-tuning-adaptation (model-adaptation)

Reviewed and failed cases from the eval workflow became a data-curation path for later fine-tuning.

Tradeoffs

Evaluation starting point

Chose Unit tests and assertions for known failures over Generic LLM judges first.

Rationale: Assertions are cheap, immediate, and fit the failure modes already observed in data.

Tooling maturity

Chose Existing CI and Metabase at the beginning over Buying specialized evaluation tooling immediately.

Rationale: The team could start measuring progress with tools it already had and upgrade later as needs matured.

Review interface

Chose A Rechat-specific annotation app over Generic trace viewers with high review friction.

Rationale: Domain-specific metadata and filters made human evaluation feasible enough to sustain.

Gotchas (what bit them)

Vibe checks hide regressions.

Root cause: Manual spot checks produced a feeling of improvement but no measured success rate across use cases.

Resolution: Add assertions, traces, dashboards, and human review so changes can be measured against known failure modes.

Trace logging alone does not create learning.

Root cause: People will avoid reviewing traces if the tool makes it hard to find Rechat-specific context.

Resolution: Build a custom viewing and labeling app to remove friction from looking at data.

LLM judges need calibration before trust.

Root cause: A model judge can disagree with human domain experts unless its critiques are aligned and checked.

Resolution: Have domain experts label critique data and iterate until the LLM judge aligns with human judgment.

Evaluation

Rechat evaluates Lucy through deterministic unit tests/assertions, trace logging, Metabase tracking, human review, synthetic test cases, and later LLM-judge alignment against human labels.

Metrics: Success rate · Failure rate · Assertion pass/fail results · Known failure-mode progress · Human-review cost · LLM judge alignment with human labels

Outcomes

• Rechat gained a workflow for knowing whether Lucy was improving rather than relying on subjective checks.
• Synthetic inputs improved test coverage across features, scenarios, and tools.
• The evaluation workflow created reusable data for fine-tuning and failed-case remediation.
• More comprehensive evaluation reduced the cost of human review over time.

The transcript gives percentage examples for unknown success rates, multiple product tool families, and a claim that human review cost falls as the evaluation framework becomes more comprehensive.