LLM-Powered Real Estate Search and Agent Matching

zillow · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real estate search support and agent matching

StreetEasy needed to answer common listing questions quickly and create more trustworthy agent introductions while controlling LLM cost, latency, hallucination risk, fair-housing compliance risk, and subjective quality in personalized communications.

Constraints:

• Shoppers spent less than 61 seconds on core property and listing pages.
• The FAQ system had to cost less than a few hundred dollars while scaling to StreetEasy/Zillow traffic.
• Instant Answers should only respond to questions answerable from available data.
• Personalized agent summaries needed to fit compact UI cards while remaining informative and unique.
• Real estate communications require Fair Housing guardrails and human oversight.

Architecture

StreetEasy built two LLM-based features. Instant Answers uses BERTopic and BERT embeddings to discover frequent buyer questions, prepares context from listing and building data, and pre-generates answers with chain-of-thought validation against context to minimize cost and latency. Easy as PIE computes structured agent-fit attributes from deal history, user preferences, and property data, then uses a chain of prompts plus LLM and human evaluation to generate concise agent bio summaries.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Topics["Historical inquiry topic model"] --> Context["Property context dataset"]
  Context --> Answers["Instant Answers generator"]
  Answers --> Cache["FAQ cache"]
  AgentAttrs["Agent-fit attribute computation"] --> BioChain["Bio summarization chain"]
  BioChain --> Evaluation["Human and automated evaluation"]
  Answers --> Evaluation
  Guardrails["Fair Housing guardrails"] --> Answers
  Guardrails --> BioChain

Components

Data flow

For Instant Answers, historical buyer questions are topic-modeled, property context is aggregated from active listings, past listings, and building pages, answerable FAQs are selected, and answers are pre-generated and validated against context. For Easy as PIE, agent deal history, inferred user preferences, and property data create suitability attributes, while LLM summarization condenses agent bios through drafting and revision prompts.

Why this architecture

StreetEasy pre-generated FAQ answers because real-time generation would be more expensive and less predictable for high-traffic listing pages. It used context validation to reduce hallucinations and used a separate summarization chain for agent bios because that task needed concise, UI-fit language judged for informativeness and creativity.

Models used: LLMs · BERTopic · BERT

Scale

• StreetEasy users spent less than 61 seconds on core property and listing pages on average.
  “shoppers spent, on average, less than 61 seconds on our core property and listing pages.”article, stated
• The FAQ topic analysis used property-for-sale-related questions submitted between December 2021 and April 2023.
  “We took all the property-for-sale-related questions that were submitted between December 2021 and April 2023 and visualized the data as word clouds.”article, stated
• The Instant Answers cost target was less than a few hundred dollars while supporting StreetEasy/Zillow traffic.
  “we wanted to build an AI system that would cost less than a few hundred dollars, while scaling to support StreetEasy/Zillow traffic.”article, stated
• Zillow's buyer survey says 9 in 10 successful buyers hire an agent.
  “the overwhelming majority of successful buyers (9 in 10) hire an agent to help with some part of their buying journey”article, stated
• The buyer survey reports 46% care about neighborhood/building expertise, 37% personal connection, and 32% deal-making experience.
  “46% of shoppers care about agent expertise in the building or neighborhood area; 37% of shoppers like the agent personally and finally 32% shoppers care and are impressed by an agent’s past deal making experience.”article, stated

Alternatives considered (roads not taken)

Traditional web FAQ search

Rejected because: It can be slow and inefficient, requiring users to navigate complex search and irrelevant results.

Real-time LLM answers to shopper inquiries

Rejected because: Pre-generation reduced computation cost and latency and made reliability easier to control.

Single-step answer generation

Rejected because: The team needed to prevent hallucinations and avoid answering unanswerable questions.

Automated summarization metrics alone for agent bios

Rejected because: Human review was needed to judge informativeness and creativity.

Patterns

latency-optimization (serving)

Instant Answers pre-generates FAQ responses from available property data to avoid real-time LLM cost and latency at serving time.

prompt-chaining (prompting)

StreetEasy uses step-by-step answer validation for Instant Answers and a chain of drafting/revision prompts for agent bios.

eval-driven-development (evaluation)

The team used faithfulness, response relevancy, design compliance metrics, LLM-based summary scoring, human review, A/B tests, and post-deployment monitoring.

human-in-the-loop-gate (quality)

Human review complements LLM evaluation for subjective agent-bio quality and sensitive personalized communications.

safety-policy-enforcement (safety)

StreetEasy uses Fair Housing guardrails, legal review, ethical AI review, and prompt boundaries to reduce discriminatory output risk.

Tradeoffs

Pre-generate property FAQs

Chose cached answers to common questions over real-time LLM answers to every inquiry.

Rationale: Pre-generation lowered computation cost and latency and improved answer reliability.

Validate generated answers against context

Chose draft-then-validate prompting over direct final-answer generation.

Rationale: The extra validation step reduced hallucination risk and prevented responses to unanswerable questions.

Use generated bio summaries in a constrained UI

Chose short LLM-generated agent summaries over long self-written agent biographies in carousel cards.

Rationale: The card format required concise but unique messages that highlighted relevant professional qualifications.

Gotchas (what bit them)

LLMs can answer when they should abstain.

Root cause: Property questions may be unanswerable from available data, and unconstrained generation can hallucinate.

Resolution: Use chain-of-thought prompting to draft, validate against context, and only return final responses after validation.

Real estate LLM features can amplify unfair or discriminatory outcomes.

Root cause: LLMs trained on broad data can perpetuate and amplify existing societal biases.

Resolution: Coordinate with ethical AI, legal, and broader AI teams and use Fair Housing guardrails in prompts.

Automated metrics do not fully capture subjective communication quality.

Root cause: Agent-introduction copy must be informative, creative, and relationship-building, not merely compliant with requirements.

Resolution: Use human-in-the-loop review alongside LLM-based evaluation.

Cross-functional alignment is a production requirement, not a nicety.

Root cause: LLM-powered user journeys must align technical feasibility, design, marketing, user needs, and business goals.

Resolution: Use iterative PEMD collaboration and continuous monitoring/evaluation/refinement.

Evaluation

StreetEasy evaluated Instant Answers with faithfulness, response relevancy, non-LLM design metrics, and answerability checks, while Easy as PIE used LLM-based requirement-compliance scoring, human review for informativeness and creativity, post-deployment A/B tests, key metric monitoring, and response monitoring.

Metrics: Faithfulness · Response Relevancy · non-LLM design compliance metrics · human informativeness and creativity ratings · A/B test metrics · LLM response monitoring

Outcomes

• StreetEasy deployed Instant Answers for quick property FAQ responses.
• StreetEasy deployed Easy as PIE for personalized buyer-agent introductions.
• Pre-generation supported a low-cost, low-latency FAQ experience.
• The work produced reusable lessons around data quality, fair-housing guardrails, human oversight, and cross-functional AI product development.

The article reports the cost target, user-page dwell-time context, historical-question window, buyer-survey percentages, and traffic-scale requirement, but not adoption or conversion results.