Multi-Agent System for Prediction Market Resolution Using LangChain and LangGraph

chaos_labs · Finance · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Prediction market resolution

Chaos Labs needed an oracle architecture for resolving prediction-market queries objectively while reducing single-model bias and grounding decisions in external data.

Constraints:

Prediction markets require transparent outcome resolution.
Relying on one model risks bias and brittle truth-seeking.
The alpha system had limited published real-world performance data.
The source is a guest post on the LangChain blog and has promotional bias.

Architecture

Edge AI Oracle uses a sequential multi-agent council built on LangChain and LangGraph. Research, scraping, bias analysis, report writing, summarization, and classification agents use multiple model providers to produce a final resolution with configurable consensus thresholds.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  research-analyst["Research Analyst"]
  web-scraper["Web Scraper"]
  document-bias-analyst["Document Bias Analyst"]
  report-writer["Report Writer"]
  summarizer["Summarizer"]
  classifier["Classifier"]
  research-analyst --> web-scraper
  web-scraper --> research-analyst
  web-scraper --> document-bias-analyst
  document-bias-analyst --> web-scraper
  document-bias-analyst --> report-writer
  report-writer --> document-bias-analyst
  report-writer --> summarizer
  summarizer --> report-writer
  summarizer --> classifier
  classifier --> summarizer

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Research Analyst`	Parses the query and identifies required data points and sources.	`Web Scraper`
`Web Scraper`	Retrieves external source and database evidence.	`Research Analyst`, `Document Bias Analyst`
`Document Bias Analyst`	Filters gathered data and checks for bias.	`Web Scraper`, `Report Writer`
`Report Writer`	Synthesizes filtered research into an initial answer.	`Document Bias Analyst`, `Summarizer`
`Summarizer`	Condenses the full report for final classification.	`Report Writer`, `Classifier`
`Classifier`	Categorizes and validates the summarized output against preset criteria.	`Summarizer`

Data flow

A research analyst parses the market query, a web scraper gathers verified data, a bias analyst filters the evidence, a report writer synthesizes findings, a summarizer condenses them, and a classifier validates the final answer.

“The multi-agent orchestration follows a directed, sequential flow with six distinct agent roles:”summary, stated

“The edge-based orchestration ensures smooth handoffs between tasks, creating what the team describes as a "cohesive and logical resolution process."”summary, stated

Why this architecture

A multi-provider council is used to counter single-model bias, while LangGraph gives the oracle a stateful workflow and LangChain supplies model, retrieval, prompt, and parser abstractions.

“the system leverages models from multiple providers including OpenAI, Anthropic, and Meta to provide diverse perspectives.”summary, stated

“LangGraph enables the graph-based, stateful workflow orchestration that connects all agents.”summary, stated

Models used: OpenAI models · Anthropic models · Meta models

Scale

The initial deployment required unanimous agreement above 95% confidence.
“the initial deployment requiring unanimous agreement with over 95% confidence from each agent”summary, stated
The workflow has six distinct agent roles.
“six distinct agent roles:”summary, stated

Alternatives considered (roads not taken)

Single-LLM oracle resolution

Rejected because: A single model can introduce bias and lacks the diverse perspectives Chaos Labs wanted for truth-seeking.

“The system addresses challenges of single-model bias and truth-seeking by orchestrating a council of AI agents powered by diverse LLM providers”summary, stated

Traditional human or rule-based market oracle

Rejected because: The product aims to provide objective, transparent, efficient resolution for varied prediction-market questions.

“Traditional oracles in prediction markets face challenges around objectivity and transparency.”summary, stated

Patterns

`multi-agent-orchestration` (architecture)

A council of specialized agents executes a sequential resolution workflow.

“The core innovation of Edge AI Oracle is its "AI Oracle Council" architecture, which orchestrates multiple agents in a sequential workflow.”summary, stated

`model-routing` (architecture)

The system uses models from OpenAI, Anthropic, and Meta to diversify perspectives.

“models from multiple providers including OpenAI, Anthropic, and Meta to provide diverse perspectives.”summary, stated

`human-in-the-loop-gate` (governance)

Prediction-market resolution remains configured around consensus and confidence thresholds before decisions are accepted.

“The architecture enables configurable consensus requirements, with the initial deployment requiring unanimous agreement with over 95% confidence from each agent”summary, stated

Tradeoffs

Use a council of diverse models

Chose Multiple providers and consensus over A single model deciding market outcomes.

Rationale: Diverse model perspectives are intended to reduce single-model bias.

“sidestep these limitations through what they describe as a "decentralized network of agents."”summary, stated

Use a sequential graph

Chose Directed handoffs through LangGraph over Unstructured agent collaboration.

Rationale: Prediction-market resolution needs an auditable chain from research through validation.

“each agent to build on the work of others in a coordinated manner.”summary, stated

Gotchas (what bit them)

The source does not explain bias-filtering methodology in depth.

Root cause: The case describes a bias analyst role but omits detailed criteria for detecting or filtering bias.

Resolution: Treat bias-filtering claims as architectural intent rather than validated effectiveness evidence.

“The inclusion of a dedicated bias-checking step is notable, though the methodology for detecting and filtering bias is not elaborated upon.”summary, stated

The alpha release lacks quantitative accuracy results.

Root cause: The case is early-stage and reports architecture more than measured production performance.

Resolution: Use the case for orchestration and consensus patterns, not as proof of accuracy improvement.

“real-world performance data is limited.”summary, stated

Evaluation

The system uses confidence thresholds and final classification but the case does not publish measured accuracy improvements.

Metrics: agent confidence · unanimous agreement

“requiring unanimous agreement with over 95% confidence from each agent, though quantitative results on accuracy improvements are not provided in the case study.”summary, stated

Outcomes

Chaos Labs released an alpha Edge AI Oracle built with LangChain and LangGraph.
The architecture can require unanimous high-confidence agreement.
No public quantitative accuracy results are reported.

The system is in alpha and reports consensus thresholds rather than production volume.

“Chaos Labs announced the alpha release of Edge AI Oracle, a multi-agent system designed to resolve queries in prediction markets.”summary, stated