Building Reliable AI Agent Systems with Effect TypeScript Framework

14.ai · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Reliable TypeScript agent systems for customer support

14.ai needed customer-support agents that interact directly with end users while remaining reliable despite unreliable APIs, complex dependencies, non-deterministic model outputs, streaming responses, testing needs, and long-running workflows.

Constraints:

• The platform uses LLMs in production and interacts directly with end users.
• External APIs and LLM providers can fail or behave unpredictably.
• Model outputs are non-deterministic and workflows can be long-running.
• Agents need deterministic workflows, actions, and sub-agents for customer-support domains.
• Streaming responses must reach users while also being captured for analytics/storage.
• Testing requires mocking LLM providers and simulating failures without changing system internals.
• Dependency injection and large Effect ecosystems add learning and tracing complexity.

Architecture

14.ai uses the Effect TypeScript library across its stack to model schemas, RPC, HTTP APIs, data processing, PostgreSQL/vector storage, and agent workflows. Agents act as planners that choose actions, workflows, or sub-agents, while deterministic workflows are expressed through a custom Effect-based DSL supporting branching, sequencing, retries, state transitions, and memory. Reliability mechanisms include multi-provider LLM fallback, stateful retry policies that avoid failed providers, duplicated token streams for user delivery plus analytics/storage, dependency injection for mocked providers and failure simulation, schema-first type/runtime validation, and OpenTelemetry observability.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  end-user[End user] --> agent-planner[Agent planner]
  agent-planner --> workflow-dsl[Workflow DSL]
  workflow-dsl --> actions[Actions]
  workflow-dsl --> sub-agents[Sub-agents]
  sub-agents --> actions
  actions --> external-systems[External systems]
  agent-planner --> llm-provider-router[LLM provider router]
  llm-provider-router --> retry-policy-state[Retry policy state]
  llm-provider-router --> llm-providers[LLM providers]
  llm-provider-router --> token-stream-duplicator[Token stream duplicator]
  token-stream-duplicator --> end-user
  token-stream-duplicator --> analytics-storage[Analytics storage]
  analytics-storage --> data-processing-engine[Data processing engine]
  effect-schemas[Effect schemas] --> effect-http-api[Effect HTTP API]
  effect-schemas --> effect-rpc-server[Effect RPC server]
  effect-rpc-server --> react-frontend[React frontend]
  effect-schemas --> workflow-dsl
  data-processing-engine --> postgresql-vector-store[PostgreSQL vector store]
  postgresql-vector-store --> effect-schemas
  dependency-injection-layer[Dependency injection layer] --> llm-provider-router
  dependency-injection-layer --> workflow-dsl
  dependency-injection-layer --> effect-runtime[Effect runtime]
  effect-runtime --> workflow-dsl
  effect-runtime --> opentelemetry-observability[OpenTelemetry observability]
  effect-http-api --> external-systems

Components

Data flow

An end user interacts with an agent planner. The planner selects an action, deterministic workflow, or sub-agent. The workflow DSL coordinates branching, sequencing, retries, state, and memory using Effect primitives. When the agent calls an LLM, a provider router tries the primary provider and falls back through stateful retry policies if it fails. Response tokens are duplicated: one stream goes to the user in real time and one goes to storage/analytics. Supporting stack services use Effect schemas, RPC/HTTP servers, a data processing engine, PostgreSQL/vector storage, DI-provided dependencies, and OpenTelemetry observability.

Why this architecture

The architecture uses Effect because plain TypeScript was not enough for unreliable APIs, complex dependencies, nondeterministic models, and long-running workflows. Effect provides type guarantees, composition, concurrency, streaming, interruptions, retry, structured error modeling, DI, schemas, and OpenTelemetry. The workflow DSL and DI approach make agent workflows explicit, testable, and swappable while preserving predictable behavior in mission-critical customer-support interactions.

Models used: LLMs in production · GPT-4 Mini · Gemini Flash 2.0 · Mock LLM providers for testing

Scale

• 14.ai describes the system as production customer support where LLM systems interact directly with end users.
  “Our systems interact directly with end users, rely on LLMs in production, and have to operate reliably under uncertain conditions.”transcript, stated
• Effect is used across the entire stack rather than only in one agent component.
  “we use effect across our entire stack.”transcript, stated
• The source names dashboards, agent interfaces, knowledge management, insights, analytics, SDKs, internal RPC, public APIs, data processing, and storage as parts of the stack.
  “We have a React front end that powers everything from dashboards to the agents ID, a knowledge management, insights, analytics, and SDKs.”transcript, stated

Alternatives considered (roads not taken)

Rely on plain TypeScript alone

Rejected because: Plain TypeScript fell short for unreliable APIs, complex dependencies, nondeterministic outputs, and long-running workflows.

Use a single LLM provider without fallback

Rejected because: Mission-critical systems need resilience when a provider fails.

Treat all agent behavior as nondeterministic model calls

Rejected because: Business-critical support flows benefit from deterministic workflows around actions and sub-agents.

Send only a single stream to the end user

Rejected because: The team also needed storage and analytics over streamed responses.

Adopt Effect all at once

Rejected because: The recommendation is incremental adoption starting from one service or endpoint.

Patterns

fallback-degradation (operations)

LLM calls fall back to another provider with similar performance characteristics when the primary provider fails.

tool-use-orchestration (agent-architecture)

Agents choose between small actions, deterministic workflows, and domain-specific sub-agents.

structured-output-contract (output-control)

Effect schemas define input, output, and error types with runtime validation, encoding, decoding, and generated documentation.

observability-tracing (operations)

Effect provides OpenTelemetry observability for production agent systems.

eval-driven-development (quality)

Dependency injection lets the team mock providers and simulate failure cases deterministically for tests.

durable-execution (orchestration)

The custom workflow DSL models retries, state transitions, memory, branching, and sequencing for long-running agent workflows.

Tradeoffs

Use Effect across the stack

Chose A consistent Effect-based frontend/backend/data/agent architecture over Plain TypeScript with ad-hoc reliability handling.

Rationale: The team wanted type safety, composability, structured errors, retry, streaming, DI, and observability across production LLM systems.

Build a custom workflow DSL

Chose Effect functional pipe-based DSL for workflows over Unstructured agent prompts for all workflow control.

Rationale: The DSL expresses deterministic and nondeterministic behavior, branching, sequencing, retries, state, and memory compositionally.

Use PostgreSQL for vectors and traditional data

Chose PostgreSQL with Effect SQL for both data and vector storage over A separate specialized vector database.

Rationale: The source presents this as part of a unified Effect stack; the likely tradeoff is operational consolidation over specialized vector infrastructure.

Duplicate token streams

Chose One stream to the user and one stream for storage/analytics over Only streaming to the end user.

Rationale: Customer-facing latency and internal analytics/storage both matter for production support agents.

Adopt incrementally

Chose Start with a single service or endpoint over Going all-in on Effect from day one.

Rationale: Effect has a real learning curve, so incremental adoption reduces risk while benefits compound.

Gotchas (what bit them)

Clean happy-path code can create a false sense of safety.

Root cause: Errors may be accidentally caught upstream or out of sight, silently losing important failures.

Resolution: Treat Effect as a disciplined tool and explicitly design error handling rather than trusting the clean flow.

Dependency injection becomes harder to trace at scale.

Root cause: Services may be provided across multiple layers and subsystems.

Resolution: Use DI deliberately and budget for tracing/understanding service provision across the system.

Effect has a real learning curve.

Root cause: It is a large ecosystem with many concepts and tools.

Resolution: Adopt incrementally and allow time for concepts to click before expecting compounding benefits.

The framework does not remove engineering judgment.

Root cause: Effect improves predictability and resilience but does not automatically solve system design.

Resolution: Continue thinking through failure modes, design choices, and reliability requirements.

Evaluation

The talk does not report quantitative production metrics. Evaluation is described through testability: dependency injection mocks LLM providers, simulates failure scenarios, guarantees required services at compile time, and supports swapping implementations without changing internal logic.

Metrics: Mocked LLM provider behavior · Simulated failure scenarios · Compile-time service provisioning · Runtime schema validation · Provider fallback behavior · Streaming delivery and storage · OpenTelemetry observability

Outcomes

• 14.ai uses Effect to build predictable, resilient, observable TypeScript LLM systems for customer support.
• The architecture supports provider fallback, stream duplication, schema validation, DI-based testing, and custom agent workflow composition.
• The source does not provide numeric reliability, latency, cost, or user-satisfaction results.

The source says 14.ai uses LLMs in production with direct end-user interaction and uses Effect across its entire stack, but it does not provide user counts, request volume, latency, or cost metrics.