Architecture Patterns for Production AI Systems: Lessons from Building and Failing with Generative AI Products

outropy · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI productivity systems

Outropy tried to build a production AI productivity tool for managers and engineers, but discovered that agentic AI systems need software-engineering architecture, domain models, memory, eventing, and resilience rather than demo-grade RAG or lab-style model experimentation.

Constraints:

• the product served real users while competing against much larger vendors
• LLMs were too weak for one-step prompt-and-context demos to become reliable products
• agents are stateful, non-deterministic, data-intensive, and poor fits for stateless microservice assumptions
• agent collaboration can become tightly coupled if agents call each other directly
• AI model APIs need resilience, bulkheads, checkpointing, and workflow durability

Architecture

Outropy evolved from a Slack chatbot into a Chrome extension built around AI pipelines and agents. The architecture treats workflows as data pipelines that transform Slack/calendar/productivity data into domain objects and treats agents like coarse-grained stateful objects. Rather than direct agent-to-agent calls, the system uses semantic events on a bus; memory is built from event-sourced facts and a probabilistic Neo4j graph; workflows are broken into smaller semantic stages with published interfaces; and durable workflow patterns such as Temporal-style checkpointing separate orchestration from side effects and protect the system around model API failures.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-productivity-inputs["slack-productivity-inputs"] --> workflow-pipeline-layer["workflow-pipeline-layer"]
  calendar-context-inputs["calendar-context-inputs"] --> workflow-pipeline-layer["workflow-pipeline-layer"]
  workflow-pipeline-layer["workflow-pipeline-layer"] --> semantic-domain-model["semantic-domain-model"]
  workflow-pipeline-layer["workflow-pipeline-layer"] --> durable-workflow-runtime["durable-workflow-runtime"]
  semantic-domain-model["semantic-domain-model"] --> agent-object-layer["agent-object-layer"]
  semantic-domain-model["semantic-domain-model"] --> memory-graph["memory-graph"]
  agent-object-layer["agent-object-layer"] --> semantic-event-bus["semantic-event-bus"]
  semantic-event-bus["semantic-event-bus"] --> memory-event-sourcing["memory-event-sourcing"]
  memory-event-sourcing["memory-event-sourcing"] --> memory-graph["memory-graph"]
  memory-graph["memory-graph"] --> agent-object-layer["agent-object-layer"]
  agent-object-layer["agent-object-layer"] --> model-api-bulkheads["model-api-bulkheads"]
  durable-workflow-runtime["durable-workflow-runtime"] --> model-api-bulkheads["model-api-bulkheads"]
  human-users["human-users"] --> workflow-pipeline-layer["workflow-pipeline-layer"]

Components

Data flow

Slack messages, calendar context, and other productivity data are ingested into workflow steps that break raw messages into conversations, deduplicate topics, rank discussions, and build a semantic domain model for personalized summaries. Agents subscribe to semantic events rather than calling each other point-to-point. User facts and event streams are compacted into memory representations and graph storage. Durable workflow orchestration handles retries, timeouts, checkpointing, and side-effect separation around AI model calls and downstream pipeline steps.

"We started as a Slack chatbot.

Eventually, we became like a Google Chrome extension. This is a screenshot from our pitch deck that shows the Chrome extension."

Why this architecture

Outropy’s lesson is that production AI systems need domain modeling and software architecture. One-step RAG demos failed because LLMs needed structured intermediate objects. Microservices were a poor conceptual fit because agents are stateful, non-deterministic, and data-intensive. Semantic events reduced agent coupling, graph/event-sourced memory handled uncertain user facts, and durable workflows provided the resilience that teams otherwise rebuild ad hoc around model APIs and long-running AI pipelines.

Models used: ChatGPT 2.5 · GPT-4 · Llama models · Neo4j · Redis · Kafka · Temporal-style durable workflows

Scale

• Outropy reached a few thousand real users.
  “one of the few actual companies to reach a few thousand people actually using the application”article, stated
• The first public beta took about six months.
  “It took about six months to get to the first public beta.”article, stated
• The product operated roughly from 2022 through early 2024.
  “This is 2023, early 2024. We're up against everybody.”article, implied
• The example architecture is acknowledged as overcomplicated for 10,000 users.
  “I'm like, how many users do we have? You say 10,000.”article, stated
• The speaker contrasts one-year and ten-month data-science project cycles with iterative product engineering.
  “it would take one year. It's like, can we have a spam classifier?”article, stated

  “We just need 10 months". They go again for 10 months”article, stated

Alternatives considered (roads not taken)

Twitter-driven development

Rejected because: Building for promised future model capabilities ignores current production limitations.

Slow data-science project mode

Rejected because: Waterfall-style lab work takes too long and may deliver weak product value after large staffing investments.

One-step RAG from raw data into the model

Rejected because: The team found raw context injection worked for demos but needed additional structuring steps in real systems.

The way that we evolved that is that instead of doing this, we actually added stuff."

— article, stated

Point-to-point agent collaboration

Rejected because: Direct agent calls create worse coupling than ordinary objects because autonomous agents decide on their own.

Deploy agents as stateless microservices

Rejected because: Agents need memory, dynamic state, and expensive context loading, breaking many Twelve-Factor assumptions.

Patterns

workflow-decomposition (workflow)

Outropy split Slack summarization into smaller semantic workflow steps such as discussion extraction, deduplication, ranking, and personalized summary.

async-message-orchestration (orchestration)

Semantic events decouple agents by publishing well-defined domain events to a bus instead of point-to-point calls.

agent-memory (state)

User and productivity facts are accumulated into memory using event sourcing and a probabilistic graph database.

durable-execution (reliability)

Durable workflows provide retries, timeouts, checkpointing, and separation between orchestration and side effects.

fallback-degradation (resilience)

The speaker recommends bulkheads between application code and AI model APIs.

structured-output-contract (data-contract)

The workflow converts raw Slack content into semantic entities with published interfaces rather than passing unstructured blobs between steps.

Tradeoffs

Treat AI products as engineering systems

Chose iterative software/product engineering over future-hype-driven or lab-style model experimentation.

Rationale: The speaker argues AI does not get a pass from iterative product engineering and that waterfall model tinkering is too slow for products.

Use semantic events rather than direct agent calls

Chose event bus with well-defined domain events over point-to-point agent collaboration.

Rationale: Events reduce coupling and avoid recreating distributed-object/web-services complexity between autonomous agents.

Delay Kafka until scale requires it

Chose SQLAlchemy events and Redis first over starting with Kafka for the semantic bus.

Rationale: The speaker recommends simpler eventing early, with Kafka only after hitting barriers.

Use durable workflow primitives

Chose Temporal-style retry/checkpoint/orchestration separation over ad hoc Redis job sidekicks and manual retry logic.

Rationale: Teams building AI/data systems tend to reinvent job splitting, checkpointing, and retry machinery if they do not adopt durable workflow frameworks.

Gotchas (what bit them)

Demo-grade RAG failed when used as the production architecture.

Root cause: Raw context injection did not create the intermediate semantic structure the product needed.

Resolution: Add workflow steps that transform messages into conversations, topics, domain objects, and personalized summaries.

Agents deployed like microservices break common operational assumptions.

Root cause: Agents are stateful, non-deterministic, data-intensive, and expensive to hydrate with context.

Resolution: Model agents as coarse-grained stateful objects/components and use durable workflows/events for orchestration.

Monolithic AI pipelines mix unrelated source concerns.

Root cause: A single pipeline had to understand Slack-specific formatting, Google Calendar context, and report generation at once.

Resolution: Break workflows into smaller steps with semantic interfaces so pieces can be swapped or reused across sources.

Memory facts extracted from natural language are uncertain.

Root cause: A user statement may be joking, misinformed, low-authority, or context-dependent.

Resolution: Represent memory in a probabilistic graph rather than treating all extracted facts as equally certain.

Evaluation

The source is a postmortem rather than an eval report. It uses production outcomes, user adoption, benchmark comparisons with Slack AI, and architectural failure analysis to judge which patterns worked. The speaker also emphasizes iterative engineering over long lab-style model experiments.

Metrics: few thousand real users · first public beta after six months · quality benchmark versus Slack AI · commercial failure despite product quality · pipeline modularity and swappability · resilience around model API calls

Outcomes

• Outropy reached a few thousand real users but failed commercially
• the team learned to model workflows as data pipelines and agents as stateful objects
• semantic eventing, graph memory, domain models, and durable workflows emerged as reusable architecture lessons
• the architecture became overcomplicated for the user count, partly because agentic systems need more state, memory, and resilience than ordinary apps

The source reports a few thousand users and discusses a final architecture that would look excessive for 10,000 users in ordinary software, but was driven by memory stores, content stores, semantic eventing, and model API resilience needs.