AI-Powered Chief of Staff: Scaling Agent Architecture from Monolith to Distributed System

outropy · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.0 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agent platform architecture for engineering leadership assistants

Outropy needed to scale an AI chief-of-staff assistant from a Slack-bot style product into reliable agent infrastructure that could handle long-running stateful workflows, user memory, tool calls, and growing customer load.

Constraints:

• agents need rich state across interactions
• LLM and tool APIs have unpredictable latency and reliability
• microservice assumptions did not fit agent behavior
• daily briefings became slow as users grew
• Python workflow tooling created integration friction

Architecture

Outropy started with a single FastAPI/asyncio ECS monolith containing a Slack-facing Chief of Staff assistant, productivity-tool connectors, inference pipelines, and stateful agent objects. Agent memory moved from mixed ORM tables to a uniform CQRS/event-sourcing model in Postgres, with a semantic event bus letting each agent decide which domain events mattered. Natural-language messages were converted into propositions for retrieval/routing instead of sending every message to every agent. As usage grew, the team added organization-based sharding, a GPTProxy for Azure GPT quota/load balancing, separate connector and agent worker components, and finally Temporal workflows plus activities for durable agent execution and LLM/tool calls.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  chief-of-staff-assistant[Chief of Staff assistant] --> productivity-connectors[Productivity connectors]
  productivity-connectors --> semantic-event-bus[Semantic event bus]
  semantic-event-bus --> agent-entities[Agent entities]
  agent-entities --> cqrs-event-store[CQRS event store]
  agent-entities --> inference-pipelines[Inference pipelines]
  inference-pipelines --> proposition-generator[Proposition generator]
  organization-sharded-ecs-app[Organization-sharded ECS app] --> chief-of-staff-assistant
  organization-sharded-ecs-app --> gptproxy[GPTProxy]
  connector-worker[Connector worker] --> semantic-event-bus
  agent-entities --> temporal-workflows[Temporal workflows]

Components

Data flow

Slack, GitHub, Docs, calendar, and other productivity events entered the assistant through connectors and a semantic event bus. Agents subscribed to state-change events, filtered what mattered, and persisted memory as CQRS commands/query models. Natural-language messages were first converted into propositions, which made retrieval/routing cheaper than invoking every agent pipeline on every message. User-facing agent work ran in the FastAPI/ECS application at first, then split into GPTProxy and connector workers, and finally into Temporal workflows where agent logic stayed in workflows while tool and model calls ran as activities.

Why this architecture

The architecture keeps conventional software where it fits and reserves agent machinery for components that need state, tool use, memory, and long-running decision processes. The team rejected stateless microservice assumptions for core agents because agent state, nondeterminism, poor data locality, and unreliable LLM/tool dependencies made fine-grained service extraction brittle. The later decomposition therefore used coarse components, explicit DTOs, shared quota services, and Temporal durability rather than pretending remote agent objects behave like local calls.

Models used: LLMs for proposition generation · LLMs for agent reasoning · embedding models

Scale

• The assistant reached 10,000 users within a year, so the architecture had to absorb real production growth rather than a demo workload.
  “Within a year, we attracted 10,000 users, [outperforming even deep-pocketed incumbents such as Salesforce and Slack AI]”article, stated
• The single-component approach was viable early: it supported growth from 8 to 2,000 users in about two months.
  “Keeping things simple allowed us to make massive progress on the product side, to a point we went from 8 to 2,000 users in about two months.”article, stated
• Daily briefings became the pressure point: as scale increased, report generation slipped from minutes to hours per user.
  “Our personal daily briefings—our flagship feature—went from taking minutes to hours per user.”article, stated
• LLM calls sat on the critical path and forced explicit rate-limit, quota, and backpressure design.
  “But in AI systems, these slow and unpredictable third-party API calls are your critical path. They make the core decisions, and this means everything is blocked until you get a response.”article, stated

Alternatives considered (roads not taken)

Stateless service-layer/microservice agents

Rejected because: The model conflicted with agents needing state, nondeterministic execution, heavy context, and slow unreliable model/tool dependencies.

An all-knowing event router that dispatches every state change to the correct agents

Rejected because: Only each agent knew what mattered; a central router would become a God object with domain logic spread across it.

Sending every Slack/GitHub/Docs/calendar item to every agent pipeline

Rejected because: It would create performance and cost problems because most content was irrelevant to most agents yet would still trigger LLM calls.

Simpler ML feature extraction for message routing

Rejected because: The team tried it but found proposition-based retrieval better for this open-ended product, while noting simpler extraction can still work in constrained domains.

Airflow, Lambda, or serverless orchestration for distributed agents

Rejected because: Those systems were optimized for stateless, scheduled, or short-lived tasks, while these agents needed durable long-running stateful workflows.

Patterns

agent-memory (context)

Outropy used CQRS and event sourcing so agents could reconstruct and query state from recorded changes.

durable-execution (orchestration)

Agent work was moved into Temporal workflows so long-running stateful computations could get retries, durability, and scalability.

rag-retrieval (retrieval)

The team converted structured messages into natural-language propositions before embedding to make vector retrieval more effective.

tool-use-orchestration (orchestration)

Outropy modeled agents as stateful objects exposing tools and decision methods through inference pipelines.

Tradeoffs

How to model agents inside the application.

Chose stateful object-like agents over stateless microservice-style service layers.

Rationale: Agents needed memory, planning state, and tool orchestration that did not match small stateless request/response services.

How to scale personalized daily briefings quickly.

Chose midnight batch processing plus organization-based sharding over per-user reports generated around each user's login time.

Rationale: The team sacrificed personalization timing to get a practical scaling path while demand kept growing.

Which workflow infrastructure to use for stateful agents.

Chose Temporal workflows over Lambda, serverless, or Airflow-style stateless scheduled tasks.

Rationale: Temporal matched the need for long-running stateful workflows, durability, retries, and multi-language support.

Whether to keep both ORM-backed simple agents and CQRS-backed complex agents.

Chose convert all agents to CQRS style while retaining ORM for non-agentic components over maintaining mixed ORM and CQRS persistence styles across agents.

Rationale: The dual model made refactorings and shared agent logic cumbersome; a uniform event-sourced agent model reduced architectural friction.

How to manage shared GPT quota at scale.

Chose extract GPT calls into GPTProxy and load-balance across Azure GPT deployments over adding distributed locks around model calls in the monolith.

Rationale: Azure per-deployment quotas gave a scalable quota unit; a dedicated proxy centralized the shared quota concern without spreading locking through the app.

Gotchas (what bit them)

Microservice architecture assumptions broke down for agents.

Root cause: Agents carry state, behave non-deterministically, process large context, and depend on slow external AI APIs.

Resolution: Model agents as stateful objects and run long-lived work through durable workflow infrastructure.

Scaling daily briefings created account-level contention.

Root cause: Large organizations could block smaller ones when all users' reports had to be generated overnight.

Resolution: Shard by organization, dedicate resources to large accounts, and prioritize active users.

Temporal solved orchestration but introduced Python integration friction.

Root cause: The Python SDK did not fit common libraries and favored data classes over the team's Pydantic-heavy stack.

Resolution: Build converters and exception wrappers around Temporal's Python SDK.

Evaluation

The post reports production scaling lessons from a live assistant with 10,000 users, plus qualitative architecture lessons from failed and successful implementation attempts.

Metrics: 10,000 users within a year · personal daily briefings went from minutes to hours per user before sharding · large organizations received dedicated resources

Outcomes

• The Chief of Staff assistant reached 10,000 users within a year.
• Demand for the underlying technology led Outropy to pivot into a developer platform.
• Temporal became core to both inference pipelines and the platform architecture.

Practitioner-authored post with unusually candid architecture lessons; some business claims are still company-authored.