Building and Deploying an Organization-Wide AI Agent with Production Security Challenges

daily.dev · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Organization-wide internal AI agent

daily.dev built Smith, a Slack-based organization-wide agent with access to production systems, then had to harden it against credential leakage, shared-runtime state bleed, memory exhaustion, and unverified autonomous behavior.

Constraints:

• Smith had broad access to databases, GitHub, browser automation, secrets, scheduled jobs, and internal APIs.
• A shared runtime created credential leakage and global CLI state hazards.
• Long conversations and large tool results could freeze the VM.
• The agent's broad autonomy made formal verification of safety impossible with current controls.

Architecture

Smith runs inside daily.dev's Slack workspace as a message-based agent. It uses GCP KMS-backed secrets with ACLs, per-turn credential injection, command sanitization, a Docker execution container with an environment allowlist, progressive tool disclosure, self-authored git-backed skills, scheduled jobs, and an MCP endpoint for delegation from local agents.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  slack-user[Slack User] --> smith-agent[Smith Agent]
  smith-agent --> tool-bundles[Tool Bundles]
  smith-agent --> secret-resolver[Secret Resolver]
  secret-resolver --> smith-exec-container[smith-exec Container]
  smith-agent --> smith-exec-container[smith-exec Container]
  smith-agent --> brain-repository[Brain Repository]
  smith-agent --> scheduler[Scheduler]
  mcp-endpoint[MCP Endpoint] --> smith-agent[Smith Agent]

Components

Data flow

A user messages Smith in Slack. Smith unlocks the needed tool bundle, resolves permitted secrets through ACL/KMS, runs commands inside the smith-exec container, redacts secret values from outputs before the LLM sees them, writes learned knowledge to a git-backed brain, and can run scheduled jobs or expose capabilities through MCP.

Why this architecture

daily.dev chose a message-first Slack agent because the bottleneck was secure access and knowledge, not dashboards. The production architecture then became defense-in-depth: ACL secrets, redaction, containers, tool gating, per-turn credentials, and memory limits, because broad internal access made security the dominant problem.

Models used: Codex · Claude Code · LLM agent runtime

Scale

• Smith shipped internally four days after the first commit.
  “We shipped the first version internally on March 12, four days after the first commit. This post isn't about those four days. It's about everything that broke once real people started using it, and the three weeks of fixes, security patches, and production incidents that followed.”article, stated
• Smith is 29,000 lines of TypeScript with 10,000 lines of tests.
  “Smith is 29,000 lines of TypeScript with 10,000 lines of tests. It was primarily written by Codex, with some touches from Claude Code. The human contribution was steering: writing specs, reviewing tests, course-correcting when the agent went in the wrong direction. Two people, 118 commits.”article, stated
• Two people made 118 commits.
  “Smith is 29,000 lines of TypeScript with 10,000 lines of tests. It was primarily written by Codex, with some touches from Claude Code. The human contribution was steering: writing specs, reviewing tests, course-correcting when the agent went in the wrong direction. Two people, 118 commits.”article, stated
• Smith has around 60 tools and 25 self-authored skills.
  “Smith has around 60 tools. Browser automation alone is 15. BigQuery is 6. Secrets management is 10. Early on, every tool was available on every turn. The system prompt ballooned. Token costs went up. The agent would reach for heavy tools when simpler ones would do, just because it could see them.”article, stated

  “There are about 25 skills right now. Every single one was written by Smith during real conversations with humans. Nobody hand-authored them.”article, stated
• One failure involved a 170-plus-message conversation and 25KB SQL MERGE statements.
  “One incident was traced to a conversation with 170+ messages. Tool results contained 25KB SQL MERGE statements. The agent's memory grew unbounded until the 15GB VM froze entirely. No swap configured. The whole machine went down, not just Smith.”article, stated

Alternatives considered (roads not taken)

Give Smith blanket GitHub organization access.

Rejected because: daily.dev instead created shared read-only, Smith-memory, and per-user OAuth tiers.

Expose all 60 tools all the time.

Rejected because: Smith now starts with 18 always-on tools and unlocks bundles on demand.

Let host environment variables enter the execution container.

Rejected because: Secrets and host configuration are stripped unless explicitly allowed.

Patterns

tool-use-orchestration (internal agent)

Smith coordinates database, browser, GitHub, Slack, cron, secrets, and internal API tools.

safety-policy-enforcement (security)

Command sanitization, ACL secrets, environment allowlists, and redaction constrain tool execution.

context-compaction (runtime)

The runtime has compaction, but a power user's pattern still bypasses it.

agent-memory (skills)

Smith writes reusable skills and documentation to a git-backed brain.

Tradeoffs

Interaction model

Chose Slack message interface over Dashboards and forms.

Rationale: The goal was to make secure internal access available through the place people already work.

Tool exposure

Chose Progressive tool disclosure over All tools in context at once.

Rationale: A smaller always-on tool set reduces cognitive/tool clutter while preserving capability bundles.

Execution isolation

Chose Docker container plus allowlisted environment over Running commands in the host process.

Rationale: The container boundary and env allowlist reduce accidental credential exposure.

Gotchas (what bit them)

Credential leakage is a category, not a single bug.

Root cause: Shared runtimes, CLI state, environment variables, and agent reasoning can all expose secrets differently.

Resolution: Use per-turn credential injection, command sanitizers, ACL secret resolution, and output redaction.

Long analytical threads can exhaust the whole VM.

Root cause: Conversation memory and large tool results grew without bounds.

Resolution: Apply systemd cgroup limits and restart cleanly on OOM.

Self-authored agent memory can drift without systematic audit.

Root cause: Smith writes its own skills and docs over time.

Resolution: Keep the brain in git and accept usage-driven correction while acknowledging the assumption.

Evaluation

daily.dev evaluates Smith through production incidents, security patches, usage by the whole team, tool-bundle adoption, crash recovery, and manual inspection of emergent skills rather than a formal benchmark.

Metrics: Production incidents · Tool count · Skill count · Crash recovery time · Credential leakage incidents · Scheduled task value

Outcomes

• Smith became a daily production agent for the whole team.
• It runs around 60 tools, about 25 self-authored skills, scheduled tasks, and an MCP endpoint.
• The team added multiple security and memory-control layers after real production failures.

The source reports four-day initial build, 29k TypeScript lines, 10k test lines, 118 commits, 60 tools, 25 skills, 170+ message failure case, 25KB SQL statements, 15GB VM freeze, and daily whole-team usage.