Building Production-Grade AI Agents with Guardrails, Context Management, and Security

portia_/riff/_okta · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production-grade enterprise agents

Portia, Riff, and Okta were addressing the gap between agent demos and production systems: agents need guardrails, context discipline, identity and access controls, human approvals, and supportable workflows before they can safely act on production data or customer systems.

Constraints:

• Only a minority of AI apps/agents in the audience were accessing production systems, illustrating the productionization gap.
• Agents can lose context over longer workflows unless anchored by explicit tasks or specifications.
• Open chat interfaces create unpredictable user behavior and can lead to unsafe actions.
• Agents that write to CRM, files, payments, or production code need explicit permissions and approval boundaries.
• Very long context windows can reduce precision; teams still need context selection, caching, tool calling, and memory discipline.

Architecture

The panel describes production agent systems built around product/task structure, conditional example retrieval, sandboxed execution, role-based access, shorter curated context, tool-calling retrieval, and human approvals. Portia/Resonant converts documents into well-specified chunks for Linear/Jira/coding agents and uses RAG-style conditional examples to keep generated tasks sane. Riff owns the app-generation environment, guides nontechnical users into task-based development, sandboxes execution, exposes many tools for codebase/package/deployment work, and asks for approval on destructive operations. Okta frames identity, OAuth delegation, and agent permissions as the next security boundary for enterprise adoption.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-document-or-chat-request["User document or chat request"] --> resonant-chunking-pipeline["Resonant chunking pipeline"]
  user-document-or-chat-request --> agent-chat-interface["Agent chat interface"]
  resonant-chunking-pipeline --> task-generation-flow["Task generation flow"]
  task-generation-flow --> conditional-example-retrieval["Conditional example retrieval"]
  task-generation-flow --> linear-or-jira-ticket-system["Linear or Jira ticket system"]
  task-generation-flow --> coding-or-app-agent["Coding or app agent"]
  agent-chat-interface --> task-generation-flow
  coding-or-app-agent --> sandboxed-development-environment["Sandboxed development environment"]
  coding-or-app-agent --> tool-catalog["Tool catalog"]
  coding-or-app-agent --> context-search-layer["Context search layer"]
  context-search-layer --> memory-and-learnings-store["Memory and learnings store"]
  offline-context-generation["Offline context generation"] --> memory-and-learnings-store
  offline-context-generation --> context-search-layer
  tool-catalog --> approval-prompt-layer["Approval prompt layer"]
  approval-prompt-layer --> audit-trail["Audit trail"]
  audit-trail --> identity-and-access-controls["Identity and access controls"]
  identity-and-access-controls --> production-systems["Production systems"]
  coding-or-app-agent --> deployment-and-hosting["Deployment and hosting"]
  deployment-and-hosting --> production-systems
  linear-or-jira-ticket-system --> human-review-layer["Human review layer"]
  approval-prompt-layer --> human-review-layer

Components

Data flow

A product manager, nontechnical employee, or enterprise user starts with a document, chat request, or problem. Portia/Resonant turns source documents into well-specified task chunks and conditionally feeds relevant example tasks back to the agent as guardrail context. Riff's app-generation agent guides users into a task flow, searches or retrieves codebase context through tools instead of cramming the entire codebase, installs packages, generates React/FastAPI apps, and runs inside an owned/sandboxed development environment. When the agent needs to delete files, pay, transfer funds, write CRM records, or access enterprise resources, approval and authorization boundaries decide whether the action is allowed and who is responsible.

Why this architecture

The architecture structures ambiguous user intent into task artifacts and controlled execution surfaces because production agents are otherwise too easy to confuse. It keeps context precise rather than huge, retrieves examples and codebase facts when needed, sandboxes what the vendor owns, and relies on RBAC, approval prompts, and identity standards when agents touch systems the vendor does not fully control.

Models used: AI agent for app generation · Coding agents · LLM tool-calling models

Scale

• The opening poll estimated that roughly 40% of AI app/agent builders in the room had production access.
  “Okay, so roughly 40% of the hands that went up earlier.”transcript, stated
• Riff reported about 150,000 people building apps that year.
  “we've had [clears throat] like 150,000 people this year building apps”transcript, stated
• One customer example reported roughly 30% savings on a custom furniture workflow step.
  “they can chat with AI instead and it will generate drawings and and and so on cutting like 30% there”transcript, stated
• Portia targets around 95% rather than 100% correctness for drafting/review workflows.
  “I'm going to aim for like 95%, but like this is still a communication tool.”transcript, stated
• Resonant aims for end-to-end latency under 30 seconds.
  “keeping the latency of the job end to end under 30 seconds.”transcript, stated
• Riff said it had been working with coding agents for a little over two years.
  “we've been doing this for two years. for two years we have like a bit more than two years we have sat and with coding agents”transcript, stated
• One agent environment had around 120 tools.
  “the agent has like 120 tools.”transcript, stated
• The panel explicitly discusses the 1 million context window as losing precision.
  “yeah like the 1 million context window it loses precision”transcript, stated

Alternatives considered (roads not taken)

Let users directly vibe-code from vague ideas

Rejected because: Riff guides nontechnical users into a task flow because well-specified tasks anchor the agent and keep it on track.

Feed broad context without conditional examples

Rejected because: Portia found that conditional example task lists let reliability be tuned to the specific agentic application.

Give agents full write access to core systems

Rejected because: The speakers say agents cannot decide their own write scope, especially for CRM enrichment and other production systems.

Use maximum-length context windows

Rejected because: Both speakers preferred shorter context for precision, with tool calling and memory/search supplying relevant context on demand.

Upload and trust stale Confluence documentation

Rejected because: Portia worries that human documentation gets stale and can flood the agent; they prefer pushing useful context into code where it is maintained.

Let agents execute risky purchases or deletions autonomously

Rejected because: The speakers describe approval prompts for payment, file deletion, travel booking, trading, and fund-transfer-style operations.

Patterns

guardrails-io-filtering (safety)

The systems use task anchoring, conditional examples, sandboxing, RBAC, approval prompts, and write-scope controls to constrain agent behavior.

rag-retrieval (context)

Portia feeds generated plans/task lists back through a RAG-like conditional-example system to supply relevant task examples.

tool-use-orchestration (execution)

Riff's app-generation agent uses many tools for codebase search, package install, documentation generation, app building, deployment, and hosting.

agent-memory (context)

Agents run refinements, ingest memory, and write learnings/checklists that can be searched later rather than relying on a full-context dump.

human-in-the-loop-gate (approval)

Humans remain responsible for shipping, payments, deletions, funds movement, travel bookings, trading, and other risky actions.

privacy-preserving-processing (security)

The identity/access-control discussion centers on preventing agents from gaining broad access to databases, APIs, documents, wiki pages, CRM records, and delegated OAuth permissions.

latency-optimization (performance)

Resonant balances live context gathering with offline cached context to keep end-to-end jobs under a 30-second attention threshold.

context-compaction (context)

The teams keep context short and retrieve/search relevant material rather than filling huge context windows.

Tradeoffs

Anchor work in product tasks.

Chose Well-specified tickets, chunks, and task flows over Unstructured idea-to-app conversations.

Rationale: Task artifacts improve agent reliability while also solving a real user pain point around writing tickets/specifications.

Use conditional examples as guardrails.

Chose RAG-like retrieval of task examples based on scope over Static prompt instructions alone.

Rationale: Conditional examples keep generated tasks sane while preserving the ability to accept broad prompts.

Sandbox what the platform owns.

Chose Owned runtime/development environment for app generation over Direct unrestricted production-system access.

Rationale: Owning the environment makes the development workflow safer, but external systems still need explicit access controls.

Keep context shorter and searchable.

Chose Tool-called search, offline context generation, and short curated context over Cramming the whole codebase or using full million-token context.

Rationale: Shorter context preserves precision and latency while tool calling retrieves what the agent needs.

Keep humans in approval loops.

Chose Prompted approvals and drafting/review UI over Autonomous execution of risky actions.

Rationale: The speakers treat payment, file deletion, production data, and generated work as requiring human responsibility even when agents do most of the work.

Use delegated permissions carefully.

Chose Agent-specific identity/authorization standards and scoped permissions over Simply handing the agent the user's full OAuth permissions.

Rationale: Traditional app boundaries disappear with nondeterministic agents that can query databases, web pages, APIs, documents, and wikis.

Gotchas (what bit them)

Productionizing agents requires more than a working demo.

Root cause: Production access introduces supportability, trust, security, and operational constraints absent from POCs.

Resolution: Design for supportability, trust, security, context, and approval boundaries before giving agents production access.

Agents lose track without task anchors.

Root cause: Extended conversations and evolving model behavior can erode context and direction.

Resolution: Guide users into well-specified tasks and chunks that anchor the agent's work.

Open chat boxes create unpredictable and unsafe user behavior.

Root cause: Users vary widely in how they instruct AI, including confusing conversations around risky workflows like crypto transfers.

Resolution: Assume arbitrary user behavior and add robust guardrails independent of user sophistication.

Agents cannot decide their own write permissions.

Root cause: CRM, databases, APIs, documents, and wiki pages expose sensitive data and side effects that current delegated access models can over-grant.

Resolution: Layer RBAC, authentication/authorization, write-scope limits, and emerging agent-specific OAuth/MCP standards.

Documentation can flood or mislead agents.

Root cause: Human-facing documentation goes stale and may be a poor representation of the executable system.

Resolution: Prefer close-to-source context, generated API docs, and documentation pushed into codebases where agents encounter it in workflows.

Long context can reduce precision.

Root cause: Very long windows are tempting but may dilute relevant signal for agent tasks.

Resolution: Keep context shorter, use tool calling/search, and cache relevant context offline.

Responsibility is unclear for agent actions without approvals.

Root cause: Agents can delete files, pay, book, trade, or transfer funds, raising auditability and liability questions.

Resolution: Use approval prompts and audit trails for who did what, who approved what, and what the agent was allowed to do.

Evaluation

The panel does not describe a formal benchmark suite. Evaluation is framed through production-readiness signals: production access, supportability, trust/security posture, user review accuracy expectations, latency under 30 seconds, observed behavior across 150,000 app builders, and whether agents stay on track with well-specified tasks.

Metrics: Production deployment/access rate · End-to-end latency under 30 seconds · Target correctness around 95% for drafting workflows · Number of active builders/users · Need for human approvals on risky actions · Context precision under shorter windows · Whether task anchoring keeps agents on track

Outcomes

• Portia launched Resonant to turn documents into well-specified chunks for Linear/Jira/coding agents.
• Riff reported an agent that generates React frontends, Python FastAPI backends, deployment, and hosting.
• Riff reported roughly 150,000 app builders that year.
• A custom furniture customer reportedly cut about 30% of a drawing/CAD-related workflow.
• The speakers converged on smaller context, task anchoring, sandboxing, and human approval as production patterns.
• Okta framed delegated agent identity and scoped authorization as an emerging standards problem.

The discussion cites roughly 40% of the audience's AI builders in production, a newly released Resonant product, 150,000 people building apps through Riff, a 30% customer workflow improvement, an agent environment with about 120 tools, a 30-second end-to-end latency target, and a 1 million context window rejected for precision.