Building an AI Innovation Team and Platform with Safeguards at Scale

twilio · Telecommunications · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise AI innovation operating model

Twilio needed to explore disruptive AI and agent products quickly while protecting customer expectations, enterprise trust, cost discipline, and product quality.

Constraints:

Enterprise agents were not yet ready for prime time because quality, cost, and regulation were unsettled.
Twilio had established product quality and customer commitments that rough prototypes could violate.
Disruptive AI opportunities were changing faster than normal roadmaps.
The team had to bridge unstructured communications data and structured customer profiles.

Architecture

Twilio's Emerging Tech and Innovation team operates as a small cross-functional group that prototypes AI systems between Communications APIs and Segment customer data. It built AI personalization and perception concepts around customer memory, then moved toward an AI assistant/agent builder. The team uses Twilio Alpha as an expectation-setting sub-brand, internal dogfooding, hackathons, rough customer previews, model/roadmap flexibility, and share-as-you-go learning.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  emerging-tech-team[Emerging Tech Team] --> customer-data-platform[Customer Data Platform]
  emerging-tech-team --> communications-platform[Communications Platform]
  customer-data-platform --> customer-memory[Customer Memory]
  communications-platform --> customer-memory[Customer Memory]
  customer-memory --> ai-personalization-engine[AI Personalization Engine]
  customer-memory --> ai-perception-engine[AI Perception Engine]
  customer-memory --> ai-assistant-builder[AI Assistant Builder]
  ai-assistant-builder --> twilio-alpha[Twilio Alpha]
  twilio-alpha --> customers[Customers]
  emerging-tech-team --> internal-dogfood-users[Internal Dogfood Users]
  emerging-tech-team --> r-and-d-teams[R and D Teams]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Emerging Tech Team`	Self-contained cross-functional team exploring disruptive AI opportunities.	`Customer Data Platform`, `Communications Platform`, `Twilio Alpha`, `Internal Dogfood Users`, `Customers`, `R and D Teams`
`Customer Data Platform`	Segment customer profile system providing structured customer data.	`Emerging Tech Team`, `Customer Memory`
`Communications Platform`	Twilio SMS, voice, email, and communication channels with unstructured engagement data.	`Emerging Tech Team`, `Customer Memory`
`Customer Memory`	Concept joining structured customer profile data with unstructured communications data.	`Customer Data Platform`, `Communications Platform`, `AI Personalization Engine`, `AI Perception Engine`, `AI Assistant Builder`
`AI Personalization Engine`	RAG concept built on top of customer profiles in Segment.	`Customer Memory`
`AI Perception Engine`	System for translating communication data into customer profile memory.	`Customer Memory`
`AI Assistant Builder`	Agent-builder concept for omnichannel customer-engagement chatbots.	`Customer Memory`, `Twilio Alpha`
`Twilio Alpha`	Sub-brand for early AI products with explicit expectations around reliability and availability.	`AI Assistant Builder`, `Customers`, `Emerging Tech Team`
`Internal Dogfood Users`	Low-risk internal use cases such as IT help desk used to gather quality data.	`Emerging Tech Team`
`Customers`	Preview and feedback participants for prototypes and Alpha releases.	`Twilio Alpha`, `Emerging Tech Team`
`R and D Teams`	Main product teams that take mature prototypes toward market.	`Emerging Tech Team`

Data flow

Customer communications data and Segment customer profiles are combined into customer memory concepts; prototypes such as personalization, perception, and assistant-building are shown to customers or internal users; feedback shapes product direction; mature concepts are handed off to R&D or shipped through Twilio Alpha with explicit reliability expectations.

“trying to rapidly prototype the bridge between our customer engagement uh like our customer data platform and our communications platform”transcript, stated

“we had conceptualized essentially two features that we showed to customers iterated on and ultimately hand it off to R&D”transcript, stated

Why this architecture

The team separated disruptive AI exploration from normal sustaining product work because enterprise AI quality and market timing were uncertain. Twilio Alpha and dogfooding let the team get real feedback before normal quality bars were satisfied, while preserving customer trust through explicit expectations.

“is agents are not ready for Enterprise prime time um primarily because the quality is not there yet right like we've we've all seen sort of uh agents going rogue partially even rag based chatbots like not even agents like you know hallucinating things like the”transcript, stated

“setting the appropriate expectations uh around for users around reliability capabilities and availability and so our solution for this was to create our own sub brand cult fio Alpha that was very focused on um you know setting the right expectations with customers but enabling”transcript, stated

Models used: Generative AI models · Claude 3.5 Sonnet

Scale

The Emerging Tech and Innovation team had 16 people across engineering, product, design, and go-to-market.
“emerging Tech and Innovation um we're a team of 16 people uh cross engineering product design and go to market and we try to operate as self-contained as possible um outside of our two more traditional trolo business units around Communications that is more focused on”transcript, stated
The team began as a three-person skunkworks team the prior year.
“original agenda was we were founded as a three-person Skunk Works team last year that was really focused on uh special projects and trying to rapidly prototype the bridge between our customer engagement uh like our customer data”transcript, stated
The team could evaluate Claude 3.5 Sonnet's relevance within a day.
“that moment so for example when 3.5 Sonic came out we were able to figure out within a day whether we wanted to pay any attention to this for now or whether we're putting it in the backlog the other side is flexibility of road map having a new team that is specifically focused on Innovation”transcript, stated

Alternatives considered (roads not taken)

Treat AI only as sustaining innovation inside existing product teams.

Rejected because: Disruptive AI quality changes quickly and may threaten incumbents before normal market signals appear.

“initially um but ignoring disruptive innovation right now can be more dangerous than ever before because the biggest difference between sustaining and disruptive innovation is the quality and the quality changes every day right like every day right now with AI there's”transcript, stated

Hand prototypes to R&D after customer demos alone.

Rejected because: The first concepts lacked quality, cost readiness, and enough market momentum for sustaining R&D prioritization.

“was uh a problem that whether quality was not quite there the cost was not quite there and primarily we were solving a niche use case like what we had made up was uh primarily helpful for this upand cominging wave of agents and generative AI Solutions in general but”transcript, stated

Expose rough AI products under normal Twilio expectations.

Rejected because: Rough prototypes could upset customers when they did not meet the usual quality bar.

“how do we keep up with a pace of Technology Without Really upsetting customers with something that doesn't meet our normal Quality Bar”transcript, stated

Operate quietly as a fully self-contained team.

Rejected because: Other teams and customers did not know what the team was learning or building.

“we basically just operated in our own Corner quietly and shared what we were working on on a need to know basis”transcript, stated

Patterns

`rag-retrieval` (customer-engagement)

The AI personalization engine is described as RAG over Segment customer profiles.

“the AI personalization engine which is a rag on top of this customer profile that is within segment”transcript, stated

`agent-memory` (customer-data)

Personalization and perception engines together create customer memory from profile and communication data.

“so we conceptualize these two systems and sort of together they make this custom customer memory”transcript, stated

`canary-rollout` (product-release)

Twilio Alpha sets expectations and lets the team ship early previews, waitlists, and internal POCs.

“by having a sub brand it allowed us to open weight lists for developer previews show our offerings and onboard people quickly”transcript, stated

`model-routing` (model-ops)

The team assumes current models may become redundant and rapidly tests whether new models address known limitations.

“we assume that every model that we're using is going to be redundant tomorrow”transcript, stated

`human-in-the-loop-gate` (feedback)

Customers and internal users receive rough prototypes and provide feedback before full productization.

“here's for one day you have access to this prototype let us know everything that is wrong with it”transcript, stated

Tradeoffs

Release expectations

Chose Twilio Alpha sub-brand over Normal Twilio product branding for rough AI previews.

Rationale: A separate sub-brand let the team ship early while setting reliability and capability expectations.

“our solution for this was to create our own sub brand cult fio Alpha”transcript, stated

Prototype validation

Chose Internal dogfooding and customer rough access over Demo-only feedback.

Rationale: Hands-on usage revealed quality challenges and structure needs that demos did not.

“what we had built previously was primarily demo driven”transcript, stated

Team hiring

Chose Curiosity and creativity over Existing AI experience as the main hiring criterion.

Rationale: The team needed people who could solve changing problems, not only execute product-defined tasks.

“Natural Curiosity within the team was more important than existing AI experience”transcript, stated

Roadmap planning

Chose Defended flexibility over Fixed commitments for fast-moving AI capabilities.

Rationale: Expectations shift quickly, while trust and safety remain stable enough for roadmap commitments.

“you will constantly have to iterate on your road map and you have to be okay with failures”transcript, stated

Gotchas (what bit them)

Customer obsession is not enough to get disruptive AI prioritized.

Root cause: The first AI memory concepts were useful for future markets but not ready on quality, cost, or market momentum.

Resolution: Build ideas further, get hands-on usage, and gather stronger evidence before R&D handoff.

“even though we had great feedback we didn't really have the momentum that helped us to prioritize this”transcript, stated

Fast GenAI prototyping conflicts with enterprise trust practices.

Root cause: Generative AI lets teams release MVPs before the data and quality confidence expected in traditional ML development.

Resolution: Use Twilio Alpha, dogfooding, and clear expectations to gather feedback without implying normal GA reliability.

“gen turned sort of the entire product development life cycle on its head”transcript, stated

A self-contained innovation team can become invisible.

Root cause: The team operated quietly and shared work on a need-to-know basis.

Resolution: Share learnings internally and externally as the work progresses.

“teams not even knowing that we existed as a team or what we were working on”transcript, stated

Evaluation

Twilio evaluated early AI concepts through customer demos, rough customer access, internal hackathons, low-risk IT help desk dogfooding, model relevance checks, and feedback loops before handoff to R&D or Alpha release.

Metrics: Customer feedback quality · Prototype readiness for R&D handoff · Quality challenges found through dogfooding · Model relevance within a day · Waitlist/developer-preview interest

“we started with a low uh lowrisk use case around it help desk”transcript, stated

“how do we quickly validate whether the next model is something that helps us in that moment”transcript, stated

Outcomes

The team handed two customer-memory concepts to R&D.
Twilio Alpha became the mechanism for early AI previews and expectation management.
Internal dogfooding and rough customer access produced more actionable quality feedback than demos alone.
The team codified principles around customer/developer obsession, shipping early, curiosity, ownership, and sharing as they go.

The talk cites a 16-person team, a prior three-person skunkworks origin, one-day customer prototype access, and one-day model relevance checks.

“these are our big learnings from the three uh you know from the last year and we actually ended up turning them into our base principles”transcript, stated

“share things as we go both internally and externally”transcript, stated