Building Domain-Native AI Organizations: A Framework for Leveraging Expertise in Vertical AI

notius_labs · Tech · 2026 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Vertical AI product organizations

Notius Labs presents a framework for building vertical AI products where the bottleneck is not only model capability but the organization's ability to operationalize domain judgment. The talk argues that many generative AI projects fail because teams automate workflows without enough understanding of domain nuance, quality criteria, and how expert review should evolve as products scale.

Constraints:

• Vertical AI products need deep understanding of customer workflows and domain-specific processes, not just stronger frontier models.
• AI quality often requires expert judgment, which may be formal expertise such as clinicians or informal expertise from people close to the workflow.
• A single domain expert can be enough early, but this can break under volume, geography, specialty, or customer-specific variation.
• Teams must decide whether quality can be measured objectively, whether manual iteration is fast enough, and whether automated improvement is needed.
• Domain experts need ownership and adjacent skills; treating them as external consultants slows the feedback loop.

Architecture

The framework defines three organizational architectures for domain expertise. In the Oracle model, a domain expert directly reviews AI outputs and traces, then improves prompts, documents, tools, or code. In the Evaluator model, domain experts define measurable quality metrics, review systems, and failure modes that engineers use to improve the product. In the Architect model, domain experts design automated improvement systems that learn from user interactions and adapt at the edge. The recommended organizational pattern is to designate a principal domain expert early, give them ownership, and let their role evolve from oracle toward evaluator, architect, or decentralized oracle as scale and measurability demand.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ai-product[AI product] --> ai-outputs-and-traces[AI outputs and traces]
  ai-outputs-and-traces --> oracle-domain-expert[Oracle domain expert]
  oracle-domain-expert --> prompts-documents-and-tools[Prompts documents and tools]
  prompts-documents-and-tools --> ai-product
  ai-outputs-and-traces --> review-dashboard[Review dashboard]
  evaluator-domain-expert[Evaluator domain expert] --> evaluation-metrics[Evaluation metrics]
  evaluator-domain-expert --> review-dashboard
  evaluation-metrics --> review-dashboard
  review-dashboard --> engineering-team[Engineering team]
  evaluation-metrics --> engineering-team
  engineering-team --> ai-product
  architect-domain-expert[Architect domain expert] --> automated-improvement-system[Automated improvement system]
  customer-usage[Customer usage] --> automated-improvement-system
  automated-improvement-system --> ai-product
  principal-domain-expert[Principal domain expert] --> oracle-domain-expert
  principal-domain-expert --> evaluator-domain-expert
  principal-domain-expert --> architect-domain-expert

Components

Data flow

AI product outputs, traces, customer feedback, and usage behavior flow to a domain expert or review system. In an Oracle setup, the expert directly diagnoses quality and updates prompts, documents, tools, or product behavior. In an Evaluator setup, the expert defines metrics, builds review dashboards or review teams, and hands structured failure data to engineers. In an Architect setup, the expert designs mechanisms that automatically learn from user interactions and deployment-specific interpretations, feeding improvements back into the AI product without constant manual intervention.

Why this architecture

The framework exists because vertical AI quality depends on expert judgment around workflows, failure modes, and customer-specific interpretations. Oracle loops are fast when quality is taste-driven or early-scale. Evaluator loops become necessary when quality can be measured and engineering iteration is still fast enough. Architect loops become necessary when customer variation and scale make manual improvement too slow, requiring automated systems that learn from usage.

Models used: LLMs · LLM-as-judge

Scale

• The talk cites vertical SaaS as a roughly $50 billion market and vertical AI as moving into a multi-trillion-dollar labor opportunity.
  “we had vertical SaaS and that was a you know, $50 billion or something market”transcript, stated

  “AI is moving into the kind of labor force and that's like a multi-trillion”transcript, stated
• The presenter cites Gartner's claim that about 50% of generative AI projects were abandoned the previous year.
  “according to Gartner, about 50% of all generative AI projects were abandoned last year.”transcript, stated
• The prior AI Engineer talk reportedly reached about 100,000 people.
  “this talk was seen by a lot of people, about 100,000 people.”transcript, stated
• Granola had recently passed a billion-dollar valuation in the example.
  “Granoola, uh for those of you who don't know, is a is a company that uh generates AI meeting notes, and recently uh kind of passed a billion in valuation.”transcript, stated
• Tandem's decentralized oracle approach had to handle many specialties, countries, note types, and thousands or more prompt variations.
  “they're serving many different specialties, many different countries, many different types”transcript, stated

  “they have many other, you know, like thousands or more of different variations on prompts”transcript, stated
• The presenter describes a failure mode where two senior clinicians left after about 12-18 months when ownership was ambiguous.
  “both of those two individuals after about like 12 18 months kind of left the company”transcript, stated

Alternatives considered (roads not taken)

Compete primarily by getting the best model.

Rejected because: The presenter argues that vertical AI differentiation comes from operationalizing domain expertise around models, not simply choosing a better model.

Use one centralized oracle for every scale and segment.

Rejected because: A single person can work early or for taste-driven outputs, but scaling across specialties, countries, customers, and prompt variations can require decentralized domain experts.

Rely on manual engineering iteration after experts define metrics.

Rejected because: Manual fixes are insufficient when customer-specific variation and policy interpretation require learning at the deployment edge.

Treat domain experts as advisors or consultants.

Rejected because: The recommended model puts them in the decision room with ownership, because advice without accountability slows the system.

Hire a generic domain credential and assume it is sufficient.

Rejected because: The presenter emphasizes direct experience with the specific workflow being automated and breadth across adjacent skills.

Patterns

human-in-the-loop-gate (organization)

The Oracle and Evaluator models deliberately put domain experts or review teams in the loop to assess AI outputs, traces, and subsets of production behavior before improvements are made.

eval-driven-development (quality)

The Evaluator role defines meaningful metrics, failure modes, and review dashboards so engineers can make changes from structured quality data.

llm-as-judge (quality)

The framework explicitly lists LLM-as-judge as one possible scalable mechanism for evaluator-style quality measurement.

prompt-versioning (operations)

The Tandem case used many prompt customizations across medical specialties, countries, and customer contexts, implying prompt variants as an operational unit.

data-quality-loop (learning-loop)

The Architect model designs automated systems that learn from user interactions and edge-case variation, turning deployment feedback into product improvement.

Tradeoffs

Start with an Oracle when quality is taste-driven or early-scale.

Chose direct expert review and improvement over premature metrics or automation.

Rationale: If there is no objective perfect output and scale is manageable, one expert can both judge outputs and directly improve prompts or product behavior.

Move to an Evaluator when objective measurement is possible and manual iteration is still fast enough.

Chose metrics and review systems feeding engineers over expert directly making every fix.

Rationale: Objective metrics let domain experts scale quality assessment while engineers continue making improvements from the resulting data.

Move to an Architect when manual engineering iteration cannot keep up.

Chose automated improvement mechanisms over centralized human fixes.

Rationale: Large customer-specific variation, such as different policy interpretations, requires systems that learn at the edge.

Name one principal domain expert instead of relying on committee consensus.

Chose single accountable owner over ambiguous shared responsibility.

Rationale: A principal owner can make quality decisions, invest in understanding AI behavior, and move faster than a committee.

Hire for breadth around domain expertise.

Chose domain expert with adjacent skills over narrow subject-matter expert only.

Rationale: Broader skills let the expert evolve from Oracle to Evaluator or Architect as the product and organization scale.

Gotchas (what bit them)

Teams can over-index on model sophistication while missing workflow nuance.

Root cause: Vertical AI products often fail because teams lack deep understanding of the workflows and expert processes they automate.

Resolution: Build an organizational system for incorporating domain insights around the model.

Hiring domain experts too late, choosing the wrong expert, or placing them poorly weakens AI quality.

Root cause: Domain expertise must be connected to quality decisions and product loops, not treated as incidental staffing.

Resolution: Answer whether an expert is needed, which kind is needed, and how the organization will leverage them.

A centralized oracle can stop scaling when product variation explodes.

Root cause: Different countries, specialties, customers, note types, and use cases may require many localized expert judgments and prompt versions.

Resolution: Use a decentralized oracle model where domain experts own subsets of customers or use cases.

Advisory-only experts create slow progress and context loss.

Root cause: Without a principal owner and decision authority, accountability becomes ambiguous and domain experts may leave.

Resolution: Define one principal domain expert, give them ownership, and put them in the decision room.

Generic credentials are not enough for vertical AI quality.

Root cause: A doctor, lawyer, or other domain expert may not know the exact workflow the AI product automates.

Resolution: Prioritize direct experience with the use case, then add adjacent skills such as prompting, data science, PM, statistics, and engineering where needed.

Evaluation

The talk evaluates the framework through decision criteria and case studies rather than a formal benchmark. It asks whether quality is objectively measurable, whether one expert is enough, whether manual iteration is fast enough, and whether automated improvement mechanisms are available. The Granola, Tandem, and Anterior examples illustrate how the same organization can begin with direct expert review, then move toward metrics, decentralized ownership, or automated improvement as scale and variation increase.

Metrics: whether AI performance can be measured with objective metrics · whether one domain expert can cover the scale · whether manual iteration is fast enough · whether customer-specific variation requires edge learning · domain-expert ownership and accountability · quality of prompt customizations · review dashboard metrics and failure modes · correct versus incorrect prior authorization decisions · ability to evolve from oracle to evaluator or architect

Outcomes

• the framework distinguishes Oracle, Evaluator, and Architect organizational roles for domain expertise
• Granola is framed as an Oracle case where a writer/journalist directly shaped prompts and AI note quality
• Tandem is framed as moving from centralized to decentralized Oracle ownership across doctors, countries, specialties, and prompt versions
• Anterior is framed as evolving from Oracle to Evaluator to Architect for prior authorization
• the talk recommends appointing a principal domain expert early and giving them ownership
• the framework explicitly treats domain-expert skill breadth as important for scaling AI quality systems

The talk cites a $50B vertical SaaS market, a multi-trillion-dollar labor opportunity, 50% abandoned generative AI projects, a prior talk seen by about 100,000 people, Granola's billion-dollar valuation, Tandem's many specialties/countries and thousands-plus prompt variations, and a 12-18 month retention failure when clinician ownership was ambiguous.