Rapid Development and Deployment of Enterprise LLM Features Through Centralized LLM Service Architecture

pagerduty · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise incident-management GenAI platform integration

PagerDuty needed to ship multiple production GenAI features into its incident-management platform within two months while standardizing LLM access, security, monitoring, provider flexibility, and product-team integration.

Constraints:

• The timeline was two months, with a motto of 'think days, not weeks.'
• Product teams needed LLM functionality without each solving access, security, monitoring, and provider complexity independently.
• Incident-management workflows require secure, reliable, observable behavior during high-pressure operations.
• The service had to support multiple LLM providers and future use cases.

Architecture

PagerDuty built a centralized LLM service running as Kubernetes microservices. Product GenAI features call service endpoints; the service authenticates the account, routes requests to an LLM manager, builds prompts, calls providers such as OpenAI, Azure OpenAI, and AWS Bedrock, applies centralized security controls, logs metrics and call details to Datadog, Sumo Logic, and S3, and returns outputs to features such as AI-generated runbooks, status updates, postmortems, Slack assistant, and automation digest.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["PagerDuty GenAI features"] --> B["Central LLM API service"]
  B --> C["Account authentication"]
  B --> D["LLM manager"]
  D --> E["Provider adapters"]
  E --> F["OpenAI"]
  E --> G["Azure OpenAI"]
  E --> H["AWS Bedrock"]
  B --> I["Security controls"]
  J["Security and legal team"] --> I
  D --> K["Datadog"]
  D --> L["Sumo Logic and S3"]

Components

Data flow

A PagerDuty GenAI feature sends an API request to the centralized LLM service. The service authenticates the account and checks GenAI access, then forwards the request to an LLM manager that builds prompts, selects the provider/model based on endpoint configuration or prompt version, calls the provider, logs metrics and call details for monitoring and debugging, and returns the generated result to the product feature.

Why this architecture

PagerDuty centralized LLM access so product teams could ship features quickly without duplicating security, monitoring, provider selection, and prompt configuration work. Kubernetes microservices and provider decoupling let teams add endpoints fast, switch providers/models, and apply security changes across all GenAI endpoints.

Models used: OpenAI · Azure OpenAI · AWS Bedrock

Scale

• PagerDuty developed and deployed GenAI features to production in two months.
  “we succeeded we developed and deployed J features into production in two months”transcript, stated
• The project followed a 'think days, not weeks' operating cadence.
  “our motto was think days not weeks”transcript, stated
• The service supported multiple providers, including OpenAI, Azure OpenAI, and AWS Bedrock.
  “multiple llm providers some of them being open AI aor open AI AWS bedrock”transcript, stated
• The centralized service produced metrics for tokens, latency, status, health, and performance.
  “status um count of to to tokens latency for monitoring health and performance”transcript, stated

Alternatives considered (roads not taken)

Let each product team solve LLM access independently

Rejected because: That would duplicate work and slow teams with LLM access, security, and monitoring complexity.

Tie product features directly to one provider or model

Rejected because: The service needed provider/model flexibility and failover.

Handle security controls per endpoint manually

Rejected because: Centralization let PagerDuty apply security changes across endpoints.

Patterns

model-routing (model-operations)

The centralized service decouples features from providers and lets product teams switch providers/models by configuration.

fallback-degradation (reliability)

The service was designed to fall back to another provider if the primary provider was down.

observability-tracing (operations)

PagerDuty sends LLM metrics to Datadog and call details to Sumo Logic/S3 for monitoring, debugging, and analysis.

safety-policy-enforcement (risk)

Security controls include account authentication, throttling, prompt-design strategy, and ongoing security/legal review.

Tradeoffs

Create a centralized LLM access service.

Chose Central dedicated LLM service/team over Each product team integrating directly with providers.

Rationale: Centralization reduced duplicated work and let product teams focus on GenAI features.

Use Kubernetes microservices.

Chose Kubernetes-based service architecture over A less scalable monolithic feature-specific integration.

Rationale: Kubernetes helped the team keep adding production endpoints quickly while maintaining scalability.

Log enough for monitoring while controlling access.

Chose Datadog plus Sumo Logic/S3 with access controls over Minimal opaque logging or unrestricted logs.

Rationale: The new technology needed visibility for debugging and analysis, while security compliance required controlled access.

Gotchas (what bit them)

Fast GenAI shipping can collide with security and legal readiness.

Root cause: LLM services create risks such as unauthorized use, prompt injection, sensitive data leakage, and misinformation.

Resolution: Work with security/legal early and centralize mandatory controls.

Provider outage could break product features.

Root cause: GenAI features depend on third-party LLM provider availability.

Resolution: Build failover to alternate providers in the centralized LLM service.

LLM observability has provider-dependent logging constraints.

Root cause: What can be logged about LLM calls depends on provider behavior and compliance boundaries.

Resolution: Define what, how, and where logs are stored and who can access them.

Evaluation

PagerDuty's reported evaluation is operational rather than benchmark-based: production deployment within two months, continuous monitoring of status/tokens/latency, issue-debugging logs, and security/legal review of risk controls.

Metrics: Two-month production deployment · LLM status counts · Token counts · Latency · Health and performance · Security risk controls

Outcomes

• PagerDuty shipped multiple GenAI features to production within two months.
• The centralized service standardized LLM access, provider support, security, and monitoring.
• Product teams could integrate through endpoints instead of solving LLM operations themselves.
• The platform supported multiple providers and future extension to additional models.

The source reports a two-month production timeline, multiple product features, multiple providers, Kubernetes scalability, centralized monitoring, and status/token/latency metrics.