Implementing LLMOps in Restricted Networks with Long-Running Evaluations

microsoft · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLMOps for Prompt Flow applications in restricted Azure networks

Microsoft's ISE team needed to run CI, continuous evaluation, and deployment for a Prompt Flow LLM chat application inside a restricted bring-your-own Azure network with private IPs, strict whitelisting, first-time Azure ML use, and a comprehensive end-to-end evaluation that took about six hours.

Constraints:

• the network used private IPs inside an Azure Virtual Network with no public IPs
• Network Security Groups allowed only trusted IPs or specific resources
• pfazure compute/runtime host resolution required FQDN configuration
• AML managed endpoints did not work properly with the private network and integrated services
• Service Principal permissions were lengthy because of customer-specific security requirements
• blob storage and other Azure resources had to resolve to private IPs with explicit no-proxy configuration
• the complete end-to-end evaluation took about six hours and could bottleneck deployment
• teams needed accountability when skipping comprehensive evaluation

Architecture

The team built a three-stage LLMOps pipeline around Prompt Flow and Azure ML in a restricted BYO network. CI runs code quality, unit tests, and Prompt Flow syntactic validation. CE sets up environments, prepares synthetic evaluation data, runs node-level and end-to-end Prompt Flow evaluation, gathers metrics, and registers the flow in Azure ML. CD versions and containerizes the Prompt Flow with Docker, pushes the image to Azure Container Registry, deploys it to Azure Web App, runs smoke tests, and monitors endpoint health and performance. Because AML managed endpoints failed in the private network, the team fell back to Docker packaging and Web App deployment. To avoid six-hour evaluation bottlenecks, a GitHub PR label controls whether the full E2E evaluation runs, while a minimal 30-minute E2E dataset still runs before every development deployment.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Restricted_Azure_Virtual_Network["Restricted Azure Virtual Network"] --> Network_Security_Groups["Network Security Groups"]
  Restricted_Azure_Virtual_Network --> Private_endpoints_and_tunnels["Private endpoints and tunnels"]
  Restricted_Azure_Virtual_Network --> Azure_Machine_Learning["Azure Machine Learning"]
  Private_endpoints_and_tunnels --> Azure_resources["Azure resources"]
  Service_Principal["Service Principal"] --> Azure_Machine_Learning
  Service_Principal --> Network_Security_Groups
  Git_Flow_branches["Git Flow branches"] --> Continuous_Integration_pipeline["Continuous Integration pipeline"]
  Git_Flow_branches --> PR_label_evaluator["PR label evaluator"]
  Prompt_Flow_application["Prompt Flow application"] --> Continuous_Integration_pipeline
  Prompt_Flow_application --> Continuous_Evaluation_pipeline["Continuous Evaluation pipeline"]
  Continuous_Evaluation_pipeline --> Azure_Machine_Learning
  Continuous_Evaluation_pipeline --> PR_label_evaluator
  Continuous_Evaluation_pipeline --> Minimal_E2E_evaluation["Minimal E2E evaluation"]
  Continuous_Evaluation_pipeline --> Complete_E2E_evaluation["Complete E2E evaluation"]
  Complete_E2E_evaluation --> AML_metrics_and_job_links["AML metrics and job links"]
  PR_label_evaluator --> Complete_E2E_evaluation
  PR_label_evaluator --> PR_template_checklist["PR template checklist"]
  Prompt_Flow_application --> Docker_packaging["Docker packaging"]
  Docker_packaging --> Azure_Container_Registry["Azure Container Registry"]
  Azure_Container_Registry --> Azure_Web_App["Azure Web App"]
  Azure_Web_App --> Smoke_tests["Smoke tests"]
  Azure_Web_App --> Endpoint_monitoring["Endpoint monitoring"]
  Continuous_Deployment_pipeline["Continuous Deployment pipeline"] --> Docker_packaging
  Continuous_Deployment_pipeline --> Smoke_tests
  Continuous_Deployment_pipeline --> Human_approval_gate["Human approval gate"]
  Human_approval_gate --> AML_metrics_and_job_links
  Minimal_E2E_evaluation --> Development_deployment["Development deployment"]

Components

Data flow

Developers branch from develop, open feature PRs, and merge after CI checks. A PR from develop to main runs CI and syntactic end-to-end Prompt Flow checks. When code reaches main, CI and CE run: the pipeline prepares synthetic evaluation data, evaluates Prompt Flow nodes and the end-to-end flow, records Azure ML jobs and metrics, and decides whether to run the six-hour complete dataset evaluation based on the PR label. The Prompt Flow is versioned, packaged as Docker, pushed to Azure Container Registry, deployed to Azure Web App, smoke-tested, monitored, and gated by human approvals for dev, non-prod, and production progression.

Why this architecture

The architecture balances enterprise security and LLM evaluation cost. Private networking protects sensitive workloads but makes AML integration, endpoint hosting, permissions, and storage resolution harder. Git Flow and CI/CE/CD provide controlled release stages across data science, engineering, and platform teams. The dual evaluation setup keeps every dev deployment covered by a minimal dataset while making the six-hour complete evaluation an explicit, accountable release decision.

Models used: Large Language Model chat application built with Prompt Flow

Scale

• The complete end-to-end evaluation takes about six hours.
  “We had also run an end-to-end evaluation on the LLM chat application, which took 6 hours.”article, stated
• The minimal end-to-end dataset run takes about 30 minutes and runs before every development deployment.
  “end-to-end run on a minimal dataset will be executed before every Dev deployment (~30 mins)”article, stated
• The complete dataset evaluation runs in parallel with development deployment and takes about six hours.
  “End-to-end on a complete dataset is executed in parallel with Dev deployment (~6hrs), and has an opt-out option as set by the PR label”article, stated
• The branching model uses develop, feature, and main branches.
  “There are three branches involved in the entire process”article, stated
• The feature branch naming convention includes developer name, issue id, and feature name.
  “This follows a naming rule developer-name/issue-id-feature-name”article, stated
• The network has no public IPs.
  “The network is fully private, with no public IPs, relying on private endpoints or secure tunnels like VPN gateways for controlled external access.”article, stated

Alternatives considered (roads not taken)

Run the six-hour E2E evaluation for every commit or release

Rejected because: The Data Science team recognized the complete E2E evaluation was unnecessary for every change and would delay simple fixes.

Use AML managed endpoints in the private network

Rejected because: Managed endpoints did not work with the private network and integrated services, so the team used Prompt Flow Docker packaging and Azure Web App.

Maintain customer-specific SDK changes for pfazure compute-host resolution

Rejected because: A customer-specific SDK version would be cumbersome and add maintenance effort.

AML flow validations through the template samples

Rejected because: The team chose command-line AML flow validation to keep the pipeline lightweight and reduce platform-side script maintenance.

Fully automated promotion after component validations

Rejected because: The team provided metrics and AML job links for manual data-scientist validation before promoting changes.

Continuous CE for every project regardless of context

Rejected because: The article recommends matching CE frequency to commit activity, test duration, infrastructure resources, and deployment cycles.

Patterns

eval-driven-development (quality)

The pipeline makes continuous evaluation a first-class stage, with node-level Prompt Flow evaluation, end-to-end simulation, metrics, and explicit decisions about when to run the complete six-hour evaluation.

privacy-preserving-processing (governance)

The LLMOps pipeline operates inside a fully private Azure network using private IPs, private endpoints, secure tunnels, and whitelisting controls.

human-in-the-loop-gate (governance)

Human approvals and manual data-scientist validation remain in the release path, especially after CE failures or before promoting evaluation results.

synthetic-data-generation (evaluation)

The CE stage prepares synthetic evaluation data and uses it for Prompt Flow node and end-to-end evaluation.

observability-tracing (operations)

The deployment stage includes endpoint health checks and performance monitoring of response time and latency.

safety-policy-enforcement (governance)

PR labels, templates, and branch controls enforce release policy around when long evaluations may be skipped.

Tradeoffs

PR-label-controlled full evaluation

Chose allow explicit opt-out from the six-hour complete E2E evaluation over mandatory complete E2E evaluation for every release.

Rationale: The six-hour run is valuable but too expensive for simple changes, so the team makes skipping it an explicit accountable decision.

Dual evaluation runs

Chose 30-minute minimal E2E before every dev deployment plus optional six-hour complete E2E in parallel over only a complete six-hour E2E gate.

Rationale: The short run maintains basic validation while the long run covers comprehensive evaluation without blocking every development deployment.

Docker/Web App deployment

Chose package Prompt Flow in Docker and deploy to Azure Web App over AML managed endpoints.

Rationale: Managed endpoints did not work with the private network and integrated services.

Command-line AML flow validation

Chose lightweight command-line validations over maintaining heavier platform-side template scripts.

Rationale: The team wanted to reduce maintenance burden on the platform side.

Git Flow release control

Chose develop, feature, and main branches with controlled release ownership over simpler branch flow with faster direct deployments.

Rationale: The team needed organized multi-team collaboration and controlled releases, accepting that Git Flow can slow simpler products and hotfixes.

Gotchas (what bit them)

Private-network Azure ML setup created non-obvious integration friction.

Root cause: The customer used Azure ML for the first time in a restricted BYO network, requiring configuration, workarounds, and service whitelisting.

Resolution: Configure FQDNs, private IP resolution, no-proxy script settings, and fall back to Docker/Web App deployment where managed endpoints do not work.

AML managed endpoints failed in the private network.

Root cause: The managed endpoint path did not work with the customer's private network and integrated services.

Resolution: Package Prompt Flow with Docker and deploy it as a web service instead.

Full E2E evaluations can block delivery when they run for every small change.

Root cause: The complete flow runs a very large simulated dataset through a detailed evaluation flow and takes six hours.

Resolution: Use a PR-label opt-out plus a shorter minimal dataset evaluation before every development deployment.

An opt-out label can be misused or applied by mistake.

Root cause: Humans may skip evaluations intentionally or unintentionally, compromising stability and reliability.

Resolution: Require training, consistent label usage, a PR template checklist, stakeholder notification, and visibility/accountability through labels.

Git Flow can slow urgent fixes.

Root cause: Hotfixes and bug fixes still need to follow the full branch/release process.

Resolution: Use Git Flow when controlled releases and multi-team coordination justify the overhead, but recognize the delay tradeoff.

Service-principal permissions can stretch timelines in enterprise deployments.

Root cause: Customer-specific security requirements made AML and resource access approval lengthy.

Resolution: Plan explicit lead time for identity, permissions, and whitelisting in restricted enterprise LLMOps projects.

Evaluation

The pipeline uses CI for code quality, unit tests, and Prompt Flow syntactic checks; CE for synthetic data preparation, node-level and end-to-end Prompt Flow evaluation, metrics, and Azure ML registration; and CD for Docker/Web App deployment, smoke tests, endpoint health checks, and performance monitoring. A minimal 30-minute E2E dataset runs before every dev deployment, while the six-hour complete E2E evaluation runs in parallel unless a PR label explicitly opts out.

Metrics: 6-hour complete end-to-end evaluation · 30-minute minimal end-to-end dataset run before every development deployment · CI code quality checks · unit tests · Prompt Flow syntactic validation on sample data · node-level Prompt Flow evaluation · node-level evaluation metrics · end-to-end flow simulation and evaluation · smoke test on test dataset · endpoint health checks · response time and latency monitoring

Outcomes

• The team established CI, CE, and CD pipelines for a Prompt Flow LLM chat application in a restricted Azure BYO network.
• The private-network deployment required FQDN configuration, whitelisting, service-principal permissions, private-IP resource resolution, no-proxy settings, and Docker/Web App fallback.
• The six-hour complete end-to-end evaluation became configuration-driven through a PR label opt-out.
• A 30-minute minimal end-to-end evaluation still runs before every development deployment.
• Git Flow helped coordinate data scientists, engineers, and platform teams around controlled releases.
• The approach balanced thorough model validation with deployment efficiency and enterprise network constraints.

This is a production LLMOps infrastructure case rather than an agentic application. Its strongest reusable lesson is release governance for long-running LLM evaluations in locked-down enterprise networks: keep a short mandatory gate, make long gates explicit and accountable, and document the risk of human opt-out misuse.