Optimizing Copilot Latency with NVIDIA TensorRT-LLM Integration

moveworks · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise copilot inference latency

Moveworks needed to reduce LLM inference latency for its enterprise AI Copilot so employee support conversations felt natural and used resources efficiently.

Constraints:

Token throughput, average latency, and time-to-first-token all affected user experience.
The copilot needed production-grade GPU serving optimizations.
The solution had to improve resource utilization, not only raw speed.

Architecture

Moveworks integrated NVIDIA TensorRT-LLM using optimized CUDA kernels, multi-GPU model parallelism, Flash-Decoding, and SmoothQuant quantization.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-ai-copilot["Enterprise AI Copilot"]
  tensorrt-llm-serving["TensorRT-LLM Serving"]
  gpu-infrastructure["GPU Infrastructure"]
  enterprise-ai-copilot --> tensorrt-llm-serving
  tensorrt-llm-serving --> gpu-infrastructure

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Enterprise AI Copilot`	Serves employee support conversations.	`TensorRT-LLM Serving`
`TensorRT-LLM Serving`	Optimizes LLM inference with CUDA, Flash-Decoding, quantization, and GPU parallelism.	`GPU Infrastructure`
`GPU Infrastructure`	Runs optimized inference for production traffic.	—

Data flow

Copilot inference requests are served through TensorRT-LLM optimized execution, which applies CUDA kernels, quantization, Flash-Decoding, and multi-GPU parallelism to reduce latency and increase throughput.

“leveraged optimized CUDA kernels, multi-GPU model parallelism, and techniques like Flash-Decoding and SmoothQuant quantization”summary, stated

“enabling more natural conversational experiences and better resource utilization for their employee support platform.”summary, stated

Why this architecture

TensorRT-LLM attacked the serving bottleneck directly at the GPU execution layer, improving throughput, latency, and first-token delay together.

“dramatically improve performance.”summary, stated

Models used: NVIDIA TensorRT-LLM · Flash-Decoding · SmoothQuant

Scale

Token throughput improved 2.3x from 19 to 44 tokens per second.
“2.3x improvement in token throughput (from 19 to 44 tokens per second)”summary, stated
Average request latency fell 2.35x from 3.4 seconds to 1.5 seconds.
“2.35x reduction in average request latency (from 3.4 seconds to 1.5 seconds)”summary, stated
Time-to-first-token improved 2.7x from 0.8 seconds to 0.3 seconds.
“2.7x improvement in time-to-first-token (from 0.8 seconds to 0.3 seconds)”summary, stated

Alternatives considered (roads not taken)

Unoptimized LLM serving

Rejected because: It had materially worse throughput, latency, and first-token time.

“LLM inference latency in their enterprise AI Copilot”summary, stated

Patterns

`cost-aware-routing` (serving-optimization)

TensorRT-LLM improved resource utilization and serving performance.

“better resource utilization for their employee support platform.”summary, stated

Tradeoffs

Integrate TensorRT-LLM

Chose GPU-level optimized serving over Baseline LLM inference stack.

Rationale: It improved throughput, average latency, and time-to-first-token simultaneously.

“integrating NVIDIA TensorRT-LLM optimization engine.”summary, stated

Gotchas (what bit them)

Copilot UX depends heavily on first-token and average latency.

Root cause: Slow inference makes employee support conversations feel unnatural.

Resolution: Optimize the GPU inference engine with TensorRT-LLM.

“enabling more natural conversational experiences”summary, stated

Evaluation

Moveworks measured token throughput, average request latency, and time-to-first-token.

Metrics: Tokens per second · Average request latency · Time to first token

“from 19 to 44 tokens per second”summary, stated

Outcomes

Token throughput improved 2.3x.
Average request latency fell 2.35x.
Time-to-first-token improved 2.7x.

Reported serving metrics moved from 19 to 44 tokens/s, 3.4s to 1.5s latency, and 0.8s to 0.3s TTFT.

“2.3x improvement in token throughput”summary, stated