Optimizing vLLM for High-Throughput Embedding Inference at Scale

snowflake · Tech · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise embedding inference throughput and cost optimization

Snowflake needed Cortex AI embedding inference to support large real-time and batch workloads with high throughput, low latency, and efficient GPU utilization, but vLLM benchmarks exposed CPU-side bottlenecks that left GPUs underused.

Constraints:

Architecture

Snowflake profiled vLLM embedding inference, found that CPU tokenization and Protobuf serialization dominated runtime, and reworked the serving path. The optimized architecture encodes embeddings as little-endian bytes, pretokenizes input so tokenization and GPU inference overlap across requests, and runs multiple identical embedding model replicas on one GPU to use idle resources. These improvements are rolling into Cortex AI and open-sourced through Arctic Inference.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Embedding_requests["Embedding requests"] --> Pretokenization_stage["Pretokenization stage"]
  Pretokenization_stage --> vLLM_embedding_inference["vLLM embedding inference"]
  vLLM_embedding_inference --> Little_endian_serialization["Little-endian serialization"]
  vLLM_embedding_inference --> Multiple_model_replicas["Multiple model replicas"]
  Multiple_model_replicas --> vLLM_embedding_inference
  Little_endian_serialization --> gRPC_frontend["gRPC frontend"]
  gRPC_frontend --> Snowflake_Cortex_AI_products["Snowflake Cortex AI products"]
  Profiling_traces["Profiling traces"] --> vLLM_embedding_inference
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Embedding requests Text inputs for semantic search, recommendations, fraud detection, agents, search, and analytics workloads. Pretokenization stage
Pretokenization stage Converts strings to token IDs before GPU inference so tokenization can overlap across requests. vLLM embedding inference
vLLM embedding inference Runs embedding model inference on GPUs from pretokenized inputs. Little-endian serialization, Multiple model replicas
Multiple model replicas Runs concurrent copies of the embedding model on one GPU to use otherwise idle resources. vLLM embedding inference
Little-endian serialization Serializes embedding tensors as raw little-endian bytes to reduce gRPC response latency. gRPC frontend
gRPC frontend Production serving layer supporting multi-tenant serving, multiple languages, and dynamic model swapping. Snowflake Cortex AI products
Snowflake Cortex AI products Downstream user-facing capabilities such as Cortex Agents, Cortex Search, and Cortex Analyst. —
Profiling traces Python runtime traces and flame graphs used to locate CPU/GPU bottlenecks. vLLM embedding inference

Data flow

Text inputs are pretokenized before vLLM, token IDs are passed into GPU inference, embedding tensors are serialized as little-endian bytes through the gRPC service, and multiple model replicas concurrently serve requests on the same GPU to increase throughput.

3 supporting quotes
“Instead of sending raw text, we pretokenize inputs and pass token IDs directly to vLLM.”article, stated
“We significantly reduced gRPC response latency by encoding the output list of floats (embedding) as raw bytes in little-endian.”article, stated
“These replicas can serve inference requests concurrently and thus increase inference throughput given the same number of GPUs.”article, stated

Why this architecture

Snowflake optimized around the actual bottlenecks revealed by profiling. GPU inference was only a small fraction of compute time, so better model kernels alone would not solve the problem; the serving path needed CPU serialization, tokenization, scheduling, and replica-level utilization changes.

3 supporting quotes
“The embed() function, which handles GPU inference, accounts for just 10% of total compute time. The remaining 90% is spent on CPU tasks”article, stated
“The flame graph attributed the CPU overhead to two sources: tokenization and data serialization.”article, stated
“Running multiple replicas of an embedding model on a single GPU increases throughput by better utilizing available GPU resources”article, stated

Models used: snowflake-arctic-embed-m-v1.5 · Embedding models

Scale

Alternatives considered (roads not taken)

Treat GPU inference as the main optimization target

Rejected because: Profiling showed only 10% of compute time was GPU inference and 90% was CPU overhead.

“The embed() function, which handles GPU inference, accounts for just 10% of total compute time. The remaining 90% is spent on CPU tasks”article, stated

Send raw strings directly to vLLM

Rejected because: Sequential CPU tokenization kept the GPU idle and created bubbles in the schedule.

“When vLLM receives prompts as strings, it would tokenize input strings on the CPU before launching inference on the GPU.”article, stated
“Because these steps are sequential, the GPU remains idle during tokenization, creating “bubbles” in the schedule and limiting throughput.”article, stated

Use Protobuf repeated float serialization for embeddings

Rejected because: Converting Python List[float32] into repeated float format added latency in the gRPC serving path.

“it introduces latency when converting embedding outputs from Python List[float32] to Protobuf’s repeated float format”article, stated

Run a single embedding model instance per GPU

Rejected because: Embedding models often leave GPU resources idle, so concurrent replicas better utilize the same hardware.

“Embedding models typically have fewer parameters and shorter runtimes than autoregressive LLMs, which means a single instance often leaves GPU resources underused”article, stated

Patterns

observability-tracing (operations)

Snowflake used runtime traces and flame graphs to find that CPU tokenization and serialization, not GPU inference, dominated embedding serving time.

“To uncover the bottlenecks, we profiled embedding inference in vLLM using Python runtime traces. Figure 1 shows a flame graph from one run — each span represents the duration of a function call.”article, stated

cost-aware-routing (optimization)

The architecture optimizes serving cost by improving throughput per GPU and showing that higher-end GPUs can be cheaper per token when fully utilized.

“by pushing the embedding throughput on the H200 GPUs, we are able to lower the cost of inference using the more expensive H200”article, stated

latency-optimization (performance)

Snowflake targeted tokenization, serialization, and GPU idle time because those serving-path delays directly affected real-time embedding products.

“The throughput gains are rolling out across Cortex, powering faster indexing and lower-latency responses across products like Cortex Agents, Cortex Search and Cortex Analyst.”article, stated

model-routing (model-operations)

Snowflake allocates A10g GPUs to embedding inference and reserves H200-class GPUs for very large models, then benchmarks price-performance tradeoffs.

“Snowflake uses A10g to serve embeddings in production. There is an industry-wide supply shortage of more powerful GPUs, such as H200, and we reserve them for models with hundreds of billions of parameters.”article, stated

Tradeoffs

Serialize embeddings as little-endian bytes.

Chose Raw bytes with NumPy vectorization over Standard Protobuf repeated float serialization.

Rationale: Raw little-endian bytes avoid copying, byte-swapping, and Python per-element overhead in the gRPC response path.

“We use little-endian because it is the native endianness on most instruction set architectures (ISA), so using little-endian avoids a memory copy or byte-swapping step, thus faster than big-endian for our use case.”article, stated
“Vectorization accelerates computation by applying operations to entire arrays at once, leveraging low-level optimizations and SIMD instructions.”article, stated

Disaggregate tokenization from GPU inference.

Chose A two-stage pretokenization and inference pipeline over Sequential string tokenization inside vLLM.

Rationale: The staged design overlaps tokenization for one request with GPU inference for another, removing idle GPU bubbles.

“To resolve this, we disaggregated tokenization and inference into a two-stage pipeline. Instead of sending raw text, we pretokenize inputs and pass token IDs directly to vLLM.”article, stated
“* Tokenization for r2 can happen while inference for r1 is running”article, stated

Run multiple replicas on one GPU.

Chose Concurrent identical embedding model replicas over One model instance per GPU.

Rationale: Embedding workloads leave GPU resources idle due to shorter runtimes and CPU/data-transfer bottlenecks, so concurrent replicas increase utilization.

“The idle GPU resources mean that we can run multiple replicas of the same model on a single GPU. These replicas can serve inference requests concurrently and thus increase inference throughput given the same number of GPUs.”article, stated

Gotchas (what bit them)

GPU underutilization was hidden behind apparently model-serving work.

Root cause: CPU tokenization and serialization consumed most compute time while the GPU waited.

Resolution: Profile the runtime path and optimize tokenization, serialization, and replica scheduling.

“When we profiled our embedding models on vLLM, the throughput and GPU utilization were far worse than what a PyTorch-native implementation could achieve.”article, stated
“The flame graph attributed the CPU overhead to two sources: tokenization and data serialization. But why did they take so long?”article, stated

Sequential tokenization created GPU schedule bubbles.

Root cause: vLLM tokenized string prompts on CPU before launching GPU inference.

Resolution: Pretokenize inputs and pass token IDs directly so tokenization and inference overlap across requests.

“The sequential process meant that the GPU would sit idle until tokenization completes, creating "bubbles" in the GPU schedule.”article, stated
“This enables pipeline parallelism; tokenization and inference can run in parallel across different requests, even though they remain sequential within each one.”article, stated

gRPC serialization introduced latency in the enterprise serving path.

Root cause: Python Protobuf conversion likely hit GIL and SIMD limitations when converting embedding float lists.

Resolution: Serialize embeddings as raw little-endian bytes and use NumPy vectorization.

“likely due to Python’s Global Interpreter Lock (GIL) and the lack of SIMD vectorization in Python Protobuf.”article, stated
“Vectorization accelerates computation by applying operations to entire arrays at once, leveraging low-level optimizations and SIMD instructions. Vectorization in NumPy is built on optimized C libraries and avoids Python’s per-element overhead.”article, stated

Evaluation

Snowflake used Python runtime traces and flame graphs to diagnose bottlenecks, then benchmarked each optimization in Snowflake Cortex on A10g and in open-source H200 comparisons against vLLM and Text Embeddings Inference.

Metrics: GPU inference only 10% of total compute time before optimization · 90% of compute time spent on CPU tasks · 3x throughput improvement in Snowflake Cortex AI · 230,000 sustained tokens per second in the final Cortex configuration · 16x higher throughput on 50-token sequences · 4.2x higher throughput on 512-token sequences · 16x cost reduction per token on short sequences

3 supporting quotes
“This section shows the incremental impact of the three optimizations — faster serialization, disaggregated tokenization and multi-replica execution — directly in Snowflake Cortex.”article, stated
“With all optimizations active, we saw 3x throughput improvement in Snowflake Cortex AI. As shown in Figure 4, each optimization step contributed to higher throughput. The final configuration delivered a sustained throughput of 230,000 tokens per second.”article, stated
“Compared to Text Embeddings Inference (TEI), we delivered up to 2.4x higher throughput on short sequences and maintained parity on long sequences.”article, stated

Outcomes

The reported production setting is Snowflake Cortex AI embedding inference at trillions of tokens per month, with A10g production serving and H200 open-source benchmarking.

3 supporting quotes
“With all optimizations active, we saw 3x throughput improvement in Snowflake Cortex AI. As shown in Figure 4, each optimization step contributed to higher throughput. The final configuration delivered a sustained throughput of 230,000 tokens per second.”article, stated
“The throughput gains are rolling out across Cortex, powering faster indexing and lower-latency responses across products like Cortex Agents, Cortex Search and Cortex Analyst.”article, stated
“we are open sourcing all vLLM-specific improvements as part of Arctic Inference”article, stated