LLM Inference Optimization: vLLM, Quantization, and Batching

Between "a large model that runs" and one that "runs fast and cheap" lies a clear inference optimization pipeline. LLM inference bottlenecks are usually memory and memory bandwidth rather than raw compute: generating each token reads the full set of weights, and with the ever-growing KV cache, GPU memory is typically the first resource to run out. Based on the official vLLM and TensorRT-LLM documentation, this article breaks down the mainstream optimization techniques.

1. Continuous Batching: The Core of Higher Throughput

Traditional batching waits for an entire batch to finish before switching to the next, leaving the GPU idle when requests vary in length. Continuous batching (in-flight batching) lets new requests join at any time and completed requests leave immediately, releasing resources, so the GPU stays saturated. vLLM reports that continuous batching can multiply throughput several-fold and up to 20x or more while cutting latency. It is a standard capability of every production-grade inference engine.

2. PagedAttention and KV Cache Management

The KV cache stores attention intermediates for generated tokens and consumes large amounts of memory as context grows. Traditional implementations reserve memory for the maximum possible length, wasting a lot. vLLM's PagedAttention borrows the idea of virtual-memory paging: it slices the KV cache into small blocks allocated on demand, dramatically improving memory utilization, which in turn supports larger batches and longer contexts.

TensorRT-LLM provides similarly fine-grained KV cache management, including cross-request KV cache reuse, KV cache quantization, and sparse attention features for long contexts.

3. Batching Strategy: Separating Prefill and Decode

Inference splits into Prefill (processing the whole input in parallel) and Decode (generating token by token), which have different resource profiles. Advanced engines support:

  • Chunked prefill / context chunking: slices long-input prefill into chunks interleaved with decode, reducing time-to-first-token.
  • Prefix caching: reuses computed KV cache for system prompts or common prefixes; the payoff is large for multi-turn chats and multi-user scenarios.
  • Disaggregated serving: places prefill and decode on separate instances that scale independently, handling high-concurrency, long-context workloads.

4. Quantization: Trading Precision for Memory and Speed

Quantization lowers weights from high precision (FP16/BF16) to INT8/INT4 or FP8, sharply reducing memory usage and accelerating compute. vLLM supports GPTQ, AWQ, FP8, INT4, GGUF, and more, with broad coverage of the open-source model ecosystem; TensorRT-LLM's quantization is deeply hardware-optimized for NVIDIA GPUs.

  • FP16/BF16: the default baseline with the best precision.
  • INT8 / FP8: minimal precision loss with clear memory and speed gains; a good production default.
  • INT4 (GPTQ/AWQ): the most memory-efficient, letting smaller GPUs run larger models; measure the accuracy trade-off yourself.
  • Advice: validate correctness at high precision first, then use quantization to fit memory, and finally verify throughput and latency with benchmarks (e.g., trtllm-bench).

5. Speculative Decoding: Accelerating Decode

Decode generates one token at a time and is the main source of latency. Speculative decoding uses a small model or draft module to produce several candidate tokens quickly, which the large model then verifies in parallel; if accepted, the whole sequence is kept, dramatically reducing serial steps of the large model. Both vLLM and TensorRT-LLM support EAGLE and other speculative decoding schemes, which help most for long outputs such as code and long-form text.

6. Model Parallelism: When a Model Does Not Fit on One GPU

Oversized models require parallelism: tensor parallelism splits each layer across GPUs, pipeline parallelism splits by layer, and expert parallelism distributes experts across GPUs for MoE models. vLLM supports tensor/pipeline/data/expert parallelism, while TensorRT-LLM adds specialized optimizations for MoE and long contexts on next-generation GPUs like Blackwell. When choosing, weigh communication overhead—optimize within a single GPU first, and only add parallelism when it no longer fits.

7. Advice from Selection to Benchmarking

  • Start with quantization plus continuous batching: these have the highest return on investment for most scenarios.
  • Let benchmarks decide: use vLLM's benchmarking or TensorRT-LLM's trtllm-bench to record throughput (tokens/s), TTFT (time to first token), and TPOT (time per output token); measure after every change.
  • Pair with hardware selection: VRAM determines the largest model you can fit; the GPU model sets the single-card throughput ceiling. See GPU Cloud Server Comparison and AI Local Deployment Hardware Guide.
  • Validate locally first: for small workloads, Ollama local deployment suffices; move to vLLM/TensorRT-LLM as you scale.

A Reproducible Optimization Order

Take a 7B model serving online, backed by a 24GB GPU. Unoptimized, single-instance throughput sits around 300-500 tokens/s, memory is tight, and high concurrency quickly triggers OOM.

Apply the steps in this order, and each one produces a measurable change:

  1. Enable continuous batching first: throughput jumps from 300 to 1200+ tokens/s because the GPU stops idling;
  2. Then quantize to FP8/INT8: memory usage drops 40-50%, the same card fits a larger max batch, and throughput roughly doubles again;
  3. Turn on prefix caching: for chat scenarios with long system prompts, repeated prefixes are no longer recomputed and TTFT can fall 30-50%;
  4. Finally evaluate speculative decoding: gains are obvious for long outputs (code, long-form text) but limited for short outputs, so measure both.

After every change, run a benchmark and record tokens/s, TTFT, TPOT and memory usage, then pin the numbers to the deployment doc. The next time you tune or scale, the team can reference historical data instead of rediscovering it.

16IDC Perspective

The essence of inference optimization is "more requests on the same GPUs, each one faster." For AI-related products, inference is often the largest operating expense, and optimization gains show up directly in gross margin. Plan the inference engine together with your AI model deployment approach: determine model size and concurrency estimates first, then pick vLLM or TensorRT-LLM, apply quantization and batching to squeeze out throughput, and let benchmark data drive scaling decisions.

Reference: vLLM performance benchmarks https://docs.vllm.ai/en/latest/performance/benchmarks/; TensorRT-LLM performance docs https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html

Source: https://docs.vllm.ai/en/latest/ and https://nvidia.github.io/TensorRT-LLM/