GPU Server Selection: Matching Compute to Workloads

GPU Server Selection: Matching Compute to Workloads

When shipping AI applications, the easiest way to waste money during server selection is to pick the wrong GPU. The reason is simple: training and inference demand very different hardware, and using a training mindset to buy an inference machine (or the reverse) burns budget. This guide starts from your workload: how to size VRAM, how to compare cards, and single vs. multi-GPU.

Training and inference are two different animals

• Training repeatedly runs forward/backward passes over billions of parameters. It is compute- and memory-intensive, typically occupies full cards for long periods, runs multi-GPU in parallel, and depends on high-bandwidth interconnects such as NVLink.
• Inference runs a single forward pass per request and cares most about throughput and latency. Many inference workloads use far less VRAM than training (no gradients to keep), so they can run on smaller, cheaper cards, or split one large card across many users with MIG.

Step 1: Calculate how much VRAM you need

VRAM is the first constraint. A useful estimate for a Transformer model at FP16:

• Model weights roughly equal parameters x 2 bytes (FP16); a 70B model is about 140GB.
• Training adds optimizer state, gradients and activations, often 2-3x the weights or more.
• Inference adds KV cache plus input/output sequence buffers.

The takeaway: a 70B model at FP16 inference needs ~141GB of memory -- exactly what a card like the NVIDIA H200 (141GB HBM3e) provides; a 7B model is about 14GB, where an L4 or L40S suffices. INT8/FP8 quantization halves the memory footprint and further lowers the card class you need. For finer estimation, pair this with LLM inference optimization.

Putting the formula on real numbers makes it clearer. Take Llama 3 8B: FP16 weights are about 16GB; add KV cache plus input/output buffers and a single 24GB L4 can run it with headroom. At the 70B level, FP16 weights are about 140GB, and with activations plus KV cache they exceed one card, so you need 8-card parallelism or an H200 (141GB) single card with quantization. If you only infer rather than fine-tune, 4-bit quantization such as AWQ/GPTQ can squeeze 70B down to about 40GB, deployable on a single 48GB L40S — at the cost of a slight drop in generation quality and speed. Start with a rough "weights + inference overhead (about 10%-20%)" estimate before choosing a card; it beats guessing.

Step 2: Understand card differences, not just "how many"

Take the H200: it is the first GPU with HBM3e, and NVIDIA reports roughly 1.9x faster Llama2 70B inference and 1.6x faster GPT-3 175B inference versus the H100, with about 1,979 TFLOPS of FP16/BF16 Tensor Core compute and MIG partitioning (up to 7 instances). For inference, memory bandwidth matters more than peak compute -- large-model inference is bandwidth-bound, which is exactly why the H200 gains so much.

How to Quantify Inference Performance

Choosing an inference machine is not just reading a spec sheet; you need two numbers: throughput (tokens/s) and time to first token (TTFT). On the same card, online chat wants low TTFT while batch offline tasks want high throughput, and the configuration differs (batch size, KV cache reuse, and so on). A common approach is to load-test the target model with an engine like vLLM and record the throughput curve at different concurrency levels: at concurrency 1, watch single-request latency; at 32-64 concurrency, check whether total throughput still scales. Many vendors publish reference numbers — for example, H200 can reach roughly 3,000 tokens/s on Llama2 70B (exact figures vary with quantization, batch size, and engine). Get those two numbers first, then talk about "is it enough", so you do not buy the wrong card.

Step 3: Single card, multi-card, or cloud GPU

• Single card is enough: small-model inference, fine-tuning and agent apps fit on one card; prioritize per-card VRAM and bandwidth, and see AI hosting and model deployment.
• Multi-card training: pre-training large models needs multi-GPU parallelism, where interconnect (NVLink/NVSwitch) decides scaling efficiency; see the hardware forms in HGX B300 server roundup.
• Cloud GPU vs. owning: use cloud GPUs for variable demand and fast iteration, with vendor comparisons in GPU cloud server comparison. Only for long, steady, high utilization and lower unit cost should you consider GPU bare metal or on-premises.

Step 4: Common mistakes

1. Using a training card for light inference: large-VRAM cards are expensive per hour; start with entry options like GPU Droplets.
2. Looking only at VRAM, not bandwidth: two 80GB cards with HBM3 vs GDDR6 differ massively in inference speed.
3. Ignoring MIG/partitioning: slicing one large card across several low-load services raises utilization.
4. Not planning expansion: size the single-card capacity first, then decide whether you need multi-card and cluster networking (e.g., 800G), to avoid rework.

A Concrete Selection Case

Suppose you are launching an enterprise customer-service bot running a 7B-class open model, expecting 50 concurrent sessions and responses under 2 seconds. A rough estimate: 7B FP16 weights are about 14GB; with KV cache, a single 24GB L4 or 48GB L40S is enough, and the key is whether memory bandwidth sustains 50 concurrent sessions. This workload does not need H100/H200 at all — the hourly cost can differ by 5-10x. Conversely, if the team fine-tunes the model weekly, the training phase needs "multi-GPU + NVLink" planning, and only then do cards like the H100 come into range. Clarify whether the load is "inference-heavy ongoing" or "training-heavy periodic", and most of the selection is done.

Quick selection guide

• Below 7B, light inference -> single L4/L40S class
• 7B-70B inference/fine-tuning -> H100/H200 single or 2-4 cards
• 70B+ pre-training -> 8x H200/B300 plus fast interconnect
• Tight budget / experiments -> start on cloud GPUs such as Hetzner AI offerings

References: https://www.nvidia.com/en-us/data-center/h200/, https://docs.vllm.ai/en/latest/features/quantization/, https://www.nvidia.com/en-us/data-center/hgx/

Source: https://www.nvidia.com/en-us/data-center/h200/