How Token Compression Changes Your GPU Sizing Math

Token compression is typically framed as a cost-per-token story. Compress your prompts, reduce your token count, pay less per API call. That framing is correct but incomplete.

The infrastructure math changes too — and most teams never recalculate it.

---

The Connection Between Tokens and GPU Memory

Every token in a request's context window requires memory on the GPU. Not compute — memory. Specifically, KV cache: the key and value vectors that the attention mechanism stores for every token, across every layer of the model.

The formula is straightforward:

KV cache per request = 2 x layers x KV heads x head dimension x bytes x context length

Context length is the only variable the application layer controls. Everything else is fixed by the model architecture.

For a 7B model (Llama 3, Mistral) at FP16:

Cut context length in half, cut KV cache per request in half. The relationship is linear and exact.

---

What This Means for GPU Sizing

GPU VRAM is split between model weights (fixed) and KV cache (variable). The model weights load once and stay resident. Everything left over is available for KV cache — which determines how many requests you can serve concurrently.

Take a 7B model on a single L4 (24GB):

• Model weights: 16 GB (FP16)
• System overhead: 2 GB
• Available for KV cache: 6 GB

At 4K average context (0.54 GB per request): 11 concurrent requests

At 1.5K average context after compression (0.20 GB per request): 29 concurrent requests

Same GPU. Same model. 2.6x more concurrency — purely from reducing context length.

---

The Tier Change Scenario

The more significant implication is GPU tier selection. Consider a 7B model workload running at 4K average context that needs to serve 20 concurrent requests reliably.

Without compression:

• 20 requests x 0.54 GB = 10.8 GB of KV cache needed
• Total VRAM needed: 16 GB (weights) + 10.8 GB (KV) + 2 GB (overhead) = 28.8 GB
• Minimum GPU: A10G (48GB) or A100 40GB
• Cost: ~$0.90/hr

With compression reducing context to 1.5K average:

• 20 requests x 0.20 GB = 4.0 GB of KV cache needed
• Total VRAM needed: 16 GB + 4 GB + 2 GB = 22 GB
• Minimum GPU: L4 (24GB) — fits comfortably
• Cost: ~$0.54/hr

The infrastructure saving: 40% on GPU cost. Every hour, every day, on top of the token savings already captured.

---

The 70B Case Is Even More Dramatic

Large models make this effect more pronounced because the model weights consume most available VRAM, leaving almost nothing for KV cache.

A 70B model (Llama 3, Qwen 72B) on 2x A100-80GB at FP16:

• Total VRAM: 160 GB
• Model weights: 140 GB
• Overhead: 2 GB
• Available for KV cache: 18 GB

At 8K average context (21.5 GB per request): zero concurrent requests — a single request at 8K context exceeds the available KV budget entirely.

At 3K average context after compression (8.0 GB per request): 2 concurrent requests — workable for many use cases.

At 1.5K average context (4.0 GB per request): 4 concurrent requests — a meaningful serving configuration.

For large models in particular, token compression is not just a cost optimization — it can be the difference between a workload being feasible at all and requiring a complete infrastructure overhaul.

---

Why Teams Miss This

The token savings from compression show up on the API bill immediately. The infrastructure implication does not — it requires someone to go back, recalculate the KV cache math with the new context length, and re-evaluate the GPU tier decision.

Most teams made their GPU tier selection once, at deployment. They sized for their original context length assumptions, chose a tier, and moved on. Token compression happened later, as an optimization. Nobody went back to revisit the infrastructure.

The result: teams are paying for a GPU tier sized for a context length they no longer have. The compression savings are real, but they are only half of the available optimization.

---

How to Recalculate

Three inputs change when context length drops:

1. Recalculate KV cache per request Use the actual new average context length after compression. Your p99 context length is the number that matters for capacity planning — use that, not the model maximum.

2. Re-evaluate max_model_len in vLLM This parameter caps the maximum context the serving engine will accept. Setting it to your actual p99 context length (rather than the model maximum) frees significant VRAM. A model with a 128K context window does not need maxmodellen=131072 if your compressed requests are averaging 1.5K tokens.

3. Re-evaluate GPU tier With the new KV cache per request, recalculate the minimum VRAM needed to serve your target concurrency. You may find that a tier one step down now fits comfortably.

Our KV Cache Calculator lets you model this directly — change the context length slider and see exactly how concurrency and cost change. The vLLM Configuration Calculator takes it further and outputs the full recommended configuration for the new parameters.

---

The Complete Picture

Token compression and GPU rightsizing are two independent optimizations that compound:

• Token compression reduces what you pay per token
• GPU rightsizing reduces what you pay per GPU hour
• Together they attack the inference bill from both sides

The teams that capture both typically see 50-70% total infrastructure cost reduction compared to an unoptimized baseline — with no change to model quality or application behavior.

The compression is the first step. The infrastructure recalculation is the second. Most teams only take the first.

Paralleliq helps you take the second. Start with the [KV Cache Calculator](https://paralleliq.ai/calculators/kv-cache) to model the impact on your specific workload, or run [piqc](https://github.com/paralleliq/piqc) against your running cluster to see what the current configuration is costing you.