CPU vs GPU Bottlenecks in Agentic AI Workloads

The Agentic Shift

Classic LLM inference is GPU-bound: a request arrives, the GPU runs a forward pass, a response is returned. The GPU is the bottleneck almost by definition.

Agentic workloads break this assumption. Between inference calls, agents execute tool calls, query databases, parse structured outputs, manage conversation state, and orchestrate downstream agents. These steps run on CPU. Depending on the workflow, the GPU may be idle for longer than it's active.

The result: GPU utilization drops, latency increases, and the bottleneck is no longer where you expect it to be.

---

How to Tell Which You Have

The quick test: Compare GPU utilization with CPU utilization during a representative agentic workflow.

If your GPU sits at 30–40% while CPU is pegged, you have a CPU bottleneck. Adding GPU capacity will not help.

---

Common CPU Bottlenecks in Agentic Workloads

1. Tool call execution

Every tool call — web search, database query, API call — runs on CPU and blocks the next inference step. If tool calls average 500ms and inference averages 200ms, the agent spends 70% of its time waiting on CPU work.

Signal: GPU idle time correlates with tool call frequency. Trace tool call duration in your observability stack.

Fix: Parallelize tool calls where the agent logic allows it. Cache deterministic tool results. Move heavy parsing to async workers.

2. Structured output parsing

Parsing JSON, XML, or function call outputs from model responses is CPU work. At scale, this adds up — especially when outputs are large or malformed and require retry logic.

Signal: CPU spikes correlate with response parsing steps in traces.

Fix: Use streaming structured output libraries (Outlines, Guidance) that constrain generation rather than parsing after the fact.

3. Context assembly

Building the next prompt — retrieving memory, formatting tool results, constructing the message history — is CPU-bound string manipulation. For long conversation histories or large tool outputs, this can take hundreds of milliseconds.

Signal: Latency between inference calls is longer than tool call duration alone explains.

Fix: Pre-format context templates. Cache rendered prompt prefixes. Use prefix caching on the inference server to avoid reprocessing repeated context.

4. Tokenization

Tokenizing long inputs is CPU-bound. For agents that repeatedly tokenize large contexts, this adds measurable overhead.

Signal: Tokenization appears as a non-trivial step in request traces.

Fix: Cache tokenized representations of static prompt components. Use the inference server's built-in tokenizer rather than a separate CPU process.

---

The CPU:GPU Ratio Shift

Traditional inference clusters were GPU-heavy: one CPU core per GPU was often sufficient. Agentic workloads are changing this ratio.

NVIDIA's GH200 and GB200 architectures reflect this shift — the Grace CPU and Blackwell GPU are co-packaged specifically because agentic workloads need more CPU capacity alongside GPU. The NVL72 rack (18 Grace-Blackwell nodes) gives a 2:1 GPU:CPU ratio by design.

For clusters not running Grace-Blackwell, the implication is practical: if you're running agentic workloads on standard GPU nodes, you may need more CPU cores per node than your current configuration provides.

Detecting the imbalance:

# CPU utilization per core during an agentic workflow
mpstat -P ALL 1 10

# GPU SM utilization simultaneously
nvidia-smi dmon -s u -d 1

If CPU cores are saturated while GPUs are idle, you need to rebalance — either by adding CPU capacity or by offloading CPU work to dedicated workers.

---

Architectural Patterns for CPU-GPU Balance

Pattern 1 — Dedicated orchestration workers Separate the agentic orchestration layer (tool calls, context assembly, routing) onto CPU-only workers. GPU nodes handle inference only. This isolates the bottlenecks and lets each tier scale independently.

Pattern 2 — Async tool execution Run tool calls asynchronously and batch inference calls when multiple tool results are ready. Reduces GPU idle time between steps.

Pattern 3 — Speculative execution For predictable agentic workflows, begin the next inference step speculatively while tool calls are in flight. Discard if the tool result changes the input.

---

What to Monitor

See how Paralleliq surfaces CPU:GPU imbalance across agentic inference fleets →

---

Next in the GPU Ops Field Guide: [How to Reduce LLM Inference Costs Without Sacrificing SLA →](/blog/gpu-ops-reduce-inference-costs)