08 Deep dive

Training and inference performance on ML workloads

Machine-learning workloads concentrate the two scheduling problems Temper addresses: training pipelines place substantial CPU work (decode, augmentation, batching) between storage and an expensive accelerator, and inference servers run small groups of hot threads whose placement relative to SMT topology dominates latency. This article consolidates the measured training and inference results by workload class — GPU-fed training under co-location, CPU training against standard Kubernetes remedies, and ONNX and llama.cpp inference with workload profiles — including the configurations in which enforcement measured no benefit.

workloads: PyTorch training · ONNX inference · llama.cpp · vLLM clusters: GKE (g2, c2, c3) · EKS (m5) records: 8 reports method: benchmark methodology

Where scheduling enters an ML pipeline

Kubernetes assigns accelerators exclusively but schedules the CPU side of the same pod under ordinary CFS weights. A training step consumes its input pipeline — DataLoader workers decoding and augmenting on CPU — before the accelerator can do useful work; if those threads are descheduled by co-located neighbors, the GPU idles while it remains reserved and billed. Inference has a different failure mode: the hot path is a handful of compute threads whose throughput depends on running on distinct physical cores, a placement property invisible to container-level metrics. The first problem is arbitration between pods and is addressed by QoS tiers; the second is structure within a pod and is addressed by workload profiles.

Training under co-location

The GPU-fed case was measured with ResNet-18 training on an NVIDIA L4 (GKE g2-standard-8) against a batch-neighbor ladder. With a Burstable trainer whose CPU demand exceeded its request (2-CPU request, ~7 vCPU demand), CFS lost 25% of training throughput at 16 neighbors (629→471 samples/s) and GPU utilization collapsed into a 0–81% band; under kernel-enforced tiers, throughput held at 636–642 samples/s and GPU utilization ~85% at every step. The effect is conditional: a control configuration with a Guaranteed trainer whose demand fit its request was flat in both arms. The full experiment, including that boundary condition, is the accelerator-utilization article.

The CPU-only case compares enforcement against the standard remedies on one node. A Guaranteed 3-CPU PyTorch trainer at density 8 (c3-standard-8) measured 14.8 samples/s under CFS, 16.5 with quota limits, 14.6 with a static kubelet cpuset — the pin was SMT-blind — and 24.7 under Temper (+67% over CFS). Whole-core, SMT-aware placement accounts for part of that margin, and the fence traded background reclaim for it (~1.9 background cores against CFS’s 5.0 on that run).

Inference: ONNX and llama.cpp

CPU inference exposes the intra-pod problem. ONNX Runtime (ResNet-50, 3 intra-op threads, Critical tier) on a 4-core c2-standard-8: tier QoS alone held throughput flat under a background ladder but paid a peak-throughput cost, because the whole-core tier layer SMT-paired the three hot threads (22.9 vs CFS’s 43–44 samples/s on an idle node). An exclusive-core workload profile — three hot threads on three whole cores, siblings idle — restored idle-node parity (44.5 vs 44.5) and held ±0.3% across the ladder while CFS degraded up to 49%.

llama.cpp behaves the same way with thread counts in place of intra-op parallelism. On c2-standard-8 with the exclusive-core profile: idle-node parity (1512 vs 1514 ms median) and worst-case drift of +3.7% under density against CFS’s +27%; at bg=8 the profile arm measured −18% median and −24% p90 relative to CFS. On a 4-core m5.2xlarge with no profile, tier QoS alone was sufficient to hold medians flat while CFS degraded +46% — SMT stacking is a property of small-core geometry, not of the allocator in general — at the cost of a residual ~12% idle-node gap. The 2-core SMT-collision analysis, the per-thread evidence from /observe, and the profile arithmetic are in the thread-profiles article.

Negative results and boundary conditions

Three configurations measured no benefit, and they delimit where enforcement applies. GPU-bound serving: vLLM serving a small model (Qwen2.5-0.5B) on the L4 measured parity in both a right-sized and a deliberately starvation-shaped configuration — tokenization at this concurrency costs a fraction of one core, so there is no CPU contention to remove. Right-sized training: a Guaranteed trainer whose demand fits its request is fully defended by kubelet’s existing QoS weighting in both arms. Small-core geometry: no profile can allocate three whole cores on a 2-core node; the configuration generator demotes such layers to Grouped, which bounds the degradation (−27% worst case on the ONNX 2-core run) but cannot restore parity — critical pods requesting more than 2 CPUs should not land on 2-core nodes.

The enforcement result applies where substantive CPU work sits between storage and the compute device — DataLoader-heavy training, preprocessing-heavy inference (long prompts, large tokenizers, multimodal encode), CPU inference generally — and does not apply to workloads whose CPU side is negligible.

Limitations

• This article consolidates existing committed records; it reports no new experiments. The per-experiment caveats in the accelerator article and the thread-profiles article apply unchanged.
• Single run per arm throughout; 60 s windows on GPU nodes, 120 s for ONNX.
• The GPU training effect is conditional on trainer CPU demand exceeding its request.
• Inference profiles are synthesized by the training pipeline; the committed pipeline cycle record (phase4) includes a mixed result in which a synthesized PyTorch profile trailed CFS on its ladder, which is why refinement retains only measured improvements.
• No distributed (multi-node) training measurement exists yet; all training runs are single-node.