Serving GLM-5.3 Across Two DGX Stations: Where vLLM's Distributed Path Gets Hard

NVIDIA’s November 13 serving walkthrough puts a 753-billion-parameter GLM-5.3 checkpoint across two DGX Stations and serves it with vLLM. The engineering question is not simply whether two machines can hold the model. It is whether the inter-node path, KV cache, and request scheduler keep enough work on each GPU to offset communication and memory pressure. ^[1]

The execution shape

The tutorial describes a roughly 464 GB checkpoint, two GB300-based systems, ConnectX-8 networking, tensor parallelism across the pair, FP8 KV cache, and prefix caching. Tensor parallelism splits layer operations across devices, so each generated token requires synchronization across the network. The link is therefore on the decoding critical path, unlike an architecture that keeps independent replicas on separate nodes.

NVIDIA reports 607.8 output tokens per second at concurrency 16, and 385.0 at concurrency 32 with mean request latency of 453.5 seconds. Those are measurements from the vendor’s setup, prompt mix, software stack, and benchmark method. They do not establish a general service rate or a user-facing latency guarantee. The fall at higher concurrency is a useful reminder that aggregate throughput and per-request completion time are different service objectives.

What operators should measure

Reproduce the exact model revision, quantization, prompt and output lengths, warm-up, and vLLM configuration before comparing results. Record prefill and decode separately; inspect tokens per second per request, tail latency, GPU utilization, KV-cache occupancy, and RDMA traffic. Prefix reuse can materially change the workload. A benchmark with repeated prefixes may tell a different story from a workload with mostly unique context.

FP8 KV storage reduces cache memory, but quality and capacity depend on the implementation and workload. Test long-context retrieval and generation quality against the intended baseline. Also test failure behavior: a link interruption or one host restart can affect a tensor-parallel replica as a unit. Two workstations are a useful lab topology, but production readiness still depends on observability, recovery, and sustained thermal and power behavior.

The takeaway is a reproducible distributed-inference recipe, not proof that adding a second station doubles capacity. The network is part of the model execution, and benchmarking must include it.

Separate prefill from decode in the benchmark

Prefill processes the input context and tends to expose large parallel matrix operations. Decode generates tokens incrementally and repeatedly touches the KV cache. A service can therefore look healthy on prompt ingestion while becoming network- or memory-bound during generation. Report both phases, and state whether throughput is aggregate output tokens per second or a per-request rate. Include percentiles: a mean can hide a subset of long prompts that occupy cache and hold up other requests.

The two-host arrangement also changes failure domains. A node restart, link reset, or RDMA configuration mismatch can make a distributed replica unavailable even when one GPU remains healthy. Test process restart, health checks, and model reload time. For a latency-sensitive endpoint, compare the two-host replica with smaller independent replicas where each request stays local to one host. The latter may hold fewer model variants, but can avoid cross-host synchronization on every decode step.

For repeatability, record GPU and NIC firmware, driver, CUDA, vLLM revision, model artifact hash, quantization format, and launch parameters. Capture actual inter-node bandwidth under the same traffic conditions, rather than substituting the link’s advertised line rate. The benchmark is useful when these details accompany it; without them, the headline throughput is difficult to transfer to another deployment.

Teams evaluating adjacent hardware tradeoffs can compare this with our Vera Rubin system overview and GPU scheduling coverage. Those systems questions affect capacity planning, but do not replace workload-specific inference measurements.

A useful comparison matrix uses one axis for context length and another for concurrent sequences. Run repeated-prefix and unique-prefix traffic, then add bursty arrivals that resemble a real queue. Use fixed request traces so changes in caching are not mistaken for a network improvement. Record errors, retries, and time spent waiting for a slot, since high tokens-per-second can coexist with poor completion behavior.

For capacity planning, estimate the KV cache required by the target context and concurrency, then leave headroom for fragmentation and scheduler behavior. The checkpoint fitting across two machines does not imply that every desired batch fits comfortably. Monitor cache eviction and preemption under long prompts, and include warm-up and model loading in recovery objectives. Compare these results with our NVIDIA GPU scheduling overview and Vera Rubin architecture coverage, while keeping each workload’s measurements separate.