GPU 加速權重傳送技術

Introduction#

Large-scale RL is different from either large-scale pretraining or large-scale inference. A key difference is that the training weights and the sampler weights should ideally match as closely as possible, which requires the trainers sending weight updates to the samplers. This creates a bottleneck: one must either use high speed intra-cluster networking to send weight updates or accept the often-meaningful delay that comes with attempting to communicate terabytes of model weights, not to mention blocking on the samplers. We introduce a faster weight update paradigm, in most cases allowing better-than-intra-cluster speeds, regardless of whether the samplers and trainers are co-located within the same cluster.

The design starts from the widely adopted idea of delta weight sync^{1Miles v0.1: Production-Level Post-Training, §4.3 Disk-Delta Updates.}. We co-designed an entire Blackwell-native compression/decompression pipeline combining the following techniques:

1. Massively parallel GPU compression, with distributed ownership and fault tolerance.
2. A curated double compression algorithm with heterogeneous CPU-GPU decompression.
3. Blackwell Decompression Engine (DE) and hardware-native decompression algorithms, fully overlapping DE with ordinary GPU compute.
4. Tiny memory buffer overheads across trainers and samplers, CPUs and GPUs.

As a result, we bring the sampler pause time below 0.23 s and the end-to-end weight sync time below 4 s in our Ampersand production RL configuration, fully saturating the Blackwell Decompression Engine pipeline and making the decompression ASIC the bottleneck for sampler pause time. The implementation is so fast in terms of both end-to-end weight sync time and sampler pause time that even intra-cluster P2P RDMA is slower for most real-world workloads.

For the rest of this article, we use the following as a consistent workload proxy. The model is NVFP4 GLM-5.3, with 445 GB of canonical weight tensors. RL training signals are sparse, with only a tiny portion of weights being updated during each training step, so we mutate 0.085% of the checkpoint as a proxy for one steady-state update. The trainer has 8 nodes with 64 B300 GPUs. Each rollout receiver has two TP4 NVFP4 GLM-5.3 engines on one 8-GPU B300 node. The diagrams use fewer GPUs and model layers for clarity.

CPU disk delta weight sync#

To explain our design progression, we need to understand the foundations of delta weight sync and why a naive implementation is not enough.

Delta update algebra#

# trainer
mask            = weights_old ⊕ weights_new

# rollout
weights_updated = weights_old ⊕ mask
                = weights_new

The core of delta weight sync is the bitwise XOR (⊕) operation. Before a training step, the rollout engines and trainer hold identical weights. The trainer finishes a training step and advances from weights_old to weights_new. The trainer performs a bitwise XOR between weights_old and weights_new to obtain a mask with exactly the same shape as the weights. The 1s in the mask indicate that the weight bit at that location has flipped, and 0s mean unchanged. The rollout engines need weights_new to sample from the new policy for the next RL step. To obtain them, the engines only need to flip the same weight bits. Thus, we transfer the mask to the rollout engines and apply bitwise XOR between their weights_old and the mask. Since the weights are viewed as a long contiguous sequence of bits, their specific data types do not matter and floating-point arithmetic or rounding is not involved.

But masks have the same size as the weights, so directly transferring masks does not save anything versus transferring the full weights. What truly makes the delta sync approach attractive is the following insight. In RL, training signals are sparse, so only a tiny portion of weights are updated during each training step. That means the masks contain mostly 0s and only a tiny portion of 1s. A lossless compression algorithm can then compress the masks into a tiny transfer payload.

Model checkpoint as a handoff#

Now let’s look at a naive but clean delta weight sync design within the RL loop. The trainer and rollout engines need to sync their checkpoints, so let’s use model checkpoints as the handoff. The trainer performs a training step, computes the mask, compresses it into a small payload, and sends the compressed mask to the rollout engines. The rollout engines load and decompress the mask, patch the checkpoint to match the new trainer weights, and reload weights from that checkpoint. Since the checkpoint is the handoff, the rollout engines reuse the same code path as server startup. We don’t need to handle how the rollout engines re-shard or map the weights.

The existing disk-delta implementation in Miles^{1Miles v0.1: Production-Level Post-Training, §4.3 Disk-Delta Updates.} largely follows this design:

From the measurement above, the compressed payload is only 0.876 GB, matching our expectations. However, this benefit has been completely offset by other significant overheads:

• Trainer compression adds latency.
• Checkpoint patching requires a disk round trip.
• Rollout engines need to reload the entire checkpoint from disk.

GPU accelerated (de)compression#

The RL stack already runs on distributed GPU clusters, so the intuitive next step is to make this pipeline GPU-accelerated, using their powerful parallel compute and high-bandwidth HBM.

A strong baseline with Blackwell-native DE#

Blackwell has a fixed-function Decompression Engine (DE) ^{2Decompression Engine FAQ.} that performs certain supported decompression algorithms extremely efficiently. To make our baseline sufficiently strong, we directly codesign the GPU-accelerated pipeline with GPU-native (de)compression algorithms and Blackwell-native dedicated hardware.

The table below compares (de)compression algorithms on GPUs^{3NVIDIA nvCOMP Quick Decision Matrix, qualitative ratings.}:

Table 1. GPU codec trade-offs and Decompression Engine support.

As shown in Fig. 4 above, the major overheads are on the receiver side, including decompression, checkpoint patching, and engine reload. Therefore, we prioritize receiver-side decompression performance, starting with LZ4/Snappy for a fast GPU codec pipeline.

LZ4 and Snappy are from the same algorithm family ^{4LZ4 and Snappy are byte-oriented LZ77 codecs: both encode literals and backward copies, without an entropy-coding stage. See the Snappy format and LZ4 block format.}, with one noticeable difference in how the copy commands are encoded. A single Snappy copy command can cover at most 64 bytes. LZ4, however, can encode a much longer region directly with fewer copy commands. Because the RL update masks are mostly filled with long sequences of zeros, LZ4 tends to compress the final payloads more compactly than Snappy. Note that the exact results can be implementation specific. Our experimental results support this intuition. On the same proxy mask, nvCOMP LZ4 compressed more compactly than Snappy. The LZ4 baseline produces a complete publication of 5.36 GB. Therefore, we choose LZ4 as the codec for this design.

Since we have decided to use GPUs for decompression, the decompressed masks are already resident in GPU HBM, and paying for a disk round trip for checkpoint patching does not make sense. Therefore, we implemented Triton kernels to apply the bitwise XOR in place in GPU HBM. The tiling and address mapping between model layers and decompressed masks can be complicated, so we prepare this metadata on the CPU and let the Triton kernel read from the prepared plan. As a result, the XOR application stage can be done extremely efficiently by a single Triton kernel launch for the entire model.

On the trainer side, we employ all the existing GPUs used by distributed training to compress the masks in parallel. Distributed training already means distributed tensor ownership, but it may still duplicate some model weights, especially with context parallelism (CP) and data parallelism (DP). For compression, we remap weight ownership so that the GPU ranks collectively cover the entire model checkpoint exactly once, with each rank owning only a subset of the weights it holds for distributed training. This has three major advantages:

1. Compression is greatly accelerated by parallelizing it across the existing training GPUs.
2. Weight-gathering traffic is greatly reduced because we only need to gather rank-local compressed payloads instead of the original model weights.
3. The memory pressure from holding earlier versions of the weights is greatly mitigated across all the existing ranks.

Putting the design together: all trainer GPUs compress concurrently, rollout GPUs decompress LZ4 directly using DE, and a single Triton kernel launch updates all model weights in HBM. All weight-sync network traffic carries compressed payloads, and disk checkpoints are completely out of the hot path. This is already an extremely strong baseline: for the same proxy payload, trainer-side preparation and compression take 0.834 s, and receiver-side work takes 6.041 s.

Working with low-precision RL#

Since compression, decompression, and bitwise XOR all treat model weights as a contiguous buffer of bits, delta weight sync works losslessly with low-precision RL.

Low-precision weights can even absorb small high-precision updates, such as changes to low mantissa bits, making the delta payload even smaller.

An interesting detail is how to work with high-performance MoE kernels like the one we described in Faster-than-light NVFP4 decode—with precision. To achieve aggressive fusion, high-performance kernels may reorder the logical weights into a different physical layout. See the example below of pairing gate and up weights in groups of 64. Using model checkpoints as handoffs dodged this complexity earlier, but we have to solve it when we want to update the weights in place.

Since the decompressed masks have the same shapes as the model weights, we can remap the masks the exact same way so they become pointwise aligned with the remapped model weights again. This adds another layer of complexity to the tiling and address mapping between model layers and decompressed masks, but all of this metadata is prepared on the CPU. The Triton kernel for XOR application only needs to read the prepared plan.

The remaining trade-offs#

We already have an extremely strong baseline: for the same proxy payload, trainer-side preparation and compression take 0.834 s, and receiver-side work takes 6.041 s. However, two critical flaws remain:

1. Payload size. On the same proxy workload, the nvCOMP LZ4 payload size is 5.36 GB, while the CPU Zstd baseline is only 0.876 GB, so LZ4 transfers 6.1x as many bytes. GPU-friendly compression algorithms do not necessarily optimize for a high compression ratio. According to Table 1, there is no single compression algorithm that combines a high compression ratio, fast decompression, and DE support.
2. GPU buffer size. GPU decompression also means GPU input and output buffers. To fit the large decompressed masks in HBM, we must evict the KV cache. First, this complicates memory lifetime management. More importantly, evicting the KV cache effectively makes overlapping rollout decoding with decompression impossible, since the KV cache is gone.

From accelerator to teleporter#

We now show how to turn GPUs from accelerators into teleporters, reducing the rollout pause for weight sync to under 0.23 s.

Double compression with heterogeneous decompression#

Compressed payloads can still contain repetition, especially with GPU-friendly algorithms that do not necessarily optimize for compression ratio. We therefore consider adding a round of Zstd compression on top of GPU-friendly codecs such as LZ4 or Snappy:

Table 2. Payload sizes before and after outer Zstd compression.

Empirically, Snappy + Zstd produces a meaningful reduction in payload size compared with the LZ4 baseline. Adding Zstd to LZ4, however, provides a smaller improvement. One possible explanation is that Snappy’s simpler copy commands are easier for Zstd to compress, even though Snappy is less efficient than LZ4 at encoding long runs of zeros in a single pass.

Double (de)compression unlocks receiver-side design opportunities beyond reducing payload size. Since the outer Zstd stage operates on small, already compressed payloads, the CPU can handle this decompression efficiently without using GPU compute or HBM. Rollout decoding can continue normally while the CPU receives the delta update payload, prepares metadata, performs the outer decompression, and stages the resulting Snappy payload in pinned CPU RAM buffers for the GPUs to consume later.

DE can directly consume pinned CPU RAM and output to GPU HBM^{5CUDA Driver API cuMemCreate, using CU_MEM_LOCATION_TYPE_HOST_NUMA and CU_MEM_CREATE_USAGE_HW_DECOMPRESS in allocFlags.usage. Ordinary pinned allocations are not sufficient; see NVIDIA’s DE memory requirements.}, which fits seamlessly into this CPU-GPU decompression pipeline. This also eliminates the need for input staging buffers in HBM, slightly reducing GPU memory pressure. We also found empirically that this approach achieves higher host-device transfer bandwidth than our own ring-buffer implementation.

Overlapping DE and SM#

To fully saturate DE, we use one final feature. Since DE is a fixed-function hardware block for decompression, its work can overlap with GPU compute on the streaming multiprocessors (SMs). For our use case, this means overlapping DE decompression with the Triton bitwise XOR kernel on the SMs. We designed a double-buffered software pipeline in which DE decompresses into one slot while the SMs read from the other. We stream all model weights through these buffers, layer by layer:

As the Nsight Systems profiling trace above shows, the DE pipeline is fully saturated, and Triton kernels are completely hidden by the DE setup-decompression loop, bringing the pause below 0.23 s.

This double-buffered streaming design also eliminates the need for a decompression output buffer for the full model. The two buffers only need to hold two model layers at a time, so they fit within the GPU memory outside the KV cache, typically reserved for activations during inference. No KV cache eviction or changes to the inference engine’s GPU memory pool are needed.

Assembling the weight teleporter#

Now it is time to assemble the complete GPU weight teleporter in our RL stack:

Since Snappy is a more GPU-friendly compression algorithm, the combined trainer-side preparation and compression time falls from 0.834 s with LZ4 to 0.789 s, even with an extra round of Zstd compression. The rollout engines take 2.355 s to pull the payload, prepare metadata, decompress the outer Zstd layer, and stage the Snappy payload. Rollout inference can continue normally because preparation uses neither GPU compute nor GPU memory. Only DE-SM streaming blocks inference, for just 219.6 ms.

Beyond the steady-state RL loop#

So far, we have mainly focused on steady-state trainer-rollout weight updates, the core RL hot path. However, this approach also provides general-purpose infrastructure for other use cases:

Eval, fault tolerance, and resuming from trained checkpoints#

Unlike steady-state trainer-rollout weight updates, where the version always increments by one, several use cases in the RL loop require updating from different weight versions:

Table 3. Base versions for evaluation, recovery, and resumed training.

These use cases require maintaining deltas against at least two additional versions: the last eval version and the rollout/eval engines’ base version. Due to our fully distributed ownership design on the trainer side, the weights are sharded and distributed across the entire training engine. Each CPU only needs to hold the extra versions of its assigned shards, without adding significant memory overhead. Also, the processing for these auxiliary deltas can be easily overlapped with the payload transmission or preparation phase on the trainer-rollout hot path, without lengthening the rollout pause, which remains below 0.23 s.

Another important detail about resuming from a trained checkpoint is that the resumed checkpoint may have advanced many steps beyond the base version and accumulated a decent amount of delta. How do we make sure the rollout engines still have enough memory buffers for staging the larger delta? The LZ4 + Zstd codec we previously did not adopt becomes a good fit here. It has a larger final payload size, but the startup transfer only happens once and can overlap with other startup work. The smaller inner payload size helps to keep the staging buffer small even with a larger accumulated delta. The trainer-rollout hot path can still use the Snappy + Zstd codec to benefit from the small final payload size.

Inference deployment and beyond#

This infrastructure also greatly simplifies inference serving and deployment. Whenever a new trained checkpoint comes out, we only need to send its tiny compressed delta across data centers instead of a full model checkpoint. The inference engine launches from a base checkpoint and simply makes a single call to /update_weights_from_gpu_delta to pull and apply the latest delta. To release resources that are no longer needed in a pure inference use case, we have implemented an opt-in release_state argument that will automatically clear any staging buffers, GPU kernels, and CPU threads related to delta updates after the delta is applied.

We also treat this infrastructure as a superior alternative to LoRA adapters to some extent. Our approach compresses full-parameter deltas into tiny final payloads, while LoRA keeps the payload small by limiting the ranks of the updates. What’s more, our approach can swap different deltas on and off during serving with just one API call and a pause of less than 0.23 s, completely imperceptible to end users.

Speeding up research iterations#

Together, these capabilities let us iterate on research experiments much more rapidly. During each training checkpoint export step, we always export the auxiliary compressed delta against the base version and save it to disk. We can immediately move it across data centers and conduct experiments without waiting for checkpoint conversion or inter-cluster checkpoint transfer. The inference endpoints can immediately pull the refreshed delta on the fly with just one API call and do not need to restart.

We have also removed a major source of network traffic both between clusters and within our cluster. As a small bonus, this also helps relieve network congestion and improve stability.

Open source#

We have open-sourced the implementation across the stack:

• Orchestration, distributed GPU compression, fault tolerance, and delta checkpointing in Miles^{6Miles GPU-delta publication and receiver coordination.}.
• The heterogeneous decompression pipeline with full DE-SM overlap, optimized Triton kernels, and the one-shot /update_weights_from_gpu_delta API in SGLang^{7SGLang GPU-delta decompression and in-place weight updates.}.

This effort uses nvCOMP heavily. We thank the NVIDIA nvCOMP team for their high-performance implementations of compression and decompression algorithms.

Conclusion#

We do not foresee further work on RL weight sync because we believe this is the optimal solution for existing hardware. The rollout pause time is so short, the CPU/GPU memory footprint is so small, and the support for fault tolerance is so complete that even the best intra-cluster P2P RDMA implementation becomes obsolete. We have used one of the fastest GPU-native codecs, fully saturated the DE pipeline, and overlapped SM work; the bottleneck for weight sync has become hardware throughput.

Citation#

Please cite this work as:

Li, Ziang and humans& ai, "GPU-accelerated Weight Teleportation",
humans& ai Blog, 2026.

Or use the BibTeX citation:

@article{li2026gpuweightteleportation,
  author = {Ziang Li and humans\& ai},
  title = {GPU-accelerated Weight Teleportation},
  journal = {humans\& ai Blog},
  year = {2026},
  note = {https://humansand.ai/blog/gpu-accelerated-weight-teleportation}
}