Lanzado el Kernel Prime Mega MoE, 3,11x más rápido

Prime Intellect has released Prime Mega MoE, a new training extension featuring a trainable fused MoE kernel that delivers a 3.11x faster forward pass. Building upon DeepSeek's open-source DeepGEMM library, which previously only supported inference for its fused Mega MoE kernel, Prime Intellect added a fast, overlapping backward pass to make the technology fully usable for training mixture-of-experts models.

In recent tests using NVIDIA H200 nodes, the implementation achieved full training steps on Qwen3 MoE models that run up to 34% faster than the prime-rl baseline and up to 55% faster than TorchTitan. The performance gains stem from overlapping communication and computation while utilizing GPU-only scratch buffers, significantly reducing idle GPU time during model training. Developers can now utilize the new kernel by setting the dispatch backend to mega_moe within the prime-rl trainer configuration.

Why it matters: This development cuts GPU hours and boosts training speed for mixture-of-experts models by making fused MoE kernels fully trainable.