12 November 2026·Engineering
Auto-probing torch._grouped_mm with a bit-compatible gather-loop fallback
Startup backend probing lets the same MoE implementation use fused grouped GEMMs on H100 builds while falling back cleanly on consumer GPUs and CPU.
Dropless MoE layers are bottlenecked by expert dispatch if implemented naively, yet hard-coding PyTorch's experimental torch._grouped_mm kernel breaks inference on older GPUs, PyTorch builds without grouped GEMM support, and CPU debugging environments.
To keep the release self-contained and reproducible—so anyone can clone the HuggingFace checkpoint and run generate.py or the Colab notebook without custom CUDA extensions—we added moe.backend: auto in the model config.
- At startup, the model probes whether torch._grouped_mm executes cleanly on the active device and dtype.
- On H100 training and supported CUDA builds, it selects the fused grouped_mm path under torch.compile.
- On unsupported GPUs or CPU, it transparently selects the portable gather-loop fallback, producing matching outputs from the same model.safetensors state_dict.