18 November 2026·Modeling
Training a 64-expert top-8 SwiGLU MoE on a single H100 with Muon + AdamW
Architecture and optimizer setup for BubblSpace MoE-0.9B-A0.2B: Qwen3-Next-inspired GQA with QK-norm, output gating, ZeroCenteredRMSNorm, and hybrid Muon + AdamW in bf16.
BubblSpace MoE-0.9B-A0.2B tests how far a clean, modern sparse architecture can be trained from scratch on a single H100 80GB GPU. The model has 915,573,760 total parameters with ~211M active per token across 16 transformer layers and 64 fine-grained experts (top-8 routing, expert hidden dimension 256).
Key architectural choices
- Attention: GQA with 16 query heads, 4 KV heads (head_dim 64), QK-norm, RoPE (theta 10,000), and attention output gating.
- Normalization & embeddings: ZeroCenteredRMSNorm (eps 1e-6) and tied input/output embeddings over a 49,280-token vocabulary (SmolLM2-360M base + BubblSpace special tokens padded to a multiple of 128).
- Routing stability: Dropless top-8 SwiGLU MoE with auxiliary load-balancing loss (0.01) and router z-loss (0.001), plus residual projection scaling std / sqrt(2 * n_layers).
Core training config (base.yaml)
yaml
optimizer:
muon:
lr: 6.0e-4
momentum: 0.95
nesterov: true
ns_steps: 5
weight_decay: 0.1
adamw:
lr: 6.0e-4
betas: [0.9, 0.95]
eps: 1.0e-8
weight_decay: 0.1
grad_clip: 1.0
schedule:
warmup_steps: 2000
decay: cosine
min_lr_ratio: 0.1
training:
dataset: HuggingFaceFW/fineweb-edu
dataset_config: sample-10BT
total_tokens: 10_000_000_000
seq_len: 2048
global_batch_tokens: 524288
micro_batch: 16
precision: bf16
compile: true