18 November 2026·Modeling

Training a 64-expert top-8 SwiGLU MoE on a single H100 with Muon + AdamW

Architecture and optimizer setup for BubblSpace MoE-0.9B-A0.2B: Qwen3-Next-inspired GQA with QK-norm, output gating, ZeroCenteredRMSNorm, and hybrid Muon + AdamW in bf16.

BubblSpace MoE-0.9B-A0.2B tests how far a clean, modern sparse architecture can be trained from scratch on a single H100 80GB GPU. The model has 915,573,760 total parameters with ~211M active per token across 16 transformer layers and 64 fine-grained experts (top-8 routing, expert hidden dimension 256).

Key architectural choices

  • Attention: GQA with 16 query heads, 4 KV heads (head_dim 64), QK-norm, RoPE (theta 10,000), and attention output gating.
  • Normalization & embeddings: ZeroCenteredRMSNorm (eps 1e-6) and tied input/output embeddings over a 49,280-token vocabulary (SmolLM2-360M base + BubblSpace special tokens padded to a multiple of 128).
  • Routing stability: Dropless top-8 SwiGLU MoE with auxiliary load-balancing loss (0.01) and router z-loss (0.001), plus residual projection scaling std / sqrt(2 * n_layers).

Core training config (base.yaml)

yaml
optimizer:
  muon:
    lr: 6.0e-4
    momentum: 0.95
    nesterov: true
    ns_steps: 5
    weight_decay: 0.1
  adamw:
    lr: 6.0e-4
    betas: [0.9, 0.95]
    eps: 1.0e-8
    weight_decay: 0.1
  grad_clip: 1.0

schedule:
  warmup_steps: 2000
  decay: cosine
  min_lr_ratio: 0.1

training:
  dataset: HuggingFaceFW/fineweb-edu
  dataset_config: sample-10BT
  total_tokens: 10_000_000_000
  seq_len: 2048
  global_batch_tokens: 524288
  micro_batch: 16
  precision: bf16
  compile: true