Mid-training eval trajectory for BubblSpace MoE-0.9B-A0.2B (steps 7,000–9,000)
Evals·Across 2,000 steps of FineWeb-Edu pretraining (~37% to ~47% of the 10B-token run), LAMBADA perplexity dropped monotonically from 58.1 to 49.3 while PIQA rose to 65.9%.