Evals·Across 2,000 steps of FineWeb-Edu pretraining (~37% to ~47% of the 10B-token run), LAMBADA perplexity dropped monotonically from 58.1 to 49.3 while PIQA rose to 65.9%.
BubblSpace MoE-0.9B-A0.2B
Building a small language model from scratch, in the open.
BubblSpace is AIEdx’s open research project. We built a 915M-parameter mixture-of-experts model, pretrained its English base on 10B tokens, and are now in Phase 2, testing what it can learn about code and tool use while retaining its English capabilities.
The training journey
Model and training stack
Built the model
We implemented the architecture, tokenizer, and training pipeline from scratch. The model routes each token through 8 of 64 experts, using roughly 211M of its 915M parameters per token.
Phase 1 · Complete
Trained the English base
We pretrained on 10B FineWeb-Edu tokens on one H100 and released the final checkpoint at step 19,073. The model could write fluent English but still struggled to associate facts and attributes with the right entities.
Phase 2 · Continual pretraining
Continued into code and tool use
We continued from the Phase 1 weights and optimizer state. The planned 12B-token mix adds code, tool-use traces, and paired examples targeting entity binding, alongside English and STEM text. The final 2B tokens shift toward tool use and binding as the learning rate falls.
What Phase 2 is showing
Early checkpoints show gains in code output prediction and tool calls. The English benchmark average fell, and the entity-binding probes have yet to show a clear gain.
| Evaluation | Phase 1 final | Phase 2 |
|---|---|---|
| Code output prediction (CRUXEval-O pass@1) | 0.0% | 8.5%Step 5,000 |
| Valid call with the expected tool name (23 held-out episodes) | 0% | 48%Step 7,000 |
| English benchmark average (6 tasks) | 47.18 | 45.40Step 5,000 |
How we train and evaluate
Profiling showed that the H100 was spending time waiting for data. Loading batches in the background raised smoke-test throughput from roughly 69–71k to 85.5k tokens per second, moving the projected Phase 2 run time from about 48 to 39 hours.
We made evaluation deterministic so repeated runs give the same scores, and added per-layer routing inspection to see which experts handle different kinds of input as training progresses.
Recent Research Notes
All notesModeling·Architecture and optimizer setup for BubblSpace MoE-0.9B-A0.2B: Qwen3-Next-inspired GQA with QK-norm, output gating, ZeroCenteredRMSNorm, and hybrid Muon + AdamW in bf16.
Engineering·Startup backend probing lets the same MoE implementation use fused grouped GEMMs on H100 builds while falling back cleanly on consumer GPUs and CPU.
Team
Founder & AI Researcher
First-principles builder and researcher behind AIEdx and the BubblSpace MoE pretraining runs.

Co-Founder & Strategy
Co-founder supporting strategy, operations, and community growth at AIEdx.
Join BubblSpace on Discord
Running a checkpoint? Bring a prompt, an eval result, or a failure case. You can also follow along as Phase 2 progresses.
Join BubblSpace DiscordFor direct collaboration, contact AIEdx.