BubblSpace MoE-0.9B-A0.2B

Building a small language model from scratch, in the open.

BubblSpace is AIEdx’s open research project. We built a 915M-parameter mixture-of-experts model, pretrained its English base on 10B tokens, and are now in Phase 2, testing what it can learn about code and tool use while retaining its English capabilities.

The training journey

  1. Model and training stack

    Built the model

    We implemented the architecture, tokenizer, and training pipeline from scratch. The model routes each token through 8 of 64 experts, using roughly 211M of its 915M parameters per token.

  2. Phase 1 · Complete

    Trained the English base

    We pretrained on 10B FineWeb-Edu tokens on one H100 and released the final checkpoint at step 19,073. The model could write fluent English but still struggled to associate facts and attributes with the right entities.

  3. Phase 2 · Continual pretraining

    Continued into code and tool use

    We continued from the Phase 1 weights and optimizer state. The planned 12B-token mix adds code, tool-use traces, and paired examples targeting entity binding, alongside English and STEM text. The final 2B tokens shift toward tool use and binding as the learning rate falls.

What Phase 2 is showing

Early checkpoints show gains in code output prediction and tool calls. The English benchmark average fell, and the entity-binding probes have yet to show a clear gain.

Evaluation snapshot through Phase 2 step 7,000 of 22,888 planned
EvaluationPhase 1 finalPhase 2
Code output prediction (CRUXEval-O pass@1)0.0%8.5%Step 5,000
Valid call with the expected tool name (23 held-out episodes)0%48%Step 7,000
English benchmark average (6 tasks)47.1845.40Step 5,000
See the full benchmark results

How we train and evaluate

Profiling showed that the H100 was spending time waiting for data. Loading batches in the background raised smoke-test throughput from roughly 69–71k to 85.5k tokens per second, moving the projected Phase 2 run time from about 48 to 39 hours.

We made evaluation deterministic so repeated runs give the same scores, and added per-layer routing inspection to see which experts handle different kinds of input as training progresses.

Recent Research Notes

All notes

Team

Amardeep Singh Sidhu

Founder & AI Researcher

First-principles builder and researcher behind AIEdx and the BubblSpace MoE pretraining runs.

Private
PrivateFablePrime

Co-Founder & Strategy

Co-founder supporting strategy, operations, and community growth at AIEdx.

Join BubblSpace on Discord

Running a checkpoint? Bring a prompt, an eval result, or a failure case. You can also follow along as Phase 2 progresses.

Join BubblSpace Discord

For direct collaboration, contact AIEdx.