Evals·Across 2,000 steps of FineWeb-Edu pretraining (~37% to ~47% of the 10B-token run), LAMBADA perplexity dropped monotonically from 58.1 to 49.3 while PIQA rose to 65.9%.
AIEdx is building open small language models from scratch.
BubblSpace is our open research effort at AIEdx. We pretrain sparse mixture-of-experts models from first principles and publish every artifact—weights, intermediate checkpoints, data pipelines, live training runs, and research notes—back to the community.
BubblSpace MoE-0.9B-A0.2B
915M total · 211M active · 10B tokensOur current open run is BubblSpace MoE-0.9B-A0.2B, a Qwen3-Next-inspired dropless top-8 SwiGLU mixture-of-experts base model (64 experts, 16 layers, GQA 16/4, QK-norm, RoPE, ZeroCenteredRMSNorm, tied embeddings) trained with Muon + AdamW in bf16 on 10B tokens of FineWeb-Edu on a single H100.
Mid-training checkpoints (lm-evaluation-harness, base model · 19,073 planned steps)
| Benchmark | Metric | step_7000 | step_8000 | step_9000 |
|---|---|---|---|---|
| PIQA | acc_norm | 64.7% | 64.5% | 65.9% |
| ARC-Easy | acc_norm | 51.1% | 50.7% | 50.8% |
| HellaSwag | acc_norm | 37.0% | 38.0% | 38.5% |
| LAMBADA | perplexity | 58.1 | 50.3 | 49.3 |
Open means we share everything
- All model weights and intermediate checkpoints on HuggingFace
- Full model architecture, training loop, tokenizer, and FineWeb-Edu data-prep code on GitHub
- Public live training curves and diagnostics on Weights & Biases
- Provisional accounts of experiments, ablations, and evals in Research Notes
Recent Research Notes
All notes →Modeling·Architecture and optimizer setup for BubblSpace MoE-0.9B-A0.2B: Qwen3-Next-inspired GQA with QK-norm, output gating, ZeroCenteredRMSNorm, and hybrid Muon + AdamW in bf16.
Engineering·Startup backend probing lets the same MoE implementation use fused grouped GEMMs on H100 builds while falling back cleanly on consumer GPUs and CPU.
Team
Founder & AI Researcher
First-principles builder and researcher behind AIEdx and the BubblSpace MoE pretraining runs.

Co-Founder & Strategy
Co-founder supporting strategy, operations, and community growth at AIEdx.
BubblSpace is developed at AIEdx. To collaborate, discuss an experiment, or follow up on a checkpoint, reach out via Contact Us or email contact@aiedx.com.