Phase 1

The 10B-token English base

We turned FineWeb-Edu into 100 shards, trained a base model on one H100, and released the checkpoint. Then we tested what its fluent completions actually meant.

Phase 1 completion excerpt
“Washington, DC.” … “This city is in California.”

Released checkpoint · step_19073

Starting point

By the end of storyboard 01 we had:

  • The architecture: a Qwen3-Next-inspired MoE, ~915.6M total / ~211M active parameters.
    • 16 layers, d_model 1024.
    • GQA 16/4 with QK-norm and an attention output gate.
    • 64 experts, top-8, SwiGLU, expert_hidden 256, dropless.
    • ZeroCenteredRMSNorm.
    • Tied embeddings over a 49,280-row vocabulary: SmolLM2's tokenizer plus 12 BubblSpace special tokens.
  • The optimizer: Muon on every hidden matrix (batched Newton-Schulz, expert stacks included), AdamW on embeddings, routers and norms.
  • The data: a debug run that passed every gate. Loss fell, load balancing stayed near 1.0, a checkpoint saved and resumed cleanly, and the torch._grouped_mm fast MoE kernel ran after the bf16 cast fix.

What we didn't have yet was a real dataset, a measured throughput, or a decision on what the first real run should actually be.

Infrastructure detour: three instances in one afternoon

You found a 2× H100 instance (the second instance) and wanted to move. Before deleting the training machine I checked what lived where:

  • Checkpoints and data were on the persistent filesystem the training filesystem, not the instance disk, so deleting the training machine lost nothing.
  • The catch: the second instance was in a different Lambda region (another region) with a different filesystem. Lambda filesystems only mount in their own region, so the second instance could never reach the training filesystem.
  • Nothing there was worth migrating anyway (debug checkpoints, 150M debug tokens re-made in ~4 minutes).

That opened the question of one GPU or two. Our code was single-GPU. Two GPUs would mean DDP or expert parallelism: real engineering, for speed rather than capability at this model size. You terminated the second instance and relaunched a single H100 in the original region. The replacement machine mounted the same filesystem, so we rebuilt the environment and reused the data.

Deciding what Phase 1 is

Three decisions, each settled with numbers.

Model size: stay at 0.9B-A0.2B

You pointed out an 80GB GPU was mostly idle and asked why not train a bigger model. My first answer quoted ~17GB of memory use, but that was the 45M-parameter debug model, and I corrected it. The real comparison was a menu of options:

Option Model Notes
1A 915M / 211M active, 1 GPU built and debug-validated, zero new engineering
1B ~3B / ~700M active, 1 GPU new shapes, re-derived Muon invariants, new debug run
2A–2C 2 GPUs DDP or expert parallelism

The deciding factor was tokens, not memory. Compute-optimal training wants roughly 20 tokens per active parameter for an MoE (active parameters drive FLOPs). 10B tokens over 211M active is ~47 tokens per active parameter, already well past the ~4.2B minimum. A 3B model would want ~14B+ tokens and proportionally more wall-clock time. You chose 1A.

Step count isn't a quality knob

You asked whether ~19,000 steps was too few. Steps are simply 10B tokens ÷ 524,288 tokens/step = 19,073. Shrinking the batch would give more steps over the same tokens: noisier, worse for MoE load balancing, and no extra learning. What matters is total tokens seen.

English first, then specialize

The most important exchange of the phase. You asked, frustrated, what a working model after 19k steps would actually be. The honest answer:

  • It would be an English base model: coherent text, general knowledge, roughly GPT-2-small/medium class.
  • It would have zero code, tool-use or reasoning ability. Not "bad at it": not at all. The special tokens existed, but no training example used them.
  • It also wouldn't chat, because there's no instruction tuning.

We agreed on the standard shape: Phase 1 = English base, proving the whole pipeline at scale, then Phase 2 = continual training on a mixed English + code + agentic stream, never a hard switch, to avoid catastrophic forgetting. I flagged then that Phase 1's cosine schedule ends at its floor, so Phase 2 would need its own warmup and decay rather than a plain resume. That turned into --init_from and the WSD schedule.

Measuring before committing

My first throughput estimate was a FLOPs-based range of 40k–150k tok/s, which put the run anywhere from 18 hours to 3 days. Too wide to plan around, so I measured instead: 30 steps of the full-size model on the debug data.

  • Steady state: ~86,300 tok/s, very stable (86,374 / 86,312 / 86,261 across steps 10–30).
  • Loss 10.99 → 10.79, grad norm ~14 pre-clip, z-loss ~19, all expected at initialization.
  • ETA: 10B ÷ 86.3k ≈ 32 hours.

Checkpoint sizing: ~7.5GB each (weights plus Muon and AdamW state), 19 of them, ~150GB total, trivial on the 1.7PB filesystem.

Tokenizing 10B tokens

src.data.prepare streamed HuggingFaceFW/fineweb-edu (sample-10BT) and encoded it with the BubblSpace tokenizer, <|endoftext|> after every document, into uint16 shards.

  • It ran at ~550–650k tok/s, about 4 hours on CPU and network.
  • nvidia-smi showing an idle GPU during this worried you. It was expected: tokenization never touches the GPU.
  • Output: 10,000,000,000 tokens in 100 shards plus meta.json on the persistent filesystem, reusable for any rerun. Phase 2 later reuses these same shards as its FineWeb-Edu replay pool.
  • The job ran detached (nohup … & disown), so it didn't depend on the Mac staying awake. You ran caffeinate so the session could launch training the moment tokenization finished.

The run

Launched Thu 19 Nov, 21:51 UTC, detached, logging to W&B run goa8jk6e.

Configuration (configs/base.yaml):

  • Global batch 524,288 tokens (micro-batch 16 × 2048 × 16 accumulation steps), bf16, torch.compile.
  • Muon (lr 6e-4, momentum 0.95, Nesterov, 5 Newton-Schulz steps, weight decay 0.1).
  • AdamW (lr 6e-4, betas 0.9/0.95, weight decay 0.1).
  • Learning rate: 2,000-step warmup, then cosine decay to min_lr_ratio 0.1 (6e-5) exactly at step 19,073.
  • Load-balance coefficient 0.01, router z-loss 0.001, grad clip 1.0.

First minutes: 71.3k tok/s, below the 86.3k measured on 2 debug shards, so I revised the ETA to ~39h. It then climbed to a steady ~85–86k tok/s once file caches warmed. Measured from checkpoint timestamps:

Interval Time s/step tok/s
step 0 → 1,000 1h 50m 6.63 ~79,100 (includes compile warmup)
1,000 → 2,000 1h 43m 6.17 ~85,050
2,000 → 3,000 1h 42m 6.12 ~85,700
3,000 → 3,890 1h 31m 6.11 ~85,800

Total: ~32.7 hours, finishing on the morning of Sat 21 Nov (UTC), within 2% of the measured estimate.

Health check at step ~3,800 (your W&B screenshots):

  • Loss 11.0 → ~3.0 by step 2k, then a slow decline.
  • z-loss ~19 → near zero by step 1k.
  • max attention logit peaked ~7.5 around step 400, then settled ~7 (no logit blow-up).
  • Load-balance loss spiked to ~1.85 near step 100 as the router left random init, then held ~1.0.
  • grad norm had one bump to ~10 around step 450, in the same window as those transients, then decayed smoothly to ~0.2.

All of it is the normal "model finding itself" pattern, not instability.

Annealing in Phase 1: you asked whether Phase 1 had one. It had a learning-rate taper but no separate anneal phase. The cosine decay is continuous from step 2,000 to 19,073, and by step 17,000 the learning rate was already ~87% of the way to its floor. There was no data-mix anneal either, since it's a single dataset. A deliberate, carved-out anneal with a data shift is what Phase 2's Stage 2B adds.

Trying the model early: inference and Hugging Face

Step 4,000. You spun up an A10 instance with its own filesystem, and we ran scripts/generate.py against a copied checkpoint without touching training. Output was fluent but factually loose, which is expected that early and with no instruction tuning.

When you asked how factual recall could improve, I first explained it as a limitation. You pushed back ("no we focus on improving model and it cant be done is not answer"). That framing, look for the mechanism instead of accepting the ceiling, came back in force after Phase 1.

Publishing:

  • Weights were exported to safetensors and pushed to TheFireHacker/bubblspace-moe0.9b-a0.2b, along with the inference code so anyone can run it.
  • You added a Colab notebook, and we fixed its cells. One long wait turned out to be a needless git lfs pull of weights already on HF.
  • The HF weights were replaced as training progressed: step 4,000 → ~10,000 → 18,000 → the final 19,073.
  • With weights public, one of your Colab prompts gave a clean answer: "Albert Einstein … the famous Einstein's theory of relativity".

Evaluation

There was no eval at all until step 7,000. You asked for the best way to start, and we decided to run on a fresh A10 rather than share the training GPU.

Building it. Our Transformer isn't a transformers model, so EleutherAI's lm-evaluation-harness can't load it. src/eval/lm_eval_adapter.py (BubblSpaceLM) answers the one question the harness asks: how likely is this continuation given this context? The harness owns datasets and metrics. Two details matter:

  • The 116 padded vocabulary rows are sliced out before the softmax, exactly as in training.
  • batch_size=1, because the attention had no padding mask.

eval-notes.md walks through every task.

Six tasks (HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande, LAMBADA), all zero-shot, all log-likelihood scored: the model never has to output "A" or "B".

Task 7k 8k 9k 18k 19,073 (final)
PIQA (acc_norm) 64.7 64.5 65.9 68.8 68.2
ARC-Easy (acc_norm) 51.1 50.7 50.8 56.6 56.1
HellaSwag (acc_norm) 37.0 38.0 38.5 42.6 42.9
LAMBADA acc 29.0 31.0 30.6 35.3 34.8
LAMBADA perplexity 58.1 50.3 49.3 31.0 32.0
WinoGrande (acc) 52.4 51.7 51.5 50.7 51.4
ARC-Challenge (acc_norm) 28.0 27.6 27.2 29.9 29.8

Reading it:

  • Steady gains on broad pattern tasks. By step 7,000 the model was already ahead of GPT-2-124M on most tasks, and competitive with fully trained Pythia-160M on ARC, despite 10B total tokens against Pythia's 300B.
  • LAMBADA moved the most. Perplexity fell 45%, from 58 to 32. That's the metric most sensitive to tokens seen.
  • The last 1,000 steps were flat, as expected with the learning rate at its floor.
  • WinoGrande sat at chance the entire run (50.7–52.4%). That one line became the most important result of Phase 1.

The comparison against GPT-2, Pythia and SmolLM2, with their training-token counts, is in eval-notes.md and benchmark/README.md.

Open-sourcing

  • Removing infrastructure details. You made the repo private on finding instance addresses in the README, and every file was scrubbed of IPs and host-specific details before you made it public again. The standing rule since: no IPs or sensitive details in docs.
  • License and disclaimer: Apache 2.0, plus a disclaimer that this is for learning and research only, with AIEDX bearing no liability for outputs.
  • Visibility: the W&B run was made public, and both READMEs link the code, weights, data pipeline and graphs. Everything is open: code, checkpoints and data recipe.

The post-mortem: fluent but loosely bound

Phase 1 ended as planned, so the question became what the model actually understood. You tested step_18000 conversationally and found the semantics "very loose, surface level". Example: asked about the Riemann hypothesis, it talked about cosmology. You ruled out "small model capacity" as an answer: "if we dont fix weak semantic understanding here it wont get magically fixed in trillion param model".

Separating format from understanding

Conversational prompts to a base model mostly measure format hijacking, so we switched to declarative completions:

Prompt Completion Failure
The capital of the United States is "Washington, DC." … "This city is in California" right fact, then loses the binding
The capital of India is "…capital of British India… the British built… the Taj Mahal" right neighbourhood, wrong relation
The Riemann hypothesis is a conjecture about "the geometry of Riemann surfaces" lexical neighbour instead of concept
In Sanskrit, the word Kama means "breath…" nearby concept bound to the wrong word

Diagnosis: the model activates the right semantic neighbourhood but fails at sharp entity–attribute binding. That's exactly what WinoGrande measures, and exactly where it sat at chance.

Building a probe for binding

scripts/probe_entity_binding.py scores WinoGrande-style minimal pairs with the same log-likelihood scorer the harness uses. Each case has two contexts that differ only in which entity is named, and two completions. It counts as correct only if the preference flips with the entity.

Version Cases Score Notes
Original 11 (spatial fit) 6/11 always preferred "too narrow" regardless of entity
v1 100, 10 categories 35/100 confounded: "had"/"was" endings let grammar pick the answer
v2 100, confound fixed 25/100 agent–patient 100% → 40%, possession 60% → 0% after the fix
v3 70, new structures 4/70 added before Phase 2 to detect template memorization

You pushed back on the first probe for being too narrow ("why only big small type"), which led to the 10-category expansion. The v1 confound was caught on review: in two categories, verb agreement alone gave away the answer. v2's category spread is the useful part:

  • spatial height 70%
  • comparative value, emotional attribution and possession/obligation at a flat zero

That gave Phase 2's contrastive corpus a precise target.

Weight-space programming (WSP): a learning experiment

You asked me to try fixing binding directly in the weights, on a copy of the final checkpoint (wsp_ prefix).

  1. Naive gain scaling. Amplify the 16 attention heads most sensitive to the entity swap by 2×. Result: 25 → 20/100. No effect on the zero categories, and collateral damage elsewhere (spatial size 40% → 10%). There was nothing to protect the cases that already worked.
  2. Closed-form ridge edit, derived from activations. You challenged the first attempt: real WSP should look at activations, not pick a multiplier ahead of time. The second version solved for the edit ΔW = R Kpᵀ(λ Kn Knᵀ + Kp Kpᵀ + μI)⁻¹ from real activations, with a protection term over the other cases. Result: 31/100. The three zero categories moved (emotional 0 → 50%, possession 0 → 30%, comparative 0 → 20%), but social attribution fell 40% → 10%.

The honest conclusion came from free generation. In your interactive REPL tests, the edited model still got cases like marathon/sprint wrong when phrased differently. Your verdict: "its not a win if final output is bad". The edit changed which of two fixed answers scored higher without changing what the model generates. WSP was kept as a diagnostic (it showed where binding information can be steered), not a fix.

Tools built along the way

  • scripts/chat_repl.py: a continuous command-line chat on the A10 with /temp, /topp, /tokens, /greedy and /sample. Your temperature question taught a useful distinction: log-likelihood benchmarks use no sampling at all, so temperature can't explain a benchmark failure. In generation, low temperature does make answers noticeably more consistent.
  • scripts/inspect_routing.py: which expert fired per layer, per token. You asked whether we could see inside the model the way interpretability work does. This was the first step, and it led to the per-layer routing analysis used throughout Phase 2.

What Phase 1 handed to Phase 2

Phase 1 finding Phase 2 response
Solid English base (PIQA 68, ARC-E 56, HellaSwag 43) continue from step_19073 with optimizer state (--init_from), keep English in every batch
Special tokens never trained, embeddings suppressed (~1.10 vs ~1.28 norm) re-warm the learning rate to 2.4e-4 so AdamW can pull them into use, plus code (FIM) and agentic data that uses them
WinoGrande at chance; binding probe 25/100 with three zero categories English rebuilt around Wikipedia / Cosmopedia / FineMath; synthetic contrastive minimal pairs aimed at the zero categories; probe v3 to catch memorization
Random 2048-token windows let documents attend across <|endoftext|> causal + same-document attention mask (flex_attention)
Benchmarks scored differently run to run near ties (found later) deterministic evals
Evaluation only at a few checkpoints an eval pipeline on every checkpoint, relayed to the A10

Lessons

  • Measure, don't estimate. The FLOPs estimate spanned 18 hours to 3 days. Thirty real steps gave 32 hours, and the run finished within 2% of it.
  • Steps aren't quality; tokens are.
  • Check regions before moving instances. Lambda filesystems are region-locked.
  • Validate probes before trusting them. The v1 grammar confound inflated two categories by 40–60 points.
  • Forced-choice wins aren't generation wins. Every binding claim now needs both a scored probe and greedy output.
  • "Small model" isn't an explanation. It's the hypothesis you test last.