Phase 2 design

Designing the next training run

Three revisions to the data mix turned the weaknesses of the English base into a plan for learning code, tool calls, and entity relationships.

Planned share by token group

Main run · 10B

Anneal · 2B

  • English and STEM41.5% to 34%
  • Code49% to 38%
  • Tool use8% to 22%
  • Entity binding1.5% to 6%

Where Phase 1 left us

Phase 1 finished at step_19073: 10B FineWeb-Edu tokens, a coherent English base model, with PIQA 68.2, ARC-Easy 56.1, HellaSwag 42.9 and LAMBADA 34.8. Phase 1 also exposed two problems that shaped every Phase 2 decision.

  1. The model had never seen code, a tool call or a reasoning trace. The 12 special tokens (FIM, <tools>, <tool_call>, <think>, …) were in the vocabulary but untrained. Worse, they were actively suppressed: on step_9000 their embedding norms measured ~1.10, against ~1.28 for normal vocabulary. Nineteen thousand steps of softmax pressure and weight decay had pushed them down with nothing pulling them up.
  2. Entity-attribute binding was weak. The model activated the right neighbourhood but bound the wrong edge. "The capital of the United States is" → "Washington, DC." followed by "This city is in California". WinoGrande stayed at chance (51%) all run. Our own 100-case minimal-pair probe scored 25/100, with three categories (comparative value, emotional attribution, possession/obligation) at a flat, confound-free zero.

You rejected "small model capacity" as an explanation: if a model can't bind entities at this scale, scale alone won't fix it. So Phase 2 had two jobs: teach code and agentic formats, and attack binding directly. All without losing the English base.

Designing the data mix

The mix went through three versions. Each was corrected by evidence, not taste.

Version 1 — the first draft (rejected)

71.5% English, about 22% code and 6.6% agentic, on a 5B budget. The audit (issue-01-phase2.md) showed this would leave the finished model at >90% English overall, and would spread 1.1B code tokens across 13 languages (85M each). Continual-pretraining practice puts replay of the old distribution at roughly 20–30%, not 70%. The audit also found the planned code datasets unusable: the-stack-v2-dedup and python-edu ship only Software Heritage blob IDs, not file text.

Version 2 — the audited plan

12B tokens: 26% English replay, 63% code and 11% agentic (plan-02). Code was narrowed to 5 languages (Python, TypeScript, JavaScript, Go, SQL/shell/config) so each gets 4–6× more signal, sourced from Avelina/python-edu-cleaned (hydrated text) and bigcode/starcoderdata.

The budget tier was an explicit cost decision. 12B is enough for each of 64 experts to see roughly 1B code tokens under top-8 routing, without a multi-day commitment. It works out to about 39 hours on one H100.

The agentic pool came in far smaller than estimated. Episodes are hard-filtered to ≤2048 tokens (dropped, never truncated), and OpenThoughts-114k kept only 8,575 of 114,000 traces because full chains of thought average 8k tokens. Two sources were added to close the gap, glaive-function-calling-v2 (58.9M tokens, 0.5% dropped) and Bespoke-Stratos-17k (5.5M), for a total of ~375M unique tokens. Rather than repeat that 6×, the agentic target was cut to 1.3B (3.5× repetition) and the difference moved to code.

Security checks changed the plan twice:

  • OpenCoder-LLM/opc-annealing-corpus was dropped entirely. Hugging Face's ClamAV scan flagged 59 of 85 sampled files in exactly the configs we'd selected.
  • starcoderdata's 4 flagged files were confirmed to sit outside our 5 languages.

Version 3 — rebalanced for binding (what's running)

The binding diagnosis changed the English pool. Version 2's English was half reused FineWeb-Edu, the same noisy web distribution that had produced the problem. Its anneal also dropped English to 16% with zero encyclopedic or STEM text. The pre-Phase 2 review (review-01-pre-phase2-semantic-diagnosis.md) rebuilt the English group around dense, contradiction-free sources:

English pool Unique tokens Within-group weight Why
Wikipedia 20231101.en 1.5B 1.5 clean entity–attribute facts (capitals, history, geography)
Cosmopedia-v2 1.52B 1.5 textbook-style explanations
FineMath-4plus 1.0B 1.0 mathematical and STEM prose, multi-step reasoning
FineWeb-Edu replay (Phase 1 shards) 10B available 0.8 anti-forgetting anchor, deliberately small

Then a fourth group was added: contrastive binding. scripts/gen_contrastive.py writes synthetic minimal-pair documents, the same fact stated with the entities swapped and the attribute following the right entity. It covers five relation families aimed at the probe's zero categories, written in six document styles. Two guards keep it from inflating the probes:

  • Any document that pairs two entities appearing together in a probe context is removed.
  • Every probe name is removed (Carlos, Emma, Grace, Priya and Wei were dropped).

The first test run was only 78% unique: the physical-fit family has only ~80 possible documents. So the generator now writes only unique documents (61.9M tokens), and repetition is controlled in one place, the sampling weight.

Final group weights (configs/phase2.yaml):

Group Stage 2A (10B tokens) Stage 2B (2B tokens)
English + knowledge + STEM 41.5% 34% (FineWeb-Edu boost 0)
Code 49% 38% (python-edu boost ×3)
Agentic 8% 22%
Contrastive 1.5% (~150M drawn) 6% (~120M drawn, ~4.4× total)

Two ideas from the review were deliberately left out:

  • Shared-expert grafting was rejected. As you pointed out, all 64 routed experts already share the residual stream and every attention layer, so there's no structural gap for a shared expert to fill.
  • A multi-token-prediction head and SmolLM2-1.7B logit distillation were kept off, as a decision for mid-run. The MTP evidence is modest and comes from much larger models. Distillation would cut throughput by 35–45%.

How the schedule is built for continual learning

Phase 2 isn't a fresh run. It continues a model that already went through a full warmup and cosine decay. The schedule exists to let new skills in without washing old ones out. That takes four mechanisms working together.

1. Start from Phase 1 with its optimizer state.

  • --init_from step_19073.pt loads the weights and the Muon momentum and AdamW moment buffers, then starts Phase 2's own step counter at 0.
  • It's separate from --resume, which would have exited immediately because Phase 1's loop was already at total_steps.
  • Carrying the optimizer state avoids the shock of an optimizer that has forgotten the loss surface it's on.

2. Re-warm, don't restart.

  • Learning rate goes from 6e-5 to 2.4e-4 over 300 steps. 6e-5 is exactly where Phase 1 ended (min_lr_ratio 0.1 × 6e-4), so there's no discontinuity.
  • The 2.4e-4 peak is 0.4× Phase 1's peak. It's large enough for Muon to re-specialize experts toward code, and for AdamW to pull the suppressed special-token embeddings back into use. It's small enough not to overwrite what Phase 1 learned.

3. A long stable plateau with replay (Stage 2A, steps 0–19,073, 10B tokens).

  • The learning rate holds at 2.4e-4.
  • Every batch is a blend of English, code, agentic and contrastive data, never 100% of anything. Every optimizer step still carries English gradient, which is the standard defence against catastrophic forgetting.
  • In an MoE it also protects the router: experts that specialized on English keep receiving English tokens instead of being reassigned wholesale to code syntax.

4. A special annealing phase (Stage 2B, steps 19,073–22,888, 2B tokens).

  • The learning rate decays linearly 2.4e-4 → 6e-6, and the data shifts at the same moment toward the highest-value material:
    • agentic 8% → 22%
    • contrastive 1.5% → 6%
    • code reweighted ×3 toward curated python-edu-cleaned
    • English kept at 34%, but only Wikipedia, Cosmopedia and FineMath; raw FineWeb-Edu is switched off
  • The reasoning comes from Warmup-Stable-Decay training (used by SmolLM2 and others): data seen during the low-learning-rate tail has an outsized effect on the final checkpoint. Updates there are small and precise, so concentrated specialized data imprints its patterns without the large updates that cause forgetting at high learning rate.
  • This is why the anneal is a separate phase rather than just the end of a curve: the plateau builds broad competence on a mixed stream, and the anneal consolidates the skills we care about most at the point where the model is most receptive and least prone to drift.

Splitting by token count: 10B in the plateau and 2B in the anneal (~17%), in line with the 10–20% decay fraction common in WSD schedules. Stage 2A and 2B are the same run with one config. The sampler reads the current step and switches mixtures at step 19,073 by itself.

What files changed

Everything is opt-in. A config without a phase2: block runs the original Phase 1 code path unchanged, and the Phase 1 tests still pass.

File Change
src/model/attention.py build_doc_block_mask(idx, eot_id): each token's document id comes from <|endoftext|> positions, and a causal + same-document BlockMask is compiled into flex_attention. Later, a KV-cache path for fast generation in evals (prefill with causal SDPA, then one token at a time against cached keys and values). With neither, the original fused causal kernel runs.
src/model/block.py, src/model/transformer.py Pass block_mask / layer_cache through; KVCache + new_kv_cache(); RoPE offset by cached length.
src/data/dataset.py MultiPoolTokenDataset: 16 named pools, stage-dependent group weights, per-pool weights and stage boost, held-out validation tail per pool. Flat pools sample random windows. Agentic pools are block-aligned (every sample starts at an episode), with pad targets set to −100. BatchPrefetcher: a background thread that produces batches in exactly the training loop's order (the profiling fix, the preflight chapter).
src/data/fim.py apply_window_fim: FIM on one document span inside the sampled 2048-token window, 50/50 PSM/SPM. Phase 1's file-level FIM split the three sentinels across windows for any file over 2048 tokens.
src/optim/schedule.py build_phase2_scheduler: re-warm, plateau, linear anneal.
scripts/train.py --init_from; Phase 2 branch with multi-pool data, prefetcher, doc mask per micro-batch, ignore_index=-100, per-pool validation every 250 steps, W&B global_step = step + 19,073.
configs/phase2.yaml, configs/phase2_smoke.yaml Full run and 100-step smoke variant.
scripts/gen_contrastive.py Contrastive binding corpus with contamination guard.
tests/test_phase2.py Doc-mask isolation (perturbing document A leaves document B's logits bit-identical), FIM invariants, block alignment and pad masking, stage mixture, schedule boundaries, prefetcher order equivalence, KV cache = full forward. 17 tests pass.

The preflight chapter covers the two sampler bugs and the smoke tests that followed.