Skip to content

bdh.py: add extend_freqs() for phase-preserving latent-width growth - #12

Open
asb-42 wants to merge 1 commit into
pathwaycom:mainfrom
asb-42:rope-prefix-growth
Open

bdh.py: add extend_freqs() for phase-preserving latent-width growth#12
asb-42 wants to merge 1 commit into
pathwaycom:mainfrom
asb-42:rope-prefix-growth

Conversation

@asb-42

@asb-42 asb-42 commented Aug 26, 2026

Copy link
Copy Markdown

Problem

RoPE frequencies are exponent-normalized by the neuron count (theta ** (-i/n) in
get_freqs). Any code that grows the latent width (progressive networks, upsizing
experiments, model editing) and simply recomputes get_freqs(n_new) therefore
**silently rewrites the phases of every existing neuron** and corrupts the trained model —
even though all actual weights are untouched.

We hit this while building a continual-learning growth mechanism on top of \BDH{}: a grown
model evaluated zero steps after width extension already behaved like an untrained model,
and the root cause was exactly this frequency-table renormalization.

Fix

extend_freqs(freqs, n_new) keeps the existing frequency table verbatim and appends
entries for new neurons computed at the new scale. Width growth becomes phase-preserving for
all pre-existing neurons by construction.

Verification

New test_bdh.py covers:

  • extended tables preserve the prefix exactly (assert_close);
  • naive recomputation at the larger count demonstrably changes the prefix (documents the pitfall);
  • Attention builds with an extended table.

All three pass.

Scope

Deliberately minimal: one helper + docstring warning on get_freqs + tests. A full
continual-learning suite (phase-ordered training, growth/routing/consolidation mechanisms)
is being maintained separately in asb-42/bdh and can be
discussed separately if of interest.

RoPE frequencies are exponent-normalized by the neuron count
(theta ** (-i/n)), so recomputing get_freqs(n_new) after growing the
latent width silently rewrites the phases of every existing neuron and
corrupts the model.

extend_freqs() keeps the existing frequency table verbatim and appends
entries for new neurons at the new scale, making width growth
phase-preserving for all pre-existing neurons. Includes tests covering
exact prefix preservation, the naive-recompute pitfall, and Attention
construction with an extended table.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant