Repository navigation
Segment a model with memory at rest points, rebuilding its memory - #223
Open
yichao-liang wants to merge 1 commit into
Open
yichao-liang wants to merge 1 commit into
yichao-liang wants to merge 1 commit into
Conversation
A model with hidden state skipped rest-point segmentation ("a rest point
does not reset a hidden process"), so every candidate parameter point
replayed each whole recording from its start: on the from-assets round
fixes_r4, fits of such models took 15 to 56 minutes, and a recording too
chaotic as a whole could not be trimmed phase by phase.
Its recordings are now truncated and segmented like any other model's.
A segment that starts mid-recording carries the actions and observations
before it in place of memory (model_state.memory_from_history), and
restoring that state runs the model's own update over them under the
replay's parameters: the memory each candidate implies, computed as the
execution-time tracker computes it from the same observations, with no
physics. On the fixes_r4 level-1 recordings this cuts Balloons seed 0
from 114 steps to one 53-step segment and Boil seed 0 to 2 segments; a
model that scores the moving robot (Bridge seed 0) finds no rest point
and still replays its recording whole.
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MODEL_STATE_INIT) skipped settled-tail truncation and rest-point segmentation ("a rest point does not reset a hidden process"), so every candidate parameter point replayed each whole recording from its start.Its recordings are now truncated and segmented like any other model's.
model_state.memory_from_history).Restoring that state (
restored_model_state, reached from_set_stateafter the replay pins its parameters) runs the model's ownupdate_model_stateover them under the replay's parameters.Each candidate point gets the memory it implies, computed as the execution-time tracker computes it from the same observations, with no physics.
Why
On the from-assets round fixes_r4, fits of hidden-state models took 15 to 56 minutes (Balloons seed 0, Boil seed 0, Bridge seed 0), and the fit gate makes the agent wait for each one after every model edit.
A recording too chaotic as a whole also could not be trimmed phase by phase: Bridge seed 0's whole 1690-step recording scored RMS 0.27 against the 0.1 trimming bar, so its first fits refused to run at all.
Scope
This changes the fit of every hidden-state model, in every arm that fits one (EMPIRIC included), not only EMPIRIC from assets.
How much it saves depends on the scored features: a model that scores the robot, which moves almost every step, finds few rest points.
On the fixes_r4 level-1 recordings:
Test plan
test_subclass_model_state.py: a state carrying a recorded history restores the memory that history implies under the env's current parameters, through_set_stateandsimulate; segments of a two-phase recording carry exactly the history before them, leave the recorded states untouched, and rebuild memory per candidate rate.simulator.py, both fits on one node): fit 450 s before, 221 s after; SSE 8.28 to 1.42 before, 10.66 to 1.25 on 2 segments after; the beliefs agree (for examplelift_H68% [1.05, 1.10] andburst_z[1.11, 1.18] in both).🤖 Generated with Claude Code