Skip to content

Segment a model with memory at rest points, rebuilding its memory - #223

Open
yichao-liang wants to merge 1 commit into
belief-coherent-misfitfrom
fit-hidden-state-segments
Open

yichao-liang wants to merge 1 commit into
belief-coherent-misfitfrom
fit-hidden-state-segments

Conversation

@yichao-liang

Copy link
Copy Markdown
Collaborator

Summary

  • A model with hidden state (MODEL_STATE_INIT) skipped settled-tail truncation and rest-point segmentation ("a rest point does not reset a hidden process"), so every candidate parameter point replayed each whole recording from its start.
    Its recordings are now truncated and segmented like any other model's.
  • A segment that starts mid-recording carries the actions and observations before it in place of memory (model_state.memory_from_history).
    Restoring that state (restored_model_state, reached from _set_state after the replay pins its parameters) runs the model's own update_model_state over them under the replay's parameters.
    Each candidate point gets the memory it implies, computed as the execution-time tracker computes it from the same observations, with no physics.
  • The physical state is re-anchored on the observation at each segment start, as for any segment.

Why

On the from-assets round fixes_r4, fits of hidden-state models took 15 to 56 minutes (Balloons seed 0, Boil seed 0, Bridge seed 0), and the fit gate makes the agent wait for each one after every model edit.
A recording too chaotic as a whole also could not be trimmed phase by phase: Bridge seed 0's whole 1690-step recording scored RMS 0.27 against the 0.1 trimming bar, so its first fits refused to run at all.

Scope

This changes the fit of every hidden-state model, in every arm that fits one (EMPIRIC included), not only EMPIRIC from assets.
How much it saves depends on the scored features: a model that scores the robot, which moves almost every step, finds few rest points.
On the fixes_r4 level-1 recordings:

recording before after
Balloons seed 0 (box, balloons) 114 steps 1 segment, 53 steps
Boil seed 0 (robot, jug, faucet) 352 steps 2 segments, 317 + 42
Bridge seed 0 (block z, robot xyz) 1690 steps 1690 steps, no rest point
Bridge seed 0, block z only 1690 steps 9 segments, 917 steps

Test plan

  • test_subclass_model_state.py: a state carrying a recorded history restores the memory that history implies under the env's current parameters, through _set_state and simulate; segments of a two-phase recording carry exactly the history before them, leave the recorded states untouched, and rebuild memory per candidate rate.
  • End to end on Balloons seed 0's fixes_r4 level 1 (scripted harness, the run's own simulator.py, both fits on one node): fit 450 s before, 221 s after; SSE 8.28 to 1.42 before, 10.66 to 1.25 on 2 segments after; the beliefs agree (for example lift_H 68% [1.05, 1.10] and burst_z [1.11, 1.18] in both).
  • Local CI replay: yapf, isort 5.10.1, docformatter, mypy and pylint pass on this commit; the 8 pytest shards pass on its first version, which differs only in one test's import line.

🤖 Generated with Claude Code

A model with hidden state skipped rest-point segmentation ("a rest point
does not reset a hidden process"), so every candidate parameter point
replayed each whole recording from its start: on the from-assets round
fixes_r4, fits of such models took 15 to 56 minutes, and a recording too
chaotic as a whole could not be trimmed phase by phase.

Its recordings are now truncated and segmented like any other model's.
A segment that starts mid-recording carries the actions and observations
before it in place of memory (model_state.memory_from_history), and
restoring that state runs the model's own update over them under the
replay's parameters: the memory each candidate implies, computed as the
execution-time tracker computes it from the same observations, with no
physics. On the fixes_r4 level-1 recordings this cuts Balloons seed 0
from 114 steps to one 53-step segment and Boil seed 0 to 2 segments; a
model that scores the moving robot (Bridge seed 0) finds no rest point
and still replays its recording whole.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant