Feature / save long-running sim outputs to cache. Manager preempts sending identical work in favor of pulling from cache - #1585

Open

jlnav wants to merge 37 commits into

feature/shelve_sims

Commits on Sep 8, 2026

initial commit, experimenting with pickling non-libE_field fields (x or f) out of a given update-index to file.
jlnav
committed
d4da77a
additional poking around and experimenting with history saving cache, and then seeing if values exist in that cache before sending sims...
jlnav
committed
786c8b9
better making of .npy database, use History attributes created upon the cache being created. iterate over the cache_keys corresponding to those in gen_specs out, then check if they're close to a ca…
jlnav
committed
5ae2d72
little note...?
jlnav
committed
8ad5f1b
comments
jlnav
committed
1ce8506
experimenting with having caching being a step of the alloc, once we've determined points_to_evaluate
jlnav
committed
1408c76
set those H entries to sim_started?
jlnav
committed
725bb1e
grab update-able indexes, then call update_history_x_out and update_history_f on/with those indexes and the associated cache values, pretending that the array of cache-values are a worker message
jlnav
committed
f342e63
moving cache logic into manager, into handle_msg_from_worker that overrides .recv, and in _send_work_order that forms a buffer from corresponding cache entries
jlnav
committed
9fcad4c
grow the manager's internal record of cache hits, instead of overwriting the same one
jlnav
committed
69918b3
save presumptive workerID for the worker that would've been given cached work. increment a cache_index for slotting into local buffer
jlnav
committed
96399bc
prevent redundant insertions into local cache retrieval. fix a bug involving gen hang. only if all sim work already in cache do we skip the send-stage. when processing cache entries, process them i…
jlnav
committed
2f61be7
refactor, and remove first draft of code that was in alloc_f
jlnav
committed
1f674e6
for now, enable cache for sims that lasted longer than a second
jlnav
committed
28c80c4
experimenting with making disk cache name match calling script plus easily accessible args
jlnav
committed
ccfd5ea
fix redundant send of work if rows send to gen. tiny test fix
jlnav
committed
9aae021
add libE_specs.cache_long_sims, plus more/better docstrings
jlnav
committed
d24fe86
manager builds libE_stats messages corresponding to cache retrievals. tiny refactors to worker.py to help reuse a method. add CACHE_RETRIEVE tag
jlnav
committed
4b57ff6
user can specify database name; trying to figure out occasionally-malformed cache data / H_rows from the cache
jlnav
committed
f268eda
fix syntax error uncaught by black and other tools?
jlnav
committed
37501d9
cache_name is only string. path not needed in specs.py
jlnav
committed
77675f2
param fix
jlnav
committed
fe3e03f
still want to send Work on persis_stop if we're doing final_gen_send
jlnav
committed
dc3202e
don't necessarily need cache collisions for these executor tests - since we're comparing to returned calc statuses
jlnav
committed
9def472
fix iterating over blank template cache entries as though they're valid data
jlnav
committed
5cca0a4
add functionality test for cache_sims
jlnav
committed
e41f006
non-existing cache already dealt with earlier?
jlnav
committed
f88a25d
add the new libe specs options to libe_specs.rst
jlnav
committed
6117b94
check that *all* fields in an outbound row match a cache row before copying the cache values into the manager's local record
jlnav
committed
7abf447
don't need cache_index since we're always updating the last row of the local record. most clarifying comments
jlnav
committed
42da8c2
cache-name joined with _. cache lives in memory for duration of run instead of saving/loading each sim. move cache location to HOME/.cache/libensemble. Fix init issue. Fix redundant uniqueness chec…
jlnav
committed
0ca88ed
update functionality test for develop
jlnav
committed
1af71db
most importantly, specify seed to gen
jlnav
committed
7e06432
cache is computed based on a hash of all the specs, all the callables, and H0. cache rebuilt upon changes to those objects
jlnav
committed
fe51a5c
bump pixi versions in ci yml files, formatting?
jlnav
committed
0ed23d2
various speed optimizations, especially around the caching sections
jlnav
committed
8b2da0c
adjust tolerance again because of flakiness of ci...
jlnav
committed
e88228e