Viewer: residual polars access violation and selection-protocol side effects after #99

[t0mdavid-m]

Follow-ups from #99 (per-session lock around render_grid + rebuilt js-component/dist from t0mdavid-m/openms-streamlit-vue-component#30), found while re-testing the branch on 2026-09-10.

1. The Viewer still crashes the server, now without overlapping script threads

#99 removes the reproduced trigger (2–4 concurrent ScriptRunner threads rendering the grid; 0 crashes in 2×60 row clicks afterwards). It does not remove the underlying native crash: during one fresh-session first load of the FLASHDeconv Viewer on fix/viewer-rerun-cascade (efa36a9) the server died again with Windows fatal exception: access violation.

Environment: Windows 11 (10.0.26200), Python 3.11.15, Streamlit 1.49.1, polars 1.44.2 (runtime rt32), pyarrow 19.0.1, numpy 2.4.6, streamlit run app.py local with -X faulthandler.

The faulthandler dump lists exactly one script thread, sitting in polars' streaming engine inside initialize_data, and no other Python thread inside polars or pyarrow (the rest are watchdog observers and the asyncio loop). There is no Current thread marker, i.e. the faulting thread was a native (non-Python) thread:

Windows fatal exception: access violation

Thread 0x000072b8 (most recent call first):
  File ".venv\Lib\site-packages\polars\lazyframe\engine.py", line 443 in collect
  File ".venv\Lib\site-packages\polars\lazyframe\frame.py", line 2591 in collect
  File "src\render\initialize.py", line 47 in initialize_data
      # data_full.select(pl.len()).collect(engine="streaming") on pl.scan_parquet(ms1_deconv_heatmap.pq)
  File "src\render\render.py", line 123 in _render_grid_unlocked
  File "src\render\render.py", line 87 in render_grid
  File "content\FLASHDeconv\FLASHDeconvViewer.py", line 117 in <module>
  ...

For comparison, the two crashes analysed for #99 (before the lock) had a Current thread in polars.DataFrame.deserialize (the st.cache_data pickle round trip of render_heatmap) while a second script thread was in a pyarrow to_table.

What was checked:

• Frequency on the fixed branch that day: 1 crash in roughly 10 fresh-session Viewer loads (all other loads and ~40 row clicks were fine). develop crashed twice more in the same session, as before.
• A standalone stress test of the same operations in one process (pl.scan_parquet(...).select(pl.len()).collect(engine="streaming"), downsample_heatmap(...).collect(engine="streaming"), DataFrame.serialize()/deserialize(), pyarrow.dataset(...).to_table(filter=...), pd.read_parquet; 150 iterations × 1 and × 4 threads, with and without import pyopenms) ran 1500 iterations with no crash. So the crash needs something the Streamlit process adds (session teardown / GC of the previous session's polars and pyarrow objects, the source watcher's sys.modules purge, cache pickling, ...) that the stress test does not have.

Ideas for narrowing it down (none verified yet):

• run the deployment with PYTHONFAULTHANDLER=1 so production crashes leave a dump;
• replace collect(engine="streaming") with the in-memory engine for the tiny row count in initialize_data and in render_heatmap, and see whether the crash site moves or disappears;
• pin polars (currently unpinned polars>=1.0.0, resolves to 1.44.2) and test an older/newer release;
• load the heatmap LazyFrames once per process (st.cache_resource) instead of once per session, so fewer polars objects are created and dropped per session;
• check whether the previous session's teardown (GC of st.session_state['plot_data']) overlaps with the new session's first render_grid.

2. Behaviour the rebuilt bundle introduces in the selection protocol

The committed bundle before #99 dropped unset selection fields (JSON.stringify drops undefined). The Vue source has serialised them as null since t0mdavid-m/openms-streamlit-vue-component#29, and #99 is the first FLASHApp bundle that ships this. Two consequences on the Python side (src/render/StateTracker.py):

• First message claims every key. updateState stores every previously unknown key unconditionally, so the first cell that reports (now with scanIndex: null, massIndex: null, ...) claims all keys as None. A later first-time value with the same counter (e.g. the Scan Table's default row selection) is then rejected by the counter >= currentStateCounter check and the Mass Table / spectra stay empty until the user clicks a row. Deterministic replay:

  other cell sends its (empty) state first          accepted=True  counter=1 state={}
  Scan Table default row: scanIndex=0 massIndex=0    accepted=False counter=1 state={}
  

  Not hit in the default FLASHDeconv layout during testing (only the Scan Table sends on load there), but any layout with a cell that writes to the store on mount is exposed.

• Spectrum reset now reaches Python. PlotlyLineplotUnified calls updateSelectedMass(undefined) (via resetManualState) when it first receives scan data; the new bundle forwards this as massIndex: null. Measured on the fix branch: two extra reruns per Viewer load, and Python's massIndex ends as None while the Mass Table still highlights row 0 (the highlight and the shared state disagree).

The counter check itself also drops any legitimate click whose counter is older than the server's (a click sent while another cell's update is in flight); the window is ~50 ms locally and grows with network latency. With echoes gone (#30), the counter is no longer needed for loop protection and per-key last-write-wins from user interactions would be safer.

Related

The "Mass Table only loads after clicking the first scan row" report has a separate, deterministic root cause (hash-equal re-render after switching to an identical experiment skips the Tabulator rebuild, so the default row selection is never sent); it predates #99 and is tracked separately.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wda4mDs1DSWJuU1beJJKvL