Add graph reinit on failure#4237
Conversation
There was a problem hiding this comment.
Pull request overview
Adds automatic MediaPipe graph reinitialization on inference failure when using the graph queue/pool, to prevent returning an errored (“poisoned”) graph instance back to the pool for reuse.
Changes:
- Introduces
GraphHelper::reinitialize()plus an RAIIGraphReinitGuardthat rebuilds a freshCalculatorGraphunless dismissed on the success path. - Hooks the guard into queue-based unary and streaming inference paths to trigger graph rebuild after failures.
- Moves per-graph shutdown logic into
GraphHelper’s destructor and simplifiesGraphQueuedestructor.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/mediapipe_internal/mediapipegraphexecutor.hpp |
Adds GraphReinitGuard around queue-path inference and dismisses it on success. |
src/mediapipe_internal/graphqueue.hpp |
Adds GraphHelper destructor, reinitialize() declaration, and new GraphReinitGuard type. |
src/mediapipe_internal/graphqueue.cpp |
Implements GraphHelper shutdown + reinit logic; defaults GraphQueue destructor. |
Comments suppressed due to low confidence (2)
src/mediapipe_internal/graphqueue.cpp:142
- reinitialize() sets graph to nullptr on ObserveOutputStream failure (graph.reset()). The next request acquiring this GraphHelper will dereference graphHelper->graph in GraphIdGuard (graph(*graphHelper->graph)), which can crash if graph is null. Ensure reinitialize() always leaves a non-null graph, or update acquisition logic to detect nullptr and rebuild/replace before use.
if (!absStatus.ok()) {
SPDLOG_ERROR("Graph reinitialize: ObserveOutputStream failed: {}", absStatus.ToString());
graph.reset();
return;
}
src/mediapipe_internal/graphqueue.cpp:162
- reinitialize() sets graph to nullptr on StartRun failure (graph.reset()). The next request acquiring this GraphHelper will dereference graphHelper->graph in GraphIdGuard (graph(*graphHelper->graph)), which can crash if graph is null. Ensure reinitialize() always leaves a non-null graph, or update acquisition logic to detect nullptr and rebuild/replace before use.
if (!absStatus.ok()) {
SPDLOG_ERROR("Graph reinitialize: StartRun failed: {}", absStatus.ToString());
graph.reset();
return;
}
| ~GraphReinitGuard() { | ||
| if (!dismissed) { | ||
| helper.reinitialize(config, sidePacketMaps); |
| auto absStatus = graph->Initialize(config); | ||
| if (!absStatus.ok()) { | ||
| SPDLOG_ERROR("Graph reinitialize: Initialize failed: {}", absStatus.ToString()); | ||
| graph.reset(); | ||
| return; |
| auto absStatus = graph->Initialize(config); | ||
| if (!absStatus.ok()) { | ||
| SPDLOG_ERROR("Graph reinitialize: Initialize failed: {}", absStatus.ToString()); | ||
| graph.reset(); | ||
| return; | ||
| } |
| try { | ||
| helper.reinitialize(config, sidePacketMaps); | ||
| } catch (const std::exception& e) { | ||
| SPDLOG_ERROR("GraphReinitGuard: reinitialize threw: {}", e.what()); |
There was a problem hiding this comment.
at this point we are in a state where we lost 1 of the graph from the pool. if it continues to throw, we might end up in a state where no more graphs are available
shouldnt we exit(1) here?
There was a problem hiding this comment.
In that case next infer request will try to reinitialize on failure. It should be pretty rare occasion as reinitialization is basically the same as creation of graph from scratch when not using graph pool.
Ticket: CVS-187323