fix: always release executors and reset state when stopping the executor manager - #3526
Draft
csviri wants to merge 1 commit into
Draft
fix: always release executors and reset state when stopping the executor manager#3526csviri wants to merge 1 commit into
csviri wants to merge 1 commit into
Conversation
…tor manager `ExecutorServiceManager.stop` had three problems, all on the interrupted path or affecting the scheduled executor. `scheduledExecutorService` was never shut down. It is created on every `start()` and exposed through a public accessor, but `stop()` only shut down the reconcile, workflow and caching executors, so the pool (and any non-daemon threads a caller created through the accessor) outlived the operator and leaked again on every restart. The helper pool leaked when interrupted. `Executors.newFixedThreadPool(3)` was created inside the `try` and only shut down on the success path, so an `InterruptedException` from `invokeAll` left three non-daemon threads behind - in a shutdown path, where they then keep the JVM alive. `started` was not reset when interrupted. It was only set to false on the success path, so after an interrupted `stop()` the executors were already shut down but `start()` would see `started == true` and do nothing. The operator then looked started while every `execute` on the terminated reconcile executor failed with `RejectedExecutionException`. Moves the cleanup into a `finally`, includes the scheduled executor in the graceful shutdown, and clears both nullable references there. Adds regression tests for the scheduled executor shutdown and for restartability; the former fails without this change.
16 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes shutdown/restart correctness in ExecutorServiceManager.stop() to prevent executor/thread leaks and ensure the manager can be restarted cleanly after a stop (including interrupted stop paths).
Changes:
- Shut down the
scheduledExecutorServicealongside other executors duringstop(). - Move helper-pool shutdown and state reset into a
finallyblock so it runs even when interrupted. - Add regression tests for scheduled-executor shutdown and manager restartability.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ExecutorServiceManager.java | Ensures shutdown logic runs reliably (including scheduled executor) and resets state in finally. |
| operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/config/ExecutorServiceManagerTest.java | Adds regression tests covering scheduled executor shutdown and restartability after stop. |
Comments suppressed due to low confidence (1)
operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/config/ExecutorServiceManagerTest.java:55
- As written, if any of the post-restart assertions fail, the test will exit before the final
manager.stop(...)call and may leave non-daemon executor threads running. Wrapping the restart assertions in atry/finallyensures cleanup even when the test fails.
manager.stop(SHUTDOWN_TIMEOUT);
manager.start(configurationService);
// start() is a no-op unless stop() reset the started flag, which would leave the manager
// handing out already terminated executors
assertThat(manager.reconcileExecutorService().isShutdown()).isFalse();
assertThat(manager.cachingExecutorService().isShutdown()).isFalse();
assertThat(manager.scheduledExecutorService().isShutdown()).isFalse();
manager.stop(SHUTDOWN_TIMEOUT);
}
Comment on lines
+162
to
+165
| parallelExec.shutdownNow(); | ||
| workflowExecutor = null; | ||
| scheduledExecutorService = null; | ||
| started = false; |
Comment on lines
+28
to
+38
| @Test | ||
| void stopShutsDownTheScheduledExecutorService() { | ||
| ConfigurationService configurationService = new BaseConfigurationService(); | ||
| var manager = configurationService.getExecutorServiceManager(); | ||
| var scheduled = manager.scheduledExecutorService(); | ||
| assertThat(scheduled.isShutdown()).isFalse(); | ||
|
|
||
| manager.stop(SHUTDOWN_TIMEOUT); | ||
|
|
||
| assertThat(scheduled.isShutdown()).isTrue(); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ExecutorServiceManager.stophad three problems, all on the interruptedpath or affecting the scheduled executor.
scheduledExecutorServicewas never shut down. It is created on everystart()and exposed through a public accessor, butstop()only shutdown the reconcile, workflow and caching executors, so the pool (and any
non-daemon threads a caller created through the accessor) outlived the
operator and leaked again on every restart.
The helper pool leaked when interrupted.
Executors.newFixedThreadPool(3)was created inside the
tryand only shut down on the success path, so anInterruptedExceptionfrominvokeAllleft three non-daemon threadsbehind - in a shutdown path, where they then keep the JVM alive.
startedwas not reset when interrupted. It was only set to false on thesuccess path, so after an interrupted
stop()the executors were alreadyshut down but
start()would seestarted == trueand do nothing. Theoperator then looked started while every
executeon the terminatedreconcile executor failed with
RejectedExecutionException.Moves the cleanup into a
finally, includes the scheduled executor in thegraceful shutdown, and clears both nullable references there.
Adds regression tests for the scheduled executor shutdown and for
restartability; the former fails without this change.
Part of #3517