Repository navigation
fix(core): keep history sampling off the execution lease - #13
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4fa9c62ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| ## Sampling ownership | ||
|
|
||
| Periodic history sampling reads the execution Worker's observed ownership state. Its short source deadlines and sweep cancellation never run database operations on the connection holding the execution lease. The Worker retains its authoritative database ownership checks before execution, publishes their observations in check order, and invalidates the observed state when it stops or loses ownership. Waiting for an ownership check and performing it share one bounded deadline; a caller cancellation while waiting never starts a database operation. An unknown, failed or stopped ownership observation prevents sampling; it never grants execution authority. |
There was a problem hiding this comment.
This paragraph says periodic sampling only reads the Worker's observed ownership state, but the same document still promises that collection runs under the database lease, checks that lease every 100 ms (lines 88–90), and performs node-history copying after a lease check (line 106); the Chinese mirror retains the same conflicting claims at lines 90–92 and 108. Since the implementation now consults runtimeHistoryOwnership rather than the lease, update those existing paragraphs instead of leaving contradictory operational semantics.
AGENTS.md reference: AGENTS.md:L58-L58
Useful? React with 👍 / 👎.
1 participant
Periodic Runtime history sampling currently sends 250 ms ownership checks and sweep-cancellation contexts to the PostgreSQL connection holding Core's execution advisory lock. Cancelling an in-flight Ping can close that connection; the execution Worker subsequently fails and Core exits, interrupting active Sessions.
Sampling now reads the Worker's observed ownership state without touching the leased connection. Execution retains authoritative database checks. Each check and its observation publish in authoritative order, with a bounded cancellable wait; an older success cannot overwrite a newer failure. Worker stop invalidates the observation before draining. Failed leased operations log bounded operation/phase, cancellation, connection state and SQLSTATE fields without SQL or error text; Worker failures identify their exit stage, and the top-level failure message no longer labels every runtime failure as startup.
Validation:
No schema/SQL, public API, Runtime/helper/node protocol or native package changes. Core/Web release only; retain the currently selected E2B template. This fixes a reproduced failure path, while the new provenance is needed to establish whether it explains every production restart. Production incident evidence includes restartCount 27 and a conn closed exit at 2026-10-07T03:57:51Z.
Production release: core-deploy run 37605037282 succeeded with beta merge commit
8aebb6c8bf531b90828bc7efd369f4e49535c67a. Core/Web images and management metrics report this exact revision. Web and health returned 200, and the public API with the required protocol header returned 401 without credentials. At 2026-10-07T10:10:14Z, both new Pods had zero restarts; execution ownership was observed, scheduler and runtime_sampler were ok, the latest sampler processed 36 objects with zero failures, and startup logs had no WARN/ERROR. E2B selection remained generation 3 and the original 10Gi PVC UID was unchanged. This is bounded release verification; end-to-end model execution is being validated separately by the user's team.