refactor: the worker runs executions without a saga, and cancel stops the pod by HardMax71 · Pull Request #428 · HardMax71/Integr8sCode

HardMax71 · 2026-10-20T21:53:28Z

Summary

The worker ran one job as four services that sent each other commands through Kafka inside one process, and their copies of an execution's state drifted apart. Cancelling a running execution only marked the saga and freed its slot (saga_orchestrator.py), so the pod ran on, the execution ended as completed and the next queued run started beside it.

Changes

A new execution scheduler keeps the Redis queue and calls the k8s worker directly. The pod monitor hands a finished pod to the result processor and frees its slot. The processor writes first and publishes only if the write applied, so a cancelled pod's late failure sends nothing. Cancel stores cancelled, then deletes the pod. The saga goes with its routes, admin page, 300 s timeout and the saga and pod command events. The worker's group keeps the name saga-orchestrator so its offsets carry over.

Warning

Stored events and DLQ rows of the removed types no longer load, and the event pages fail on them until they're gone. Right after the deploy, on prod MongoDB:

const removed = ["saga_started", "saga_completed", "saga_failed", "saga_cancelled", "saga_compensating", "saga_compensated", "create_pod_command", "delete_pod_command", "allocate_resources_command", "release_resources_command"];
db.events.deleteMany({ event_type: { $in: removed } });
db.dlq_messages.deleteMany({ "event.event_type": { $in: removed } });
db.sagas.drop();

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update
Refactoring (no functional changes)
CI/CD or tooling changes

Test Plan

Backend unit tests pass (cd backend && uv run pytest tests/unit)
Frontend unit tests pass (cd frontend && npm run test)
Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
Frontend E2E tests pass (cd frontend && npm run test:e2e)
API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
Manual testing performed

Validation: backend e2e on k3s with this branch's worker gives 463 passed, 1 skipped (main collects 494; the 30 saga tests went). The cancel test now checks the stored cancelled. CI's e2e-ready also waits for a consumer on execution_requested, since the worker's two subscribers join seconds apart. The unit, contract, Vitest, lint, type and docs gates pass.

Checklist

My code follows the project's style guidelines
I have run pre-commit run --all-files and all checks pass
I have added tests that prove my fix/feature works
I have updated documentation if needed
My changes don't introduce new warnings

Summary by cubic

The worker no longer runs executions through a saga. A new execution scheduler pulls from the Redis queue and tells the k8s worker to start a pod directly; the pod monitor hands finished pods to the result processor, which writes the result before publishing anything and then frees its slot. Before, the four services inside the worker passed commands to each other over Kafka and kept drifting copies of the same execution state.

Cancelling a run previously only marked the saga and freed its slot, so the pod ran to completion. Cancel now stores the execution as cancelled and deletes its pod, ending the run at once. The saga orchestrator, its API and admin pages, and the saga and pod command events are removed; the worker's group keeps the name saga-orchestrator so its Kafka offsets carry over.

Migration

Stored events and DLQ rows of the removed saga types no longer load, and the event pages fail on them. After deploying, run on MongoDB:

const removed = ["saga_started", "saga_completed", "saga_failed", "saga_cancelled", "saga_compensating", "saga_compensated", "create_pod_command", "delete_pod_command", "allocate_resources_command", "release_resources_command"];
db.events.deleteMany({ event_type: { $in: removed } });
db.dlq_messages.deleteMany({ "event.event_type": { $in: removed } });
db.sagas.drop();

^{Written for commit d4edbe3. Summary will update on new commits.}

… the pod The saga orchestrator, k8s worker, result processor and pod monitor passed commands to each other through Kafka inside one process. The new execution scheduler keeps the Redis queue and calls the k8s worker directly; the pod monitor hands finished pods to the result processor, which stores the result before it publishes anything, then frees the slot. Cancel now stores the execution as cancelled and deletes its pod. Before, only the saga was marked and the pod ran to the end. The saga documents, routes, admin page, saga and pod command events go. The worker's consumer group keeps the name saga-orchestrator so its committed offsets carry over.

…duler The two shared the name scheduler in start_jobs. The entrypoint test now checks the function each job calls.

…s group The worker's two subscribers join saga-orchestrator seconds apart, so the group can be Stable before the request consumer has partitions, and a request sent then is skipped on a fresh Kafka.

coderabbitai · 2026-10-20T21:53:37Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults
Review profile: CHILL
Plan: Advanced
Run ID: d81fe435-06fe-44e0-9331-273cf935dfc1

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

sonarqubecloud · 2026-10-20T21:54:13Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

codecov-commenter · 2026-10-20T21:55:04Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 81.50685% with 27 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
backend/app/services/k8s_worker/worker.py	26.66%	11 Missing ⚠️
backend/app/services/result_processor/processor.py	72.72%	9 Missing ⚠️
backend/app/services/pod_monitor/monitor.py	80.95%	4 Missing ⚠️
backend/app/events/handlers.py	75.00%	2 Missing ⚠️
backend/app/core/providers.py	75.00%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Flag	Coverage Δ
backend-e2e	`83.04% <31.50%> (+0.76%)`	⬆️
backend-unit	`75.77% <81.50%> (+1.14%)`	⬆️
frontend-unit	`92.12% <ø> (-0.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
backend/app/core/container.py	`87.50% <ø> (ø)`
backend/app/db/__init__.py	`100.00% <ø> (ø)`
backend/app/db/docs/__init__.py	`100.00% <ø> (ø)`
backend/app/db/repositories/__init__.py	`100.00% <ø> (ø)`
backend/app/db/repositories/user_repository.py	`95.89% <ø> (-0.06%)`	⬇️
backend/app/dlq/models.py	`91.54% <100.00%> (+1.13%)`	⬆️
backend/app/domain/enums/__init__.py	`100.00% <ø> (ø)`
backend/app/domain/enums/events.py	`100.00% <ø> (ø)`
backend/app/domain/events/__init__.py	`100.00% <ø> (ø)`
backend/app/domain/events/typed.py	`100.00% <ø> (ø)`
... and 17 more

... and 1 file with indirect coverage changes

Components	Coverage Δ
Backend	`93.15% <81.50%> (+0.58%)`	⬆️
Frontend	`92.12% <ø> (-0.15%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

HardMax71 added 4 commits October 20, 2026 23:30

docs: the system diagram shows the execution scheduler in the worker

c240335

fix: the worker's queue job calls the execution scheduler, not APSche…

a3a9acb

…duler The two shared the name scheduler in start_jobs. The entrypoint test now checks the function each job calls.

ci: e2e waits until execution_requested has a consumer in the worker'…

d4edbe3

…s group The worker's two subscribers join saga-orchestrator seconds apart, so the group can be Stable before the request consumer has partitions, and a request sent then is skipped on a fresh Kafka.

HardMax71 enabled auto-merge (squash) October 20, 2026 21:53

HardMax71 merged commit 872baf9 into main Oct 20, 2026
23 checks passed

HardMax71 deleted the refactor/worker-without-saga branch October 20, 2026 21:59

Navigation Menu

Repository navigation

refactor: the worker runs executions without a saga, and cancel stops the pod - #428

refactor: the worker runs executions without a saga, and cancel stops the pod#428
HardMax71 merged 4 commits into
mainfrom
refactor/worker-without-saga

HardMax71 commented Oct 20, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 20, 2026

Uh oh!

sonarqubecloud Bot commented Oct 20, 2026

Uh oh!

codecov-commenter commented Oct 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

HardMax71 commented Oct 20, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Type of Change

Test Plan

Checklist

Summary by cubic

Uh oh!

coderabbitai Bot commented Oct 20, 2026

Uh oh!

sonarqubecloud Bot commented Oct 20, 2026

Quality Gate passed

Uh oh!

codecov-commenter commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HardMax71 commented Oct 20, 2026 •

edited by cubic-dev-ai Bot

Loading

codecov-commenter commented Oct 20, 2026 •

edited

Loading