Skip to content

refactor: one worker process runs the consumers, the pod watch and the scheduled jobs - #374

Merged
HardMax71 merged 1 commit into
mainfrom
refactor/one-worker
Oct 13, 2026
Merged

HardMax71 merged 1 commit into
mainfrom
refactor/one-worker

Conversation

@HardMax71

@HardMax71 HardMax71 commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Six worker containers each loaded the whole app to run one or two Kafka subscribers or a timer. On prod they sat at 146 to 150 MiB of their 160 MiB limits at under 3% CPU, about 890 MiB together.

python -m app.worker (backend/app/worker.py) now runs all of it in one container: the k8s worker, result processor and saga routers, the pod watch, and the saga queue, saga timeout and DLQ retry jobs. Each consumer keeps its group id (k8s-worker, result-processor, saga-orchestrator), so Kafka delivers exactly what it did before and the committed offsets carry over. The service code doesn't change, only the wiring: one container factory instead of six, one config.worker.toml instead of six override files, and no backend/workers/ package. app.worker joins app.main as an entry point in the orphan check.

The event replay worker is gone rather than merged. Its cleanup job ran against an empty session map, since replays live in the API, so all it did was delete finished replay_sessions older than 48 hours. A TTL index on completed_at does that now. On first boot MongoDB will drop every finished session that ended more than 48 hours ago.

The worker has to stay at one replica, as the pod monitor did, since the pod watch and the jobs would run once per replica. Compose gives it 256m; it idles near 150 MiB.

Tested on a local k3s in a container, with Kueue, Mongo, Redis and Kafka on fresh state for each run: the backend e2e suite gives 491 passed, 1 skipped for this branch with the merged worker, and the same for main with its six workers. The merged worker used 146 to 150 MiB during the run, against 145 MiB for each of main's six. Ruff, mypy, the orphan check, unit tests, the Grafana contract and the strict docs build all pass.

Deploy note: the nightly release runs docker compose up --remove-orphans, which removes the six old containers.

Docs follow the change, including the system diagram (one worker box, replay edges moved to the backend) and the event replay page, now at components/event-replay.md.


Summary by cubic

Replaces the six worker containers with one worker process that runs the k8s worker, result processor and saga Kafka routers, the pod watch, and the saga queue, saga timeout and DLQ retry jobs. Each consumer keeps its consumer group, so Kafka delivery and committed offsets don't change; the six containers used about 890 MiB together, while the single process idles near 150 MiB.

Migration

  • The worker must run at one replica, since the pod watch and scheduled jobs would run once per replica.
  • The event replay worker is removed; a MongoDB TTL index on completed_at now drops finished replay sessions 48 hours after they end.
  • On the first boot after this change, MongoDB deletes every finished session older than 48 hours.
  • The nightly release runs docker compose up --remove-orphans, which removes the six old containers.

Written for commit 2816136. Summary will update on new commits.

View guided diff Turn on auto-fix

…e scheduled jobs

The six worker containers each loaded the whole app to run one or two subscribers or a timer, about 145 MiB apiece. app/worker.py now runs the k8s worker, result processor and saga routers, the pod watch, and the saga and DLQ jobs in one process, with each consumer keeping its consumer group. The event replay worker is gone: replay sessions live in the API, and a TTL index on completed_at drops finished ones after 48 hours.
@HardMax71
HardMax71 enabled auto-merge (squash) October 13, 2026 20:38
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 13, 2026
@coderabbitai

coderabbitai Bot commented Oct 13, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 13e4547d-10d9-4c53-a748-fb5b6a299b13
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area: api FastAPI routes, auth, rate limiting, settings area: execution Queue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes area: events Kafka consumers and middlewares, producer, idempotency, DLQ, replay area: deploy Compose, deploy.sh, images, release workflow, production host area: ci GitHub workflows, test infrastructure, repo tooling labels Oct 13, 2026
@sonarqubecloud

Copy link
Copy Markdown

@codecov-commenter

codecov-commenter commented Oct 13, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Flag Coverage Δ
backend-e2e 82.33% <4.54%> (-0.55%) ⬇️
backend-unit 73.24% <100.00%> (+0.04%) ⬆️
frontend-unit 92.37% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
backend/app/core/container.py 87.50% <100.00%> (+20.83%) ⬆️
backend/app/db/docs/replay.py 97.87% <ø> (ø)
backend/app/settings.py 98.21% <ø> (ø)
backend/app/worker.py 76.05% <100.00%> (ø)

... and 2 files with indirect coverage changes

Components Coverage Δ
Backend 91.73% <100.00%> (+0.01%) ⬆️
Frontend 92.37% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HardMax71
HardMax71 merged commit cc8e5f5 into main Oct 13, 2026
25 checks passed
@HardMax71
HardMax71 deleted the refactor/one-worker branch October 13, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: api FastAPI routes, auth, rate limiting, settings area: ci GitHub workflows, test infrastructure, repo tooling area: deploy Compose, deploy.sh, images, release workflow, production host area: events Kafka consumers and middlewares, producer, idempotency, DLQ, replay area: execution Queue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants