Repository navigation
refactor: one worker process runs the consumers, the pod watch and the scheduled jobs - #374
Conversation
…e scheduled jobs The six worker containers each loaded the whole app to run one or two subscribers or a timer, about 145 MiB apiece. app/worker.py now runs the k8s worker, result processor and saga routers, the pod watch, and the saga and DLQ jobs in one process, with each consumer keeping its consumer group. The event replay worker is gone: replay sessions live in the API, and a TTL index on completed_at drops finished ones after 48 hours.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes
🚀 New features to boost your workflow:
|



Six worker containers each loaded the whole app to run one or two Kafka subscribers or a timer. On prod they sat at 146 to 150 MiB of their 160 MiB limits at under 3% CPU, about 890 MiB together.
python -m app.worker(backend/app/worker.py) now runs all of it in one container: the k8s worker, result processor and saga routers, the pod watch, and the saga queue, saga timeout and DLQ retry jobs. Each consumer keeps its group id (k8s-worker,result-processor,saga-orchestrator), so Kafka delivers exactly what it did before and the committed offsets carry over. The service code doesn't change, only the wiring: one container factory instead of six, oneconfig.worker.tomlinstead of six override files, and nobackend/workers/package.app.workerjoinsapp.mainas an entry point in the orphan check.The event replay worker is gone rather than merged. Its cleanup job ran against an empty session map, since replays live in the API, so all it did was delete finished
replay_sessionsolder than 48 hours. A TTL index oncompleted_atdoes that now. On first boot MongoDB will drop every finished session that ended more than 48 hours ago.The worker has to stay at one replica, as the pod monitor did, since the pod watch and the jobs would run once per replica. Compose gives it 256m; it idles near 150 MiB.
Tested on a local k3s in a container, with Kueue, Mongo, Redis and Kafka on fresh state for each run: the backend e2e suite gives 491 passed, 1 skipped for this branch with the merged worker, and the same for main with its six workers. The merged worker used 146 to 150 MiB during the run, against 145 MiB for each of main's six. Ruff, mypy, the orphan check, unit tests, the Grafana contract and the strict docs build all pass.
Deploy note: the nightly release runs
docker compose up --remove-orphans, which removes the six old containers.Docs follow the change, including the system diagram (one worker box, replay edges moved to the backend) and the event replay page, now at components/event-replay.md.
Summary by cubic
Replaces the six worker containers with one
workerprocess that runs the k8s worker, result processor and saga Kafka routers, the pod watch, and the saga queue, saga timeout and DLQ retry jobs. Each consumer keeps its consumer group, so Kafka delivery and committed offsets don't change; the six containers used about 890 MiB together, while the single process idles near 150 MiB.Migration
completed_atnow drops finished replay sessions 48 hours after they end.docker compose up --remove-orphans, which removes the six old containers.Written for commit 2816136. Summary will update on new commits.