Skip to content

fix: a replay with short waits between events no longer stalls after the first one - #403

Merged
HardMax71 merged 1 commit into
mainfrom
fix/replay-dispatch-stall
Oct 19, 2026
Merged

HardMax71 merged 1 commit into
mainfrom
fix/replay-dispatch-stall

Conversation

@HardMax71

@HardMax71 HardMax71 commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Summary

Small bug found while checking the replay metrics in #401: a replay session whose next event is due at once, or within a few milliseconds, stops after one event and stays running. That covers every replay at speed 100 and anything replaying events with equal or very close timestamps. Locally a 30-event session at speed 100 sat at running with 1 event sent, and the scheduler logged "skipped: maximum number of running instances reached (1)".

Changes

_dispatch_next sent one event and then added its own APScheduler job again for the next one. When that run came due while the current run still counted as running, APScheduler skipped it, and nothing scheduled another. Now one job, _dispatch_events, loops over the events and awaits each delay itself. Pause and cancel stop it through the status check as before. A resume that lands while the loop still waits is skipped by the same rule, and the loop carries on.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: the new test replays 25 events at speed 100 and 99 through start_session and times out on main's code; with the fix it passes. Unit 572 passed, e2e 488 passed and 1 skipped. The 30-event session at speed 100 now completes in 0.055 s.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes replay sessions stalling after the first event when waits between events are zero or within a few milliseconds, which covers every replay at speed 100 and any session with equal or very close event timestamps.

  • Scheduling one APScheduler job per event meant the next run could come due while the current one still counted as running, so it was skipped under max_instances=1 and nothing scheduled another, leaving the session running with one event sent.
  • A single job, _dispatch_events, now loops over the events and awaits each delay itself.
  • Pause and cancel stop the loop through the status check, as before.

Written for commit 79d41f9. Summary will update on new commits.

View guided diff Turn on auto-fix

…the first one

_dispatch_next sent one event and then re-added its own APScheduler job for the next. When the wait was zero or shorter than what was left of the running job, the new run came due while that job still counted as running, so APScheduler skipped it under max_instances=1 and nothing scheduled another: the session sat in running with one event sent. One job now loops over the events and awaits each wait itself. Pause and cancel stop it through the status check, as before.
@HardMax71
HardMax71 enabled auto-merge (squash) October 18, 2026 23:58
@coderabbitai

coderabbitai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: af251b23-43d4-461f-b1e7-6167d5c92073
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the area: events Kafka consumers and middlewares, producer, idempotency, DLQ, replay label Oct 18, 2026
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
B Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@codecov-commenter

codecov-commenter commented Oct 18, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 69.69697% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ackend/app/services/event_replay/replay_service.py 69.69% 10 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Flag Coverage Δ
backend-e2e 82.93% <63.63%> (+0.16%) ⬆️
backend-unit 74.29% <69.69%> (+0.40%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ackend/app/services/event_replay/replay_service.py 86.53% <69.69%> (+0.86%) ⬆️

... and 1 file with indirect coverage changes

Components Coverage Δ
Backend 92.20% <69.69%> (+0.04%) ⬆️
Frontend 92.60% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HardMax71
HardMax71 merged commit ed735da into main Oct 19, 2026
20 of 21 checks passed
@HardMax71
HardMax71 deleted the fix/replay-dispatch-stall branch October 19, 2026 00:03
HardMax71 added a commit that referenced this pull request Oct 19, 2026
…their own methods (#405)

SonarCloud failed #403 on python:S3776: wrapping the old dispatch body in a loop put _dispatch_events at cognitive complexity 21 against the allowed 15. _next_event loads the next batch when the loaded one runs out, and _dispatch_event sends one event and returns the wait before the next; the loop only joins them. Behaviour is the same.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: events Kafka consumers and middlewares, producer, idempotency, DLQ, replay

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants