Skip to content

fix(buffer): persist to Redis off the request path, bounded and non-recursive - #81

Merged
rennf93 merged 4 commits into
masterfrom
fix/agent-buffer-redis-deadlock
Oct 15, 2026
Merged

rennf93 merged 4 commits into
masterfrom
fix/agent-buffer-redis-deadlock

Conversation

@rennf93

@rennf93 rennf93 commented Oct 15, 2026

Copy link
Copy Markdown
Member

Closes #80

  • The buffer lock now covers only the in-memory append. The Redis write runs as a tracked background task, so the request path never waits on Redis, and stop_auto_flush drains pending writes.
  • A _PERSISTING context flag stops a Redis failure from re-entering persistence, which removes the deadlock.
  • In-flight writes are capped at buffer_size. Past the cap, an item stays in memory only, counts as a persist failure, and logs one warning per backlog episode. With a 2 s Redis and 20000 events, in-flight tasks went from 20000 (164 MB peak) to at most 1000.
  • An item flushed or evicted before its write starts is never written. One flushed during the write has its key deleted rather than orphaned. Both checks use an O(1) pending-id map. The first version scanned the buffer per task, which stalled the event loop 21.5 s at buffer_size=30000; now it is linear (843 ms, master 582 ms on the same harness).
  • A failed write always releases its pending id. With Redis down, 10000 events leave 0 entries behind.
  • Unpadded base64url keys are accepted, and the startup encryption error includes the underlying cause.

Gate: pre-commit clean, 545 passed, 100% coverage.

@github-actions github-actions Bot added area: buffer Touches guard_agent/buffer.py (event/metric batching) area: encryption Touches guard_agent/encryption.py (payload encryption) tests Test suite changes labels Oct 15, 2026
@rennf93
rennf93 merged commit e3ff1b0 into master Oct 15, 2026
12 checks passed
@rennf93
rennf93 deleted the fix/agent-buffer-redis-deadlock branch October 15, 2026 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: buffer Touches guard_agent/buffer.py (event/metric batching) area: encryption Touches guard_agent/encryption.py (payload encryption) tests Test suite changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Event buffering can block every request when Redis is slow or failing

1 participant