Repository navigation
buffer: add evacuate_limit_size to keep evacuated chunks from filling the disk - #5473
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new buffer configuration option to cap (or disable) chunk evacuation during Buffer#clear_queue!, preventing unbounded disk growth when an output stays down and the retry limit is reached (Fixes #5352).
Changes:
- Introduces
evacuate_limit_size(nil= unbounded/current behavior,0= disable evacuation, positive = cap total evacuated bytes perclear_queue!pass). - Updates
Buffer#clear_queue!to conditionally evacuate queued chunks based on the configured limit. - Adds unit tests covering default behavior, disabled evacuation (
0), and bounded evacuation.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| lib/fluent/plugin/buffer.rb | Adds evacuate_limit_size config and enforces it during queue clearing/evacuation. |
| test/plugin/test_buffer.rb | Adds tests validating evacuation behavior under different evacuate_limit_size settings. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Watson1978
previously requested changes
Aug 31, 2026
|
Is there any update, when can we expect this to go live as this is a major issue when used at larger scale and there is no current ways supported to disable this Chunk's Backup Completely. |
|
This PR has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this PR will be closed in 7 days |
When an output stays unreachable past the retry limit, queued chunks get evacuated to the backup dir and can fill up the disk. evacuate_limit_size caps the total evacuated size: once the budget is used up the remaining chunks are purged without evacuating, and setting it to 0 turns evacuation off entirely. Default stays nil (no limit) so existing behavior doesn't change. Signed-off-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
The budget was reset on every clear_queue! call. A long outage hits the retry limit many times, so the evacuated files kept growing past the limit. Now the budget is the limit minus the size of the files already in the evacuation directory. Moving the files back for recovery frees the budget again. Evacuation also stops at the first chunk that does not fit, and a warning is logged for each chunk purged without being evacuated. Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Skip only the chunks that do not fit, so more data is kept under the same limit. - Log one warning per clear_queue! instead of one per chunk, and none when the limit is 0. - Do not let an error while counting the evacuated files escape clear_queue!. It stopped the flush thread and left the queue. - Reject a negative limit, and warn when @id is not configured or the limit is smaller than chunk_limit_size. Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nted - Read the file names in the encoding of the evacuation directory, so non-ASCII paths can be counted under LANG=C. - Purge chunks without evacuation if the evacuated files cannot be counted, so evacuate_limit_size keeps bounding the disk usage. - Ignore evacuate_limit_size for buffers which do not evacuate chunks, such as the memory buffer. - Fix the warnings about @id and chunk_limit_size to match the actual behavior. Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The total size may exceed the limit due to chunk metadata or concurrent evacuation by multiple workers. Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Watson1978
force-pushed
the
evacuate-limit-size
branch
from
October 20, 2026 02:38
a509101 to
bf4e37f
Compare
Watson1978
changed the title
Contributor
|
Thanks for the PR, and sorry for the slow review on our side. Since there was no update for a while, I took it over and pushed to your branch, as you have "allow edits by maintainers" enabled.
|
Contributor
Watson1978
dismissed
their stale review
October 20, 2026 02:39
I took over this PR and addressed my review in the commits above.
Without @id, a graceful reload also changes the plugin_id, and then the files evacuated before are not counted. Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kenhys
approved these changes
Oct 20, 2026
Contributor
|
Thanks |
Watson1978
added a commit
to fluent/fluentd-docs-gitbook
that referenced
this pull request
Oct 23, 2026
* buffer: document evacuate_limit_size fluent/fluentd#5473 adds evacuate_limit_size to limit the total size of the evacuated chunk files. Describe how chunks are chosen, why the total can exceed the limit, and why @id is needed. Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * buffer: describe that evacuate_limit_size 0 works as before v1.19.0 "disable the evacuation" did not say what happens to the queued chunks. Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-authored-by: Kentaro Hayashi <kenhys@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Shizuo Fujita <fujita@clear-code.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Kentaro Hayashi <kenhys@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) this PR fixes:
Fixes #5352
What this PR does / why we need it:
When the retry limit is reached, all queued chunks are evacuated to
${root_dir}/buffer/${plugin_id}/, and a long outage repeats this until the disk is full (#5352). This PR addsevacuate_limit_size, which limits the total size of the files in that directory, including the ones evacuated before. Chunks that do not fit are purged without being evacuated,0disables evacuation, and the defaultnilkeeps the current behavior.Docs Changes:
Describe
evacuate_limit_sizein "Handling Successive Failures" of buffer/README.md. It is an approximate limit. The total can exceed it by the.metafiles of the evacuated chunks, or when multiple workers of the same plugin evacuate chunks at the same time. Chunks are evacuated from the oldest, so once the directory reaches the limit, newer chunks are not evacuated until the evacuated files are moved or removed.@idis recommended, and the parameter is ignored by the memory buffer. I will open a PR on fluentd-docs-gitbook.fluent/fluentd-docs-gitbook#715
Release Note:
buffer: add
evacuate_limit_sizeto limit the total size of the evacuated chunk files. Set0to disable evacuation.🤖 Generated with Claude Code