From 92239766e434a2e0da8f45bfde79a7f0c495421c Mon Sep 17 00:00:00 2001 From: Shizuo Fujita Date: Tue, 20 Oct 2026 11:43:50 +0900 Subject: [PATCH 1/2] buffer: document evacuate_limit_size fluent/fluentd#5473 adds evacuate_limit_size to limit the total size of the evacuated chunk files. Describe how chunks are chosen, why the total can exceed the limit, and why @id is needed. Signed-off-by: Shizuo Fujita Co-Authored-By: Claude Opus 5.5 --- buffer/README.md | 50 +++++++++++++++++++++++++ configuration/buffer-section.md | 3 ++ plugin-development/api-plugin-buffer.md | 6 +++ 3 files changed, 59 insertions(+) diff --git a/buffer/README.md b/buffer/README.md index c46d2e3a..bcf13db1 100644 --- a/buffer/README.md +++ b/buffer/README.md @@ -109,6 +109,56 @@ Please see [How to Write Buffer Plugin](../plugin-development/api-plugin-buffer. Please note that [`buf_memory`](memory.md) does not support this feature. {% endhint %} +#### Limiting the size of evacuated chunk files + +Since v1.20.0, you can limit the total size of the evacuated chunk files with `evacuate_limit_size` in the `` section. +It prevents the evacuated chunk files from filling the disk when the retry limit is reached again and again during a long outage. +`buf_memory` ignores this parameter. + +```text + + @type forward + @id forward_app + + host 192.168.1.3 + + + @type file + path /var/log/fluent/buf + evacuate_limit_size 1g + + +``` + +When the retry limit is reached, Fluentd evacuates the queued chunks that fit within the remaining capacity, in the order of the queue. +This is usually from the oldest chunk. +The chunks that do not fit are discarded without being evacuated. +If `evacuate_limit_size` is smaller than `chunk_limit_size`, full chunks may not be evacuated, and Fluentd warns about it at startup. + +All files directly under the directory are counted, including the ones evacuated before. +Files in subdirectories are not counted. +So once the directory reaches the limit, newer chunks are not evacuated until you move or remove the evacuated chunk files. +If Fluentd cannot get the size of the evacuated chunk files, for example because the directory is not readable, it discards the queued chunks without evacuating them. + +Set `0` to disable the evacuation. +The default is no limit. + +{% hint style='warning' %} +`evacuate_limit_size` is an approximate limit. +Fluentd counts the size of the directory only when the evacuation starts, so the total size may exceed the limit in the following cases. + +* `buf_file` also evacuates the `.meta` file of each chunk. These files are not counted until the next evacuation. +* All workers of the same plugin share the directory. If their evacuations overlap, the total size can reach about the number of workers times the limit. + +The total size is counted again at the next retry limit. +If it already exceeds the limit, no chunks are evacuated. +{% endhint %} + +{% hint style='info' %} +Configure [@id](../configuration/plugin-common-parameters.md#id) for the output plugin. +Without `@id`, `${plugin_id}` can change when the configuration is changed or reloaded, and the chunk files evacuated before are not counted. +{% endhint %} + ### Handling Unrecoverable Errors Not all errors are recoverable in nature. For example, if the content of a chunk file gets corrupted, you obviously cannot fix anything just by redoing the write operation. Rather, a blind retry attempt will just make the situation worse. diff --git a/configuration/buffer-section.md b/configuration/buffer-section.md index b875aa05..766f0c91 100644 --- a/configuration/buffer-section.md +++ b/configuration/buffer-section.md @@ -595,6 +595,9 @@ Following are the flushing parameters for chunks to optimize performance \(laten * Instead of storing unrecoverable chunks in the backup directory, just discard them. This option is new in Fluentd v1.2.6. +* `evacuate_limit_size` \[size\] + * Default: `nil` \(no limit\) + * The approximate size limit of the evacuated chunk files in the evacuation directory. When the retry limit is reached, queued chunks that fit within the remaining capacity are evacuated, and the others are discarded without being evacuated. Set `0` to disable the evacuation. It is available for `buf_file` and `buf_file_single`. This option is new in Fluentd v1.20.0. See [Limiting the size of evacuated chunk files](../buffer/#limiting-the-size-of-evacuated-chunk-files). With `exponential_backoff`, `retry_wait` interval will be calculated as below: diff --git a/plugin-development/api-plugin-buffer.md b/plugin-development/api-plugin-buffer.md index 27040870..d2e80382 100644 --- a/plugin-development/api-plugin-buffer.md +++ b/plugin-development/api-plugin-buffer.md @@ -24,3 +24,9 @@ rescue => e log.error "unexpected error while evacuating chunk files.", error: e end ``` + +Since v1.20.0, the base class decides which chunks to pass to this method by `evacuate_limit_size`. +It counts the files directly under the directory used in the sample code above, so evacuate the chunk files to this directory. +Since v1.20.0, `evacuate_dir` returns this directory. +The base class subtracts only `chunk.bytesize` from the remaining capacity, so other files written by this method are not counted until the next evacuation. +The base class ignores `evacuate_limit_size` if `persistent?` of the buffer plugin returns `false`. From d18d8c4583ecbd772ff82bcc3662d3c15e07bbd0 Mon Sep 17 00:00:00 2001 From: Shizuo Fujita Date: Tue, 20 Oct 2026 16:35:26 +0900 Subject: [PATCH 2/2] buffer: describe that evacuate_limit_size 0 works as before v1.19.0 "disable the evacuation" did not say what happens to the queued chunks. Signed-off-by: Shizuo Fujita Co-authored-by: Kentaro Hayashi Co-Authored-By: Claude Opus 5.5 --- buffer/README.md | 2 +- configuration/buffer-section.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/buffer/README.md b/buffer/README.md index bcf13db1..624f4ff1 100644 --- a/buffer/README.md +++ b/buffer/README.md @@ -140,7 +140,7 @@ Files in subdirectories are not counted. So once the directory reaches the limit, newer chunks are not evacuated until you move or remove the evacuated chunk files. If Fluentd cannot get the size of the evacuated chunk files, for example because the directory is not readable, it discards the queued chunks without evacuating them. -Set `0` to disable the evacuation. +Set `0` to discard the queued chunks without evacuating them, the same as the behavior before v1.19.0. The default is no limit. {% hint style='warning' %} diff --git a/configuration/buffer-section.md b/configuration/buffer-section.md index 766f0c91..96b07be9 100644 --- a/configuration/buffer-section.md +++ b/configuration/buffer-section.md @@ -597,7 +597,7 @@ Following are the flushing parameters for chunks to optimize performance \(laten discard them. This option is new in Fluentd v1.2.6. * `evacuate_limit_size` \[size\] * Default: `nil` \(no limit\) - * The approximate size limit of the evacuated chunk files in the evacuation directory. When the retry limit is reached, queued chunks that fit within the remaining capacity are evacuated, and the others are discarded without being evacuated. Set `0` to disable the evacuation. It is available for `buf_file` and `buf_file_single`. This option is new in Fluentd v1.20.0. See [Limiting the size of evacuated chunk files](../buffer/#limiting-the-size-of-evacuated-chunk-files). + * The approximate size limit of the evacuated chunk files in the evacuation directory. When the retry limit is reached, queued chunks that fit within the remaining capacity are evacuated, and the others are discarded without being evacuated. Set `0` to discard the queued chunks without evacuating them, the same as the behavior before v1.19.0. It is available for `buf_file` and `buf_file_single`. This option is new in Fluentd v1.20.0. See [Limiting the size of evacuated chunk files](../buffer/#limiting-the-size-of-evacuated-chunk-files). With `exponential_backoff`, `retry_wait` interval will be calculated as below: