From ef64b6bfde8aa8d4d588886aa6df6e22b595e807 Mon Sep 17 00:00:00 2001 From: SagiROosto Date: Tue, 12 May 2026 13:48:04 +0300 Subject: [PATCH 1/2] docs: outputs: file: add log rotation support and configuration examples Signed-off-by: SagiROosto --- pipeline/outputs/file.md | 62 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/pipeline/outputs/file.md b/pipeline/outputs/file.md index 4b05cf381..f4b47c53e 100644 --- a/pipeline/outputs/file.md +++ b/pipeline/outputs/file.md @@ -11,7 +11,11 @@ The _File_ output plugin lets you write the data received through the input plug | Key | Description | Default | | :--- | :--- | :--- | | `file` | Set filename to store the records. If not set, the filename will be the `tag` associated with the records. | _none_ | +| `files_rotation` | Enable size-based [log rotation](#log-rotation). When enabled, files that exceed `max_size` are rotated and optionally compressed. | `false` | | `format` | The [format](#format) of the file content. | _none_ | +| `gzip` | Compress rotated files using gzip. Only applies when `files_rotation` is enabled. | `true` | +| `max_files` | Maximum number of rotated files to retain per output file. Oldest files are deleted first. Must be `1` or greater. Only applies when `files_rotation` is enabled. | `7` | +| `max_size` | Maximum size of the active output file before rotation is triggered. Supports size suffixes: `k` (kilobytes), `m` (megabytes), `g` (gigabytes). Only applies when `files_rotation` is enabled. | `100m` | | `mkdir` | Recursively create output directory if it doesn't exist. Permissions set to `0755`. | `false` | | `path` | Directory path to store files. If not set, Fluent Bit will write the files in its own working directory. | _none_ | | `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` | @@ -112,6 +116,64 @@ You will get the following output: 1564462620.000254 used=1045448 free=31760160 total=32805608 ``` +## Log rotation + +The File output plugin supports size-based log rotation. + +When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the next flush rotates the file by renaming it with a timestamp suffix in the format `.`. The `YYYYMMDD_HHMMSS` is machine local timestamp of the time the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second. + +If `gzip` is enabled (the default), rotated files are compressed with gzip and stored with an additional `.gz` extension (for example, `cpu.log.20260512_134500_a1b2c3d4.gz`). + +The plugin retains up to `max_files` rotated files per output file. When the limit is reached, the oldest rotated files are deleted automatically. + +Log rotation works with all supported output [formats](#format): `plain`, `CSV`, `LTSV`, `template`, and `msgpack`. File operations are thread-safe, so rotation can be used alongside multiple [workers](../../administration/multithreading.md#outputs). + +### Log rotation example + +The following configuration writes CPU metrics to file with rotation enabled. Files are rotated at 50 MB and the five most recent rotated files are retained with gzip compression: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: cpu + tag: cpu + + outputs: + - name: file + match: '*' + path: /var/log/fluent-bit + file: cpu.log + files_rotation: true + max_size: 50m + max_files: 5 + gzip: true +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name cpu + Tag cpu + +[OUTPUT] + Name file + Match * + Path /var/log/fluent-bit + File cpu.log + Files_Rotation true + Max_Size 50m + Max_Files 5 + Gzip true +``` + +{% endtab %} +{% endtabs %} + ## Get started You can run the plugin from the command line or through the configuration file. From 0fa881537cb7a9f72b083398fbf64f32bbea25b5 Mon Sep 17 00:00:00 2001 From: SagiROosto Date: Thu, 30 Jul 2026 16:16:00 +0300 Subject: [PATCH 2/2] docs: outputs: file: align rotation options with out_file rotate_* keys Co-authored-by: Cursor --- pipeline/outputs/file.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/pipeline/outputs/file.md b/pipeline/outputs/file.md index f4b47c53e..6e092dede 100644 --- a/pipeline/outputs/file.md +++ b/pipeline/outputs/file.md @@ -11,13 +11,13 @@ The _File_ output plugin lets you write the data received through the input plug | Key | Description | Default | | :--- | :--- | :--- | | `file` | Set filename to store the records. If not set, the filename will be the `tag` associated with the records. | _none_ | -| `files_rotation` | Enable size-based [log rotation](#log-rotation). When enabled, files that exceed `max_size` are rotated and optionally compressed. | `false` | | `format` | The [format](#format) of the file content. | _none_ | -| `gzip` | Compress rotated files using gzip. Only applies when `files_rotation` is enabled. | `true` | -| `max_files` | Maximum number of rotated files to retain per output file. Oldest files are deleted first. Must be `1` or greater. Only applies when `files_rotation` is enabled. | `7` | -| `max_size` | Maximum size of the active output file before rotation is triggered. Supports size suffixes: `k` (kilobytes), `m` (megabytes), `g` (gigabytes). Only applies when `files_rotation` is enabled. | `100m` | | `mkdir` | Recursively create output directory if it doesn't exist. Permissions set to `0755`. | `false` | | `path` | Directory path to store files. If not set, Fluent Bit will write the files in its own working directory. | _none_ | +| `rotate` | Enable size-based [log rotation](#log-rotation). When enabled, files that exceed `rotate_max_size` are rotated and optionally compressed. | `false` | +| `rotate_gzip` | Compress rotated files using gzip. Only applies when `rotate` is enabled. | `true` | +| `rotate_max_files` | Maximum number of rotated files to retain per output file. Oldest files are deleted first. Must be `1` or greater. Only applies when `rotate` is enabled. | `7` | +| `rotate_max_size` | Maximum size of the active output file before rotation is triggered. Supports size suffixes: `k` (kilobytes), `m` (megabytes), `g` (gigabytes). Only applies when `rotate` is enabled. | `100M` | | `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` | ## Format @@ -120,13 +120,13 @@ You will get the following output: The File output plugin supports size-based log rotation. -When `files_rotation` is enabled, the plugin monitors the size of each output file. Once a file exceeds `max_size`, the next flush rotates the file by renaming it with a timestamp suffix in the format `.`. The `YYYYMMDD_HHMMSS` is machine local timestamp of the time the rotation occurred, and `XXXXXXXX` is a random identifier to guarantee unique filenames if multiple rotations happen within the same second. +When `rotate` is enabled, the plugin monitors the size of each output file. Once a file exceeds `rotate_max_size`, the next flush rotates the file by renaming it with a timestamp suffix in the format `.`. The `YYYYMMDD_HHMMSS` is the machine-local timestamp of the rotation, and `XXXXXXXX` is a random hex identifier that guarantees unique filenames if multiple rotations happen within the same second. -If `gzip` is enabled (the default), rotated files are compressed with gzip and stored with an additional `.gz` extension (for example, `cpu.log.20260512_134500_a1b2c3d4.gz`). +If `rotate_gzip` is enabled (the default), rotated files are compressed with gzip and stored with an additional `.gz` extension (for example, `cpu.log.20260512_134500_a1b2c3d4.gz`). -The plugin retains up to `max_files` rotated files per output file. When the limit is reached, the oldest rotated files are deleted automatically. +The plugin retains up to `rotate_max_files` rotated files per output file. When the limit is reached, the oldest rotated files are deleted automatically. -Log rotation works with all supported output [formats](#format): `plain`, `CSV`, `LTSV`, `template`, and `msgpack`. File operations are thread-safe, so rotation can be used alongside multiple [workers](../../administration/multithreading.md#outputs). +Log rotation works with all supported output [formats](#format): default (`out_file`), `plain`, `csv`, `ltsv`, `template`, and `msgpack`. File operations are thread-safe, so rotation can be used alongside multiple [workers](../../administration/multithreading.md#outputs). ### Log rotation example @@ -146,10 +146,10 @@ pipeline: match: '*' path: /var/log/fluent-bit file: cpu.log - files_rotation: true - max_size: 50m - max_files: 5 - gzip: true + rotate: true + rotate_max_size: 50M + rotate_max_files: 5 + rotate_gzip: true ``` {% endtab %} @@ -161,14 +161,14 @@ pipeline: Tag cpu [OUTPUT] - Name file - Match * - Path /var/log/fluent-bit - File cpu.log - Files_Rotation true - Max_Size 50m - Max_Files 5 - Gzip true + Name file + Match * + Path /var/log/fluent-bit + File cpu.log + Rotate true + Rotate_Max_Size 50M + Rotate_Max_Files 5 + Rotate_Gzip true ``` {% endtab %}