Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion appendix/td-agent-v2-vs-v3-vs-v4.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,5 +59,5 @@ This is for v0.12 and old distribution users. We don't recommend this version fo
* [macOS](../installation/install-by-dmg-td-agent-v4.md)
* [RubyGems](../installation/install-by-gem.md)

If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0)
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under [the Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0)

8 changes: 4 additions & 4 deletions configuration/buffer-section.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,16 +405,16 @@ Following are the configuration parameters for buffer plugin and its chunks:

* `chunk_limit_size` \[size\]
* Default: 8MB \(memory\) / 256MB \(file\)
* The max size of each chunks: events will be written into chunks until
* The max size of each chunk: events will be written into chunks until

the size of chunks become this size
the size of chunks becomes this size
* `chunk_limit_records` \[integer\]
* Optional
* The max number of events that each chunks can store in it
* The max number of events that each chunk can store in it
* `total_limit_size` \[size\]
* Default: 512MB \(memory\) / 64GB \(file\)
* The size limitation of this buffer plugin instance
* Once the total size of stored buffer reached this threshold, all append
* Once the total size of stored buffer reaches this threshold, all append

operations will fail with error \(and data will be lost\)
* `queue_limit_length` \[integer\]
Expand Down
6 changes: 3 additions & 3 deletions configuration/config-file-yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ See also: [Lifecycle of a Fluentd Event](../quickstart/life-of-a-fluentd-event.m

Fluentd starts to support YAML configuration format but this is not 1-by-1 correspondence for Fluentd config file syntax.

Normal Fluentd configuration syntax has the following the list of directives:
Normal Fluentd configuration syntax has the following list of directives:

1. **`source`** directives determine the input sources
2. **`match`** directives determine the output destinations
Expand All @@ -42,7 +42,7 @@ Under `config` object, Fluentd will handle the following elements:
### Special YAML elements

1. **`!include`** defines including rules for other files
2. **`!fluent/s`** defines Fluentd string format that is equivalent for double quoted string
2. **`!fluent/s`** defines Fluentd string format that is equivalent to double quoted string
3. **`!fluent/json`** defines Fluentd JSON format that is used for Hash type object
4. **`$tag`** defines tag for output plugin
5. **`$label`** defines label routes for input plugin
Expand Down Expand Up @@ -243,7 +243,7 @@ If `$label` with `$name: '@ERROR'` is set, the events are routed to this label

The `@ROOT` label is a builtin label used for getting root router by plugin's `event_emitter_router` API.

This label is introduced since v1.14.0 to assign a label back to the default route. For example, timed-out event records are handled by the concat filter can be sent to the default route.
This label is introduced since v1.14.0 to assign a label back to the default route. For example, timed-out event records handled by the concat filter can be sent to the default route.

## 6. Limit to specific workers: the `worker` element

Expand Down
4 changes: 2 additions & 2 deletions configuration/config-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ $ sudo vi /etc/fluent/fluentd.conf
#### td-agent (EOL)

{% hint style='warning' %}
As `td-agent` had already reached EOL, we recommend to use `fluent-package` (the successor to `td-agent`).
As `td-agent` had already reached EOL, we recommend using `fluent-package` (the successor to `td-agent`).

* [fluent-package v5 vs td-agent v4](../quickstart/fluent-package-v5-vs-td-agent.md)
{% endhint %}
Expand Down Expand Up @@ -289,7 +289,7 @@ If `<label @ERROR>` is set, the events are routed to this label when the related

The `@ROOT` label is a builtin label used for getting root router by plugin's `event_emitter_router` API.

This label is introduced since v1.14.0 to assign a label back to the default route. For example, timed-out event records are handled by the concat filter can be sent to the default route.
This label is introduced since v1.14.0 to assign a label back to the default route. For example, timed-out event records handled by the concat filter can be sent to the default route.

## 6. Limit to specific workers: the `worker` directive

Expand Down
2 changes: 1 addition & 1 deletion configuration/parse-section.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ For the `types` parameter, the following types are supported:

* **`time_format_fallbacks`** \(\) \(optional\): uses the specified time format as a fallback in the specified order.

You can parse undetermined time format by using `time_format_fallbacks`. This options is enabled when `time_type` is `mixed`.
You can parse undetermined time format by using `time_format_fallbacks`. This option is enabled when `time_type` is `mixed`.

* Default: `nil`

Expand Down
2 changes: 1 addition & 1 deletion configuration/routing-examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Label reduces complex `tag` handling by separating data pipelines.

## Reroute Event by Tag

Use [fluent-plugin-route](https://github.com/tagomoris/fluent-plugin-route) plugin. This plugin rewrites `tag` and re-emit events to other `match` or Label.
Use [fluent-plugin-route](https://github.com/tagomoris/fluent-plugin-route) plugin. This plugin rewrites `tag` and re-emits events to other `match` or Label.

```text
<match worker.**>
Expand Down
2 changes: 1 addition & 1 deletion configuration/transport-section.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ The max size of socket receive buffer for TCP/UDP. This is used in `SO_RCVBUF` s
* Default: `false`
* Version: 1.18.0
* Specifies whether it must use FIPS mode with OpenSSL. If `true`, Fluentd will check FIPS mode is supported in your environment, if not, just aborts.
If `false`, it does nothing and don't care FIPS mode with OpenSSL.
If `false`, it does nothing and doesn't care about FIPS mode with OpenSSL.

If you want to accept multiple TLS protocols, use `min_version`/`max_version` instead of `version`. To support the old style, fluentd accepts `TLS1_1` and `TLSv1_1` values.

Expand Down
2 changes: 1 addition & 1 deletion container-deployment/docker-compose.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This article explains how to collect [Docker](https://www.docker.com/) logs and

[Elasticsearch](https://www.elastic.co/products/elasticsearch) had been an open-source search engine known for its ease of use. [Kibana](https://www.elastic.co/products/kibana) had been an open-source Web UI that makes Elasticsearch user-friendly for marketers, engineers and data scientists alike.

NOTE: Since v7.11, These products are distributed under non open-source license (Dual licensed under Server Side Public License and Elastic License)
NOTE: Since v7.11, These products are distributed under a non open-source license (Dual licensed under Server Side Public License and Elastic License)


By combining these three tools EFK \(Elasticsearch + Fluentd + Kibana\) we get a scalable, flexible, easy to use log collection and analytics pipeline. In this article, we will set up four \(4\) containers, each includes:
Expand Down
10 changes: 5 additions & 5 deletions container-deployment/docker-logging-driver.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Docker Logging Driver

The article describes how to implement a unified logging system for your [Docker](http://www.docker.com) containers. An application in a production environment requires to register certain events or problems during its runtime.
The article describes how to implement a unified logging system for your [Docker](http://www.docker.com) containers. An application in a production environment needs to register certain events or problems during its runtime.

The old-fashioned way is to write these messages into a log file, but that inherits certain problems. Specifically, when we try to perform some analysis over the registers, or on the other hand, if the application has multiple instances running, the scenario becomes even more complex.

Expand Down Expand Up @@ -130,7 +130,7 @@ Then you provide the log message with JSON format:
$ docker run --log-driver=fluentd --log-opt tag=docker ubuntu echo "{\"key\":\"value\"}"
```

About `--log-opt tag=...`, please refer at [Driver Options](#driver-options) section.
About `--log-opt tag=...`, please refer to the [Driver Options](#driver-options) section.

Original Event (without filter plugin):

Expand All @@ -148,7 +148,7 @@ Filtered Event:

The application log is stored in the `log` field of the record. You can concatenate these logs by using [`fluent-plugin-concat`](https://github.com/fluent-plugins-nursery/fluent-plugin-concat) filter before sending it to the destinations.

At first, you need to create custom docker image due to install the `fluent-plugin-concat` gem in the Fluentd container.
At first, you need to create a custom docker image to install the `fluent-plugin-concat` gem in the Fluentd container.

Create `Dockerfile` with the following content:

Expand Down Expand Up @@ -197,7 +197,7 @@ Launch the Fluentd container:
$ docker run -it -p 24224:24224 -v $(pwd)/demo.conf:/fluentd/etc/demo.conf -e FLUENTD_CONF=demo.conf fluentd-test
```

Then you provide the log message contains newlines:
Then you provide the log message containing newlines:

```text
$ docker run --log-driver=fluentd --log-opt tag=docker ubuntu echo "-e:2:in \`/'"$'\n'"-e:2:in \`do_division_by_zero'"$'\n'"-e:4:in \`<main>'"
Expand Down Expand Up @@ -262,7 +262,7 @@ $ docker run --log-driver=fluentd --log-opt fluentd-address=192.168.2.4:24225 ub
$ docker run --log-driver=fluentd --log-opt tag=docker.my_new_tag ubuntu echo "..."
```

Additionally, this option allows to specify some internal variables such as `{{.ID}}`, `{{.FullID}}` or `{{.Name}}` like this:
Additionally, this option allows you to specify some internal variables such as `{{.ID}}`, `{{.FullID}}` or `{{.Name}}` like this:

```text
$ docker run --log-driver=fluentd --log-opt tag=docker.{{.ID}} ubuntu echo "..."
Expand Down
8 changes: 4 additions & 4 deletions container-deployment/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Before getting started, make sure you understand or have a basic idea about the

> A DaemonSet ensures that all \(or some\) nodes run a copy of a pod. As nodes are added to the cluster, pods are added to them. As nodes are removed from the cluster, those pods are garbage collected. Deleting a DaemonSet will clean up the pods it created...

Since applications runs in Pods and multiple Pods might exists across multiple nodes, we need a specific Fluentd-Pod that takes care of log collection on each node: Fluentd DaemonSet.
Since applications run in Pods and multiple Pods might exist across multiple nodes, we need a specific Fluentd-Pod that takes care of log collection on each node: Fluentd DaemonSet.

## Fluentd DaemonSet

Expand All @@ -41,7 +41,7 @@ The following steps will focus on sending the logs to an Elasticsearch Pod:

### Get Fluentd DaemonSet sources

We have created a Fluentd DaemonSet that have the proper rules and container image ready to get started:
We have created a Fluentd DaemonSet that has the proper rules and container image ready to get started:

* [https://github.com/fluent/fluentd-kubernetes-daemonset](https://github.com/fluent/fluentd-kubernetes-daemonset)

Expand All @@ -53,7 +53,7 @@ $ git clone https://github.com/fluent/fluentd-kubernetes-daemonset

### DaemonSet Content

The cloned repository contains several configurations that allow to deploy Fluentd as a DaemonSet. The Docker container image distributed on the repository also comes pre-configured so that Fluentd can gather all the logs from the Kubernetes node's environment and append the proper metadata to the logs.
The cloned repository contains several configurations that allow you to deploy Fluentd as a DaemonSet. The Docker container image distributed on the repository also comes pre-configured so that Fluentd can gather all the logs from the Kubernetes node's environment and append the proper metadata to the logs.

This repository has several presets for Alpine/Debian with popular outputs:

Expand Down Expand Up @@ -100,7 +100,7 @@ This YAML file contains two relevant environment variables that are used by Flue
| :--- | :--- | :---: |
| `FLUENT_ELASTICSEARCH_HOST` | Specify the host name or IP address. | `elasticsearch-logging` |
| `FLUENT_ELASTICSEARCH_PORT` | Elasticsearch TCP port | 9200 |
| `FLUENT_ELASTICSEARCH_SSL_VERIFY` | Whether verify SSL certificates or not. | `true` |
| `FLUENT_ELASTICSEARCH_SSL_VERIFY` | Whether to verify SSL certificates or not. | `true` |
| `FLUENT_ELASTICSEARCH_SSL_VERSION` | Specify the version of TLS. | `TLSv1_2` |

Any relevant change needs to be done in the YAML file before deployment. The defaults assume that at least one Elasticsearch Pod **elasticsearch-logging** exists in the cluster.
Expand Down
2 changes: 1 addition & 1 deletion deployment/command-line-option.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Command Line Option

This article describes the command-line tools and its options in `fluentd` project.
This article describes the command-line tools and their options in `fluentd` project.

## `fluentd`

Expand Down
2 changes: 1 addition & 1 deletion deployment/fluentd-ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
* View Fluentd log with simple error viewer

{% hint style='danger' %}
`fluentd-ui` does not maintained anymore. It won't work with `fluent-package`. This content is for v0.12 for now.
`fluentd-ui` is not maintained anymore. It won't work with `fluent-package`. This content is for v0.12 for now.
{% endhint %}

## Enterprise
Expand Down
2 changes: 1 addition & 1 deletion deployment/high-availability.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ To configure Fluentd for high-availability, we assume that your network consists

![Fluentd&apos;s High-Availability Overview](../.gitbook/assets/fluentd_ha%20%281%29%20%281%29%20%281%29.png)

'**Log forwarders**' are typically installed on every node to receive local events. Once an event is received, they forward it to the 'log aggregators' through the network. For log forwarders, [fluent-bit](https://fluentbit.io/) is also good candidate for light-weight processing.
'**Log forwarders**' are typically installed on every node to receive local events. Once an event is received, they forward it to the 'log aggregators' through the network. For log forwarders, [fluent-bit](https://fluentbit.io/) is also a good candidate for light-weight processing.

'**Log aggregators**' are daemons that continuously receive events from the log forwarders. They buffer the events and periodically upload the data into the cloud.

Expand Down
4 changes: 2 additions & 2 deletions deployment/linux-capability.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Linux capabilities grant privileges to processes and executables that are otherw

Fluentd uses the [`capng_c` gem](https://github.com/fluent-plugins-nursery/capng_c) to handle Linux capabilities.

Add this line to your Fluentd' or td-agent's Gemfile:
Add this line to your Fluentd's or td-agent's Gemfile:

```ruby
gem 'capng_c'
Expand Down Expand Up @@ -183,7 +183,7 @@ If this article is incorrect or outdated, or omits critical information, please

## Capability handling on docker container
If you would like to collect logs from a file as a non-root user, you can use `CAP_DAC_READ_SEARCH` Linux capabilities.
However, `CAP_DAC_READ_SEARCH` now cannot be used on docker container by default.
However, `CAP_DAC_READ_SEARCH` now cannot be used on a docker container by default.

When using `CAP_DAC_READ_SEARCH` in a Docker container, you need to add the `--cap-add DAC_READ_SEARCH` option to the `docker run` command.
Or, if you are using `docker compose`, you need to add `cap_add` to the service definition.
Expand Down
6 changes: 3 additions & 3 deletions deployment/logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ If you do not specify the `@log_level` parameter, the plugin will use the global

## Log Format

Following format are supported:
Following formats are supported:

* `text` \(default\)
* `json`
Expand Down Expand Up @@ -122,7 +122,7 @@ Fluentd provides two parameters to suppress log/stacktrace messages
</system>
```

Under high loaded environment, output destination sometimes becomes unstable and it causes lots of same log message. This parameter mitigates such situation.
Under high loaded environment, output destination sometimes becomes unstable and it causes lots of same log messages. This parameter mitigates such situation.

### `ignore_same_log_interval`

Expand Down Expand Up @@ -158,7 +158,7 @@ $ fluentd -o /path/to/log_file

### By Config File

Since v1.18.0, You can also configure the log file path using the `<log>` directive under `<system>`:
Since v1.18.0, you can also configure the log file path using the `<log>` directive under `<system>`:

```text
<system>
Expand Down
4 changes: 2 additions & 2 deletions deployment/multi-process-workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This feature can simply replace `fluent-plugin-multiprocess`.

By default, one instance of `fluentd` launches a supervisor and a worker. A worker consists of input/filter/output plugins.

The **multi-process workers** feature launches multiple workers and use a separate process per worker. `fluentd` provides several features for multi-process workers.
The **multi-process workers** feature launches multiple workers and uses a separate process per worker. `fluentd` provides several features for multi-process workers.

![Multi-process Workers](../.gitbook/assets/multi-process-workers%20%281%29%20%281%29.png)

Expand Down Expand Up @@ -211,7 +211,7 @@ You may see following error in the fluentd logs:
2018-10-01 10:00:00 +0900 [error]: config error file="/path/to/fluentd.conf" error_class=Fluent::ConfigError error="Plugin 'tail' does not support multi workers configuration (Fluent::Plugin::TailInput)"
```

This means that the configured plugin does not support multi-process workers. All configured plugins must support multi-process workers. See "Multi-Process Worker and Plugins" section above.
This means that the configured plugin does not support multi-process workers. All configured plugins must support multi-process workers. See "Multi-Process Workers and Plugins" section above.

If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

6 changes: 3 additions & 3 deletions deployment/plugin-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ Any fluentd plugin can unknowingly break fluentd completely (and possibly break
There is no way to block this kind of situation. This is because the problem itself is derived from plug-in mechanism, and that's Lightweight Language.
One solution is "Do not use unreliable plugins".

Generally speaking, plug-in mechanism can break core functionality not only Fluentd but also in most other software.
Generally speaking, plug-in mechanism can break core functionality not only in Fluentd but also in most other software.
We shouldn't use unreliable plugins in any software.

We recommend to send feedback to plugin owner if you faced such a fault in plugins.
We recommend sending feedback to plugin owner if you faced such a fault in plugins.

### If Using `td-agent`, Use `/usr/sbin/td-agent-gem`

Expand Down Expand Up @@ -80,7 +80,7 @@ For `td-agent`, Fluentd uses the `/etc/td-agent/plugin` directory instead of `/e

Fluentd and plugins are evolving so you may hit an unexpected error with the latest version e.g. regression by a new feature, remove a deprecated parameter, change library dependency, etc. To avoid these problems, we recommend fixing fluentd and plugin version on production. If you want to update fluentd or plugins, check the behavior first on your test environment. For example, td-agent fixes fluentd and plugins version in each release.

Fluentd plugins are RubyGems and RubyGems installs the latest version by default. So we don't recommend to execute following commands on production:
Fluentd plugins are RubyGems and RubyGems installs the latest version by default. So we don't recommend executing the following commands on production:

* `gem install fluentd`
* `gem install fluent-plugin-elasticsearch`
Expand Down
2 changes: 1 addition & 1 deletion deployment/signals.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ See [Zero-downtime restart](zero-downtime-restart.md) for details.

**Comparison with SIGHUP**

`SIGHUP` gracefully restarting the worker process to reload.
`SIGHUP` gracefully restarts the worker process to reload.

This method does not cause socket downtime, so if there is no need to restart the supervisor, `SIGHUP` is a lighter zero-downtime restart method.

Expand Down
Loading