From bef9b6fa2c1ae29808fcbfe197c6eb8baf452f39 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 29 Oct 2026 08:51:40 -0500 Subject: [PATCH 1/2] Documented cf-reactor watch methods --- .../reference/components/cf-reactor.markdown | 55 ++++++++++++++++++- .../reference/promise-types/events.markdown | 44 +++++++++++++++ 2 files changed, 98 insertions(+), 1 deletion(-) diff --git a/content/reference/components/cf-reactor.markdown b/content/reference/components/cf-reactor.markdown index 76d44f55b..cc0fbf2ea 100644 --- a/content/reference/components/cf-reactor.markdown +++ b/content/reference/components/cf-reactor.markdown @@ -88,11 +88,64 @@ The daemon is then signaled to handle the event and run the bundles of the `then **Notes:** -- Event detection is based on polling, but `cf-reactor` does not use busy-waiting. +- Event detection is based on polling, sped up by [watch methods](#watch-methods), but `cf-reactor` does not use busy-waiting. Instead, the polling thread sleeps between polling cycles and remains dormant when no events are registered. - `cf-reactor` does not guarantee that an event will be handled immediately after it occurs. The delay depends on factors such as the polling interval, policy reloads, and slow code execution. However, `cf-reactor` guarantees that changes will be detected as fast as possible and the corresponding code executed, provided that execution does not fail. +- After a watch method wakes it up, `cf-reactor` waits 200 milliseconds before checking, so that a burst of changes is checked once. +- An event that fires more than 10 times in 60 seconds is delayed, starting at 10 seconds and doubling up to 5 minutes, until it calms down. + The changes it holds back are handled in one run when the delay ends. + Meanwhile, and while its bundles run, the event is not checked again: it is checked once when it can run. + +### Watch methods + +A watch method wakes an event's watcher up as soon as the operating system reports a change to the watched file or its directory, so that `cf-reactor` checks the event right away instead of at its next [`interval`][events#interval]. +Each event uses the first available method of its [`watch_methods`][events#watch_methods] attribute. +Methods only decide _when_ an event is checked, never _whether_ it happened, and polling continues underneath all of them. +Changing the method never changes which events fire, only how soon. + +- `ebpf` (Linux, Enterprise) loads a small program into the kernel that matches changes against the watched files' inodes before telling `cf-reactor`. + It needs kernel 5.8 or later (or the RHEL 8 kernel) with BTF, and `CAP_SYS_ADMIN` (or `CAP_BPF` and `CAP_PERFMON`). + It has no per-directory watch limits, and watches up to 65,536 files; further events use their next watch method, and `cf-reactor` logs a warning. +- `fanotify` and `inotify` (Linux, Enterprise) watch each file for writes and changes of its permissions or owner, and its parent directory for files being created, deleted and renamed. + Files created and deleted in that directory are reported to `cf-reactor`, which ignores the ones for other files. + `inotify` is limited by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances`, and `fanotify` by `fs.fanotify.max_user_marks`. + When out of watches or marks, they watch files through their directories only, and `cf-reactor` logs a warning; a write through a file descriptor opened before the file was renamed into place is then only seen at the next interval. + Raise the limit to watch more than about half as many files as it allows. +- `usn` (Windows, Enterprise) reads the NTFS change journal, and `rdcw` (Windows, Enterprise) uses `ReadDirectoryChangesW`. +- `poll` only checks at the interval. It is the only method in CFEngine Community. + +**Cost:** away from watched files, no method adds measurable CPU time to the rest of the system. +In a directory that holds a watched file, each file operation costs about 1 µs more with `ebpf`, `fanotify` or `inotify` (about 10%), counting the process making the change and `cf-reactor`. +At 1,000 file operations per second in one watched directory this is about 0.1% of a CPU core, so the choice of method makes no practical difference to CPU time. +Where the methods differ: + +| Method | Wakes the watcher after a change | Kernel resources | +| ---------- | -------------------------------- | ------------------------------------------------------------------------ | +| `ebpf` | under 2 ms | up to 65,536 files, about 0.7 MB of locked memory for 1,000 | +| `fanotify` | about 20 ms | one mark per file and one per directory (`fs.fanotify.max_user_marks`) | +| `inotify` | about 20 ms | one watch per file and one per directory (`fs.inotify.max_user_watches`) | +| `poll` | at the next interval | none | + +`cf-reactor` then waits 200 milliseconds before checking, so in practice the native methods react equally fast. +`fanotify` and `inotify` gather the changes of 20 milliseconds into one wakeup. +Measured on Ubuntu 24.04 (kernel 6.8) in a virtual machine. + +`file_content_modified` hashes the whole file when its size or modification time changed. +For a large file that is written to all the time, such as a log, this costs about as much with every method, since the event is limited to 10 runs a minute (see above) and checked once per run: hashing a 200 MB file appended to 20 times a second took about 20% of a CPU core. +Use `file_modified` for such files, which only compares the modification time. + +**Changes only seen at the next interval:** + +- writes through a memory mapping (`mmap`) that stays open, with every method +- with `ebpf`, writes on file systems that don't report them to the kernel's generic write path (for example some network file systems) until the file is closed +- with every method, changes on network file systems made by other hosts + +**Notes:** + +- On CFEngine Enterprise, the method each event uses is reported to the hub, in the `reactor_watchers` table of the reporting database. +- SELinux or other mandatory access control may deny `cf-reactor` a method; it then falls back to the next one in the list. ## Enterprise hub tasks diff --git a/content/reference/promise-types/events.markdown b/content/reference/promise-types/events.markdown index f9c34b9d8..77c0f348f 100644 --- a/content/reference/promise-types/events.markdown +++ b/content/reference/promise-types/events.markdown @@ -119,6 +119,7 @@ bundle reactor main Each event has its own interval. A lower value means the event is noticed sooner but checked more often. +A [watch method](#watch_methods) can wake the check up earlier, so with one the interval mostly bounds how late a missed change is noticed. For how polling works, see [Under the hood][cf-reactor#Under the hood]. ```cf3 @@ -135,6 +136,49 @@ body when deleted_slow(file) **Default value:** `1` +#### watch_methods + +**Description:** The watch methods that may wake this event's watcher up before its next interval, in order of preference. + +A watch method is a notification from the operating system that the watched file, or its directory, changed. +It makes `cf-reactor` check the event right away instead of at the next interval. +It never decides whether the event happened: the check does, so the same events fire whatever the method. +Polling at the [`interval`](#interval) continues underneath every method, and catches what a method misses. + +`cf-reactor` uses the first method in the list that is available on the host and can watch the event's target. +Without `watch_methods`, it tries `ebpf`, `fanotify`, `inotify`, `usn`, `rdcw` and `poll`, in that order. +CFEngine Community only has `poll`. +If none of the listed methods is available, the event is not watched and `cf-reactor` logs an error; list `poll` last to fall back to polling instead. + +| Method | Platform | Edition | +| ---------- | -------- | ---------- | +| `ebpf` | Linux | Enterprise | +| `fanotify` | Linux | Enterprise | +| `inotify` | Linux | Enterprise | +| `usn` | Windows | Enterprise | +| `rdcw` | Windows | Enterprise | +| `poll` | All | All | + +For how the methods differ, see [Watch methods][cf-reactor#Watch methods]. + +```cf3 +body when passwd_modified +{ + file_content_modified => "/etc/passwd"; + interval => 300; + watch_methods => { "inotify", "poll" }; +} +``` + +**Type:** `slist` + +**Allowed input range:** `poll`, `inotify`, `fanotify`, `ebpf`, `usn`, `rdcw` + +**Notes:** + +- changing `watch_methods` does not reset the event: on policy reload, a watcher keeps its state when only its watch methods change. +- with `watch_methods => { "poll" }` the event is only checked every `interval` seconds. + #### file_deleted **Description:** Creates an event when the file at the given path is deleted. From e485fff92e1a3987a4942ecc47b426496bcec9f5 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Sat, 31 Oct 2026 13:10:31 -0500 Subject: [PATCH 2/2] Documented the Windows cf-reactor watch methods --- content/reference/components/cf-reactor.markdown | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/content/reference/components/cf-reactor.markdown b/content/reference/components/cf-reactor.markdown index cc0fbf2ea..58f8eebbe 100644 --- a/content/reference/components/cf-reactor.markdown +++ b/content/reference/components/cf-reactor.markdown @@ -113,7 +113,14 @@ Changing the method never changes which events fire, only how soon. `inotify` is limited by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances`, and `fanotify` by `fs.fanotify.max_user_marks`. When out of watches or marks, they watch files through their directories only, and `cf-reactor` logs a warning; a write through a file descriptor opened before the file was renamed into place is then only seen at the next interval. Raise the limit to watch more than about half as many files as it allows. -- `usn` (Windows, Enterprise) reads the NTFS change journal, and `rdcw` (Windows, Enterprise) uses `ReadDirectoryChangesW`. +- `usn` (Windows, Enterprise) reads the change journal of NTFS and ReFS volumes, waiting for new records as they are written. + It sees changes made while `cf-reactor` was stopped, and doesn't lose changes during bursts, so it comes before `rdcw`. + It needs administrator rights and a change journal on the volume; `cf-reactor` never creates one. + If the journal is deleted, recreated or overwritten past the last record read, every event on that volume is checked. +- `rdcw` (Windows, Enterprise) uses `ReadDirectoryChangesW` on the watched files' directories, and works on every local file system, including FAT32 and exFAT. + When its buffer overflows during a burst, the events of that directory are checked. + It isn't told when a watched directory, or one above it, is renamed or moved; `cf-reactor` notices within 5 seconds and watches the new location. + Changes made while `cf-reactor` was stopped are found by the first check after it starts. - `poll` only checks at the interval. It is the only method in CFEngine Community. **Cost:** away from watched files, no method adds measurable CPU time to the rest of the system. @@ -126,11 +133,13 @@ Where the methods differ: | `ebpf` | under 2 ms | up to 65,536 files, about 0.7 MB of locked memory for 1,000 | | `fanotify` | about 20 ms | one mark per file and one per directory (`fs.fanotify.max_user_marks`) | | `inotify` | about 20 ms | one watch per file and one per directory (`fs.inotify.max_user_watches`) | +| `usn` | under 2 ms | one reader per volume, none per file | +| `rdcw` | under 3 ms | one directory handle per watched directory | | `poll` | at the next interval | none | `cf-reactor` then waits 200 milliseconds before checking, so in practice the native methods react equally fast. `fanotify` and `inotify` gather the changes of 20 milliseconds into one wakeup. -Measured on Ubuntu 24.04 (kernel 6.8) in a virtual machine. +Measured on Ubuntu 24.04 (kernel 6.8) and Windows Server 2019, in virtual machines. `file_content_modified` hashes the whole file when its size or modification time changed. For a large file that is written to all the time, such as a log, this costs about as much with every method, since the event is limited to 10 runs a minute (see above) and checked once per run: hashing a 200 MB file appended to 20 times a second took about 20% of a CPU core.