Support debounce while watching configuration files and directories #17047
Description
Use Case
We have a large number of telegraf configuration files(~500) created programmatically and telegraf is running with --watch-config option turned on. Telegraf detects the file modifications, and gracefully sends a SIGHUP to reload its configuration. However, this is done for each event. Therefore, we have a situation where telegraf is getting reloaded 500 times! This situation is not optimal and we need a better way to tackle it.
Expected behavior
Telegraf can introduce a debounce where it keeps resetting the timer until it elapses and only then send a SIGHUP.
Actual behavior
Reloads on each event.
➜ telegraf git:(master) ✗ go run . --config-directory debounce --watch-config=notify --config agent.conf
2025-05-21T16:44:16Z I! Loading config: agent.conf
2025-05-21T16:44:16Z I! Loading config: debounce/file0.conf
2025-05-21T16:44:16Z I! Loading config: debounce/file1.conf
2025-05-21T16:44:16Z I! Loading config: debounce/file2.conf
2025-05-21T16:44:16Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:44:16Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:44:16Z I! Loaded inputs: mem (3x)
2025-05-21T16:44:16Z I! Loaded aggregators:
2025-05-21T16:44:16Z I! Loaded processors:
2025-05-21T16:44:16Z I! Loaded secretstores:
2025-05-21T16:44:16Z I! Loaded outputs: file (3x)
2025-05-21T16:44:16Z I! Tags enabled: host=hostname
2025-05-21T16:44:16Z I! Config watcher started for agent.conf
2025-05-21T16:44:16Z I! [agent] Config: Interval:120s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:44:16Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-05-21T16:44:16Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:44:16Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:44:16Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:44:16Z I! Config watcher started for debounceLets modify the config files now. We get 3+ reloads in telegraf.
2025-05-21T16:44:36Z I! Config file/directory "debounce" modified
2025-05-21T16:44:36Z I! Reloading Telegraf config
2025-05-21T16:44:36Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-05-21T16:44:36Z I! [agent] Stopping running outputs
2025-05-21T16:44:36Z I! Loading config: agent.conf
2025-05-21T16:44:36Z I! Config watcher started for agent.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:44:36Z I! Loading config: debounce/file0.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce
2025-05-21T16:44:36Z I! Loading config: debounce/file1.conf
2025-05-21T16:44:36Z I! Loading config: debounce/file2.conf
2025-05-21T16:44:36Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:44:36Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:44:36Z I! Loaded inputs: mem (3x)
2025-05-21T16:44:36Z I! Loaded aggregators:
2025-05-21T16:44:36Z I! Loaded processors:
2025-05-21T16:44:36Z I! Loaded secretstores:
2025-05-21T16:44:36Z I! Loaded outputs: file (3x)
2025-05-21T16:44:36Z I! Tags enabled: host=hostname
2025-05-21T16:44:36Z I! [agent] Config: Interval:120s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:44:36Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!2025-05-21T16:44:36Z I! Config file/directory "debounce" modified
2025-05-21T16:44:36Z I! Reloading Telegraf config
2025-05-21T16:44:36Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-05-21T16:44:36Z I! [agent] Stopping running outputs
2025-05-21T16:44:36Z I! Config watcher started for agent.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:44:36Z I! Loading config: agent.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce
2025-05-21T16:44:36Z I! Loading config: debounce/file0.conf
2025-05-21T16:44:36Z I! Loading config: debounce/file1.conf
2025-05-21T16:44:36Z I! Loading config: debounce/file2.conf
2025-05-21T16:44:36Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:44:36Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:44:36Z I! Loaded inputs: mem (3x)
2025-05-21T16:44:36Z I! Loaded aggregators:
2025-05-21T16:44:36Z I! Loaded processors:
2025-05-21T16:44:36Z I! Loaded secretstores:
2025-05-21T16:44:36Z I! Loaded outputs: file (3x)
2025-05-21T16:44:36Z I! Tags enabled: host=hostname
2025-05-21T16:44:36Z I! [agent] Config: Interval:120s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:44:36Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!2025-05-21T16:44:36Z I! Config file/directory "debounce" modified
2025-05-21T16:44:36Z I! Reloading Telegraf config
2025-05-21T16:44:36Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-05-21T16:44:36Z I! [agent] Stopping running outputs
2025-05-21T16:44:36Z I! Config watcher started for agent.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:44:36Z I! Loading config: agent.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:44:36Z I! Config watcher started for debounce
2025-05-21T16:44:36Z I! Loading config: debounce/file0.conf
2025-05-21T16:44:36Z I! Loading config: debounce/file1.conf
2025-05-21T16:44:36Z I! Loading config: debounce/file2.conf
2025-05-21T16:44:36Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:44:36Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:44:36Z I! Loaded inputs: mem (3x)
2025-05-21T16:44:36Z I! Loaded aggregators:
2025-05-21T16:44:36Z I! Loaded processors:
2025-05-21T16:44:36Z I! Loaded secretstores:
2025-05-21T16:44:36Z I! Loaded outputs: file (3x)
2025-05-21T16:44:36Z I! Tags enabled: host=hostname
2025-05-21T16:44:36Z I! [agent] Config: Interval:120s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:44:36Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!Additional info
No response