Skip to content

feat(agent): Add debounce for watch events #17048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 16, 2025
Merged

feat(agent): Add debounce for watch events #17048

merged 3 commits into from
Jun 16, 2025

Conversation

neelayu
Copy link
Contributor

@neelayu neelayu commented May 21, 2025 •
edited by skartikey
Loading

Summary

This PR implements a debounce mechanism for configuration file watching that batches multiple file changes into a single reload operation.

Changes Made

  • Added --watch-debounce-interval CLI flag to configure the debounce duration (default: 0s for backward compatibility)
  • Implemented timer-based debouncing in watchLocalConfig() function that:
    • Starts a timer when the first file change is detected
    • Resets the timer if additional changes occur during the wait period
    • Only triggers reload (SIGHUP) after the timer expires without interruption
  • Enhanced state tracking with reloadPending flag to prevent unnecessary operations
  • Applied debouncing to all file events: Modified, Deleted, Truncated, and Created
  • Proper resource cleanup with timer management in all exit paths

Behavior Change

  • Before: 500 config file changes → 500 separate reloads
  • After: 500 config file changes → 1 batched reload (when debounce interval > 0)
  • Backward Compatible: Default behavior unchanged (0s debounce = immediate reload)

Usage Example

# Enable debouncing with 5-second wait period
telegraf --config-directory /configs --watch-config=notify --watch-debounce-interval=5s

Checklist

  • No AI generated code was used in this PR

Related issues

resolves #17047

➜  telegraf git:(debounce-reload) ✗ go run . --config-directory debounce --watch-config=notify --config agent.conf --watch-debounce-interval=10s
2025-05-21T16:52:12Z I! Loading config: agent.conf
2025-05-21T16:52:12Z I! Loading config: debounce/file0.conf
2025-05-21T16:52:12Z I! Loading config: debounce/file1.conf
2025-05-21T16:52:12Z I! Loading config: debounce/file2.conf
2025-05-21T16:52:12Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:52:12Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:52:12Z I! Loaded inputs: mem (3x)
2025-05-21T16:52:12Z I! Loaded aggregators:
2025-05-21T16:52:12Z I! Loaded processors:
2025-05-21T16:52:12Z I! Loaded secretstores:
2025-05-21T16:52:12Z I! Loaded outputs: file (3x)
2025-05-21T16:52:12Z I! Tags enabled: host=hostname
2025-05-21T16:52:12Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:52:12Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-05-21T16:52:12Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:52:12Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:52:12Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:52:12Z I! Config watcher started for agent.conf
2025-05-21T16:52:12Z I! Config watcher started for debounce
2025-05-21T16:52:28Z I! Config file/directory "debounce" modified
2025-05-21T16:52:28Z I! Config file/directory "debounce" modified
2025-05-21T16:52:28Z I! Config file/directory "debounce" modified
2025-05-21T16:52:38Z I! Debounce period elapsed, triggering config reload for "debounce"
2025-05-21T16:52:38Z I! Reloading Telegraf config
2025-05-21T16:52:38Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-05-21T16:52:38Z I! [agent] Stopping running outputs
2025-05-21T16:52:38Z I! Config watcher started for agent.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:52:38Z I! Loading config: agent.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce
2025-05-21T16:52:38Z I! Loading config: debounce/file0.conf
2025-05-21T16:52:38Z I! Loading config: debounce/file1.conf
2025-05-21T16:52:38Z I! Loading config: debounce/file2.conf
2025-05-21T16:52:38Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:52:38Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:52:38Z I! Loaded inputs: mem (3x)
2025-05-21T16:52:38Z I! Loaded aggregators:
2025-05-21T16:52:38Z I! Loaded processors:
2025-05-21T16:52:38Z I! Loaded secretstores:
2025-05-21T16:52:38Z I! Loaded outputs: file (3x)
2025-05-21T16:52:38Z I! Tags enabled: host=hostname
2025-05-21T16:52:38Z I! [agent] Config: Interval:120s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:52:38Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!

Essentially, the config watcher will wait for 10 seconds after the last change to the config directory before reloading the config. This is useful when you have multiple files in a directory and you want to avoid reloading the config multiple times if they are changed in quick succession.

@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label May 21, 2025
skartikey
skartikey reviewed May 23, 2025
View reviewed changes
@telegraf-tiger
Copy link
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

👍 This pull request doesn't change the Telegraf binary size

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

@skartikey
Copy link
Contributor

@neelayu Could you please rebase your branch on top of master? The changes to tls.go are already present in the master branch.

@skartikey skartikey self-assigned this May 28, 2025
neelayu and others added 3 commits May 28, 2025 21:04
@neelayu neelayu force-pushed the debounce-reload branch from 687c44b to aa9f773 Compare May 28, 2025 15:34
@neelayu
Copy link
Contributor Author

neelayu commented May 28, 2025

Yes. I think I added it to check if tests were passing or not. Rebased now

skartikey
skartikey approved these changes Jun 13, 2025
View reviewed changes
Copy link
Contributor

@skartikey skartikey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neelayu Nice work!

@skartikey skartikey added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Jun 13, 2025
@skartikey skartikey assigned srebhan and mstrandboge and unassigned skartikey Jun 13, 2025
srebhan
srebhan approved these changes Jun 16, 2025
View reviewed changes
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @neelayu!

@srebhan srebhan merged commit f432a16 into influxdata:master Jun 16, 2025
23 of 24 checks passed
@github-actions github-actions bot added this to the v1.35.0 milestone Jun 16, 2025
@neelayu
Copy link
Contributor Author

neelayu commented Jun 16, 2025

Thank you @skartikey
Had a lot of learnings in this PR.

@neelayu neelayu mentioned this pull request Jul 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skartikey skartikey skartikey approved these changes

@srebhan srebhan srebhan approved these changes

Assignees

@srebhan srebhan

@mstrandboge mstrandboge

Labels
area/agent feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Milestone
v1.35.0
Development

Successfully merging this pull request may close these issues.

Support debounce while watching configuration files and directories
4 participants
@neelayu @skartikey @srebhan @mstrandboge