feat(outputs.health): Add max time between metrics check #16646

tulku · 2025-03-17T09:34:32Z

Summary

This adds a new check to the health output plugin that can detect moments when no metrics are flowing through telegraf. For example, when impacted by issues like #16070

This is disabled by default, and can be enabled using the new configuration options:

[[outputs.health]]
  service_address = "http://:8080"
  max_time_between_metrics = "1s"
  unhealthy_if_metrics_are_delayed = true

Besides the included unittests, this change was validated running telegraf with the following configuration:

[[inputs.internal]]
  interval = "20s"

[[outputs.health]]
  service_address = "http://:8080"
  max_time_between_metrics = "1s"
  unhealthy_if_metrics_are_delayed = true

And querying the health status using:

while true; do curl -X GET localhost:8080; sleep 0.01; done

Telegraf starts healthy.
Once 1s passes after the first internal metrics are written, the status changes to unhealthy
After that, every 20s, it reports OK for 1s and then goes back to unhealthy.

This PR replaces the previous implementation attempt done in #16212
For more context, refer to this slack thread

Checklist

No AI generated code was used in this PR

Related issues

resolves #16645

srebhan

Thanks @tulku for your contribution! I think the idea is nice. Some suggestions for simplification in the code. The tests need to be adapted accordingly...

plugins/outputs/health/README.md

plugins/outputs/health/health.go

srebhan · 2025-03-20T15:28:22Z

plugins/outputs/health/health.go

+	if h.UnhealthyIfMetricsAreDelayed && h.firstMetricReceived {
+		healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)
+	}


Suggested change

if h.UnhealthyIfMetricsAreDelayed && h.firstMetricReceived {

healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)

}

if h.MaxTimeBetweenMetrics > 0 {

healthy = healthy && time.Since(h.lastMetricTime) < time.Duration(h.MaxTimeBetweenMetrics)

}

Here I added a check for the lastMetricTime being non zero also, so the status is healthy before the first metrics are written.

Hmmm shouldn't we rather set the h.lastMetricTime to now in Connect as otherwise the output stays healthy if no metric arrives at all...

I want the output to be healthy if no metric is received. This is because I want to avoid the case where telegraf is restarted (because of being unhealthy) on startup.

with the last change, lastMetricTime is set to now in Connect.

srebhan · 2025-03-20T15:29:23Z

plugins/outputs/health/health.go

+	h.firstMetricReceived = true
+	h.lastMetricTime = time.Now()


How about initializing the lastMetricTime in Connect?

This is because Connect is guaranteed to be called before the first Write, correct?

I left the initialization in NewHealth (which is called from init). If Connect is preferred, I can move it there.

Yes, Connect would be better. The init function is called when importing this plugin so you might get timeouts if setting up Telegraf takes long. Connect is called after Init but before the first Write and is the better place I think.

With the latest change, lastMetricTime is initialized as 0 and it is only updated in Write. Whe can add an initialization to now on Connect. This will help catch cases where telegraf starts but no metric is written before the max time elapses.
The current behaviour waits for the first metric to be written to start checking if the time between metrics is longer than expected. This was by design, to prevent the health output to fail during startup. However, if Connect is called 'close enough' to when everything should already be working, we can set lastMetricTime = now in connect also.

with the last change, lastMetricTime is set to now in Connect.

plugins/outputs/health/health.go

plugins/outputs/health/sample.conf

tulku · 2025-03-27T17:02:41Z

Thanks for the review @srebhan ! The code is now simpler, removing the two boolean flags. I took the liberty to initialize lastMetricTime in NewHealth instead of Connect. Let me know if this is a problem.

srebhan

Thanks @tulku! One more comment... Please also check my responses to the unresolved previous comments...

srebhan · 2025-04-09T14:13:17Z

plugins/outputs/health/health.go

+		MaxTimeBetweenMetrics: 0,
+		healthy:               true,
+		lastMetricTime:        time.Time{},


You don't need to initialize the two fields as Golang always uses the nil value when creating new instances. So go with

Suggested change

MaxTimeBetweenMetrics: 0,

healthy: true,

lastMetricTime: time.Time{},

healthy: true,

srebhan · 2025-04-09T14:14:59Z

@tulku please also run make lint on your machine and fix the linter issues...

srebhan

Thanks @tulku! Please set the initial h.lastMetricTime in Connect() to avoid cases where no metric arrives at all and the plugin is in healthy state.

plugins/outputs/health/health.go

telegraf-tiger · 2025-04-10T15:09:37Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

srebhan

Thanks a lot @tulku!

skartikey

@tulku Thanks for your contribution!

telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Mar 17, 2025

tulku force-pushed the feat/health_timeout_check branch from 4df75f2 to f83e744 Compare March 17, 2025 09:55

srebhan reviewed Mar 20, 2025

View reviewed changes

srebhan changed the title ~~feat(health): Add max time between metrics check~~ feat(outputs.health): Add max time between metrics check Mar 20, 2025

telegraf-tiger bot added the plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins label Mar 20, 2025

srebhan self-assigned this Mar 20, 2025

tulku force-pushed the feat/health_timeout_check branch from f83e744 to d5dc1a1 Compare March 27, 2025 16:57

tulku requested a review from srebhan March 27, 2025 17:02

tulku force-pushed the feat/health_timeout_check branch from d5dc1a1 to c42dcf0 Compare April 8, 2025 16:15

srebhan reviewed Apr 9, 2025

View reviewed changes

tulku force-pushed the feat/health_timeout_check branch from c42dcf0 to cf1f78d Compare April 9, 2025 14:55

tulku requested a review from srebhan April 10, 2025 08:58

srebhan reviewed Apr 10, 2025

View reviewed changes

plugins/outputs/health/health.go Outdated Show resolved Hide resolved

plugins/outputs/health/health.go Show resolved Hide resolved

tulku force-pushed the feat/health_timeout_check branch from cf1f78d to 38cfe92 Compare April 10, 2025 14:04

tulku requested a review from srebhan April 10, 2025 14:38

feat(health): Add max time between metrics check

8d25ea4

tulku force-pushed the feat/health_timeout_check branch from 38cfe92 to 8d25ea4 Compare April 10, 2025 14:39

srebhan approved these changes Apr 14, 2025

View reviewed changes

srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Apr 14, 2025

srebhan assigned skartikey and mstrandboge and unassigned srebhan Apr 14, 2025

skartikey approved these changes Apr 14, 2025

View reviewed changes

skartikey merged commit be9e5bf into influxdata:master Apr 14, 2025
27 checks passed

github-actions bot added this to the v1.35.0 milestone Apr 14, 2025

feat(outputs.health): Add max time between metrics check #16646

feat(outputs.health): Add max time between metrics check #16646

Uh oh!

Conversation

tulku commented Mar 17, 2025

Summary

Checklist

Related issues

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tulku commented Mar 27, 2025

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srebhan commented Apr 9, 2025

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

telegraf-tiger bot commented Apr 10, 2025

Artifact URLs

Uh oh!

srebhan left a comment

Choose a reason for hiding this comment

Uh oh!

skartikey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!