Add host health metrics gauge #7728
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
laniehei
commented
May 8, 2025
| HEALTH_STATE_NOT_SERVING = 2; | ||
| HEALTH_STATE_DECLINED_SERVING = 3; | ||
| // The host is unhealthy through external observation. | ||
| HEALTH_STATE_NOT_SERVING = 2; |
There was a problem hiding this comment.
This change didn't get in to the previous PR somehow.
yux0
reviewed
May 8, 2025
| @@ -202,15 +202,18 @@ func (h *Handler) DeepHealthCheck( | |||
| return nil, err | |||
| } | |||
| if status.Status != healthpb.HealthCheckResponse_SERVING { | |||
| metrics.HistoryHostHealthGauge.With(h.metricsHandler).Record(float64(enumsspb.HEALTH_STATE_DECLINED_SERVING)) | |||
There was a problem hiding this comment.
yes! Because this is on the handler, metrics get scraped by prometheus on per-cell and per-host information. We'll be able to create a graph with this on per-cluster per-host checks
yux0
approved these changes
May 12, 2025
josesa
added a commit
to josesa/temporal
that referenced
this pull request
May 12, 2025
* main: (22 commits) Add host health metrics gauge (temporalio#7728) add rule expiration check (temporalio#7749) Add activity options to the pending activity info (temporalio#7727) Enable DLQ V2 for replication (temporalio#7746) chore: be smarter about when to use Stringer vs String (temporalio#7743) versioning entity workflows: enabling auto-restart pt1 (temporalio#7715) Refactor code generators (temporalio#7734) allow passive to generate replication tasks (temporalio#7713) Validate links in completion callbacks (temporalio#7726) CHASM: Engine Update/ReadComponent implementation (temporalio#7696) Enable transition history in dev env and tests (temporalio#7737) chore: Add Stringer tags (temporalio#7738) Add internal pod health check to DeepHealthCheck (temporalio#7709) Rename internal CHASM task processing interface (temporalio#7730) [Frontend] Log slow gRPC requests (temporalio#7718) Remove cap for dynamic config callback pool (temporalio#7723) Refactor updateworkflowoptions package (temporalio#7725) Remove a bunch of redundant utf-8 validation (temporalio#7720) [CHASM] Pure task processing - GetPureTasks, ExecutePureTasks (temporalio#7701) Send ActivityReset flag to the worker in heartbeat response (temporalio#7677) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
Add a new metrics gauge to track host health
Why?
Visibility into which hosts are failing when
How did you test it?
Potential risks
Adding Sean to the PR to make sure I didn't miss anything obvious in regards to how we want to display this.