sentinel: Add Prometheus metrics #656
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is what the metrics look like in action, with 3 sentinels running: |
sgotti
requested changes
Jun 3, 2019
cmd/sentinel/cmd/metrics.go
Outdated
| @@ -0,0 +1,73 @@ | |||
| // Copyright 2017 Sorint.lab | |||
|
@benwh Can you please squash in a single commit? |
These metrics provide the ability to build alerts that tell us whether the sentinels are operating as expected: - The last time that the sentinel successfully processed the clusterdata. - Whether the sentinel is a leader. - The number of times that the sentinel has been elected leader.
|
@sgotti I'd originally kept it separate as it was also changing a file outside of the scope of this PR. But happy to do so, squashed now. |
|
@benwh Oh I haven't noticed that you also changed the keeper metrics file. Anyway it's not a big styling issue. I'm going to merge it. Thanks again! |
|
Excellent, thanks very much for the speedy review! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These metrics provide the ability to build alerts that tell us whether the sentinels are operating as expected:
This follows on from the keeper metrics in #639.
We've found these sentinel metrics to be extremely useful, upon discovering that our sentinels could occasionally all become stuck - and no decisions would be made across the cluster - when there are issues communicating with etcd (we're planning to upstream a patch for that issue in the near future!)