keeper Prometheus metrics #1
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
054cd12 to
2e5f4f1
Compare
2e5f4f1 to
cc9a680
Compare
lawrencejones
commented
Apr 30, 2019
| @@ -75,6 +76,20 @@ func AddCommonFlags(cmd *cobra.Command, cfg *CommonConfig) { | |||
| } | |||
| } | |||
|
|
|||
| var ( | |||
| clusterIdentifier = prometheus.NewGaugeVec( | |||
There was a problem hiding this comment.
This is the metric we (GoCardless) have been using to join various time series together. By having all components report this in a consistent fashion (see here for the stolon-pgbouncer definition) it becomes possible to join series from totally different process/infrastructures on the cluster_name and store_prefix labels.
lawrencejones
commented
Apr 30, 2019
| @@ -1391,6 +1482,10 @@ func (p *PostgresKeeper) postgresKeeperSM(pctx context.Context) { | |||
| targetRole := db.Spec.Role | |||
| log.Debugw("target role", "targetRole", string(targetRole)) | |||
|
|
|||
| // Set metrics to power alerts about mismatched roles | |||
There was a problem hiding this comment.
These metrics are particularly useful to power an up-to-date view of keeper state. It looks like this in a dashboard:
lawrencejones
commented
Apr 30, 2019
| @@ -1770,6 +1877,7 @@ func (p *PostgresKeeper) generateHBA(cd *cluster.ClusterData, db *cluster.DB, on | |||
| func sigHandler(sigs chan os.Signal, cancel context.CancelFunc) { | |||
| s := <-sigs | |||
| log.Debugw("got signal", "signal", s) | |||
| shutdownSeconds.SetToCurrentTime() | |||
There was a problem hiding this comment.
Useful for detecting a pending shutdown.
lawrencejones
commented
Apr 30, 2019
| @@ -42,6 +42,13 @@ const ( | |||
| RoleStandby Role = "standby" | |||
| ) | |||
|
|
|||
| // Roles enumerates all possible Role values | |||
There was a problem hiding this comment.
This list needs maintaining to ensure we can set all possible label values to 0 on stolon_keeper_{local,target}_role before setting our active metric to be 1. This becomes clearer when you see the output of a single instances metrics:
Add a collection of Prometheus metrics to the keeper. The metrics are aimed to expose errors in the keeper sync loop, providing enough visibility to detect when the sync is failing (and some insight into why).
cc9a680 to
b937766
Compare
lawrencejones
added a commit
to gocardless/stolon-pgbouncer
that referenced
this pull request
Apr 30, 2019
[^1]: gocardless/stolon#1 This commit is associated with an open PR [^1] to stolon that adds Prometheus metrics to the keeper. The changes here include adding a keeper dashboard that can visualise the keeper statuses and a couple of essential alerts for keeper health. We update the playground environment so developers can explore these metrics and make use of the dashboard. This includes a new docker image, which has been pushed to Docker hub.
lawrencejones
added a commit
that referenced
this pull request
May 9, 2019
keeper Prometheus metrics
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a collection of Prometheus metrics to the keeper. The metrics are
aimed to expose errors in the keeper sync loop, providing enough
visibility to detect when the sync is failing (and some insight into
why).
This commit can be tested in the stolon-pgbouncer setup gocardless/stolon-pgbouncer#29