Problem with promtheus, cadvisor and node exporter in docker swarm with no Leader role node #16596

ImanoLCarbonell · 2024-06-06T13:05:51Z

ImanoLCarbonell
Jun 6, 2024

What did you do?

Hello everyone !! I recently encountered the following problem with promtheus, cadvisor and node exporter on the docker swarm clusters we have.

I have Prometheus, Cadvisor and Node Exporter deployed in the clusters as services to collect information about the containers and hosts. This Prometheus is federated with another one that we have on another host to centralize all the metrics in one and in this way, have a single graph to be able to see them.

This architecture only works well for us if we deploy the prometheus, cadvisor and node exporter stack on the node that has the Leader role.
When, for any reason, this node loses the Leader role and therefore, the containers are deployed on another node, we stop receiving metrics from the cadvisor and node exporter of the new node that has the Leader role and lines appear in the Prometheus logs indicating that it cannot connect to the cadvisor and node exporter.

We do not understand the reason for this problem.

Has anyone ever had this happen to them? Do you know the cause of this problem?

In this scheme it works well.

ID                            HOSTNAME          STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION                                                                                            
8bov7bneixt8d8p7ug6hb76rz *   VMSMOUSWARMPRE1   Ready     Active         Leader           26.1.3                                                                                                    
kxzvj12733og33hjgb67rx6ks     VMSMOUSWARMPRE2   Ready     Active         Reachable        26.1.3                                                                                                    
15ka0dj36qim2unut5dg4ti4f     VMSMOUSWARMPRE3   Ready     Active         Reachable        26.1.3                                                                                                    
c07aqzyqgmo942330rpayzamn     VMSMOUSWARMPRE4   Ready     Active                          26.1.3

ID             NAME                                          MODE         REPLICAS   IMAGE                              PORTS                                                                       
wxtolvkuzovl   prometheus_federation_cadvisor                global       4/4        gcr.io/cadvisor/cadvisor:v0.49.1                                                                               
xhx9mpkjyvk3   prometheus_federation_node-exporter           global       4/4        prom/node-exporter:v1.8.1                                                                                      
ect9ph1lb6w7   prometheus_federation_prometheus-federation   replicated   1/1        prom/prometheus:v2.52.0            *:9090->9090/tcp

Not in this one

ID                            HOSTNAME          STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION                                                                                            
8bov7bneixt8d8p7ug6hb76rz *   VMSMOUSWARMPRE1   Ready     Active         Reachable        26.1.3                                                                                                    
kxzvj12733og33hjgb67rx6ks     VMSMOUSWARMPRE2   Ready     Active         Reachable        26.1.3                                                                                                    
15ka0dj36qim2unut5dg4ti4f     VMSMOUSWARMPRE3   Ready     Active         Leader           26.1.3                                                                                                    
c07aqzyqgmo942330rpayzamn     VMSMOUSWARMPRE4   Ready     Active                          26.1.3

What did you expect to see?

These are the metrics of Prometheus when it works well.

ID                            HOSTNAME          STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION                                                                                            
8bov7bneixt8d8p7ug6hb76rz *   VMSMOUSWARMPRE1   Ready     Active         Leader           26.1.3                                                                                                    
kxzvj12733og33hjgb67rx6ks     VMSMOUSWARMPRE2   Ready     Active         Reachable        26.1.3                                                                                                    
15ka0dj36qim2unut5dg4ti4f     VMSMOUSWARMPRE3   Ready     Active         Reachable        26.1.3                                                                                                    
c07aqzyqgmo942330rpayzamn     VMSMOUSWARMPRE4   Ready     Active                          26.1.3

What did you see instead? Under which circumstances?

These are the metrics of Prometheus when it does not work well when deployed with this scheme.

ID                            HOSTNAME          STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION                                                                                            
8bov7bneixt8d8p7ug6hb76rz *   VMSMOUSWARMPRE1   Ready     Active         Reachable        26.1.3                                                                                                    
kxzvj12733og33hjgb67rx6ks     VMSMOUSWARMPRE2   Ready     Active         Reachable        26.1.3                                                                                                    
15ka0dj36qim2unut5dg4ti4f     VMSMOUSWARMPRE3   Ready     Active         Leader           26.1.3                                                                                                    
c07aqzyqgmo942330rpayzamn     VMSMOUSWARMPRE4   Ready     Active                          26.1.3

System information

Linux 3.10.0-1160.118.1.el7.x86_64 x86_64

Prometheus version

prometheus, version 2.52.0 (branch: HEAD, revision: 879d80922a227c37df502e7315fad8ceb10a986d)                                                                                                       
  build user:       root@1b4f4c206e41                                                                                                                                                               
  build date:       20240508-21:56:43                                                                                                                                                               
  go version:       go1.22.3                                                                                                                                                                        
  platform:         linux/amd64                                                                                                                                                                     
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

Here are the Prometheus stack and configuration:

services:                                                                                                                                                                                           
                                                                                                                                                                                                    
  prometheus-federation:                                                                                                                                                                            
    image: prom/prometheus:v2.52.0                                                                                                                                                                  
    deploy:                                                                                                                                                                                         
      placement:                                                                                                                                                                                                                                                                                                                              
        constraints: [node.role == manager ]                                                                                                                                                        
      resources:                                                                                                                                                                                    
        limits:                                                                                                                                                                                     
          memory: 300M                                                                                                                                                                              
    user: root                                                                                                                                                                                      
    volumes:                                                                                                                                                                                        
      - ./prometheus.yml:/etc/prometheus/prometheus.yml                                                                                                                                             
      - /var/run/docker.sock:/var/run/docker.sock:ro                                                                                                                                                
    ports:                                                                                                                                                                                          
      - 9090:9090                                                                                                                                                                                   
    entrypoint:                                                                                                                                                                                                                                                                                                                                            
      - /bin/prometheus \                                                                                                                                                                          
         --config.file=/etc/prometheus/prometheus.yml \                                                                                                                                             
         --storage.tsdb.retention.time=90d \                                                                                                                                                        
         --storage.tsdb.path=/prometheus \                                                                                                                                                          
         --log.level=debug                                                                                                                                                                                                                                                                                                        
    networks:                                                                                                                                                                                       
      - smou_network                                                                                                                                                                                
                                                                                                                                                                                                    
  cadvisor:                                                                                                                                                                                         
     image: gcr.io/cadvisor/cadvisor:v0.49.1                                                                                                                                                        
     deploy:                                                                                                                                                                                        
       mode: global                                                                                                                                                                                 
       resources:                                                                                                                                                                                   
         limits:                                                                                                                                                                                    
           memory: 150M 
     labels:                                                                                                                                                                                        
       prometheus-job: cadvisor                                                                                                                                                                     
     ports:                                                                                                                                                                                         
       - target: 8080                                                                                                                                                                               
         mode: host                                                                                                                                                                                 
     command: -docker_only -v 4                                                                                                                                                                     
     volumes:                                                                                                                                                                                       
       - /:/rootfs:ro                                                                                                                                                                               
       - /var/run:/var/run:ro                                                                                                                                                                       
       - /dev/disk:/dev/disk:ro                                                                                                                                                                     
       - /sys:/sys:ro                                                                                                                                                                               
       - /var/lib/docker:/var/lib/docker:ro                                                                                                                                                         
       - /dev/kmsg:/dev/kmsg:ro                                                                                                                                                                                                                                                                                                                                                       
     networks:                                                                                                                                                                                      
       - bridge                                                                                                                                                                                     
                                                                                                                                                                                                    
  node-exporter:                                                                                                                                                                                    
    image: prom/node-exporter:v1.8.1                                                                                                                                                                
    deploy:                                                                                                                                                                                         
      mode: global                                                                                                                                                                                  
      resources:                                                                                                                                                                                    
        limits:                                                                                                                                                                                     
          memory: 150M                                                                                                                                                                              
    labels:                                                                                                                                                                                         
      prometheus-job: node-exporter                                                                                                                                                                 
    ports:                                                                                                                                                                                          
      - target: 9100                                                                                                                                                                                
        mode: host                                                                                                                                                                                  
    command: --log.level=debug                                                                                                                                                                                                                                                                                                                                                          
    networks:                                                                                                                                                                                       
      - bridge                                                                                                                                                                                      
                                                                                                                                                                                                    
networks:                                                                                                                                                                                           
  bridge:                                                                                                                                                                                           
    external: true                                                                                                                                                                                  
  smou_network:                                                                                                                                                                                     
    external: true	


  - job_name: 'swarm-cdavisor-node-exporter'                                                                                                                                                        
    dockerswarm_sd_configs:                                                                                                                                                                         
      - host: unix:///var/run/docker.sock                                                                                                                                                           
        role: tasks                                                                                                                                                                                 
        filters:                                                                                                                                                                                    
          - name: service                                                                                                                                                                           
            values:                                                                                                                                                                                 
              - prometheus_federation_cadvisor                                                                                                                                                      
              - prometheus_federation_node-exporter                                                                                                                                                 
    relabel_configs:                                                                                                                                                                                
      - source_labels: [__meta_dockerswarm_container_label_prometheus_job]                                                                                                                          
        target_label: job                                                                                                                                                                           
      - source_labels: [__meta_dockerswarm_node_hostname]                                                                                                                                           
        target_label: instance

  - job_name: 'federate-smou-p0-api-web'                                                                                                                                                            
    scrape_interval: 15s                                                                                                                                                                            
    honor_labels: true                                                                                                                                                                              
    metrics_path: '/federate'                                                                                                                                                                       
    params:                                                                                                                                                                                         
      'match[]':                                                                                                                                                                                    
         - '{job=~".+"}'                                                                                                                                                                            
    static_configs:                                                                                                                                                                                 
     - targets:                                                                                                                                                                                     
        - '172.17.18.160:9090'                                                                                                                                                                      
        - '172.17.18.162:9090'

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

Logs when it works fine:

prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.824Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(ver
sion=2.52.0, branch=HEAD, revision=879d80922a227c37df502e7315fad8ceb10a986d)"                                                                                                                       
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.824Z caller=main.go:622 level=info build_context="(go=go1.22.3, platform=linux/amd64, user=ro
ot@1b4f4c206e41, date=20240508-21:56:43, tags=netgo,builtinassets,stringlabels)"                                                                                                                    
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.824Z caller=main.go:623 level=info host_details="(Linux 3.10.0-1160.118.1.el7.x86_64 #1 SMP T
hu Apr 4 03:33:23 EDT 2024 x86_64 62bb34dd6339 (none))"                                                                                                                                             
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.824Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)"                  
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.824Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)"              
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.827Z caller=web.go:568 level=info component=web msg="Start listening for connections" address
=0.0.0.0:9090                                                                                                                                                                                       
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.827Z caller=main.go:1129 level=info msg="Starting TSDB ..."                                  
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.830Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9090  
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.830Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false add
ress=[::]:9090                                                                                                                                                                                      
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.837Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chun
ks if any"                                                                                                                                                                                          
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.837Z caller=head.go:703 level=info component=tsdb msg="On-disk memory mappable chunks replay 
completed" duration=9.277µs                                                                                                                                                                         
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.837Z caller=head.go:711 level=info component=tsdb msg="Replaying WAL, this may take a while" 
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.837Z caller=head.go:783 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegme
nt=0                                                                                                                                                                                                
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.837Z caller=head.go:820 level=info component=tsdb msg="WAL replay completed" checkpoint_repla
y_duration=56.947µs wal_replay_duration=499.491µs wbl_replay_duration=188ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=9.277µs total_replay_duration=601.778µs                      
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.841Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC                                  
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.841Z caller=main.go:1153 level=info msg="TSDB started"                                       
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.841Z caller=main.go:1154 level=debug msg="TSDB options" MinBlockDuration=2h MaxBlockDuration=
9d MaxBytes=0B NoLockfile=false RetentionDuration=90d WALSegmentSize=0B WALCompression=true                                                                                                         
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.841Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus
/prometheus.yml                                                                                                                                                                                     
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:170 level=debug component="discovery manager scrape" msg="Starting pro
vider" provider=dockerswarm/0 subs=[swarm-cdavisor-node-exporter]                                                                                                                                   
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:170 level=debug component="discovery manager scrape" msg="Starting pro
vider" provider=dockerswarm/1 subs=[smou-maps-api-crons]                                                                                                                                            
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:170 level=debug component="discovery manager scrape" msg="Starting pro
vider" provider=dockerswarm/2 subs=[smou-maps-api-v1]                                                                                                                                               
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:170 level=debug component="discovery manager scrape" msg="Starting pro
vider" provider=dockerswarm/3 subs=[smou-maps-api-data-apps-v1]                                                                                                                                     
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:170 level=debug component="discovery manager scrape" msg="Starting pro
vider" provider=dockerswarm/4 subs=[smou-maps-api-traffic-v1]                                                                                                                                       
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:170 level=debug component="discovery manager scrape" msg="Starting pro
vider" provider=dockerswarm/5 subs=[smou-maps-api-oauth-v1]                                                                                                                                         
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:170 level=debug component="discovery manager scrape" msg="Starting pro
vider" provider=static/6 subs=[prometheus]                                                                                                                                                          
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/e
tc/prometheus/prometheus.yml totalDuration=1.739687ms db_storage=7.111µs remote_storage=1.852µs web_handler=928ns query_engine=1.548µs scrape=517.484µs scrape_sd=389.002µs notify=1.107µs notify_sd
=993ns rules=4.574µs tracing=6.156µs                                                                                                                                                                
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=main.go:1114 level=info msg="Server is ready to receive web requests."           
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:188 level=debug component="discovery manager scrape" msg="Discoverer c
hannel closed" provider=static/6                                                                                                                                                                    
prometheus_federation_prometheus-federation.1.fz2362iut9xs@VMSMOUSWARMPRE1    | ts=2024-06-06T11:55:36.843Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..."


Logs when it doesn't work well:


prometheus_federation_prometheus-federation.1.a9yd4ebzfy6z@VMSMOUSWARMPRE1    | ts=2024-06-06T12:03:36.108Z caller=scrape.go:1331 level=debug component="scrape manager" scrape_pool=swarm-cdavisor-
node-exporter target=http://0.0.0.0:32773/metrics msg="Scrape failed" err="Get \"http://0.0.0.0:32773/metrics\": dial tcp 0.0.0.0:32773: connect: connection refused"

machine424 · 2025-05-13T11:46:45Z

machine424
May 13, 2025
Maintainer

This seems like a support request, converting it into a discussion.
Don't hesitate to use the other channel that are more suited for such requests; https://prometheus.io/community/#slack-channel (the slack one is more appropriate for example)

Also, you maybe need to reach out to the swarm community, as dial tcp 0.0.0.0:32773: connect: connection refused suggests there could be a networking issue that occurs when the leader changes.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Problem with promtheus, cadvisor and node exporter in docker swarm with no Leader role node #16596

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Problem with promtheus, cadvisor and node exporter in docker swarm with no Leader role node #16596

Uh oh!

Uh oh!

ImanoLCarbonell Jun 6, 2024

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

System information

Prometheus version

Prometheus configuration file

Alertmanager version

Alertmanager configuration file

Logs

Replies: 1 comment

Uh oh!

machine424 May 13, 2025 Maintainer

ImanoLCarbonell
Jun 6, 2024

machine424
May 13, 2025
Maintainer