-
Notifications
You must be signed in to change notification settings - Fork 1k
Description
Describe the bug
An image specifies the health check with various arguments, e. g. the following health check:
HEALTHCHECK --interval=5s --timeout=2s --start-period=10s --retries=3 CMD [ "/bin/bash", "/healthcheck.sh" ]
container: ($> docker inspect kjn2kj3498
)
The image author updates the health check parameters, e. g. timeout will be increased to 15s, and retries to 4:
HEALTHCHECK --interval=5s --timeout=15s --start-period=10s --retries=4 CMD [ "/bin/bash", "/healthcheck.sh" ]
Watchtower updates the container by pulling the new image and recreate a new container of the new image. But the new container runs with the old healthcheck arguments instead of the updated ones:
$> docker inspect ui43j3na
// ...
"Healthcheck": {
"Test": [
"CMD",
"/bin/bash",
"/healthcheck.sh"
],
"Interval": 5000000000,
"Timeout": 2000000000,
"StartPeriod": 10000000000,
"Retries": 3
},
If you manually re-create the container, e. g. using docker-compose pull
and docker-compose up -d
, the updated health check parameters will be applied to the new container:
$> docker inspect asdn2904ds
// ...
"Healthcheck": {
"Test": [
"CMD",
"/bin/bash",
"/healthcheck.sh"
],
"Interval": 60000000000,
"Timeout": 15000000000,
"StartPeriod": 600000000000,
"Retries": 4
},
Steps to reproduce
- Create a simplified dockerfile:
FROM ubuntu
HEALTHCHECK --interval=5s --timeout=2s --start-period=10s --retries=3 CMD [ "/usr/bin/sleep", "1s" ]
ENTRYPOINT [ "/usr/bin/sleep", "10m" ]
- build the image:
docker build --no-cache -t healthcheck-image .
- publish the image:
docker push healthcheck-image
- Create a container + watchtower with the following compose file:
version: '3'
services:
healthcheck-test:
image: healtcheck-image
labels:
- com.centurylinklabs.watchtower.enable=true
watchtower:
image: containrrr/watchtower
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: --interval 10
environment:
- WATCHTOWER_LABEL_ENABLE=true
- Inspect the container to check the healthcheck configuration
$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 2s 10s 3}
- Update the healthcheck
timeout
andretries
Parameter
FROM ubuntu
HEALTHCHECK --interval=5s --timeout=15s --start-period=10s --retries=4 CMD [ "/usr/bin/sleep", "1s" ]
ENTRYPOINT [ "/usr/bin/sleep", "10m" ]
- rebuild and republish the image
- Wait for watchtower to update the container
- re-check the container healthcheck configuration
$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 2s 10s 3}
Expected:
$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 15s 10s 4}
Do the same with docker-compose up -d --force-recreate
. The expected healthcheck configuration will be applied.
Expected behavior
The container should be recreated with the updated healtcheck configuration
Screenshots
No response
Environment
docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.16.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Containers: 9
Running: 2
Paused: 0
Stopped: 7
Images: 347
Server Version: 23.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 5.10.102.1-microsoft-standard-WSL2
Operating System: Ubuntu 20.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 12.27GiB
Name: Wehrstedt
ID: DDGC:2DRA:UQYD:5LEP:PAUW:AA4Y:XA6G:MJD5:SCWX:U6MM:3D5Z:Z2IV
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 172.17.0.0/12, Size: 20
Base: 192.168.0.0/16, Size: 24
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
Your logs
Attaching to watchtower-test_watchtower_1, watchtower-test_healthcheck-test_1
watchtower_1 | time="2023-03-21T11:43:49Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
watchtower_1 | time="2023-03-21T11:43:50Z" level=debug msg="Making sure everything is sane before starting"
watchtower_1 | time="2023-03-21T11:43:50Z" level=debug msg="Retrieving running containers"
watchtower_1 | time="2023-03-21T11:43:50Z" level=debug msg="There are no additional watchtower containers"
watchtower_1 | time="2023-03-21T11:43:50Z" level=debug msg="Watchtower HTTP API skipped."
watchtower_1 | time="2023-03-21T11:43:50Z" level=info msg="Watchtower 1.5.3"
watchtower_1 | time="2023-03-21T11:43:50Z" level=info msg="Using no notifications"
watchtower_1 | time="2023-03-21T11:43:50Z" level=info msg="Only checking containers using enable label"
watchtower_1 | time="2023-03-21T11:43:50Z" level=info msg="Scheduling first run: 2023-03-21 11:44:00 +0000 UTC"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Checking containers for updated images"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Retrieving running containers"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Trying to load authentication credentials." container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Loaded auth credentials for user <user>, on registry <registry-url>/healtcheck-image:latest, from file /config.json"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Got image name: <registry-url>/healtcheck-image:latest"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Credentials loaded"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Checking if pull is needed" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Building challenge URL" URL="https://<registry-url>/v2/"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://<registry-url>/jwt/auth\",service=\"container_registry\"" status="401 Unauthorized"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Checking challenge header content" realm="https://<registry-url>/jwt/auth" service=container_registry
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Setting scope for auth token" image=<registry-url>/healtcheck-image scope="repository:<registry-url>/healtcheck-image:pull"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Credentials found."
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Parsing image ref" host=<registry-url> image=healtcheck-image normalized="<registry-url>/healtcheck-image:latest" tag=latest
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://<registry-url>/v2/healtcheck-image/manifests/latest"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Could not do a head request for \"<registry-url>/healtcheck-image:latest\", falling back to regular pull." container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://<registry-url>/jwt/auth\\\",service=\\\"container_registry\\\",scope=\\\"repository:healtcheck-image:pull\\\",error=\\\"insufficient_scope\\\"\"" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1 | time="2023-03-21T11:44:30Z" level=debug msg="Pulling image" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1 | time="2023-03-21T11:44:30Z" level=info msg="Found new <registry-url>/healtcheck-image:latest image (feb17fe278fe)"
watchtower_1 | time="2023-03-21T11:44:30Z" level=info msg="Stopping /watchtower-test_healthcheck-test_1 (f29ee537d965) with SIGTERM"
watchtower_1 | time="2023-03-21T11:44:40Z" level=debug msg="Skipped another update already running."
watchtower_1 | time="2023-03-21T11:44:40Z" level=debug msg="Scheduled next run: 2023-03-21 11:44:50 +0000 UTC"
watchtower_1 | time="2023-03-21T11:44:40Z" level=debug msg="Removing container f29ee537d965"
watchtower-test_healthcheck-test_1 exited with code 137
watchtower_1 | time="2023-03-21T11:44:41Z" level=info msg="Creating /watchtower-test_healthcheck-test_1"
watchtower_1 | time="2023-03-21T11:44:41Z" level=debug msg="Starting container /watchtower-test_healthcheck-test_1 (de887b1bc2fc)"
watchtower_1 | time="2023-03-21T11:44:41Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=1 notify=no
Additional context
No response