Fix docker compose up --wait failing when Trillian server isn't healthy #2473
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
haydentherapper
added a commit
to sigstore/fulcio
that referenced
this pull request
May 6, 2025
Same as documented in sigstore/rekor#2473, the MySQL healthcheck was inaccurately reporting healthy when localhost was used. Switching to an address seems to fix it. Signed-off-by: Hayden B <haydentherapper@users.noreply.github.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2473 +/- ##
===========================================
- Coverage 66.46% 25.17% -41.29%
===========================================
Files 92 191 +99
Lines 9258 24790 +15532
===========================================
+ Hits 6153 6240 +87
- Misses 2359 17784 +15425
- Partials 746 766 +20
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
28b880a to
54495cb
Compare
cpanato
previously approved these changes
May 6, 2025
haydentherapper
added a commit
to sigstore/fulcio
that referenced
this pull request
May 6, 2025
Same as documented in sigstore/rekor#2473, the MySQL healthcheck was inaccurately reporting healthy when localhost was used. Switching to an address seems to fix it. Signed-off-by: Hayden B <haydentherapper@users.noreply.github.com>
bobcallaway
reviewed
May 6, 2025
54495cb to
7750b7a
Compare
haydentherapper
added a commit
to sigstore/fulcio
that referenced
this pull request
May 6, 2025
As noted in sigstore/rekor#2473 (comment), this did not properly populate username and password. Signed-off-by: Hayden B <haydentherapper@users.noreply.github.com>
haydentherapper
added a commit
to sigstore/fulcio
that referenced
this pull request
May 6, 2025
As noted in sigstore/rekor#2473 (comment), this did not properly populate username and password. Signed-off-by: Hayden B <haydentherapper@users.noreply.github.com>
8 tasks
cpanato
previously approved these changes
May 6, 2025
7750b7a to
496e046
Compare
As noted in docker/compose#12424, compose --wait doesn't seem to honor healthchecks with restart:always, when the server crashes and restarts a few times and eventually becomes healthy. This was happening with Rekor: * MySQL was not yet healthy because the healthcheck wasn't working as expected. docker-library/mysql#930 (comment) suggested using 127.0.0.1 instead of localhost * trillian-log-server was not yet healthy even when MySQL reported as healthy, causing trillian-log-server to crash and restart a few times. There was no healthcheck for either Trillian service because the image we're using is based on Distroless, which has no curl/wget. * rekor-server tried to start up with an unhealthy trillian-log-server, and crashed. The healthcheck reported as unhealthy, and even though the server eventually became healthy because of the restart:always policy, the healthcheck reported the startup as unhealthy. This change adds healthchecks to trillian-log-server and log-signer by pulling the binaries out of the images and putting them into Debian 12 containers that include curl, so we can curl the /healthz endpoint. This also fixes the MySQL healthcheck as noted above. Now, docker compose up --wait properly waits for a healthy MySQL before starting trillian-log-server, and a healthy Trillian before starting Rekor. Also fix minor Dockerfile linting errors. Signed-off-by: Hayden B <8418760+haydentherapper@users.noreply.github.com>
496e046 to
ed660e5
Compare
cpanato
approved these changes
May 14, 2025
haydentherapper
added a commit
to sigstore/rekor-monitor
that referenced
this pull request
May 20, 2025
With sigstore/rekor#2473, healthchecks have been added for Trillian's log signer and log server, so there are now 5 services rather than 3 to wait for. E2E tests failed because there will never be just 3 healthy services, there will be 5. This e2e script is a little brittle and can be cleaned up and simplified to use `compose up --wait`, will do in a later PR. Signed-off-by: Hayden B <haydentherapper@users.noreply.github.com>
haydentherapper
added a commit
to sigstore/rekor-monitor
that referenced
this pull request
May 20, 2025
With sigstore/rekor#2473, healthchecks have been added for Trillian's log signer and log server, so there are now 5 services rather than 3 to wait for. E2E tests failed because there will never be just 3 healthy services, there will be 5. This e2e script is a little brittle and can be cleaned up and simplified to use `compose up --wait`, will do in a later PR. Signed-off-by: Hayden B <haydentherapper@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As noted in docker/compose#12424, compose --wait doesn't seem to honor healthchecks with restart:always, when the server crashes and restarts a few times and eventually becomes healthy. This was happening with Rekor:
This change adds healthchecks to trillian-log-server and log-signer by pulling the binaries out of the images and putting them into Debian 12 containers that include curl, so we can curl the /healthz endpoint. This also fixes the MySQL healthcheck as noted above. Now, docker compose up --wait properly waits for a healthy MySQL before starting trillian-log-server, and a healthy Trillian before starting Rekor.
Also fix minor Dockerfile linting errors.
Summary
Release Note
Documentation