Skip to content

Issue #2835 - Fix Tekton task "func-buildpacks" prepare/results steps use non-multi-arch bash:5.1.4 image, causing exec format errors on ARM64 #2842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 27, 2025

Conversation

luciantin
Copy link
Contributor

Changes

  • 🐛 Tekton task "func-buildpacks" prepare/results steps use non-multi-arch bash:5.1.4 image, causing exec format errors on ARM64

/kind bug

Fixes #2835

fix: fixes issue with func in-cluster build/deploy pipelines to work on ARM64

@knative-prow knative-prow bot added the kind/bug Categorizes issue or PR as related to a bug. label May 26, 2025
@knative-prow knative-prow bot requested review from nainaz and vyasgun May 26, 2025 18:42
Copy link

linux-foundation-easycla bot commented May 26, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: luciantin / name: Tin (b7ec376)

Copy link

knative-prow bot commented May 26, 2025

Welcome @luciantin! It looks like this is your first PR to knative/func 🎉

Copy link

knative-prow bot commented May 26, 2025

Hi @luciantin. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@knative-prow knative-prow bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 26, 2025
@luciantin
Copy link
Contributor Author

Some more info to help reproduce the errors.

When using buildpack to build Go on an ARM64 host, the Tekton "prepare" and "results" steps fail with:

Error executing command: fork/exec /tekton/scripts/script-0-....: exec format error

Running the image in docker :

$ docker run --rm -it docker.io/library/bash:5.1.4@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6 sh

Unable to find image 'bash:5.1.4@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6' locally
docker.io/library/bash@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6: Pulling from library/bash
801bfaa63ef2: Pull complete
19e45ec58f41: Pull complete
6435bc57893e: Pull complete
Digest: sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6
Status: Downloaded newer image for bash@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/local/bin/docker-entrypoint.sh: exec format error
Solution was to untag the following images :

image: docker.io/library/bash:5.1.4@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6

image: docker.io/library/bash:5.1.4@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6

Fyi

Running the 'TarImage' and 'FuncUtilImage' from main branch, i.e. "ghcr.io/knative/func-utils:v2"

Such as :

kubectl run func-shell --rm -it \
  --image=ghcr.io/knative/func-utils:v2 \
  -- sh

Or the specific arm64 :

kubectl run func-shell --rm -it \
  --image=ghcr.io/knative/func-utils:v2@sha256:d26fc93fa14f3f1fe34cc1b320676dbbeed9b4ff662de44146f92ba856b2e02b \
  -- sh

Both return :

exec /sh: exec format error                                                                                                              
stream closed EOF for default/func-shell (func-shell) 

The image that is used is :
ID: ghcr.io/knative/func-utils@sha256:45b3f8f66d354585c78ac290f723de264d42c857da737e620c5b314ad3610cdf

I confirmed that k8s deployed the specific image, also tried on an ARM64 instance with Docker :

$ docker run --rm -it ghcr.io/knative/func-utils:v2@sha256:d26fc93fa14f3f1fe34cc1b320676dbbeed9b4ff662de44146f92ba856b2e02b sh

Unable to find image 'ghcr.io/knative/func-utils:v2@sha256:d26fc93fa14f3f1fe34cc1b320676dbbeed9b4ff662de44146f92ba856b2e02b' locally
ghcr.io/knative/func-utils@sha256:d26fc93fa14f3f1fe34cc1b320676dbbeed9b4ff662de44146f92ba856b2e02b: Pulling from knative/func-utils
ff0713626554: Pull complete
63e7b8aa440c: Pull complete
Digest: sha256:d26fc93fa14f3f1fe34cc1b320676dbbeed9b4ff662de44146f92ba856b2e02b
Status: Downloaded newer image for ghcr.io/knative/func-utils@sha256:d26fc93fa14f3f1fe34cc1b320676dbbeed9b4ff662de44146f92ba856b2e02b
exec /sh: exec format error

The working one I am using is from branch 'release-1.17' -> 'ghcr.io/knative/func-utils:latest'

https://github.com/luciantin/func-dev/blob/02912c0dc8860c33a56bfbdaa160b735e577936a/pkg/k8s/persistent_volumes.go#L70

As far as the volume-uploader image is concerned, my conclusion is that the ARM64 image does not work on ARM64
https://github.com/knative/func/pkgs/container/func-utils/424105624?tag=v2

Havent looked further, might be tied to kubernetes-sigs/metrics-server#882

Not included in pull, prob should be fixed on the image building side, havent looked yet, but func wont deploy functions on ARM64 with the latest v2 version.

Also, the main branch deploys functions correctly, on ARM64 through buildpack, after the following images are reverted (including the changes in the PR), no other changes were necessary.
(probably also kills s2i if volume-uploader image isnt working)

var DeployerImage = "ghcr.io/knative/func-utils:v2"

var TarImage = "ghcr.io/knative/func-utils:v2"

I can look it up if you'd like, and add it to the PR (if its in the same repo, idk...)

@matejvasek
Copy link
Contributor

ghcr.io/knative/func-utils:v2 is definitely mullti-arch.

@matejvasek
Copy link
Contributor

But for whatever reason it contains amd64 binary, weird.

@matejvasek
Copy link
Contributor

matejvasek commented May 26, 2025

Also note that sh in func image is not really a shell.
See:

func/cmd/func-util/main.go

Lines 189 to 202 in 8e0b752

func sh(ctx context.Context) error {
if !slices.Equal(os.Args[1:], []string{"-c", "umask 0000 && exec tar -xmf -"}) {
return fmt.Errorf("this is a fake sh (only for backward compatiblility purposes)")
}
wd, err := os.Getwd()
if err != nil {
return fmt.Errorf("cannot get working directory: %w", err)
}
unix.Umask(0)
return tar.Extract(os.Stdin, wd)
}

Copy link

codecov bot commented May 26, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.49%. Comparing base (0ce5759) to head (b7ec376).
Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2842      +/-   ##
==========================================
+ Coverage   62.66%   63.49%   +0.82%     
==========================================
  Files         127      127              
  Lines       15545    15545              
==========================================
+ Hits         9742     9870     +128     
+ Misses       4892     4736     -156     
- Partials      911      939      +28     
Flag Coverage Δ
e2e-tests 43.63% <100.00%> (+1.42%) ⬆️
integration-tests 49.45% <100.00%> (+1.79%) ⬆️
unit-tests 49.53% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@matejvasek
Copy link
Contributor

matejvasek commented May 26, 2025

OK I know where the problem is 🤦‍♂️ .
I updated the func-util image manually now. I will create PR latter.

@matejvasek
Copy link
Contributor

#2843

@matejvasek
Copy link
Contributor

@luciantin The failing test is not your fault. Please just rebase to include fix for it.

@matejvasek
Copy link
Contributor

/ok-to-test
/approve
/lgtm

@lkingland @dsimansk please override the failing test if possible.

@knative-prow knative-prow bot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 26, 2025
@knative-prow knative-prow bot added lgtm Indicates that a PR is ready to be merged. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 26, 2025
Copy link

knative-prow bot commented May 26, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: luciantin, matejvasek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed lgtm Indicates that a PR is ready to be merged. labels May 26, 2025
…eps use non-multi-arch bash:5.1.4 image, causing exec format errors on ARM64
@matejvasek
Copy link
Contributor

/lgtm

@knative-prow knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label May 27, 2025
@matejvasek matejvasek requested review from gauron99 and removed request for nainaz and vyasgun May 27, 2025 11:56
@knative-prow knative-prow bot merged commit 3150fed into knative:main May 27, 2025
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tekton task "func-buildpacks" prepare/results steps use non-multi-arch bash:5.1.4 image, causing exec format errors on ARM64
2 participants