Failed to pull image that is pushed with minikube image build #16036
Description
What Happened?
I am running latest minikube on Ubuntu 22.04 on rootless docker started with:
$> minikube start --container-runtime=containerd --driver=docker
😄 minikube v1.29.0 on Ubuntu 22.04
✨ Using the docker driver based on user configuration
📌 Using rootless Docker driver
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=3900MB) ...
📦 Preparing Kubernetes v1.26.1 on containerd 1.6.15 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎 Verifying Kubernetes components...
🌟 Enabled addons: storage-provisioner, default-storageclass
💡 kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
I tried to push a local image to minikube with the image build command. I followed the guide: https://minikube.sigs.k8s.io/docs/handbook/pushing/
The image is built correctly and it is shown if I run:
minikube image ls
However, when trying to create a pod with it like:
minikube kubectl -- run testapp2 --image=testapp2 --image-pull-policy=Never --restart=Never
I get:
"PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/library/testapp2:latest\": failed to resolve reference \"docker.io/library/testapp2:latest\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" image="testapp2:latest"
Same image, but build on the host with docker build and then loaded minikube image load works exact as expected for pods started with the same command. The whole scenario is in the attached log file.
An interesting thing, maybe related, is that the image pushed with minikube image build cannot be removed with minikube image rm.
registry.k8s.io/pause:3.9
registry.k8s.io/kube-scheduler:v1.26.1
registry.k8s.io/kube-proxy:v1.26.1
registry.k8s.io/kube-controller-manager:v1.26.1
registry.k8s.io/kube-apiserver:v1.26.1
registry.k8s.io/etcd:3.5.6-0
registry.k8s.io/coredns/coredns:v1.9.3
gcr.io/k8s-minikube/storage-provisioner:v5
docker.io/library/testapp2:latest
docker.io/kindest/kindnetd:v20221004-44d545d1
test>minikube image rm testapp2
❗ Failed to remove images for profile minikube error removing images: crictl: sudo /usr/bin/crictl rmi testapp2: Process exited with status 1
stdout:
stderr:
time="2023-03-13T10:41:39Z" level=error msg="no such image testapp2"
time="2023-03-13T10:41:39Z" level=fatal msg="unable to remove the image(s)"```
Exact same command works fine with images pushed by 'minikube image load'
### Attach the log file
[minikube.log](https://github.com/kubernetes/minikube/files/10956082/minikube.log)
### Operating System
Ubuntu
### Driver
Docker