-
Notifications
You must be signed in to change notification settings - Fork 41.3k
Description
/kind bug
What happened:
When a cluster is provisioned using a containerised kubelet using the following images:
gcr.io/google-containers/hyperkube-amd64:v1.8.0-beta.0
gcr.io/google-containers/hyperkube-amd64:v1.8.1-beta.0
Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned thindiskpod to node3
1m 1m 1 {kubelet node3} Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-5h5x6"
1m 1m 5 {kubelet node3} Warning FailedMount MountVolume.MountDevice failed for volume "pvc-4dc51e93-82c5-11e7-a497-0050569c48bf" : executable file not found in $PATH
This executable is mkfs.ext4
. This problem can be resolved by mounting paths from the host into the rkt container, specifically /usr/sbin
and /usr/lib
.
Is this expected? @saad-ali mentioned in #50802, that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary.
If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this?
If it's not, I'll leave this open as a tracking issue.
What you expected to happen:
The hyperkube image to work without lots of host mounts.
How to reproduce it (as minimally and precisely as possible):
Run kubelet with rkt in one of the above listed images (see CoreOS' kubelet-wrapper).
Environment:
- Kubernetes version (use
kubectl version
): v1.8.0.beta-1 - Cloud provider or hardware configuration**: OpenStack
- OS (e.g. from /etc/os-release):
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1520.3.0
VERSION_ID=1520.3.0
BUILD_ID=2017-09-15-2017
PRETTY_NAME="Container Linux by CoreOS 1520.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
- Kernel (e.g.
uname -a
):
Linux jamie-terraform-worker-1 4.13.2-coreos #1 SMP Fri Sep 15 20:02:58 UTC 2017 x86_64 AMD Opteron 23xx (Gen 3 Class Opteron) AuthenticAMD GNU/Linux
- Install tools: tectonic-installer