Skip to content

v1.8 hyperkube images don't allow the kubelet to mount #52789

@jamiehannaford

Description

@jamiehannaford

/kind bug

What happened:

When a cluster is provisioned using a containerised kubelet using the following images:

gcr.io/google-containers/hyperkube-amd64:v1.8.0-beta.0
gcr.io/google-containers/hyperkube-amd64:v1.8.1-beta.0

Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries:

Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----			-------------	--------	------			-------
  1m		1m		1	{default-scheduler }			Normal		Scheduled		Successfully assigned thindiskpod to node3
  1m		1m		1	{kubelet node3}				Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "default-token-5h5x6" 
  1m		1m		5	{kubelet node3}				Warning		FailedMount		MountVolume.MountDevice failed for volume "pvc-4dc51e93-82c5-11e7-a497-0050569c48bf" : executable file not found in $PATH

This executable is mkfs.ext4. This problem can be resolved by mounting paths from the host into the rkt container, specifically /usr/sbin and /usr/lib.

Is this expected? @saad-ali mentioned in #50802, that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary.

If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this?
If it's not, I'll leave this open as a tracking issue.

What you expected to happen:

The hyperkube image to work without lots of host mounts.

How to reproduce it (as minimally and precisely as possible):

Run kubelet with rkt in one of the above listed images (see CoreOS' kubelet-wrapper).

Environment:

  • Kubernetes version (use kubectl version): v1.8.0.beta-1
  • Cloud provider or hardware configuration**: OpenStack
  • OS (e.g. from /etc/os-release):
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1520.3.0
VERSION_ID=1520.3.0
BUILD_ID=2017-09-15-2017
PRETTY_NAME="Container Linux by CoreOS 1520.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
  • Kernel (e.g. uname -a):
Linux jamie-terraform-worker-1 4.13.2-coreos #1 SMP Fri Sep 15 20:02:58 UTC 2017 x86_64 AMD Opteron 23xx (Gen 3 Class Opteron) AuthenticAMD GNU/Linux
  • Install tools: tectonic-installer

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.sig/nodeCategorizes an issue or PR as relevant to SIG Node.sig/storageCategorizes an issue or PR as relevant to SIG Storage.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions