Skip to content

kube-apiserver failing to start on slow hardware #72163

@rkojedzinszky

Description

@rkojedzinszky

What happened:
I've bootstrapped a cluster with kubeadm on a banana pi, it is up and running. Upon restarting, some process is slow that kube-apiserver does not boot up, exits with last lines:

I1218 19:19:04.831034       1 log.go:172] http: TLS handshake error from 10.0.0.39:43582: EOF
I1218 19:19:04.867159       1 log.go:172] http: TLS handshake error from 10.0.0.39:43598: EOF
I1218 19:19:04.932248       1 log.go:172] http: TLS handshake error from 10.0.0.39:43552: EOF
I1218 19:19:05.169874       1 log.go:172] http: TLS handshake error from 10.0.0.39:43578: EOF
W1218 19:19:05.534620       1 storage_scheduling.go:95] unable to get PriorityClass system-node-critical: Get https://[::1]:6443/apis/scheduling.k8s.io/v1beta1/priorityclasses/system-node-critical: net/http: TLS handshake timeout. Retrying...

What you expected to happen:
Expected the kube-apiserver to start up.

How to reproduce it (as minimally and precisely as possible):
On a banana pi (A20) bootstrap a cluster with kubeadm, add kube-router cni, and then restart kubelet.

Anything else we need to know?:
The version below contains a proposed fix, that is why the GitVersion is dirty.

Environment:

  • Kubernetes version
    Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/arm"}
    Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.2-beta.0.1+598a01989dcd06-dirty", GitCommit:"598a01989dcd06ec776248506fe6eb32f499bd38", GitTreeState:"dirty", BuildDate:"2018-12-18T19:10:37Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/arm"}

  • Cloud provider or hardware configuration:
    Banana pi m1 plus

  • OS (e.g. from /etc/os-release):
    PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
    NAME="Debian GNU/Linux"
    VERSION_ID="9"
    VERSION="9 (stretch)"
    ID=debian
    HOME_URL="https://www.debian.org/"
    SUPPORT_URL="https://www.debian.org/support"
    BUG_REPORT_URL="https://bugs.debian.org/"

  • Kernel (e.g. uname -a):
    Linux k8s-master 4.9.0-8-armmp Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Debian 4.9.130-2 (2018-10-27) armv7l GNU/Linux

  • Install tools:
    kubeadm

  • Others:
    Seems that in pkg/registry/scheduling/rest/storage_scheduling.go AddSystemPriorityClasses() adds a function which uses wait.Poll. On line 95 it logs 'Retrying...', howewer the return statement tells wait.Poll() to stop polling.

/kind bug

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.sig/api-machineryCategorizes an issue or PR as relevant to SIG API Machinery.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions