This issue was reported in the Kubernetes Security Audit Report

Description
When kubelet attempts to load a Pod manifest, it will read the manifest entirely into memory in an attempt to parse and validate the manifest. Due to this behavior, if a large manifest is provided to kubelet it will cause an Out-Of-Memory (OOM) error. This could cause other services on the system upon which kubelet is running to fail outside of the scope of Kubernetes.

func (s *sourceFile) extractFromDir(name string) ([]*v1.Pod, error) {
	dirents, err := filepath.Glob(filepath.Join(name, "[^.]*"))
	...
	sort.Strings(dirents)
	for _, path := range dirents {
		statInfo, err := os.Stat(path)
		...
		switch {
		...
		case statInfo.Mode().IsRegular():
			pod, err := s.extractFromFile(path)
			...
		...
		}
	}
	return pods, nil
}

Figure 14.1: The extractFromDir function which finds all manifest files within a manifest directory and extracts Pods using extractFromFile.

func (s *sourceFile) extractFromFile(filename string) (pod *v1.Pod, err error) {
	...
	file, err := os.Open(filename)
             ...
	defer file.Close()

	data, err := ioutil.ReadAll(file)
	if err != nil {
		return pod, err
	}
	...
}

Figure 14.2: The extractFromFile function, which attempts to read all of the manifest files.

Additionally, due to kubelet causing a system OOM, the kubelet process will be killed and restarted. This appears to prevent kubelet from realizing that it has failed to start a Pod despite the backoff period, leading to an infinite loop of OOM, ultimately rendering the system unresponsive.

Exploit Scenario
Alice configures kubelet to pull Pods from a manifest directory. Eve identifies Alice’s kubelet manifest directory and has sufficient permissions to place a file in the manifest directory. Eve places a malicious manifest file within the kubelet manifest directory. kubelet’s restart policy and process-ephemeral backoff period causes the machine to lock in an OOM loop.

Recommendation
Avoid loading arbitrary data into memory regardless of size. Limit the size of a valid manifest or inform the user when it consumes a substantial amount of memory, especially for manifests that are fetched from remote endpoints. Consider persisting backoff periods for each Pod to allow for consistency between restarts of kubelet.

Anything else we need to know?:

See #81146 for current status of all issues created from these findings.

The vendor gave this issue an ID of TOB-K8S-019 and it was finding 14 of the report.

The vendor considers this issue Medium Severity.

To view the original finding, begin on page 46 of the Kubernetes Security Review Report

Environment:

• Kubernetes version: 1.13.4