Skip to content

Conversation

Kevinz857
Copy link
Contributor

PR Description: Add LimitMEMLOCK=infinity to containerd systemd service

Summary

Add LimitMEMLOCK=infinity configuration to the containerd systemd service file to remove memory locking limitations and support advanced container workloads.

Background

The current containerd systemd service configuration has a default memory lock limit that can prevent certain workloads from running properly. This limitation affects:

  • GPU workloads: NVIDIA/AMD GPU containers require locked memory for device mappings
  • eBPF programs: Network policies, monitoring, and security features need to lock eBPF maps in memory
  • High-performance computing: Applications requiring large amounts of locked memory for performance
  • Real-time applications: Containers that need guaranteed memory residency to avoid swap latency

Changes Made

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
+LimitMEMLOCK=infinity
TasksMax=infinity
OOMScoreAdjust=-999

Problem Solved

Before: Containers requiring memory locking would fail with errors like:

failed to create containerd task: OCI runtime create failed: 
runc create failed: unable to start container process: 
error during container init: operation not permitted: mlock failed

After: containerd can successfully manage containers with memory locking requirements.

Use Cases Enabled

  1. GPU Containers

    docker run --gpus all nvidia/cuda:11.8-runtime nvidia-smi
  2. eBPF-enabled Networking

    # Cilium, Calico with eBPF dataplane
    kubectl apply -f cilium-config.yaml
  3. High-Performance Applications

    docker run --cap-add=IPC_LOCK --ulimit memlock=-1 hpc-workload
  4. Database Optimization

    # PostgreSQL with huge pages and memory locking
    docker run -e POSTGRES_PASSWORD=pass --ulimit memlock=-1 postgres:15

Compatibility

  • Backward Compatible: ✅ Existing containers continue to work unchanged
  • Kubernetes: ✅ No impact on existing Kubernetes workloads
  • Docker: ✅ Compatible with all Docker container types
  • System Requirements: ✅ Works on all supported Linux distributions

Performance Impact

  • Memory Usage: Minimal impact - only locks memory when needed by specific workloads
  • System Stability: Improved stability for GPU and HPC workloads
  • Performance: Eliminates page faults for critical container operations

References

Reviewer Notes

This change aligns containerd with other container runtimes (Docker, CRI-O) that commonly use unlimited memory locking. The configuration is widely adopted in production Kubernetes clusters running GPU workloads.

Copy link

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubesphere-prow kubesphere-prow bot added do-not-merge/release-note-label-needed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 11, 2025
…ervice

Signed-off-by: Kevinz <ruoshuidba@gmail.com>
@Kevinz857 Kevinz857 force-pushed the support-containerd-memlock branch from 0845eea to b8b40c0 Compare June 11, 2025 06:14
Copy link

@pixiake pixiake requested a review from Copilot June 11, 2025 06:16
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Add the LimitMEMLOCK=infinity directive to the containerd systemd service configuration to enable workloads that require locked memory.

  • Insert LimitMEMLOCK=infinity in the static service file
  • Update the Go template to emit the same directive and adjust import order

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
pkg/service/containermanager/templates/containerd.service Add LimitMEMLOCK=infinity under resource limits
cmd/kk/pkg/container/templates/containerd_service.go Reorder imports and include LimitMEMLOCK=infinity in the generated template
Comments suppressed due to low confidence (2)

cmd/kk/pkg/container/templates/containerd_service.go:45

  • [nitpick] Consider adding a unit test to verify that the generated service template includes the LimitMEMLOCK=infinity directive.
LimitMEMLOCK=infinity

cmd/kk/pkg/container/templates/containerd_service.go:22

  • The import "github.com/lithammer/dedent" is no longer used in this file and will cause a compile error; please remove it or apply it to the template string if needed.
"github.com/lithammer/dedent"

@pixiake
Copy link
Collaborator

pixiake commented Jun 11, 2025

/lgtm
/approve

@kubesphere-prow kubesphere-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jun 11, 2025
Copy link

LGTM label has been added.

Git tree hash: eea7411d512c4ede8d4aa45aab0b596e44f720b8

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Kevinz857, pixiake

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubesphere-prow kubesphere-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 11, 2025
@kubesphere-prow kubesphere-prow bot merged commit e9a8ad5 into kubesphere:master Jun 11, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/release-note-label-needed lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants