I’d like to bring up a discussion about Sandbox API in containerd.

Apparently we can not deny growth of popularity of containers with various flavors, such as Pods (e.g. a group of containers with shared namespace) or secure environments (aka micro VMs), like
fircracker-containerd, EKS on Fargate, or Kata containers.

On the other hand we need a path forward for higher level entities (container managers and orchestrators) to be able to run these container extensions transparently.

Today there is no defined way to do this, so everyone has to build own solution and solve same problems (how to manage microVM lifecycle? how to pass resource requirements? how to keep it orchestrator agnostic?).

This proposal introduces a notion of a group of containers in containerd - a “sandbox”. It aims to add a low-level lifecycle and resource management capabilities for containers that run inside of some environment (where "some" is defined and implemented by a client).

The sandbox concept has the following properties in relation to containers it hosts:

• Sandbox acts as a parent entity for containers, e.g. it starts first and ends last (this typically useful in (micro)VM environments, where VMs are required to be started before any other entities).
• Sandbox acquires resources needed for running child containers (for instance Kubernetes creates "pause" containers to acquire IP and network namespace for child containers).

The API aims to be implementation and orchestrator independent, as low level as possible, and introduces no dependencies to other Go packages.

It adds one more proxy plugin type - "sandbox", which implements Controller interface (similarly to snapshotters). Controller is responsible for platform specific sandbox implementation. It knows how to create/start/stop/delete a sandbox instance, check lifetime, report status, gather metadata, etc. containerd keeps track of running instances (in metadata store), generates proper lifecycle events, and translates client calls to a proper proxy plugin. Client can manage sandboxes via client API (example).

From orchestrator perspective there will be required a “bridge”, that translates orchestrator calls to sandbox API (e.g. cri-containerd, but implementation agnostic), sandbox controllers remain interchangeable.

Currently the sandbox API implemented in this fork (master...mxpv:sandbox) and exists as proof of concept in sandbox branch of firecracker-container repo.

cc: @samuelkarp @micahhausler @egernst

Also there is a good Twitter discussion about Kubernetes / Firecracker challenges working together: https://twitter.com/micahhausler/status/1238496944684597248