adding @AkihiroSuda into cc as he worked on similar issues on https://github.com/AkihiroSuda/filegrain

🎉
Does it work with "non-thin" images?

Also, can we have ctr images pull --unpack-cvmfs that creates thin.json for non-thin images, although I'm not sure it should be in this repo?

@AkihiroSuda, at the moment this snapshotter works only with thin images, however it is trivial to make it works with normal one. Here is a little tricky to decide what to allow, I am sure that someone would prefer a very rigid interface that works only with thin images while someone else would prefer something more pragmatic allowing the use also of regular images.

Most likely, if we want to support regular images we will based the implementation on the overlayfs snapshotter.

Internally at CERN we are developing utilities to translate normal images into thin one, it is still a work in progress and it is in its infancy but it is open source: https://github.com/cvmfs/docker-graphdriver/tree/devel/daemon

Also @stevvooe as we've discussed this a couple times in the past

I'm extremely confused: this PR has a snapshotter that doesn't really do anything but call into the transactional store. It looks like a clone of overlay, but I am not understanding the differences.

I think it would be most helpful you could par down this proposal to the components that affect containerd and snapshotters. For example, we know what containers are, so no need to explain that. Specifically, I would focus the following questions:

1. Why can't this be done with the existing snapshotters mounted on the fuse filesystem backed by cern vm fs (or other backend)?
2. Can anyone run a cern vm fs? If so, how? What is the current adoption rate? Consider that it might be better to preserve this as a specific build of containerd for use at cern.
3. How are we going to test this in CI?
4. Let's see some real performance numbers: I do think you can reduce disk space with this approach, but there is no way you are getting native performance with FUSE.
5. Let's look at this in the context of filegrain (someone already mentioned this). Are there are approaches to thin containers that don't rely on cernvmfs? Can we make this more generic so that others can get benefits?

That said, I think this is a good direction and having support at this level will be a huge benefit. However, we need to make sure the cvmfs is the right direction and won't be incur baggage that other users have to carry.

@stevvooe Many thanks for looking into this! I think I can address some of your points, for the implementation details I'll leave it to @siscia.

Our intention is not necessarily to make a cvmfs specific snapshotter. Cvmfs in this PR is mainly a demonstrator. We'd like to have a snapshotter that can start a container with a root file system in a mounted read-only directory -- in the same way you'd do a chroot. The directories with the container root file systems could then sit on any local or network mount point.

Regarding cvmfs itself, it is used mostly in the scientific world. Fully open sourced, BSD licensed, on github, documented. Besides high-energy physics, used for instance also by LIGO (gravitational wave detectors), the EUCLID space mission, bio science, some industry users etc. From the numbers we know, more then 1 billion files under management, order of 100k world-wide installed clients.

Fuse performance is less of an issue than one might think. Most of the calls are stat() and they are covered by the Linux VFS buffers. Also, the performance penalty is limited to the startup period, after which the software is in memory. After cache warm-up, the penalty is practically negligible (early measurements, more recent ones).

But again, we'd be happy with anything that lets us operate containerd on container images extracted in a directory.

Hi @stevvooe,

the PR was indeed based on the overlay snapshotter, I was in doubt if actually just call the overlay snapshotter methods here and there but at the end I opted to duplicate them, we can fix it.

The main difference is in the function, (o *snapshotter) mounts on line 415 which calls (o *snapshotter) mountCVMFS L:308, the interesting part of this function is how it get the lowerdir to mount using overlay, indeed on line 341 it use getLowerFilesystem define on L:292

getLowerFilesystem is where the real magic happens, it read the thin.json file inside the directory and from that file, it infers where the actual content of the layers is and mounts those directories as lowerdir.

Those directories are where the cvmfs file system is mounted, so /cvmfs/REPO/layers as a convention.

So, instead of getting the directories as tarball, decompress them inside $containderd_root/io.containerd.snapshotter.v1.overlayfs/... we already have those directories under /cvmfs/REPO/layers/..., the advantage of that is that everything that is mounted and used via CVMFS has a filegrain level. We are not downloading anymore the whole tarball, but just and only the file we are interested in.

Let me now answer your question.

1. Why can't this be done with the existing snapshotters mounted on the fuse filesystem backed by cern vm fs (or other backend)?

On the client, so the machine that read the files inside CVMFS, we only have read-only rights, this makes the whole concept possible if we allowed also to write file it would be a NFS which all the problem that it implies. However, containerd requires to write in it's root. A posible solution would be to use overlayfs to create a writeable layer on top of the files provided by CVMFS, but then you would have two layers of overlayfs running one of top of the other, which is not possible.

2. Can anyone run a cern vm fs? If so, how? What is the current adoption rate? Consider that it might be better to preserve this as a specific build of containerd for use at cern.

As jakob already mentioned, yes, it can be run by anybody.

3. How are we going to test this in CI?

We would need a server running the CVMFS server and connect the CVMFS client, I do agree that it is some work, but it is not impossible.

4. Let's see some real performance numbers: I do think you can reduce disk space with this approach, but there is no way you are getting native performance with FUSE.

We are definitely getting comparable performance with FUSE when the "cache is warm", when the data are not yet in the system and need to be retrievied via the network of course it is slower, but this should be compare with starting a container that is not in the system yet and that needs to be downloaed.

Are you looking for specific benchmarks? I can definitely prepare some for the sake of discussion. Even though I have already linked some in the first poster

5. Let's look at this in the context of filegrain (someone already mentioned this). Are there are approaches to thin containers that don't rely on cernvmfs? Can we make this more generic so that others can get benefits?

If we really want to go filegrain we need to leave behind the concept of layers and tarballs.

Our intereopable solution it's been to create this thin layers, that are just like normal layer but they do not contains the "data" but they are the "recipe" to get the data.

Can this approach be made more general to work with a wider range of technologies? The answer is clearly a yes.

I believe it could help the discussion to make clear what really is a thin layer.

The json below is the thin layer for redis:4

{
  "version": "1.0",
  "origin": "library/redis:4@https://registry-1.docker.io/v2",
  "layers": [
    {
      "digest": "683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e",
      "url": "cvmfs://cd.cern.ch/layers/683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e"
    },
    {
      "digest": "259238e792d86e23dab13fbcfcadf090333328ad9f80894544316437461f0d1b",
      "url": "cvmfs://cd.cern.ch/layers/259238e792d86e23dab13fbcfcadf090333328ad9f80894544316437461f0d1b"
    },
    {
      "digest": "78399601c709f0e252523e534db03152a1b3f017c9f7c756d68791ef07bc5d0b",
      "url": "cvmfs://cd.cern.ch/layers/78399601c709f0e252523e534db03152a1b3f017c9f7c756d68791ef07bc5d0b"
    },
    {
      "digest": "f397da4746012e36f5550d3eb830d81a522a1910417d6b3e1bd1ba046dfb8133",
      "url": "cvmfs://cd.cern.ch/layers/f397da4746012e36f5550d3eb830d81a522a1910417d6b3e1bd1ba046dfb8133"
    },
    {
      "digest": "c57de4edc390ed763d75e015294194b9147207a5ad77b6a3d01cd7ee22b0b010",
      "url": "cvmfs://cd.cern.ch/layers/c57de4edc390ed763d75e015294194b9147207a5ad77b6a3d01cd7ee22b0b010"
    },
    {
      "digest": "b2ea05c9d9a1b925610552e146549112aa893634e6a25ad18fd0cc295aec1cac",
      "url": "cvmfs://cd.cern.ch/layers/b2ea05c9d9a1b925610552e146549112aa893634e6a25ad18fd0cc295aec1cac"
    }
  ]
}

for each layer in the normal docker image we have have an URL, what the snapshotter does is to identify where the layers are in the filesystem and provide them inside a mountpoint.

The code itself needs to be cleaned up, and I am going to work on that quite soon, but I would really appreciate this conversation to keep moving forward.

Please, if I weren't clear on any particular point let me know.

follow on #2943