I would prefer this to be ImageStore rather then PullStore. You even differentiate between the two within the code.

imagestore SGTM I'll amend this.

Will the RO layer be found when Image Store is used?

e.g. another client is configured to not use the Image Store, and will create all the layers in the regular store. How would it work when the Image Store is set

I tried replaying this scenario, no RO layer will be not present if user switches to --imagestore in between, since --imagestore switches to a completely new image store. In such cases if users wants to use layers from previous store he should use additional-image-store and use original store as an additional-image-store manually.

Will the RO layer be found when Image Store is used?
e.g. another client is configured to not use the Image Store, and will create all the layers in the regular store. How would it work when the Image Store is set

I tried replaying this scenario, no RO layer will be not present if user switches to --imagestore in between, since --imagestore switches to a completely new image store. In such cases if users wants to use layers from previous store he should use additional-image-store and use original store as an additional-image-store manually.

I'd prefer the opposite to happen. If a --imagestore is specified, then images are pulled there and it is internally handled as an additional image store. The lookup would work in the same way it does now, first looking in the current layers then looking it up in the additional image stores.

When I build from a Containerfile, where would the layers end up?

@giuseppe I made a modification where when --image-store is used exisintg graphRoot is automatically used as an secondary additionalStore internal allowing users to use images both from newly added image-store and the original graphRoot, I have a added an integration test: split-store - use graphRoot as an additional store by default which tests this behavior. Do you think this suffice the use-case mentioned above.

Tagging @mrunalp for inputs.

Looks fine to me. Did we test with two disks?

I think it should work without any issues but let me test it out with two separate disk to confirm.

Okay I did test it with imagestore on a different disk than my original graphroot and it works fine. I used an external usb stick as imagestore and my current disk as original graphroot.

# Check if we mounted external usb
[fl@fedora ~]$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0  32.3M  1 loop /var/lib/snapd/snap/snapd/12704
loop1         7:1    0  55.4M  1 loop /var/lib/snapd/snap/core18/2128
loop2         7:2    0  13.9M  1 loop /var/lib/snapd/snap/universal-ctags/552
sdb           8:16   1  14.3G  0 disk 
├─sdb1        8:17   1  14.3G  0 part /run/media/fl/12a7bc12-59b6-4aa2-b40a-76f67faf28b7
└─sdb2        8:18   1     1M  0 part /run/media/fl/UEFI_NTFS
zram0       252:0    0     8G  0 disk [SWAP]
nvme0n1     259:0    0 476.9G  0 disk 
├─nvme0n1p1 259:1    0   600M  0 part /boot/efi
├─nvme0n1p2 259:2    0     1G  0 part /boot
└─nvme0n1p3 259:3    0 475.4G  0 part /home

# Check if external use contains directory for podman-storage
[fl@fedora ~]$ ls  /run/media/fl/12a7bc12-59b6-4aa2-b40a-76f67faf28b7
lost+found  podman-storage

# Check our storage.conf
[fl@fedora ~]$ cat /home/fl/.config/containers/storage.conf
[storage]

imagestore = "/run/media/fl/12a7bc12-59b6-4aa2-b40a-76f67faf28b7/podman-storage"

# First set imagestore to `none` and pull ubuntu
# Second set imagestore to `usb` and pull alpine

# podman images should show image from both original graph root and usb-stick
[fl@fedora bin]$ ./podman images
WARN[0000] The storage 'driver' option should be set in /home/fl/.config/containers/storage.conf. A driver was picked automatically. 
REPOSITORY                TAG         IMAGE ID      CREATED      SIZE        R/O
docker.io/library/alpine  latest      9ed4aefc74f6  11 days ago  7.34 MB     false
docker.io/library/ubuntu  latest      08d22c0ceb15  4 weeks ago  80.3 MB     true

# run container where image is not on imagestore but on original graph root
[fl@fedora bin]$ ./podman run --pull=never -it --rm ubuntu sh
WARN[0000] The storage 'driver' option should be set in /home/fl/.config/containers/storage.conf. A driver was picked automatically. 
# exit
[fl@fedora bin]$ 

# run container where image is on imagestore but not on original graph root
[fl@fedora bin]$ ./podman run --pull=never -it --rm alpine sh
WARN[0000] The storage 'driver' option should be set in /home/fl/.config/containers/storage.conf. A driver was picked automatically. 
/ # exit
[fl@fedora bin]$ 

@giuseppe @mrunalp ^

You need to update the containers-storage.conf.5.md

I am probably too sleep-deprived but I just can’t understand how this works, starting with dir2.

Why are all callers of dir2 special-casing this? I would expect a single location in this package to contain some code to find an existing image, which callers can then use without any conditionals. And I would expect a single location in this package to contain some code to determine where to create new images or store more data, which callers can then use without any conditionals.

This is clearly not that, and I can’t understand how this is intended to work.

Again, maybe it is just me. But please someone take another look.

Looking at it now, I don't think I fully understand it, either. With the option set, when the overlay driver creates and removes layers under the graph root, continuing to record things like the list of lowers there, it also seems to be creating and removing directories under the image store location, and it's neither obvious nor commented why it puts things in one place or the other. If my mostly-guesswork view of what the overlay driver is doing is correct, then the overlay driver is not correctly setting disk quotas.
I definitely don't understand what's supposed to be happening differently in the vfs driver.