Docker in Docker doesn't seem to find / forward the volume mounts #1283
Description
Checklist
- I've looked through the issues and pull requests for similar reports
Describe your issue
Using a clean Ubuntu Server 22.04.2 LTS system, running cross inside of one Docker container gives me the same error Build fails with sh: 1: cargo: not found listed in #260.
If /var/lib/docker is not bind mounted to the outer Docker instance, when cross goes to forward some of those mounts, there is nothing to forward to the inner Docker instance:
rustuser@rust-build:~/workspace$ cross build -vv --target aarch64-unknown-linux-gnu
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ /usr/bin/docker
+ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build -vv --target aarch64-unknown-linux-gnu'
sh: 1: cargo: not found
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
If you run the container above and drop into a shell, you can examine the paths available in /var/lib and see what is missing.
rustuser@rust-build:~/workspace$ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh
$ ls
target
$ id
uid=1000 gid=1000 groups=1000
$ ls /
bin boot cargo common.sh dev etc home lib lib.sh lib64 linux-image.sh linux-runner media mnt opt proc qemu root run rust sbin srv sys target tmp usr var xargo
rustuser@rust-build:~/workspace$ ls -l /var/lib
total 48
drwxr-xr-x 1 root root 4096 Jun 28 14:25 apt
drwxr-xr-x 1 root root 4096 Jun 28 14:26 dpkg
drwxr-xr-x 2 root root 4096 Feb 22 10:51 git
drwxr-xr-x 2 root root 4096 Apr 2 13:55 misc
drwxr-xr-x 2 root root 4096 Jun 12 02:00 pam
drwxr-xr-x 2 root root 4096 Jun 13 05:29 python
drwxr-xr-x 3 root root 4096 Jun 28 14:25 sudo
drwxr-xr-x 1 root root 4096 Jun 13 05:29 systemd
drwxr-xr-x 3 root root 4096 Jun 13 05:29 ucf
rustuser@rust-build:~/workspace$ ls -l /var/lib/docker
ls: cannot access '/var/lib/docker': No such file or directory
I don't have a solution to this yet. It should be possible to bind mount /var/lib/docker, but the root:root permissions will be an issue.
System Info
Host system:
$ uname -a
Linux boombox 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
$ docker --version
Docker version 24.0.2, build cb74dfc
Outer Docker is rust:latest (debian:11, 11.7, bullseye, bullseye-20230612):
$ docker --version
Docker version 20.10.5+dfsg1, build 55c4c88
$ cross --version
cross 0.2.5
[cross] note: Falling back to `cargo` on the host.
cargo 1.70.0 (ec8a8a0ca 2023-04-25)
Inner Docker is the aarch64-unknown-linux-gnu image.
What target(s) are you cross-compiling for?
aarch64-unknown-linux-gnu
Which operating system is the host (e.g computer cross is on) running?
- macOS
- Windows
- Linux / BSD
- other OS (specify in description)
What architecture is the host?
- x86_64 / AMD64
- arm32
- arm64 (including Mac M1)
What container engine is cross using?
- docker
- podman
- other container engine (specify in description)
cross version
cross 0.2.5
Example
Examining the Docker command line that is run by cross, it finds the /var/lib/docker mounts, I believe by querying the docker.sock, but because the outer Docker container does not mount /var/lib/docker, the inner Docker can't then access any of those mounts.
It would be an issue anyways as the outer Docker container marks all mounts root:root in /var/lib/docker.
Additional information / notes
I'm trying to think of workarounds.
One of which would be to run rustup in my non-admin account in the host system, and then run the aarch64-unknown-linux-gnu container.
Another workaround might be Cross.toml offering a way to specify what folders to mount for which tools.
Another workaround might be to use a Docker managed volume for the tools and cross.
Final option would be to reproduce the mounts in a docker-compose.yml and reuse the cross-rs containers from the host system.