Skip to content

Conversation

smoser
Copy link
Contributor

@smoser smoser commented Apr 25, 2023

stacker was only using anuvu/squashfs import to get the size of the squashfs file (from its superblock, rather than from 'stat').

That go library used libsquashfs-tools-ng as a c-go dependency, which meant a build and runtime dependency on it.

The change here is to drop that import and instead use some native go.

@smoser smoser requested review from rchincha and hallyn as code owners April 25, 2023 20:05
@smoser smoser force-pushed the feature/no=more-anuvu-squashfs-dep branch 2 times, most recently from f330865 to 4d059dd Compare April 25, 2023 20:29
@smoser
Copy link
Contributor Author

smoser commented Apr 25, 2023

Given the superblock.go is copied from diskfs, I wonder if I need to add something like this:

diff --git a/pkg/squashfs/superblock.go b/pkg/squashfs/superblock.go
index a90ab5b..1934e23 100644
--- a/pkg/squashfs/superblock.go
+++ b/pkg/squashfs/superblock.go
@@ -1,3 +1,10 @@
+/* This file was initially copied from go-diskfs [1].  The copied portion is
+   Copyright (c) 2017 Avi Deitcher and licensed under the terms of the MIT
+   license [2].
+    [1] https://github.com/diskfs/go-diskfs/filesystem/squashfs/superblock.go
+    [2] https://opensource.org/licenses/MI
+*/
+
 package squashfs
 
 import (

@smoser smoser force-pushed the feature/no=more-anuvu-squashfs-dep branch 2 times, most recently from 437083b to 6c4b972 Compare April 25, 2023 21:07
@tych0
Copy link
Collaborator

tych0 commented Apr 25, 2023

My only thought is that it would be nice to import superblock.go instead of copying it, but I suppose you tried that and it introduces some other C deps that you have to build?

@smoser
Copy link
Contributor Author

smoser commented Apr 25, 2023

The next trick to drop dependencies would be to just "shell out" to cryptsetup and dmsetup.
Normally I would argue heavily against shelling out. But I think it is different here for a couple reasons:

  • 'dmsetup' and 'cryptsetup' seem to be reasonable dependencies to me . They are very mature cli interfaces that are very standard on linux.
  • dmsetup and cryptsetup cover the vast majority of the dynamic dependencies that stacker needs to use.

when atomfs builds, it gets a dependency (from ldd) like the following. I marked the ones that come from cryptsetup with a '-'.

 /lib64/ld-linux-x86-64.so.2 (0x00007f8a04774000)
-libargon2.so.1 => /lib/x86_64-linux-gnu/libargon2.so.1
-libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1
-libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3
-libcryptsetup.so.12 => /lib/x86_64-linux-gnu/libcryptsetup.so.12
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
-libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
-libjson-c.so.5 => /lib/x86_64-linux-gnu/libjson-c.so.5
 liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1
 liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 
-libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
-libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 
-libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 
-libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 
-libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 
 libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 
 linux-vdso.so.1 

So basically if we shelled out, i think we could maybe get atomfs's deps down to only those compression libraries.

(note, stacker probably has others, i was just looking at atomfs now, and it
only imports stackerbuild.io/stacker/pkg/atomfs and
stackerbuild.io/stacker/pkg/mount, but a lot of functionality comes from just that)

@smoser smoser force-pushed the feature/no=more-anuvu-squashfs-dep branch from 6c4b972 to b54b54c Compare April 25, 2023 21:25
@smoser
Copy link
Contributor Author

smoser commented Apr 26, 2023

My only thought is that it would be nice to import superblock.go instead of copying it, but I suppose you tried that and it introduces some other C deps that you have to build?

All diskfs exposes is a Filesystem interface. You do a Read and get one back, but then you don't have any way to get 'size' or anything even though they're in the struct.

I could propose an upstream change to expose them in a consistent manner for each of the Filesystem interface implementations; this is just lazy.

@smoser smoser force-pushed the feature/no=more-anuvu-squashfs-dep branch from b54b54c to 25cd403 Compare April 26, 2023 01:26
The superblock.go comes from
https://github.com/diskfs/go-diskfs/blob/master/filesystem/squashfs/

diskfs is MIT licensed and this one Apache, so copy is fine.

The project doesn't expose any of the superblock internals (even size),
so we can't just go import it.

The only change from diskfs's current tip is to replace fmt.Errorf with
errors.Errorf.

Signed-off-by: Scott Moser <smoser@brickies.net>
@smoser smoser force-pushed the feature/no=more-anuvu-squashfs-dep branch from 25cd403 to 46f2fc3 Compare April 26, 2023 01:36
@tych0
Copy link
Collaborator

tych0 commented Apr 26, 2023 via email

@smoser
Copy link
Contributor Author

smoser commented Apr 26, 2023

You could downcast to the right one if the individual implentations are public.

squashfs.Filesystem is what you get back from the Read. It has a '[superblock]' of type *superblock member that is not exposed (lower case). The superblock type has a non-exposed uint64 size.

Maybe i could manage to get it through an import, but not in any way that was clean or imo preferable to this one file copy.

@smoser
Copy link
Contributor Author

smoser commented Apr 26, 2023

oh for pete's sake. test taht failed is tests/caching.bats "can read previous version's cache".
That test does a clone of stacker and attempts to build it. I can't build the previous version of stacker because I don't have one of it's dependencies (libsquashfs1 -- thats what i'm tryint to fix).

smoser added 2 commits April 25, 2023 22:12
This test won't work because it tries to build an old
(upstream/main) version of stacker.  The system is missing
the libsquashfs1 dependency, so this would not be expected
to work.

Signed-off-by: Scott Moser <smoser@brickies.net>
After removing the anuvu/squashfs import we no longer need to build
or install libsquashfs1 from squashfs-tools-ng.

Signed-off-by: Scott Moser <smoser@brickies.net>
@smoser smoser force-pushed the feature/no=more-anuvu-squashfs-dep branch from 46f2fc3 to 8fdbbe1 Compare April 26, 2023 02:12
@tych0
Copy link
Collaborator

tych0 commented Apr 26, 2023

Yeah, you'd have to do reflection, and then you're at risk of some non-public API change. Another option would be to fork + go replace, but copy seems fine too.

@smoser
Copy link
Contributor Author

smoser commented Apr 26, 2023

I think

The next trick to drop dependencies would be to just "shell out" to cryptsetup and dmsetup. Normally I would argue heavily against shelling out. But I think it is different here for a couple reasons:
...

I guess there are the following cases

  1. kernel 'mount' of a squashfs+dmverity image
  2. creation of verity data and root hash

It seems that mount is pretty straightforward with dmsetup or cryptsetup. creation might be a little more difficult.
if we separated use and creation into separate directories, then a consumer (such as 'machine' or 'atomfs') that only needed mount would not end up with the library dep.

Copy link
Contributor

@rchincha rchincha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants