Squashfs is a modern archive format that supports mounting and random access. It is efficient and endorsed by Linux kernel. For archival use, it is considerably better in both accessibility and space: even if sacrificing the random access by using .sqfs.xz, it can be >15% smaller than .tar.xz on my real data.

The downsides of it are:

1. It collects metadata together rather than scattering and streaming them like TAR. Thus we need to maintain metadata in memory. Though it typically does not exceed MB range.
2. It stores some metadata in header which can only be know after completion. It means we need Seek + Write for the output file, or to use fallocate() with FALLOC_FL_INSERT_RANGE (Linux-specific) to finalize the output file. libarchive seems to have troubles with this. It's also non-trivial if we are chaining compression on the output side (.sqfs.xz).

There are a few libraries for squashfs read and write. The most popular one seems to be backhand