fex: Add support for library forwarding #413255
Conversation
pkgs/by-name/fe/fex/package.nix
Outdated
|
|
||
| src = fetchFromGitHub { | ||
| owner = "FEX-Emu"; | ||
| owner = "neobrain"; |
There was a problem hiding this comment.
This is currently required for neobrain/FEX@5811914 , which will be included in the next FEX release (2506).
There was a problem hiding this comment.
It looks like quite a simple patch. Would it be possible to use substituteInPlace for that instead, like this? Or is there something else in that branch that is needed?
substituteInPlace ThunkLibs/GuestLibs/CMakeLists.txt \
--replace-fail "--target=i686-linux-unknown" "--target=i686-linux-gnu"
There was a problem hiding this comment.
The patch just got merged upstream and a release is scheduled later this week, so we can drop the use of my fork soon :)
There was a problem hiding this comment.
I compiled it and ran the tests, seems to work fine. Tested running basic binaries too. However I'm not able to test the library forwarding functionality as I don't (currently) have any graphical aarch64 host. I'll try to set up a VM to test it, but for now here is some feedback on just the package file :)
pkgs/by-name/fe/fex/package.nix
Outdated
|
|
||
| src = fetchFromGitHub { | ||
| owner = "FEX-Emu"; | ||
| owner = "neobrain"; |
There was a problem hiding this comment.
It looks like quite a simple patch. Would it be possible to use substituteInPlace for that instead, like this? Or is there something else in that branch that is needed?
substituteInPlace ThunkLibs/GuestLibs/CMakeLists.txt \
--replace-fail "--target=i686-linux-unknown" "--target=i686-linux-gnu"
pkgs/by-name/fe/fex/package.nix
Outdated
| llvmPackages.stdenv.mkDerivation (finalAttrs: { | ||
| pname = "fex"; | ||
| version = "2505"; | ||
| # version = "2505"; | ||
| version = "5811914a784cc6fdfdc90242ab7f8ea6ff39ec38"; |
There was a problem hiding this comment.
Maybe it's better to prefix it with 2505 or 2506 to at least indicate what base version it is? I'm not sure having just a raw commit hash as a version is good...
There was a problem hiding this comment.
IMO we should just wait for 2506 to be released
|
Oh also nixfmt is failing, please run |
Thanks for the suggestions and for testing so quickly! FWIW I think for Vulkan you should be able to run |
Could elaborate a bit more on this? What paths should be set exactly? Are the defaults appropriate? For me this PR works and I can run Portal 2 fine, but this feels a bit too easy... How can I verify that it's actually running with the forwarded libraries? Also I noticed that the
Could you explain in a little more detail how all this is supposed to work? Here is my current understanding:
|
Thanks for checking back on this: The defaults will not enable any library forwarding, so instead do the following:
If set up properly and library forwarding successfully kicks in, FEX should log a
You got it mostly right. We can't entirely rely on magic though, so each of the layers is a bit thicker than you might expect:
Due to how flexible the system needs to be to deal with all the oddities in each library, I implemented custom tooling for FEX to automate part of the boilerplate required in all this process:
It gets more complicated once you involve things like function pointers, but as far as packaging is concerned I've already gone into more detail than you probably cared about :)
I'm not familiar with RPATH |
FEX's CMake files use EDIT: Using |
|
I've been looking some more into the dlopen issue. As suspected,
Why though? It's the emulated application that depends on Vulkan, not FEX itself, so FEX seems like the wrong place to inject a
Sadly using
Now that I understand a little bit better what RPATH is: Does this require adding EDIT: Actually, even when using |
Well the emulated application depends on the x86 vulkan-loader. How would you guarantee that an aarch64 vulkan-loader is even present on the system? I don't see any other way than just adding it as a dep of FEX.
Indeed that seems like an issue. For now I would just kick this issue down the road, and focus on the currently available forwarded libraries.
Nothing guarantees that
Yes, I think for now we should add vulkan-loader as a dep. |
Seems like an okay (if unsatisfactory) solution for NixOS deployments. What would non-NixOS users do, where vulkan-loader will still fail? |
|
With the latest update, FEXConfig will properly initialize HostThunks and GuestThunks automatically if |
What would be a good way to do this? Is it possible to know and query the final store path of FEX itself and patch its own sources with that? Otherwise, I think I would have to fall back to inferring the location at runtime based on the FEX executable path or similar. |
If you are talking about patching FEX sources so it has a reference to the store path where it's installed, you can try |
|
|
||
| # Add include paths for thunkgen invocation | ||
| substituteInPlace ThunkLibs/HostLibs/CMakeLists.txt \ | ||
| --replace-fail "-- " "-- $(cat ${llvmPackages.stdenv.cc}/nix-support/libc-cflags) $(cat ${llvmPackages.stdenv.cc}/nix-support/libcxx-cxxflags) $NIX_CFLAGS_COMPILE" |
There was a problem hiding this comment.
If I move from stdenvGcc13 to stdenv(Gcc14), this fails to find C++ headers (which are located e.g. in 5nyv2zzsm5ckfrxji7jgmhf5nb31snwq-x86_64-unknown-linux-gnu-gcc-13.3.0/include/c++/13.3.0/type_traits). Any advice how I can retrieve that path symbolically? I tried using gcc-unwrapped, but that is somehow not recognized within the llvmPackages context.
There was a problem hiding this comment.
Actually turns out the HostLibs build is fine and it was the GuestLibs that were failing. I'm added the aarch64 libstdc++ header paths to the latter and it seems to compile fine (presumably it's the same set of headers as in pkgsCross.gnu64/32 anyway).
|
Just as a comment. I'm running on a apple silicon linux with 16k pages and I get an error when building this PR about a jemalloc pagesize mismatch: |
AFAIK the official upstream stance is that only 4k pages are supported by FEX. |
|
Right when running programs. I would expect to build this at least to be able to use muvm to run fex. But if the case that a build does not work it will mean the entire fex build with need to be done in muvm as well. |
You would have to set |
Perfect, thanks! The correct paths are now baked into the FEX executables, so now the only configuration needed is ticking the FEXConfig checkboxes for libraries that should be forwarded. |
476ba94 to
fb7f952
Compare
Thanks for re-checking! I rebased and squashed the patches in the PR now (no other change). I've been thinking some more about the "dlopen won't find /usr/lib64/libXYZ.so" problem some more. I think the most reasonable (if unsatisfying) option is to offload that problem to the user, who need to ensure the libraries are visible at runtime. For NixOS, this could be done using nixGL for OpenGL and Vulkan, and If that's okay, this should be ready to merge now! |
In NixOS this can probably be abstracted with a module and FHS environment for user-specified packages that should be wrapped, similar to how Steam is wrapped, although I'm not sure. I'll try and implement it after this PR is merged. |
|
Currently testing on non-NixOS (postmarketOS), with a flake that generates a dev shell, I want it to just use Mesa from nixpkgs: devShells.aarch64-linux.default =
armpkgs.mkShell { packages = [ armpkgs.fex armpkgs.mesa pkgs.vulkan-tools ]; };Qt's X11 backend doesn't like freedreno(?)So I've also had to add Then in FEXConfig I ticked logging + host libraries (drm, Vulkan, WaylandClient, GL) and saved the config. Somehow the config is ignored, there's no logging nor are thunks used..?! Looking at strace, config is read, thunk libs are checked.. but x86-64 libvulkan is indeed loaded without replacement: Adding the explicit path in a {
"DB": {
"Vulkan": {
"Library": "libvulkan-guest.so",
"Overlay": [
"/nix/store/nszik1q8ffmvsqk54kbc75dwyxwvi2nm-vulkan-loader-1.4.313.0/lib/libvulkan.so",
"/nix/store/nszik1q8ffmvsqk54kbc75dwyxwvi2nm-vulkan-loader-1.4.313.0/lib/libvulkan.so.1",
"/nix/store/nszik1q8ffmvsqk54kbc75dwyxwvi2nm-vulkan-loader-1.4.313.0/lib/libvulkan.so.1.4.313",
"@PREFIX_LIB@/libvulkan.so",
"@PREFIX_LIB@/libvulkan.so.1",
"@HOME@/.local/share/Steam/ubuntu12_32/steam-runtime/pinned_libs_64/libvulkan.so.1"
]
}
}
}makes it attempt the replacement… and crash :((same with libwayland-client first if I also add that) …wait, does the thunk even have a way to find the Nix aarch64 libs? uuuhh.. hm. X11 is busted (not good for Steam…) but let's see: Success! \o/ (Still crashes after listing all the info) vkcube next?Oops, let's add wat??? Hm, let's try something else- Lovely. Missing the newest changes I see. And without host libwayland? …actually, vkmark with X11 backend works! So, it seems like FHS wrapping might not be necessary, if you make sure the |
pkgs/by-name/fe/fex/package.nix
Outdated
| --replace-fail "FEX_CONFIG_OPT(ThunkGuestLibs32, THUNKGUESTLIBS32);" "fextl::string ThunkGuestLibs32() { return \"$out/share/fex-emu/GuestThunks_32/\"; }" | ||
| substituteInPlace Source/Tools/LinuxEmulation/Thunks.cpp \ | ||
| --replace-fail "FEX_CONFIG_OPT(ThunkHostLibsPath, THUNKHOSTLIBS);" "fextl::string ThunkHostLibsPath() { return \"$out/lib/fex-emu/HostThunks/\"; }" \ | ||
| --replace-fail "FEX_CONFIG_OPT(ThunkHostLibsPath32, THUNKHOSTLIBS32);" "fextl::string ThunkHostLibsPath32() { return \"$out/lib/fex-emu/HostThunks/\"; }" |
There was a problem hiding this comment.
uh oh! should be ../HostThunks_32/ here!!
[ERROR] Tried to call guest function with arguments packed to a 64-bit address
Illegal instruction
There was a problem hiding this comment.
Oof yes, thanks for catching this!
EDIT: Fixed!
There was a problem hiding this comment.
heh, looks like the just-released 2507 also changed from having a separate option for 32-bit to appending _32 to the ThunkHostLibsPath, which would've fixed this anyway!
fb7f952 to
2099d24
Compare
Awesome, thanks for testing!
Yeah we're basically stumbling upon all the same reasons out of which nixGL was developed. Sadly we can't use the store paths without adding false dependencies in FEX to each library that it supports forwarding for. I like andre4ik3's idea of using a module on NixOS to transparently add an FHS env, but non-NixOS users will probably be stuck with nixGL and manually juggling |
well, nixGL does a bunch of stuff we don't care about (nvidia, vdpau…) and like 2-5 lines that actually matter. To make the host GLVND libGL find Mesa, all we need is either an LD_LIBRARY_PATH to the host Mesa, or a manual ad-hoc hacks turned into nix code that modifies the fex package from here let
thunksConf = armpkgs.writeText "nixthunks.json" (builtins.toJSON { DB = {
GL = {
Library = "libGL-guest.so";
Overlay = builtins.map (m: "${m.pre}/${m.name}") (armpkgs.lib.cartesianProduct {
pre = ["${x64pkgs.libglvnd}/lib" "${i686pkgs.libglvnd}/lib" "@PREFIX_LIB@"];
name = ["libGL.so" "libGL.so.1" "libGL.so.1.2.0" "libGL.so.1.7.0"];
});
};
Vulkan = {
Library = "libvulkan-guest.so";
Overlay = builtins.map (m: "${m.pre}/${m.name}") (armpkgs.lib.cartesianProduct {
pre = [
"${x64pkgs.vulkan-loader}/lib" "${i686pkgs.vulkan-loader}/lib"
# does not actually catch it by that path:
"/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu"
"/usr/lib/pressure-vessel/overrides/lib/i386-linux-gnu"
"@HOME@/.local/share/Steam/ubuntu12_32/steam-runtime/usr/lib/x86_64-linux-gnu"
"@HOME@/.local/share/Steam/ubuntu12_32/steam-runtime/usr/lib/i386-linux-gnu"
"@PREFIX_LIB@"
];
name = ["libvulkan.so" "libvulkan.so.1" "libvulkan.so.1.3.239" "libvulkan.so.${x64pkgs.lib.getVersion x64pkgs.vulkan-loader}"];
});
};
drm = {
Library = "libdrm-guest.so";
Overlay = builtins.map (m: "${m.pre}/${m.name}") (armpkgs.lib.cartesianProduct {
pre = ["${x64pkgs.libdrm}/lib" "@PREFIX_LIB@"];
name = ["libdrm.so" "libdrm.so.2" "libdrm.so.2.4.0" "libdrm.so.${x64pkgs.lib.getVersion x64pkgs.libdrm}"];
});
};
asound = {
Library = "libasound-guest.so";
Overlay = builtins.map (m: "${m.pre}/${m.name}") (armpkgs.lib.cartesianProduct {
pre = ["${x64pkgs.alsa-lib}/lib" "@PREFIX_LIB@"];
name = ["libasound.so" "libasound.so.2" "libasound.so.2.0.0"];
});
};
WaylandClient = {
Library = "libwayland-client-guest.so";
Overlay = builtins.map (m: "${m.pre}/${m.name}") (armpkgs.lib.cartesianProduct {
pre = ["${x64pkgs.wayland}/lib" "${i686pkgs.wayland}/lib" "@PREFIX_LIB@"];
name = ["libwayland-client.so" "libwayland-client.so.0" "libwayland-client.so.0.20.0"
"libwayland-client.so.0.${x64pkgs.lib.removePrefix "1." (x64pkgs.lib.getVersion x64pkgs.wayland)}"];
});
};
}; });
# at least with turnip, steamwebhelper using vulkan results in a lot of glitches
# XXX: Vulkan=0 does not apply from there?? have to copy the file to ~/.config/.fex-emu/AppConfig/ ????
steamwebConf = armpkgs.writeText "steamweb.json" (builtins.toJSON { ThunksDB = { GL = 0; Vulkan = 0; }; });
patchedFex = armpkgs.fex.overrideAttrs (old: {
postFixup = ''
# GL already includes the path to host libglvnd!
patchelf --add-rpath ${armpkgs.lib.makeLibraryPath [armpkgs.vulkan-loader]} \
$out/lib/fex-emu/HostThunks/libvulkan-host.so \
$out/lib/fex-emu/HostThunks_32/libvulkan-host.so
patchelf --add-rpath ${armpkgs.lib.makeLibraryPath [armpkgs.libdrm]} \
$out/lib/fex-emu/HostThunks/libdrm-host.so
patchelf --add-rpath ${armpkgs.lib.makeLibraryPath [armpkgs.wayland]} \
$out/lib/fex-emu/HostThunks/libwayland-client-host.so \
$out/lib/fex-emu/HostThunks_32/libwayland-client-host.so
patchelf --add-rpath ${armpkgs.lib.makeLibraryPath [armpkgs.alsa-lib]} \
$out/lib/fex-emu/HostThunks/libasound-host.so
cp ${thunksConf} $out/share/fex-emu/ThunksDB.json
cp ${steamwebConf} $out/share/fex-emu/AppConfig/steamwebhelper.json
'';
});
in ……and I got Steam with Portal 2 (GL) working eventually! However, the one really cursed issue I've encountered is that the steamapps' shebangs are not respected, everything is forcibly launched with a Nix bash, so I've had to do awful things like adding to |
|
Not entirely familiar with the nixpkgs process - what's left to be done for this to be merged? Besides the now-fixed 32-bit issue, this feature seems to be working fine for everyone who tested (thanks again!). The runtime setup needs followup work but that's fine since libraries need to be manually enabled after installation. @valpackett Pretty cool you got it working in the end! I'm assuming the main intent was to document your research and progress here, but if I've overlooked proposed changes to the PR let me know. |
@RossComputerGuy Sounds good, thanks - happy to adapt as needed whenever you find time for the review. |
2099d24 to
469fab0
Compare
|
Rebased on the recently merged FEX 2507.1 update, which includes my upstream changes that allow cutting down the |
|
Sorry to ask in this post, but do we need a NixOS rootfs to run OpenGL/Vulkan-related applications? |
FEX isn't designed to work without a RootFS, so this is expected. Library forwarding only removes the need for having x86 GL/Vulkan drivers (and other forwarded libraries) in the rootfs. |
Any prebuilt rootfs for nixos? A Nixos rootfs stores almost everything in /nix/store, is it suffice to provide a rootfs containing /run/opengl-driver. |
|
FEX's RootFS feature is only really needed for resolving the same paths differently based on emulated vs. native execution. FEX absolutely works fine without a RootFS in a properly mixed/multilib system – I managed to get it to run Steam games both using Nix-on-non-NixOS (see research above) and Flatpak with hacks. @qbisi what you're probably missing for library forwarding is making ThunksDB.json point to the Nix store paths of the x86_64 libGL that your glxinfo loads, see #413255 (comment) above. (Specifically for forwarding, you're not required to have an …ThunksDB is actually the most "painful" part of using Nix for this. Maybe the package could provide a ThunksDB generator function that takes in x86/64
|
|
|
Fex uses jemalloc which doesn't like my desktop's 64k page size... |
The FEX derivation has a note about non-4k systems, and this is already an issue in the existing FEX derivation: # Unsupported on non-4K page size kernels (e.g. Apple Silicon)
doCheck = true;I'm not very familiar with nixpkgs-review, but perhaps you can post the thoughts you had without the tool since your system does not support running the tests? |
enables* (It used to be explicitly disabled I think) But yes, on non-4k pagesize you need to do |
Oops, yes I was just remembering the comment in the derivation and thought it was still disabled hence, but you're right. Thanks 😅 |
Library forwarding (misleadingly often called "thunks") is a FEX feature that redirects API calls to specific ARM64 host libraries instead of emulating the full x86 builds of those libraries.
This has multiple benefits:
To test this feature locally, make sure to run FEXConfig and enable the various libraries in the Libraries tab.
Things done
The new derivation successfully builds support for all libraries FEX can forward by creating a development RootFS containing glibc/stdlibc++ headers and some additional library headers. As far as the build-side is concerned, this is all perfectly working. (I was actually surprised how easy nix makes this!)
Unfortunately there is a fundamental roadblock I'm unsure how to resolve: To enable library forwarding (for example for libvulkan), FEX uses a
libvulkan-host.sowrapper library thatdlopen()s the true ARM64 libvulkan.so installed on the host. Currently that dlopen call fails, since it only searches the nix store. Looking in the nix store doesn't really make sense here though, since FEX is more of a runtime and the true library consumer is the emulated application (which itself is typically not built by nix).This problem isn't exclusive to Vulkan but probably applies to all other libraries FEX can forward. I'm guessing there is a "right way" to do this on NixOS, but ideally a non-NixOS nix-shell should be able to just load an externally managed
/usr/lib64/libvulkan.sowithout further ado.Does anyone with more nix experience than me have any advice on how to proceed here?
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.