NixOS test framework: Add macOS VMs #429189

roberth · 2025-07-28T20:49:01Z

Goal: enable automated testing of Nix on macOS

Done

nix run -f . nixosTests.nixos-test-driver.multi-os.driverInteractive launches macOS

TODO:

add the testing backdoor service somehow
requiredFeatures = ["apple-branded-computer"]; to support license compliance
do not use getFlake somehow (not sure if those parts can be in-tree)
factor generic parts out of qemu-vm.nix

Things done

Add a 👍 reaction to pull requests you find important.

This is horrible code, and it doesn't have a backdoor service yet. Maybe it needs to be an SSH connection? Would be nice to send commands to an offline machine though.

Mic92 · 2025-07-29T08:02:18Z

nixos/tests/nixos-test-driver/multi-os/qemu-vm.nix

+          ln -s ${hostPkgs.writeScript "run-nixos-vm" startVM} $out/bin/run-${config.system.name}-vm
+        '';
+    virtualisation.qemu.options = [
+      "-enable-kvm" # ??


Is this intended to run on Linux or on macOS? There is no such thing as kvm on macOS.

You're allowed to use Linux as a hypervisor of sorts on your Apple computer.

I think these settings will not work on macOS hosts, though? That would significantly diminishes the usefulness given the fact that you’d need macOS to build a nix‐darwin configuration to run in the VMs to begin with. We have darwin.linux-builder to solve that bootstrapping problem one way, but it’s less tractable the other way around. It’s also not really practical to get cloud Macs running Linux.

Ah, and I should mention that running aarch64-darwin guests on anything other than aarch64-darwin is much less of a thing than even with x86_64-darwin, so the aarch64-linux to aarch64-darwin mapping elsewhere in the PR is dubious. Apple do ship a kernel variant for VMs that doesn’t rely on their proprietary ISA extensions and I believe people have gotten something running, but in practical terms you need to be using the system Virtualization framework. Even QEMU’s Apple Silicon guest support requires an Apple Silicon host, albeit with the lower‐level Hypervisor framework and more bespoke implementation of the macOS VM device model.

Mic92 · 2025-07-29T08:04:30Z

nixos/tests/nixos-test-driver/multi-os.nix

+  nixThePlanet = builtins.getFlake "github:MatthewCroughan/NixThePlanet/c9d159dc2e0049e7b250a767a3a01def52c3412b";
+
+   # no specific commit in particular
+  nix-darwin = builtins.getFlake "github:nix-darwin/nix-darwin/e04a388232d9a6ba56967ce5b53a8a6f713cdfcf";


I am wondering if the correct place of this test framework would be nix-darwin itself? Afaik all the module system of the test framework should be accessible from there as well.

Yes, I was thinking to move things when it works, and the refactoring is done.
Refactoring will be important, because when the code is split out, it will need a reasonably clean interface to attach to.

I suspect some parts should live in NixThePlanet and some parts will make more sense to be put into nix-darwin where they can be shared between both x86 and ARM VMs

Mic92 · 2025-07-29T08:05:06Z

nixos/tests/nixos-test-driver/multi-os.nix

+    # A macOS VM
+    daisy = {
+      system.stateVersion = 6;
+      virtualisation.diskImage = "${nixThePlanet.packages.${hostPkgs.stdenv.hostPlatform.system}.macos-ventura-image}";


Could this be an FOD?

It's an installation process into a disk image, and it's not byte for byte reproducible, so we can't slap a hash on it.

emilazy

Thanks for working on this! I have wanted VM tests for nix‐darwin for a long while now. There is some prior art by @Enzime (nix-darwin/nix-darwin#1000) and @winterqt (I believe unpublished), but those didn’t use the full NixOS VM test framework. I would love to be able to use the existing mechanisms to test nix‐darwin, and remember looking into the possibility of using QEMU to make that possible; it’s very cool to see it proven out this far and I’m curious what led you to doing this work.

However, I have significant concerns about this approach. The code here only works with x86_64-darwin; the latest macOS won’t even run on that platform in a year’s time, and although we’re still working out the exact deprecation timeline, the platform clearly has a limited remaining lifespan, and it’s only still built on Hydra with slow user‐space emulation on aarch64-darwin hardware.

There has been work to make QEMU able to host Apple Silicon macOS guests, and I see that at least some of it has been merged upstream. However, there are severe limitations: it cannot handle installing a new VM and it doesn’t support macOS ≥ 13 guests. macOS 12 is already end‐of‐life, and the plan is for Nixpkgs 25.11 to support only macOS ≥ 14, as macOS 13 will be end‐of‐life by the time of its release. So using QEMU for aarch64-darwin guests seems like a non‐starter at present.

I don’t think we would want to pursue x86_64-darwin‐only functionality at this juncture, and I wouldn’t expect that QEMU’s aarch64-darwin guest support will become production‐ready any time soon. Therefore, it doesn’t seem like a QEMU‐based solution is the way to go, at least for now.

The reason QEMU’s support for this is so patchy is that it builds on top of the raw Hypervisor framework rather than the higher‐level Virtualization framework, which has first‐class support for macOS guests. There are tools like Tart that use the Virtualization framework to manage automation of macOS guests on Apple Silicon, including things like automated macOS installation; that’s what was used in @Enzime’s earlier work.

Unfortunately, Tart is non‐free, although of course given that macOS is as well, that may not be a hard blocker; I think there are alternatives, but I haven’t done a deep dive on the options. It only supports Apple Silicon, but that doesn’t seem like as big of a problem these days. In any case, the test framework would require further adaptation to use with any non‐QEMU VMM, of course, but I don’t see any other practical option for macOS VM tests at this point.

FWIW, the macOS EULA only allows using macOS on two guest VMs simultaneously on a given host (and this is technologically enforced by the Virtualization framework). So we will be limited to two nodes for macOS tests, which would be rather unfortunate perf‐wise in terms of running a full nix‐darwin test suite, but oh well.

I’ve left some additional comments on the implementation inline, but they’re probably not too relevant given the above issues with the QEMU‐based approach.

emilazy · 2025-07-30T03:01:45Z

nixos/tests/nixos-test-driver/multi-os/qemu-vm.nix

+          ln -s ${hostPkgs.writeScript "run-nixos-vm" startVM} $out/bin/run-${config.system.name}-vm
+        '';
+    virtualisation.qemu.options = [
+      "-enable-kvm" # ??


I think these settings will not work on macOS hosts, though? That would significantly diminishes the usefulness given the fact that you’d need macOS to build a nix‐darwin configuration to run in the VMs to begin with. We have darwin.linux-builder to solve that bootstrapping problem one way, but it’s less tractable the other way around. It’s also not really practical to get cloud Macs running Linux.

emilazy · 2025-07-30T03:23:02Z

nixos/tests/nixos-test-driver/multi-os.nix

+
+let
+
+  nixThePlanet = builtins.getFlake "github:MatthewCroughan/NixThePlanet/c9d159dc2e0049e7b250a767a3a01def52c3412b";


I don’t think we should use builtins.getFlake in Nixpkgs, especially to a repository that contains a bunch of legally dubious stuff. (But I assume this is just a proof‐of‐concept and the intention was to pull the relevant functionality in‐tree.)

emilazy · 2025-07-30T03:24:47Z

nixos/tests/nixos-test-driver/multi-os.nix

+    # A macOS VM
+    daisy = {
+      system.stateVersion = 6;
+      virtualisation.diskImage = "${nixThePlanet.packages.${hostPkgs.stdenv.hostPlatform.system}.macos-ventura-image}";


Ventura is going to be EOL by the release of Nixpkgs 25.11.

emilazy · 2025-07-30T03:30:22Z

nixos/tests/nixos-test-driver/multi-os/qemu-vm.nix

+      "-device" "usb-ehci,id=ehci"
+      "-device" "nec-usb-xhci,id=xhci"
+      "-global" "nec-usb-xhci.msi=off"
+      "-device" ''isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"''


This string has, of course, been the subject of substantial legal dispute. I don’t know if there’s any consensus position on whether there is a problematic copyright or DMCA aspect to including it here. Psystar did lose their case, though.

(It’s only relevant for x86_64-darwin QEMU hacks, not aarch64-darwin using the Virtualization framework, so I think it probably isn’t relevant these days, but I figured we would require end users to supply this string as an assertion that they accept that it’s their responsibility to comply with the EULA.)

After reading about the Psystar Corporation, it sounds like they were grifters and this string is now publicly available in court filings:

https://web.archive.org/web/20201108001726/https://www.rcfp.org/wp-content/uploads/imported/20120105_202426_apple_sealing.pdf

Some relevant links:

https://dortania.github.io/OpenCore-Install-Guide/why-oc.html#legality-of-hackintoshing
https://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/#sec_4
https://en.wikipedia.org/wiki/Hackintosh#Legal_issues_and_Apple's_objections
https://en.wikipedia.org/wiki/Psystar_Corporation

emilazy · 2025-07-30T03:33:42Z

nixos/tests/nixos-test-driver/multi-os.nix

+        "-drive" "if=pflash,format=raw,readonly=on,file=${nixThePlanet.legacyPackages.${hostPkgs.stdenv.hostPlatform.system}.osx-kvm}/OVMF_CODE.fd"
+        "-drive" "if=pflash,format=raw,readonly=on,file=${nixThePlanet.legacyPackages.${hostPkgs.stdenv.hostPlatform.system}.osx-kvm}/OVMF_VARS-1920x1080.fd"
+        "-drive" "id=OpenCoreBoot,if=virtio,snapshot=on,readonly=on,format=qcow2,file=${nixThePlanet.legacyPackages.${hostPkgs.stdenv.hostPlatform.system}.osx-kvm}/OpenCore/OpenCore.qcow2"


OpenCore is, of course, open, and we have from‐source builds of EDK II and OVMF. It would be unfortunate to consume it in the form of opaque binary blobs from the somewhat messy OSX-KVM repository that contains things like decompiled macOS code. (Again only relevant for x86_64-darwin anyway, though.)

roberth · 2025-07-30T06:30:51Z

Hey @emilazy, thank you so much for your elaborate feedback.
I approached it from the test framework angle, so the actual macOS stuff is very proof of concept.

I'd adopted the qemu approach because I could reuse a working example, but I don't think it needs to be qemu.
The coupling with qemu doesn't seem all that bad. I added most of the qemu related options because having them seemed useful, not because the were required by the test framework. The test driver python code can be adjusted by subclassing Machine per virtualisation technology. (Something like #139788)

only allows using macOS on two guest VMs simultaneously

I hope the Virtualization Framework has similar networking functionality or is compatible with qemu. I guess this is a good use case for mixing macOS and NixOS VMs: one or two client VMs, and however many application / backend / infra VMs.
(A true client OS. I suppose the irony of their marketing's appeal to creativity isn't lost on anyone.)

Both technologically and legally I suppose it would be simpler to start running BSDs using this, but those and macOS can both be done using code like this.

hsjobeki

i kind of like this change because it opens up the possibility to properly mimic the open submodules type. freeformType is nice, but has its problems, with infinite recursion when refering to config.

The same change could also be applied to listOf but thats a bit harder because of the list merging not ensuring the index stays constant after merging 🤔

roberth · 2025-08-19T18:52:18Z

properly mimic the open submodules type.

Needing this feels like bad design, but you're right, and it is entirely possible to, for example, implement type checked "Hungarian notation" with this.

freeformType is nice, but has its problems, with infinite recursion when refering to config.

I guess that's inherently like attrsOf instead of lazyAttrsOf. Here it is configurable.
types.record may be an alternate solution to that problem.

The same change could also be applied to listOf

Probably best solved by avoiding lists (which are also not overridable, and whose ordering is susceptible to changes in imports structure), but it could be done if someone wants to bother with that.

roberth added 2 commits July 28, 2025 22:43

WIP: lib.types.attrsWith: Add getElemType argument for name param

4cd56ed

WIP: launch macOS from NixOS test framework

e71cdb2

This is horrible code, and it doesn't have a backdoor service yet. Maybe it needs to be an SSH connection? Would be nice to send commands to an offline machine though.

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Jul 28, 2025

roberth mentioned this pull request Jul 28, 2025

VM tests nix-darwin/nix-darwin#1552

Open

Mic92 reviewed Jul 29, 2025

View reviewed changes

Mic92 mentioned this pull request Jul 29, 2025

Run on macos numtide/nix-vm-test#97

Open

emilazy reviewed Jul 30, 2025

View reviewed changes

roberth mentioned this pull request Aug 4, 2025

mention what versions of MacOS are supported NixOS/nix#13675

Open

2 tasks

hsjobeki reviewed Aug 15, 2025

View reviewed changes

nixpkgs-ci bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 30, 2025


		let

		nixThePlanet = builtins.getFlake "github:MatthewCroughan/NixThePlanet/c9d159dc2e0049e7b250a767a3a01def52c3412b";

Uh oh!

NixOS test framework: Add macOS VMs #429189

Are you sure you want to change the base?

NixOS test framework: Add macOS VMs #429189

Uh oh!

Conversation

roberth commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

Mic92 Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emilazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roberth commented Jul 30, 2025

Uh oh!

hsjobeki left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roberth commented Aug 19, 2025

Uh oh!

Uh oh!

roberth commented Jul 28, 2025 •

edited

Loading

Mic92 Jul 29, 2025 •

edited

Loading

hsjobeki left a comment •

edited

Loading