mpi is essentialy a buildInput by providing linking library -lmpi, though cmake find its library via checking mpicc/mpiexec in the path.
I would suggest not using strictdeps and put mpi in buildInputs.

honestly i have to disagree. If things are broken with strictDeps, then they should be fixed. Removing strictDeps shouldn't be the solution, especially because there are efforts to make strict deps a global default.

I am not against using strictdeps, but we should resolve the dependency of mpi first. Current status of this pr: mpi as a nativeBuildInput is in the nar closure of scotch. It's common we have nativeBuildInputs bash/perl in the nar closure since we patch SheBang, but is it good with nativeBuildInputs mpi if we actually link to libmpi.so?

it'd very much explode on cross. But i am not sure how to fix this properly.. Maybe our resident strictDeps @quantenzitrone has some insights?

i don't really know a lot about strictDeps
however according to cmake docs, if i understand them correctly (i have never worked with cmake), we should be able to fix this by setting MPI_HOME

Oh wait sorry, i confused you with someone else, sorry! Should have done the PR lookup. I meant @FliegendeWurst, the other nix human i met at GPN! Sorry for the noise...

But i'll look into MPI_HOME, that is still a good pointer.

the thing that has me confused: openmpi supplies things like mpiCC, which is a C compiler. It already does some tricks to provide the correct compiler with the right target platform. But then its linking libraries... I also can't seem to make it work setting MPI_HOME.

i also tried around a bit and neither MPI_HOME as cmakeFlag or environment variable nor MPI_C_LIBRARIES and MPI_C_ADDITIONAL_INCLUDE_DIRS seem to work

can we make ptscotch a separate variant of scotch by setting the make target?
This approach should involve treewide change in mumps/petsc.

we cant. scotch is build always, because PtScotch depends on it. There is no supported way to build just ptscotch.

we need either

mpiSupport ? lib.meta.availableOn stdenv.hostPlatform mpi,
withPtScotch? mpiSupport,

or

diff --git a/pkgs/by-name/pe/petsc/package.nix b/pkgs/by-name/pe/petsc/package.nix
index 8a2d78fc3226..417559da86e5 100644
--- a/pkgs/by-name/pe/petsc/package.nix
+++ b/pkgs/by-name/pe/petsc/package.nix
@@ -27,10 +27,10 @@
   # External libraries options
   withHdf5 ? withCommonDeps,
   withMetis ? withCommonDeps,
-  withZlib ? (withP4est || withPtscotch),
+  withZlib ? (withP4est || withPtScotch),
   withScalapack ? withCommonDeps && mpiSupport,
   withParmetis ? withFullDeps, # parmetis is unfree
-  withPtscotch ? withCommonDeps && mpiSupport,
+  withPtScotch ? withCommonDeps && mpiSupport,
   withMumps ? withCommonDeps,
   withP4est ? withFullDeps,
   withHypre ? withCommonDeps && mpiSupport,
@@ -68,7 +68,7 @@ assert withP4est -> (mpiSupport && withZlib);
 # Package parmetis depend on metis and mpi support
 assert withParmetis -> (withMetis && mpiSupport);
 
-assert withPtscotch -> (mpiSupport && withZlib);
+assert withPtScotch -> (mpiSupport && withZlib);
 assert withScalapack -> mpiSupport;
 assert (withMumps && mpiSupport) -> withScalapack;
 assert withHypre -> mpiSupport;
@@ -82,6 +82,7 @@ let
       python3Packages
       # global override options
       mpiSupport
+      withPtScotch
       fortranSupport
       pythonSupport
       precision
@@ -141,7 +142,7 @@ stdenv.mkDerivation (finalAttrs: {
   ++ lib.optional withP4est petscPackages.p4est
   ++ lib.optional withMetis petscPackages.metis
   ++ lib.optional withParmetis petscPackages.parmetis
-  ++ lib.optional withPtscotch petscPackages.scotch
+  ++ lib.optional withPtScotch petscPackages.scotch
   ++ lib.optional withScalapack petscPackages.scalapack
   ++ lib.optional withMumps petscPackages.mumps
   ++ lib.optional withHypre petscPackages.hypre
@@ -192,7 +193,7 @@ stdenv.mkDerivation (finalAttrs: {
   ++ lib.optional pythonSupport "--with-petsc4py=1"
   ++ lib.optional withMetis "--with-metis=1"
   ++ lib.optional withParmetis "--with-parmetis=1"
-  ++ lib.optional withPtscotch "--with-ptscotch=1"
+  ++ lib.optional withPtScotch "--with-ptscotch=1"
   ++ lib.optional withScalapack "--with-scalapack=1"
   ++ lib.optional withMumps "--with-mumps=1"
   ++ lib.optional (withMumps && !mpiSupport) "--with-mumps-serial=1"

to make petsc.petscPackages.scotch in consistent with petsc.petscPackages.mumps

What the hell is petscPackages? This is the first time i am hearing about it.

Further: mpi is broken on musl, because libpsm2 is broken on musl. lib.meta.availableOn does not return false if something is broken, and it certainly does not return false if a dependency of something is broken. It only respects badPlatforms of the package itself. We do not have a way to do badPlatforms for musl afaik.

Also i'd rather not change the override interface here. That sounds like a bad idea asking for peoples overrides to explode.

I did add f2390fb, which fixes pkgsMusl.petsc.petscPackages.mumps and pkgsMusl.petsc.petscPackages.scotch. But i will not change the override interface, and i will not switch to availableOn because it doesn't work for this.

If openmpi is broken on musl, we can mark it broken when stdenv.hostPlatform.isMusl and use lib.meta.availableOn stdenv.hostPlatform mpi here.

actually we can probably delete this assert completely. mpi is already marked broken (by transitive dep)

Same as #431612 (comment). availableOn does not really work here.

actually we can probably delete this assert completely

Then let's just remove it? Or add a comment explaining why so we know to remove it later when mpi is fixed.

Can you fix withPtscotch-> withPtScotch in petsc and inherit withPtScotch in petscPackages, then you dont have to set extra override arguments (e.g here inherit mpiSupport; in petscPackages.mumps is redundant since self.callPackages will pass mpiSupport to mumps)

As i said, that would break the override interface of petsc, on which people might be relying. I see no good reason to actively break peoples setup with a somewhat pointless rename like this. See #431612 (comment)

i am also not super convinced callPackage is the best approach here, it probably should be override. But that is a different topic, and would balloon this PR out of proportion.

Ok, just remove this commit in petsc, i will open a pr fixing it seprately.

Do we needs the overridden Scotch here — for pkgsMusl.mumps or mumps?
IMO whether the package Scotch should be built with PtScotch by default is handled on each platform, so we don’t need the override.

openmpi is introducing a perl dependency. If we build mumps without PtScotch, we shouldn't add it to scotch to avoid that perl dependency in runtime closure. In fact this was my key motivation to even start looking into this! This should stay for the sake of perlless-ness.

I see, we need to minimize the closure size of mumps/scotch when we dont need mpi. Now only petsc/mumps depend on scotch, I will open a pr to set withPtScotch by default false.