ENH:MAINT: sparse.linalg: Rewrite PROPACK in C #23531
Open
+7,657
−701
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It finds half of the
Yep, I am blind that's a typo and I was staring at it for hours. |
tylerjereddy
reviewed
Sep 1, 2025
There was a problem hiding this comment.
I seem to remember some propack-specific issues in tests, but the one skip I tried to purge out has the same behavior here I think:
--- a/scipy/sparse/linalg/_eigen/tests/test_svds.py
+++ b/scipy/sparse/linalg/_eigen/tests/test_svds.py
@@ -738,9 +738,6 @@ class SVDSCommonTests:
and k == min(A.shape) - 1):
pytest.skip("#16725")
- if self.solver == 'propack':
- pytest.skip("PROPACK failures unrelated to PR #16712")There was, at one time, a mass of propack problems on Windows + conda-forge IIRC, that even blocked a release for a month or so? It might be worth touching base with @h-vetinari to see if there's a desire to take this branch for a spin over there? Or maybe those problems are all long gone anyway..
|
If someone knows how to tame conda to use MKL instead of openblas in scipy builds, I can check these remaining 3 failing jobs. The instructions are a bit outdated and not geared towards The sparse.linalg test suite apparently grew a bit organically over time and has lots of conditional skips that I can't always read the intention behind it. Some might be actual shortcomings of PROPACK. |
|
With https://github.com/rgommers/pixi-dev-scipystack you can Just avoid |
From https://github.com/scipy/scipy/blob/main/.github/workflows/linux_blas.yml#L60 Note 1) need the numpy fork of meson, and 2) need to |
No luck unfortunately.
This worked after adding ninja, But now the test does not fail, it might not be a code issue |
Thanks both! I just checked and pointing to NumPy's fork of meson and passing I will make a PR to fix this in the Pixi workspace this week. |
|
Great to see this!
MKL version issue perhaps? CI pulls 2025.2.0 from PyPI, and that isn't yet available in conda-forge. Probably easiest to check by downgrading versions in |
I can test this PR in our CI if it helps. Unfortunately, the BLAS+openmp situation is in pretty bad shape in conda-forge ATM. MKL 2025 has been blocked for about a year, which caused us to globally cap to Furthermore, we switched from I've pushed pretty hard on unblocking MKL myself, and asked for help in various places, but there's no cavalry on the horizon. Due to all of this (mostly MKL, and how long they take to update their compatibility with reference lapack), we're also still stuck on LAPACK 3.9, which is by now getting quite long in the tooth. |
|
In the meantime I mishmashed a conda-forge environment to have pip installed MKL2025.2. The test still passes (scipy-numpy-mkl) ➜ scipy git:(rewrite_propack) spin test -t scipy.sparse.linalg.tests.test_propack::test_svdp\[LM-True-float64-array\] -v -v -v -- --capture=sys
Invoking `build` prior to running tests:
$ meson compile -j 16 -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/ninja -C /home/ilayn/Documents/mkl-venv/scipy/build -j 16
ninja: Entering directory `/home/ilayn/Documents/mkl-venv/scipy/build'
[2/338] Generating subprojects/highs/src/HConfig.h with a custom command
$ meson install --only-changed -C build --destdir ../build-install --tags=runtime,python-runtime,tests,devel
$ export PYTHONPATH="/home/ilayn/Documents/mkl-venv/scipy/build-install/usr/lib/python3.13/site-packages"
$ /home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/python3.13 -P -c 'import scipy'
$ cd /home/ilayn/Documents/mkl-venv/scipy/build-install/usr/lib/python3.13/site-packages
$ /home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/python3.13 -P -m pytest -v -m 'not slow' --capture=sys --pyargs 'scipy.sparse.linalg.tests.test_propack::test_svdp[LM-True-float64-array]'
===================================================================================================================================================== test session starts =====================================================================================================================================================
platform linux -- Python 3.13.5, pytest-8.4.1, pluggy-1.6.0 -- /home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/python3.13
cachedir: .pytest_cache
hypothesis profile 'default'
rootdir: /home/ilayn/Documents/mkl-venv/scipy
configfile: pytest.ini
plugins: hypothesis-6.138.11, xdist-3.8.0
collected 1 item
scipy/sparse/linalg/tests/test_propack.py::test_svdp[LM-True-float64-array] PASSED [100%]
====================================================================================================================================================== 1 passed in 0.16s ======================================================================================================================================================
(scipy-numpy-mkl) ➜ scipy git:(rewrite_propack) cd build-install/usr/lib/python3.13/site-packages
(scipy-numpy-mkl) ➜ site-packages git:(rewrite_propack) python
Python 3.13.5 | packaged by conda-forge | (main, Jun 16 2025, 08:27:50) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy
>>> scipy.show_config()
/home/ilayn/Documents/mkl-venv/scipy/build-install/usr/lib/python3.13/site-packages/scipy/__config__.py:164: UserWarning: Install `pyyaml` for better output
warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
"Compilers": {
"c": {
"name": "gcc",
"linker": "ld.bfd",
"version": "14.3.0",
"commands": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/x86_64-conda-linux-gnu-cc",
"args": "-march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include",
"linker args": "-Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now, -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -Wl,-rpath-link,/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -L/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include"
},
"cython": {
"name": "cython",
"linker": "cython",
"version": "3.1.3",
"commands": "cython"
},
"c++": {
"name": "gcc",
"linker": "ld.bfd",
"version": "14.3.0",
"commands": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/x86_64-conda-linux-gnu-c++",
"args": "-fvisibility-inlines-hidden, -fmessage-length=0, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include",
"linker args": "-Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now, -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -Wl,-rpath-link,/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -L/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -fvisibility-inlines-hidden, -fmessage-length=0, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include, -DNDEBUG, -D_FORTIFY_SOURCE=2, -O2, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include"
},
"fortran": {
"name": "gcc",
"linker": "ld.bfd",
"version": "14.3.0",
"commands": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/x86_64-conda-linux-gnu-gfortran",
"args": "-march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include, -I/home/ilayn/miniforge3/envs/scipy-numpy-mkl/include",
"linker args": "-Wl,-O2, -Wl,--sort-common, -Wl,--as-needed, -Wl,-z,relro, -Wl,-z,now, -Wl,--disable-new-dtags, -Wl,--gc-sections, -Wl,--allow-shlib-undefined, -Wl,-rpath,/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -Wl,-rpath-link,/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -L/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib, -march=nocona, -mtune=haswell, -ftree-vectorize, -fPIC, -fstack-protector-strong, -fno-plt, -O2, -ffunction-sections, -pipe, -isystem, /home/ilayn/miniforge3/envs/scipy-numpy-mkl/include, -I/home/ilayn/miniforge3/envs/scipy-numpy-mkl/include"
},
"pythran": {
"version": "0.18.0",
"include directory": "../../../../miniforge3/envs/scipy-numpy-mkl/lib/python3.13/site-packages/pythran"
}
},
"Machine Information": {
"host": {
"cpu": "x86_64",
"family": "x86_64",
"endian": "little",
"system": "linux"
},
"build": {
"cpu": "x86_64",
"family": "x86_64",
"endian": "little",
"system": "linux"
},
"cross-compiled": false
},
"Build Dependencies": {
"blas": {
"name": "mkl-dynamic-lp64-seq",
"found": true,
"version": "2025.2",
"detection method": "pkgconfig",
"include directory": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib/pkgconfig/../../include",
"lib directory": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib/pkgconfig/../../lib",
"openblas configuration": "unknown",
"pc file directory": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib/pkgconfig"
},
"lapack": {
"name": "mkl-dynamic-lp64-seq",
"found": true,
"version": "2025.2",
"detection method": "pkgconfig",
"include directory": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib/pkgconfig/../../include",
"lib directory": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib/pkgconfig/../../lib",
"openblas configuration": "unknown",
"pc file directory": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/lib/pkgconfig"
},
"pybind11": {
"name": "pybind11",
"version": "3.0.1",
"detection method": "config-tool",
"include directory": "unknown"
}
},
"Python Information": {
"path": "/home/ilayn/miniforge3/envs/scipy-numpy-mkl/bin/python3.13",
"version": "3.13"
}
}
>>> All versions seem to be similar to BLAS job including the numpy-meson. I am starting to fear that this is another Ubuntu problem similar to that in #23161 that I still could not figure out why. |
passes locally in an environment with and One other difference from #23531 (comment) is the python version. My local environment is for python 3.12. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference issue
Towards #18566
What does this implement/fix?
?larnvto anxoshiro256+implementation whose seeds can be controlled by NumPy PRNGs.cdotcandzdotcand rewrote as standalone functions to avoid the occasional segfaults.common.cAdditional information
Since this was previously a submodule, if merged, this code will create some annoyance in your local fork due to remaining Fortran77 files from earlier cloning. After using
git clean -xdfin the repo root should clean up the remaining fortran pieces and scipy can be rebuilt.