[WIP] NumPy 1.24 #8620

gmarkall · 2022-11-24T15:22:50Z

Current status: All is well with NumPy 1.24, so this needs some tidying up and other accompanying changes to be ready for review. To-do list:

Update docs
Reconfigure CI appropriately
Rewrite history for coherency
Create new PR for review, linking to this for history / edits

Original, outdated description follows:

For testing against NumPy 1.24.0rc1. Presently all seems to be good (all tests passing with NumPy 1.24) so it is not expected that any major changes are needed, assuming there are no further NumPy changes requiring attention.

I will keep this up to date with main and update to use newer RCs when they appear - when the final release is available, I will make this Ready For Review in order to support NumPy 1.24.

The use of resolve_dtypes will be added as a separate PR, because it is not needed to make all existing functionality in Numba work with 1.24 - its use will eventually simplify Numba' ufunc typing logic.

np.bool was removed in NumPy 1.24.

This always produced invalid results (though they were consistent between Numba and NumPy) but now this fails in NumPy 1.24 with an exception: ``` TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit. You can avoid this error by using the scalar types `np.float64` or the dtype string notation. ``` Note that the exception message is misleading, and using the dtype string notation does not provide a workaround.

gmarkall · 2022-11-24T15:24:56Z

/azp run

azure-pipelines · 2022-11-24T15:25:04Z

Azure Pipelines failed to run 1 pipeline(s).

gmarkall · 2022-11-24T15:26:54Z

Azure Pipelines failed to run 1 pipeline(s).

Well, I know exactly what went wrong now. Thanks, bot!

gmarkall · 2022-11-24T15:27:42Z

Guessing I've ended up with duplicate matrix entries or something like that.

I believe this was written in error and should always have been float16.

The modified regex matches the existing message produced by NumPy < 1.24, and the new improved message in 1.24.

gmarkall · 2022-11-24T16:14:05Z

gpuci run tests

gmarkall · 2022-11-24T16:14:31Z

Oops, I haven't updated the gpuCI test scripts to test with NumPy 1.24.

gmarkall · 2022-11-24T16:56:33Z

gpuci run tests

Setting an array element with a sequence is removed in NumPy 1.24.

gmarkall · 2022-11-24T21:40:55Z

gpuci run tests

This is so that a proper solution can be implemented. This reverts commit 08416ad.

`init_ufunc_dispatch()` should set an exception if it's going to fail. Without setting an exception, an unrecognized ufunc method (e.g. `resolve_dtypes()` in NumPy 1.24) results in: ``` SystemError: initialization of _internal failed without raising an exception ``` This commit modifies it to set an exception notifying the user of the error. Now when an unrecognized ufunc method is encountered, the traceback ends in: ``` from numba.np.ufunc import _internal RuntimeError: Unexpected ufunc method resolve_dtypes() ```

The Windows build did not like a variable-length stack array.

gmarkall · 2022-12-22T09:40:43Z

Looks like the failure persists after going back to rc2, so it's not related to the 1.24 release.

gmarkall · 2022-12-22T12:50:58Z

seems that the sign of some inputs is getting flipped.

I wrongly assumed that this test only checked inputs, but it also checks outputs, and it is the output check that is failing. This test probably needs updating to check using reconstruction (check that A ~= U*S*V**H) if the plain match fails, like test_linalg_svd does.

This reverts commit 22980fe.

This test only checked for a plain match when comparing outputs. However, in some cases a reconstruction check can be necessary, as in `test_linalg_svd`.

gmarkall · 2022-12-22T15:01:23Z

gpuci run tests

gmarkall · 2023-01-02T11:56:11Z

gpuci run tests

bnavigator · 2023-01-03T12:35:51Z

numba/tests/test_np_functions.py

@@ -4893,6 +4893,7 @@ def create_harcoded_variant(self, basefunc, ty):
        eval(compile(funcstr, '<string>', 'exec'))
        return locals()['foo']

+    @unittest.skipIf(numpy_version >= (1, 24), "NumPy < 1.24 required")


This also fails for me on numpy 1.23.4 (openSUSE Tumbleweed packaging):

[ 2303s] ====================================================================== [ 2303s] ERROR: test_MachAr (numba.tests.test_np_functions.TestNPMachineParameters) [ 2303s] ---------------------------------------------------------------------- [ 2303s] Traceback (most recent call last): [ 2303s] File "/usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py", line 4783, in test_MachAr [ 2303s] self.check(machar, attrs) [ 2303s] File "/usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py", line 4762, in check [ 2303s] got = cfunc(*args) [ 2303s] File "/usr/lib/python3.8/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args [ 2303s] error_rewrite(e, 'typing') [ 2303s] File "/usr/lib/python3.8/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite [ 2303s] raise e.with_traceback(None) [ 2303s] numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) [ 2303s] Use of unsupported NumPy function 'numpy.MachAr' or unsupported use of the function. [ 2303s] [ 2303s] File "../../../../../../usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py", line 102: [ 2303s] def machar(*args): [ 2303s] return np.MachAr() [ 2303s] ^ [ 2303s] [ 2303s] During: typing of get attribute at /usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py (102) [ 2303s] [ 2303s] File "../../../../../../usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py", line 102: [ 2303s] def machar(*args): [ 2303s] return np.MachAr() [ 2303s] ^ [ 2303s] [ 2303s] [ 2303s] ====================================================================== [ 2303s] FAIL: test_np_MachAr_deprecation_np122 (numba.tests.test_np_functions.TestNPMachineParameters) [ 2303s] ---------------------------------------------------------------------- [ 2303s] Traceback (most recent call last): [ 2303s] File "/usr/lib/python3.8/site-packages/numba/tests/support.py", line 599, in inner [ 2303s] self.subprocess_test_runner(test_module=self.__module__, [ 2303s] File "/usr/lib/python3.8/site-packages/numba/tests/support.py", line 584, in subprocess_test_runner [ 2303s] self.assertEqual(status.returncode, 0, streams) [ 2303s] AssertionError: 1 != 0 : [ 2303s] captured stdout: [ 2303s] captured stderr: E [ 2303s] ====================================================================== [ 2303s] ERROR: test_np_MachAr_deprecation_np122 (numba.tests.test_np_functions.TestNPMachineParameters) [ 2303s] ---------------------------------------------------------------------- [ 2303s] Traceback (most recent call last): [ 2303s] File "/usr/lib/python3.8/site-packages/numba/tests/support.py", line 607, in inner [ 2303s] func(self) [ 2303s] File "/usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py", line 4836, in test_np_MachAr_deprecation_np122 [ 2303s] f() [ 2303s] File "/usr/lib/python3.8/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args [ 2303s] error_rewrite(e, 'typing') [ 2303s] File "/usr/lib/python3.8/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite [ 2303s] raise e.with_traceback(None) [ 2303s] numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) [ 2303s] Use of unsupported NumPy function 'numpy.MachAr' or unsupported use of the function. [ 2303s] [ 2303s] File "../../../../../../usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py", line 4835: [ 2303s] def test_np_MachAr_deprecation_np122(self): [ 2303s] <source elided> [ 2303s] category=NumbaDeprecationWarning,) [ 2303s] f = njit(lambda : np.MachAr().eps) [ 2303s] ^ [ 2303s] [ 2303s] During: typing of get attribute at /usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py (4835) [ 2303s] [ 2303s] File "../../../../../../usr/lib/python3.8/site-packages/numba/tests/test_np_functions.py", line 4835: [ 2303s] def test_np_MachAr_deprecation_np122(self): [ 2303s] <source elided> [ 2303s] category=NumbaDeprecationWarning,) [ 2303s] f = njit(lambda : np.MachAr().eps) [ 2303s] ^ [ 2303s] [ 2303s] [ 2303s] ---------------------------------------------------------------------- [ 2303s] Ran 1 test in 0.271s [ 2303s] [ 2303s] FAILED (errors=1) [ 2303s] [ 2303s] [ 2303s] ---------------------------------------------------------------------- [ 2303s] Ran 9684 tests in 2243.604s

I could have messed something up for old NumPy versions because I've only been testing with 1.24 in the matrix here - I've opened #8690 to run CI on this branch but with the other NumPy versions to check for this and maybe any other problems.

@bnavigator thanks for the report, this seems a bit strange as NumPy 1.23 definitely has MachAr still. Maybe there's an issue with Numba's overload construction for it that is triggered in the openSUSE builds. Perhaps open a new issue to discuss/track? Thanks!

I could have messed something up for old NumPy versions because I've only been testing with 1.24 in the matrix here - I've opened #8690 to run CI on this branch but with the other NumPy versions to check for this and maybe any other problems.

I think there's potentially something else going on. Looking at the NumPy source, there's MachAr exported in both 1.23 and 1.24.

There's no np.MachAr in 1.24, but np.core.MachAr is there - it's been left in but is not public / documented, AIUI. I think that will be removed too in a future release.

Ouch. Looks like I have some outdated patch which still includes

--- a/numba/np/arraymath.py +++ b/numba/np/arraymath.py @@ -4203,7 +4207,7 @@ def _gen_np_machar(): return impl -_gen_np_machar() +# _gen_np_machar() def generate_xinfo(np_func, container, attr):

@gmarkall Thanks, I was indeed looking under numpy.core, where it still exists.

@bnavigator Ah, thanks for checking, guess that's the root cause of the problem!

Sorry for the noise. My mistake. I can confirm that the current PR with 7d3826e as the latest commit works for both NumPy 1.23.4 and after an update to 1.24.1 when packaging numba 0.56.4 for openSUSE Tumbleweed.

https://build.opensuse.org/request/show/1046565

@bnavigator no worries, thanks for checking that it's now all working! This is essentially an independent verification of the update in this patch, which is really useful to have!

numba/np/arraymath.py

From PR numba#8691 feedback.

PR numba#8691 feedback.

gmarkall · 2023-01-11T16:28:30Z

gpuci run tests

stuartarchibald · 2023-01-25T12:16:46Z

@gmarkall could this WIP PR now be closed in favour of #8691?

gmarkall · 2023-01-25T22:26:34Z

@stuartarchibald This PR contains the same changes as #8691, but tests all slices with NumPy 1.24 (therefore demonstrating that all tests pass with 1.24) - I'd like to keep this open until #8691 can run on the buildfarm, so if further changes are made I can validate (to some degree) they don't break anything at all with NumPy 1.24.

stuartarchibald · 2023-01-26T14:07:14Z

@gmarkall ah yes, thanks for reminding me, good idea to keep this open for the reasons noted.

gmarkall · 2023-02-17T11:05:36Z

Now that #8691 is ready to go except for CI on that branch itself, I think this is no longer needed.

gmarkall added 5 commits November 24, 2022 09:57

Quick workaround for new ufunc methods

08416ad

Overload np.MachAr only for NumPy < 1.24

3a3b0d7

Replace use of deprecated np.bool with np.bool_

0e50ac6

np.bool was removed in NumPy 1.24.

Use CI for testing NumPy 1.24.0rc1

9649f46

gmarkall added the 2 - In Progress label Nov 24, 2022

gmarkall mentioned this pull request Nov 24, 2022

NumPy 1.24 support #8464

Closed

6 tasks

gmarkall added 4 commits November 24, 2022 15:29

Try to fix azure by renaming duplicate checks

026ef43

Use Python >= 3.8

8b840c7

CUDA intrinsics tests: correct np.float -> np.float16

04e235e

I believe this was written in error and should always have been float16.

test_hypot: Tweak regex so it matches NumPy 1.24

a6a58c6

The modified regex matches the existing message produced by NumPy < 1.24, and the new improved message in 1.24.

gmarkall added 2 commits November 24, 2022 16:48

Avoid use of np.bool in stencilparfor.py

64c460d

Use NumPy 1.24.0rc1 on gpuCI

482c8b5

test_comp_nest_with_dependency: skip on NumPy 1.24

38194fa

Setting an array element with a sequence is removed in NumPy 1.24.

gmarkall added the numpy label Nov 24, 2022

gmarkall added 7 commits November 25, 2022 09:33

Revert "Quick workaround for new ufunc methods"

5b98548

This is so that a proper solution can be implemented. This reverts commit 08416ad.

init_ufunc_dispatch: Use a constant size for exception message buffer

b2684d2

The Windows build did not like a variable-length stack array.

Another attempt to fix Windows build

292ce9d

Use PyErr_Format instead

6bb635b

init_ufunc_dtypes: Set exception when fastcall flags are inconsistent

e8956b4

Ignore resolve_dtypes() and private methods

23b8ccd

MarcoGorelli mentioned this pull request Dec 22, 2022

CLN remove unnecessary numba-compat code pandas-dev/pandas#50392

Closed

gmarkall added 2 commits December 22, 2022 13:02

Revert "Ttry with rc2 on Linux py39"

a16abe6

This reverts commit 22980fe.

TestLinalgSvd.test_no_input_mutation: use reconstruction if necessary

a3f1990

This test only checked for a plain match when comparing outputs. However, in some cases a reconstruction check can be necessary, as in `test_linalg_svd`.

dwhswenson mentioned this pull request Dec 26, 2022

Unpin numpy when possible openpathsampling/openpathsampling#1127

Open

martinvonk mentioned this pull request Dec 28, 2022

Implement Type Hints pastas/pastas#441

Merged

4 tasks

jGaboardi mentioned this pull request Dec 31, 2022

changing configuration for the testing action pysal/giddy#181

Merged

Update to NumPy 1.24.1

81c750c

metab0t mentioned this pull request Jan 3, 2023

Lift numpy version constraint conda-forge/numba-feedstock#110

Closed

5 tasks

tomwhite mentioned this pull request Jan 3, 2023

Upstream failure: SystemError: initialization of _internal failed without raising an exception sgkit-dev/sgkit#979

Open

bnavigator reviewed Jan 3, 2023

View reviewed changes

Update NumPy version support documentation

7d3826e

gmarkall mentioned this pull request Jan 3, 2023

[CI only] Testing #8620 with NumPy versions < 1.24 #8690

Closed

stuartarchibald reviewed Jan 3, 2023

View reviewed changes

numba/np/arraymath.py Show resolved Hide resolved

gmarkall mentioned this pull request Jan 3, 2023

NumPy 1.24 #8691

Merged

gmarkall added 2 commits January 11, 2023 16:19

Docs: Correct NumPy upper bound for 0.57 release

e880be8

From PR numba#8691 feedback.

Update comment on skipped test

d9aacd5

PR numba#8691 feedback.

jmahlik mentioned this pull request Jan 11, 2023

[BUG] Can't install on python 3.11 mlflow/mlflow#7681

Closed

2 tasks

bnavigator mentioned this pull request Jan 14, 2023

numpy 1.24 deprecations fail quimb jcmgray/autoray#11

Closed

bnavigator mentioned this pull request Jan 28, 2023

Change dtype repr for Numpy 1.24 quantumlib/Cirq#5991

Merged

gmarkall closed this Feb 17, 2023

gmarkall added the abandoned PR is abandoned (no reason required) label Feb 17, 2023

[WIP] NumPy 1.24 #8620

[WIP] NumPy 1.24 #8620

Uh oh!

Conversation

gmarkall commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmarkall commented Nov 24, 2022

Uh oh!

azure-pipelines bot commented Nov 24, 2022

Uh oh!

gmarkall commented Nov 24, 2022

Uh oh!

gmarkall commented Nov 24, 2022

Uh oh!

gmarkall commented Nov 24, 2022

Uh oh!

gmarkall commented Nov 24, 2022

Uh oh!

gmarkall commented Nov 24, 2022

Uh oh!

gmarkall commented Nov 24, 2022

Uh oh!

gmarkall commented Dec 22, 2022

Uh oh!

gmarkall commented Dec 22, 2022

Uh oh!

gmarkall commented Dec 22, 2022

Uh oh!

gmarkall commented Jan 2, 2023

Uh oh!

bnavigator Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

gmarkall Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

stuartarchibald Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

stuartarchibald Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

gmarkall Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

bnavigator Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stuartarchibald Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

bnavigator Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stuartarchibald Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gmarkall commented Jan 11, 2023

Uh oh!

stuartarchibald commented Jan 25, 2023

Uh oh!

gmarkall commented Jan 25, 2023

Uh oh!

stuartarchibald commented Jan 26, 2023

Uh oh!

gmarkall commented Feb 17, 2023

Uh oh!

Uh oh!

gmarkall commented Nov 24, 2022 •

edited

Loading

bnavigator Jan 3, 2023 •

edited

Loading

bnavigator Jan 3, 2023 •

edited

Loading