pytest would complain with the previous method of adding custom markers to conftest.py, because I added uses of pytest.mark.array_api_backends in conftest.py before conftest.py::pytest_configure() could be executed.

This sounds like a circular import of sorts, and those typically signal that something is wrong with the design.
I actually don't know enough about pytest: what is the recommended way, conftest.py or pytest.ini?

If the former, the next question is what is so special about array_api_backends that it breaks the recommended pattern; if the latter, then moving markers to pytest.ini is a clean-up?

This sounds like a circular import of sorts, and those typically signal that something is wrong with the design.

I agree, I doubt that pytest markers were ever thought by pytest designers to be added at runtime. You can, in some cases, but I don't think you should.

I actually don't know enough about pytest: what is the recommended way, conftest.py or pytest.ini?

This is the very first time, across all projects I've worked on, that I see this much pytest configuration defined at runtime in conftest.py - and unnecessarily so. I believe that the current industry standard is to put as much as possible in pyproject.toml (or specific config files missing pyproject.toml).

This is akin to the shift from setup.py to setup.cfg to pyproject.toml we have witnessed in the last few years.

Could not find a use case for this

Read comment in pytest.ini

Are there any tests that don't use the xp fixture that would be important to run with SCIPY_ARRAY_API=1?

probably fine to just run those with xp for now. We can always expand coverage in the future.

size 0 is covered by test_empty (and fails on pytorch)

Do you know why this one was passing before? When converting shape tests like this, I know we typically use plain assert, and I thought that was because xp_assert_equal would fail due to neither of the results being arrays.

It was passing because the scipy variant of array_namespace() applied to a tuple returns numpy:

So _assert_matching_namespace was observing that numpy is the namespace for both for the actual and the desired array.

FWIW, I played a bit with removing this np.asanyarray because in array-api-compat==1.10, array_namespace accepts and ignores python scalars and Nones. This turned out to be an unpleasant exercise, and this asanyarray is being relied on in several scenarios.

Why not skip other backends? Also, I think the intent here (for better or for worse) was to check for a Python scalar input to variation, which produces a NumPy scalar output.

Are you suggesting there is value in explicitly testing an input that is an actual pure-python scalar instead of a scalar numpy array?

I didn't investigate the code, but the fact that array-api-strict fails too strongly hints at a bug in the scipy.stats code itself

Are you suggesting there is value in explicitly testing an input that is an actual pure-python scalar instead of a scalar numpy array?

I don't think there's value in the behavior to begin with. This is a reducing function. If the input is 0d and you reduce along axis 0 (which doesn't exist), what should happen?

I commented because I was trying to understand the motivation for the change in what was being tested. We try to preserve existing behavior with Python floats, but I don't think unusual behavior needs to be extended to Array API 0d arrays. So maybe yes, it seems just a tiny bit better to leave the test with Python floats and NumPy only.

I didn't investigate the code, but the fact that array-api-strict fails too strongly hints at a bug in the scipy.stats code itself

Maybe take that up with NumPy and JAX first:

import numpy as np
np.mean(np.asarray(1.), axis=0)
# numpy.exceptions.AxisError: axis 0 is out of bounds for array of dimension 0

import jax.numpy as xp
xp.mean(xp.asarray(1.), axis=0)
# IndexError: tuple index out of range

LMK what they say. But I don't think reducing functions should need to be able to reduce along an axis that doesn't exist. I wonder if it was intentional in torch.

import torch
torch.mean(torch.asarray(1.), axis=0)
# tensor(1.)

I don't think reducing functions should need to be able to reduce along an axis that doesn't exist.

Definitely they shouldn't

Maybe take that up with NumPy and JAX first:

Why does the test pass with a 0d numpy array though?

Anyway, I've reverted the test

To make sure I understand what's going on here (am being dense, please bear with me):

• the first if xp is None, line 253, is decided by whether xp_assert_close was called with an explicit xp or the default xp=None?
• the inner if xp is None is decided by a module-level contextvar, which, in turn, gets the value from the xp fixture. The fixture may invoke the context manager, which, in turn, is decided by whether the pytest invocation has SCIPY_ARRAY_API env variable set or not. Like this?

If the above is somewhat correct:

1. Is there a way to move entring the context manager closer to _strict_check? This would both make the fixture a little less magical, and make it easier to reason about what's going on in general. If not --- and I suspect the problem is that we want both def test_function(xp) and xp_assert_close(x, y) without an explicit xp argument --- please add copious code comments to explain what comes from where and how.

2. It seems that the namespace is always wrapped: if SCIPY_ARRAY_API is not set, we still get np_compat as the namespace? If yes, is this a change of behavior or it was always like this? My impression is that's the former, and I'd like to understand why it's harmless. Or maybe if it is, maybe we want to wrap earlier, and for the duration of the xp_assert_* body?

3. xp namespace is now derived from desired, while previously it was derived from actual (line 296 below). Why this change, is it necessary here and is this what we actually want? Or maybe it's not relevant because namespaces must match in via _assert_matching_namespace. I'm on record of wanting actual and desired to be as symmeric as possible, so preserving or improving symmetry would be a nice bonus.

the first if xp is None, line 253, is decided by whether xp_assert_close was called with an explicit xp or the default xp=None?
the inner if xp is None is decided by a module-level contextvar, which, in turn, gets the value from the xp fixture. The fixture may invoke the context manager, which, in turn, is decided by whether the pytest invocation has SCIPY_ARRAY_API env variable set or not. Like this?

All correct. I've changed the inner test for none for a test for LookupError, to make it easier to understand that we're testing for the default state of the contextvar.

Is there a way to move entering the context manager closer to _strict_check?

No. That's the whole use case for contextvars. We're saying, "within the context of this test, it's unacceptable to observe an array whose namespace differs from xp". The context manager is set by the xp fixture and the observation for the arrays is done every time you call *assert_*.

It seems that the namespace is always wrapped: if SCIPY_ARRAY_API is not set, we still get np_compat as the namespace?

Correct, you always get np_compat.

If yes, is this a change of behavior or it was always like this?

It was always like this:

xp namespace is now derived from desired, while previously it was derived from actual

It was derived from actual in the individual check function, and then immediately tested to be identical to the one derived from desired on line 259. By the time _assert_matching_namespace exits, it does not matter anymore where you derived it from. I simply cleaned the confusion up. At the end of the day I don't care which of the two arrays we derive it from, as long as we only derive it once.

I'm unsure what the point of having the option to pass check_namespace=False, as from my understanding the array API spec is quite adamant that the various libraries are not meant to be interoperable. There is only one place in which it's called, test_signaltools.py (#20772). I've now removed the flag from the test and it's still passing. should I also remove the flag from the assert functions too?

check_namespace=False

I would keep it for now, even if we don't end up using it. The xp_assert functions are an improvement over the offerings from np.testing, and even if we never turn off check_namespace in SciPy, maybe some other library would find it useful. That said, that seems like a niche use-case, so I don't feel too strongly about it.

is there a way to move entering the context manager closer to _strict_check?

No. That's the whole use case for contextvars. We're saying, "within the context of this test, it's unacceptable to observe an array whose namespace differs from xp". The context manager is set by the xp fixture and the observation for the arrays is done every time you call assert_.

Meant this: applying

$ git diff
diff --git a/scipy/_lib/_array_api.py b/scipy/_lib/_array_api.py
index cb9fb05eda..212ba8e3be 100644
--- a/scipy/_lib/_array_api.py
+++ b/scipy/_lib/_array_api.py
@@ -257,7 +257,8 @@ def _strict_check(actual, desired, xp, *,
             xp = array_namespace(xp.asarray(0))
  
     if check_namespace:
-        _assert_matching_namespace(actual, desired, xp)
+        with default_xp(xp):
+            _assert_matching_namespace(actual, desired, xp)
 
     # only NumPy distinguishes between scalars and arrays; we do if check_0d=True.
     # do this first so we can then cast to array (and thus use the array API) below.
diff --git a/scipy/conftest.py b/scipy/conftest.py
index a0891e7ff0..9d3f7556e1 100644
--- a/scipy/conftest.py
+++ b/scipy/conftest.py
@@ -213,8 +213,8 @@ def xp(request):
         # xp_assert_* functions, test that the array namespace is xp in both the
         # expected and actual arrays. This is to detect the case where both arrays are
         # erroneously just plain numpy while xp is something else.
-        with default_xp(request.param):
-            yield request.param
+  #      with default_xp(request.param):
+        yield request.param
     else:
         yield request.param
 
diff --git a/scipy/signal/tests/test_signaltools.py b/scipy/signal/tests/test_signaltools.py
index a472d763d1..d6b4b1b4f9 100644
--- a/scipy/signal/tests/test_signaltools.py
+++ b/scipy/signal/tests/test_signaltools.py
@@ -4429,3 +4429,7 @@ class TestUniqueRoots:
         unique, multiplicity = unique_roots(p, 2)
         assert_almost_equal(unique, [np.min(p)], decimal=15)
         xp_assert_equal(multiplicity, [100])
+
+
+def test_namespace(xp):
+    xp_assert_close(xp.arange(3), np.arange(3))   # should fail the namespace check

and running

$ python dev.py test -t scipy/signal/tests/test_signaltools.py -b all -v -- -k'namespace'
💻  ninja -C /home/br/repos/scipy/scipy/build -j4
ninja: Entering directory `/home/br/repos/scipy/scipy/build'
[2/338] Generating subprojects/highs/src/HConfig.h with a custom command
Build OK

... snip...

collected 36975 items / 36972 deselected / 3 selected                                               

scipy/signal/tests/test_signaltools.py::test_namespace[numpy] PASSED                          [ 33%]
scipy/signal/tests/test_signaltools.py::test_namespace[array_api_strict] FAILED               [ 66%]
scipy/signal/tests/test_signaltools.py::test_namespace[torch] FAILED                          [100%]

============================================= FAILURES ==============================================
_________________________________ test_namespace[array_api_strict] __________________________________
scipy/signal/tests/test_signaltools.py:4435: in test_namespace
    xp_assert_close(xp.arange(3), np.arange(3))   # should fail the namespace check
E   AssertionError: Namespace of actual and desired arrays do not match.
E   Actual: array_api_strict
E   Desired: scipy._lib.array_api_compat.numpy
        xp         = <module 'array_api_strict' from '/home/br/repos/array-api-strict/array_api_strict/__init__.py'>
_______________________________________ test_namespace[torch] _______________________________________
scipy/signal/tests/test_signaltools.py:4435: in test_namespace
    xp_assert_close(xp.arange(3), np.arange(3))   # should fail the namespace check
E   AssertionError: Namespace of actual and desired arrays do not match.
E   Actual: scipy._lib.array_api_compat.torch
E   Desired: scipy._lib.array_api_compat.numpy
        xp         = <module 'torch' from '/home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/torch/__init__.py'>
====================================== short test summary info ======================================
FAILED scipy/signal/tests/test_signaltools.py::test_namespace[array_api_strict] - AssertionError: Namespace of actual and desired arrays do not match.
FAILED scipy/signal/tests/test_signaltools.py::test_namespace[torch] - AssertionError: Namespace of actual and desired arrays do not match.
=========================== 2 failed, 1 passed, 36972 deselected in 3.39s ===========================

produces the desired result: two out of three tests fail, as they should.

Unless I'm missing something, so this seems to imply that

• the xp fixture in conftest.py can flip back to be a pytest.mark.parametrize
• the default_xp context manager can be contained to _lib/_array_api.py

The goal is to avoid increasing our use of pytest black magic (what on earth is request.param), avoid the coupling of conftest.py and _lib/_array_api.py, and making it generally a little easier to trace what comes from where.

There will still be a bit of bend-over-backwards passing the xp module variable to assertions via the contextvar, but this indeed seems not avoidable. Let's at least contain the magic and spooky action on a distance.

The patch you propose reverts to what you already have in main, nothing more, nothing less. There is no point in setting a context variable two lines ahead of testing it; you might as well continue passing xp=xp as a parameter.

def test_namespace(xp):
    xp_assert_close(xp.arange(3), np.arange(3))   # should fail the namespace check

This is not what this PR aims to fix. This already works in main.

This PR aims to fix these use cases:

def test1(xp):
    xp_assert_close(np.arange(3), [0, 1, 2])  # completely forgot to use xp

def test2(xp):
    x = xp.asarray([1,2,3])
    actual = foo(x)  # incorrectly returns a numpy array
    desired = bar(x)  # ALSO incorrectly returns a numpy array
    xp_assert_close(actual, desired)

The goal is to avoid increasing our use of pytest black magic (what on earth is request.param)

It's a feature of pytest that is extensively documented in the pytest documentation. I strongly disagree on calling it "black magic".

Okay, thanks. In my patch prototype the context manager is entered too late indeed. Then the right thing is probably to do something like

with default_xp(xp):
    ... = _strict_check(....)

Like I said, what I'm trying to achieve is to avoid coupling between conftest.py and _lib/_array_api.py.

It's a feature of pytest that is extensively documented

Sure it is. Does not make it less magic (magick) from where I stand :-). Anyhow, this is about tastes which differ, let's not let it distract from the PR itself.

Then the right thing is probably to do something like

with default_xp(xp):
  ... = _strict_check(....)

I'm sorry, but this still makes no sense. What's the value of the xp parameter to default_xp if you call it one line before _strict_check? And why are you not just calling _strict_check(..., xp=xp) (which is exactly what main does)?

Okay, I'm going to stop trying to offer half-solutions and circle back to my main point then: I suspect the code will be simpler if we decouple conftest.py from _lib/_array_api.py. Is there a way to have that?

I don't see any.

That's too bad. I'll keep thinking about it , but my thinking does not need to block this PR.

This sounds like a circular import of sorts, and those typically signal that something is wrong with the design.
I actually don't know enough about pytest: what is the recommended way, conftest.py or pytest.ini?

If the former, the next question is what is so special about array_api_backends that it breaks the recommended pattern; if the latter, then moving markers to pytest.ini is a clean-up?

FWIW, I played a bit with removing this np.asanyarray because in array-api-compat==1.10, array_namespace accepts and ignores python scalars and Nones. This turned out to be an unpleasant exercise, and this asanyarray is being relied on in several scenarios.