cc @tomwhite @jakirkham

@mdhaber FYI there are a few places in _chandrupatla and related infra that used xp.full_like with passing an array to fill_value. As per the spec, fill_value should be a Python scalar. array-api-strict does not catch this, but Dask does, albeit indirectly via a FutureWarning about np.copyto being used along the way.

(I'll fix it here, but just in case you were going on to write something similar before this is merged)

@ev-br could you suggest how to fix

for

TypeError: Array.__new__() missing 2 required positional arguments: 'name' and 'chunks'

?

What is result and is type(result)?

What is result and is type(result)?

result is a dask array, type(result) is dask.array.Array

_lazywhere is tripped up by Dask's limited boolean indexing capability:

In [19]: da.asarray(0.)[True]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[19], line 1
----> 1 da.asarray(0.)[True]

File ~/dev/pixi-dev-scipystack/scipy/.pixi/envs/default/lib/python3.12/site-packages/dask/array/core.py:1975, in Array.__getitem__(self, index)
   1967     index = (index,)
   1969 from dask.array.slicing import (
   1970     normalize_index,
   1971     slice_with_bool_dask_array,
   1972     slice_with_int_dask_array,
   1973 )
-> 1975 index2 = normalize_index(index, self.shape)
   1976 dependencies = {self.name}
   1977 for i in index2:

File ~/dev/pixi-dev-scipystack/scipy/.pixi/envs/default/lib/python3.12/site-packages/dask/array/slicing.py:789, in normalize_index(idx, shape)
    787 idx = idx + (slice(None),) * (len(shape) - n_sliced_dims)
    788 if len([i for i in idx if i is not None]) > len(shape):
--> 789     raise IndexError("Too many indices for array")
    791 none_shape = []
    792 i = 0

IndexError: Too many indices for array

result is a dask array, type(result) is dask.array.Array

Then the fix is to wrap the # lists/tuples stanza in an explicit if isinstance(result, (list, tuple)

cc @phofl (for awareness)

@mdhaber does the diff in e63b914 look okay to you? I've been experimenting with Dask and a SciPy nightly, and this diff allows e.g. special.rel_entr to stay native to Dask via the generic implementation, so things can be kept parallel.

If so, I think I'd like to merge that separately before this PR.

@mdhaber does the diff in e63b914 look okay to you? I've been experimenting with Dask and a SciPy nightly, and this diff allows e.g. special.rel_entr to stay native to Dask via the generic implementation, so things can be kept parallel.

If so, I think I'd like to merge that separately before this PR.

Just to follow up here @mdhaber - that change isn't specific to Dask.

@ev-br do you have any idea why we are hitting "dask.array has no argsort" errors for some ndimage.{maximum, minimum} tests? As far as I understand, we shouldn't be calling dask.array.argsort anywhere? Are we accidentally calling np.argsort on a Dask array maybe when we should be converting to np first?

Looks like indeed some np.asarray / np.asanyarray calls are missing and internal functions receive mixtures of numpy and dask arrays:

917  ->	def _select(input, labels=None, index=None, find_min=False, find_max=False,
918  	            find_min_positions=False, find_max_positions=False,
919  	            find_median=False):
920  	    """Returns min, max, or both, plus their positions (if requested), and
921  	    median."""
922  	
(Pdb) n
> /home/ev-br/repos/scipy/build-install/lib/python3.11/site-packages/scipy/ndimage/_measurements.py(923)_select()
-> input = np.asanyarray(input)
(Pdb) p input
dask.array<array, shape=(3, 4), dtype=int8, chunksize=(3, 4), chunktype=numpy.ndarray>

...

(Pdb) a
input = array([[5, 4, 2, 5],      # have been converted, but lables and index have not been
       [3, 7, 8, 2],
       [1, 5, 1, 1]], dtype=int8)
labels = dask.array<array, shape=(4,), dtype=float64, chunksize=(4,), chunktype=numpy.ndarray>
index = dask.array<array, shape=(2,), dtype=float64, chunksize=(2,), chunktype=numpy.ndarray>
find_min = False
find_max = False
find_min_positions = False
find_max_positions = True
find_median = False

Makes sense. I wonder why this doesn't trip up PyTorch CPU but does for Dask.

Planning on helping give this a little nudge.

Using the newly wrapped fft module in array-api-compat, we can get down to 11 failures on fft at least - mainly regarding dtype differences.
(some updates to the tests were required though, I'll try to push something up tomorrow)

I haven't looked at any of the other failures yet.

EDIT 1: Have gotten special to work (other than the issue with suppressing the runtime warnings from dask/numpy)

feel free to make a PR to my branch @lithomas1 !

I made a draft PR here
lucascolley#14
(I'll add some more comments/insights on why I changed what lines of code later)

Could you pull main on your branch later if you have more time?
(the diff on that PR is quite large since this PR is a bit behind main)

thanks @lithomas1 , that definitely helped to bring the number of errors down. Of the remaining errors, some look trivial, others I'm not so sure.

@lithomas1 do you think you could resolve the merge conflicts? @ev-br also updated array-api-compat in gh-21796

Apologies for the long silence here.

I did some digging into the remaining fft/special failures and other than some small issues with extra warnings coming from dask, I think I have those two modules passing locally.

One source of failures that I've noticed is due to us passing the "naked" dask.array namespace (as opposed to the array-api-compat wrapped version) in as the xp fixture in tests even though it is not really array API compatible. This is something that we do for the other libraries too (e.g. torch), even though those libraries also seem not to be fully array API compatible.

The problem is worse for dask since oftentimes in the there is an xp.asarray(..., copy=True) call, and I think the default dask asarray doesn't support copy as a parameter (it accepts it as a kwarg silently and then ignores it I think).

This is probably why we had the issue with the copy not happening in logsumexp I think too I think.
(I haven't verified this particular case, but I've tested fft and using array_api_compat.dask.array in the xp_available_backends variable solves the remaining failing tests in fft which were previously failing due to copies not happening/dtype difference)

Now, I should have all of fft/linalg/special passing.

stats looks do-able, the main changes seem to be silencing/ignoring the RuntimeWarnings from dask and figuring out how to make _lazy_where not use dynamic shapes for dask/jax and co.

I've fixed up the merge conflicts in lucascolley#15, however the diff is very large, so I've split that out of my dask changes.

(Maybe this is because I do merge commits instead of rebasing to keep up with main?)

thanks @lithomas1, I rebased on main here. The CI log is quite messy due to an error in TestLazywhere in _lib.tests.test__util:

TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

thanks @lithomas1, I rebased on main here. The CI log is quite messy due to an error in TestLazywhere in _lib.tests.test__util:

Thanks for the update.

I've attempted to cherry-pick my changes on top of your branch in lucascolley#16.

(The history is still a little messed up - but I think it should be OK if you're fine with squashing my PR into a single commit).

continued in #22240