I suppose we don't need to fix gh-19683 (causing the CI failures) before this, in which case this is ready for review @tylerjereddy (no rush of course)

I've reverted the fix for gh-19683 as that can be done separately, I think this should pass CI now

@tylerjereddy I've fixed the granularity problem by using a pytest fixture and a new mark. I've also consolidated all of the decorators into one, so that now we just have two additional marks whenever any skipping is used.

Sorry, the diff has increased in size by quite a bit, but I think this is the right direction.

Okay, this is ready for a look @tylerjereddy . I've had some success with the rework - the granularity issue is fixed, the skips can be applied to classes now, and overall verbosity in the test files is reduced greatly.

cc @tupui as original author of the infra in case you're interested

Thanks Tyler!

I think I'm in favor of merge once the failing test is marked appropriately. Perhaps we could start a fresh issue for the array-like/empty list input business since going through this PR multiple rounds can take a fair bit of time, and that was only a few cases I think.

If you don't mind, when I find the time, I'll go ahead with my suggestion of:

• When a parameter takes (e.g.) either an ndarray or an int, we should have tests for both types, for all backends.

• When a parameter takes just an ndarray (but e.g. ints happen to work with NumPy), we do not support array-likes for alternative backends. xp.asarray should be used in all tests, and a singular separate test, just for NumPy, should check array-like input.

and make sure that is consistent here. I'm pretty sure that makes sense and is better than what we currently have.

I'll also clear up any unresolved comments and let you know when I think this looks ready 👍

@tylerjereddy I think this is ready to go now. I believe that the test coverage is now strictly increased in this PR. The lint failures are unrelated (will send a separate PR). oops, lint error is related, fixing now

@rgommers merge conflict resolved, at your leisure :)

Still one numpy.array_api reference lurking in the docs.

I spent quite some time reading every line of the diff again and checking the arithmetic on the skip/pass results for a cuda-enabled full array API test suite assessment of this branch vs. main. I think this is the second time I've done this now, and the remaining cleanups/comments don't justify a block IMO. Much easier to review those smaller things in a follow-up.

This is a clear improvement in the array API testing infrastructure so I'm going to merge it now. The commit history is clean/reasonable for the size of the PR, so I'll keep it. This is mostly internal testing infrastructure improvements, and has been open for long enough to give folks time to comment if they wanted to.

It is true that we're getting deeper into pytest + decorator magic, but hard to avoid for all the backend orchestration.

Thanks!

thanks for the thorough reviews Tyler! Will send a follow-up for the few remaining bits in the coming days 👍