ENH: `stats.zmap`/`zscore`/`gzscore`: add array API support #21068

mdhaber · 2024-06-27T21:21:24Z

Reference issue

Toward gh-20544

What does this implement/fix?

Adds array API support to zmap, zscore, and gzscore.

scipy/stats/_stats_py.py

scipy/stats/tests/test_stats.py

mdhaber · 2024-06-28T13:02:06Z

Array API strict failures seem to be unrelated (gh-21073).

j-bowhay · 2024-06-29T08:36:20Z

scipy/stats/_stats_py.py

+
+def _convert_common_float(*arrays, xp=None):
+    xp = array_namespace(*arrays) if xp is None else xp
+    arrays = [xp.asarray(array) for array in arrays]


I think this might need to be

Suggested change

arrays = [xp.asarray(array) for array in arrays]

arrays = [_asarray(array, subok=True) for array in arrays]

otherwise line 3158 will break subclasses for gzscore.

Also with this change _zmap_array_api will preserve subclasses so could we only keep the new shorter implementation as it seems much cleaner and by the test comment seems to fix a bug or is it backwards incompatible?

Mmm maybe. I'll investigate. But in the meantime I've followed the approach of zmap and just branched here depending on SCIPY_ARRAY_API to avoid any possible unintended interactions.

@j-bowhay the second to last commit removes the old implementation. Besides using _asarrray, I needed to replace a use of xp.where, which doesn't preserve array subclasses.

Note that _xp_mean and _xp_var "just worked" as far as the existing unit tests are concerned, but note that they weren't written with masked arrays in mind, and the unit tests are very weak. I don't have a preference if you're willing to test/review more carefully to check that removing the old implementation is safe (including for other subclasses, like matrix), but personally, I'd just as soon keep both implementations and remove the old one when we no longer have to support subclasses.

Ok thanks I will look more closely over the next couple of days

scipy/stats/_stats_py.py

[lint only]

mdhaber · 2024-06-30T18:35:53Z

scipy/stats/tests/test_stats.py

        z = stats.gzscore(mx)
        desired = ([-1.526072095151, -0.194700599824, np.inf, 0.584101799472,
                    1.136670895503])
-        assert_allclose(desired, z)


This test had the actual and desired values mixed up, and it was checking for an inf where the element should be masked. (I can't see why inf would be the right answer anyway - probably was just a fill_value that got put in during the calculation?) I corrected the test to consider the fact that the output should be a masked array.

scipy/stats/_stats_py.py

j-bowhay

Having played with this a bit I think the best option might be to revert to 396e12c as the like_zscore branch in _zmap doesn't not preserve subclasses, presumably due to xp.where not preserving subclasses.

In [1]: import numpy as np

In [2]: class ExampleTensor(np.ndarray):
   ...:     def __new__(cls, input_array):
   ...:         return np.asarray(input_array).view(cls)
   ...:
   ...:     def __array_finalize__(self, obj) -> None:
   ...:         if obj is None: return
   ...:         # This attribute should be maintained!
   ...:         default_attributes = {"attr": 1}
   ...:         self.__dict__.update(default_attributes)  # another way to set attributes
   ...:

In [3]: a = ExampleTensor([0.5, 2.0, 2.5, 3])

In [4]: from scipy.stats import zmap

In [5]: zmap(a, a)
Out[5]: array([-1.60356745,  0.        ,  0.53452248,  1.06904497])

In [6]: b = ExampleTensor([0, 1, 2, 3, 4])

In [7]: zmap(a, b)
Out[7]: ExampleTensor([-1.06066017,  0.        ,  0.35355339,  0.70710678])

Sorry for the wild goose chase, I am just keen to keep the implementation as clean as possible where reasonable to do so.

mdhaber · 2024-07-07T13:35:18Z

Well, I can probably try using item assignment there instead? JAX is already skipped sometimes.

j-bowhay · 2024-07-07T13:50:15Z

Well, I can probably try using item assignment there instead? JAX is already skipped sometimes.

Yes I think that would also work

mdhaber · 2024-07-07T14:19:59Z

OK, tried that so you can test more thoroughly. Also replaced the use of NumPy to work with axis. Ternary version provided as a comment ; )

Please comment on the differences the things the old code took extra care on, such as warnings (or lack thereof) and results when elements of a slice are not unique.

j-bowhay

I tried zmap with some examples with nans and nan_policy="omit" and saw no issues. Also tried an a which was constant along axis and that worked fine so I think this is ok to proceed.

I assume we can now just move the contents of _zmap into zmap?

mdhaber · 2024-07-18T18:17:09Z

@melissawm any guess on whether the PosixPath('/home/circleci/repo/doc/source/_contents/hypothesis_bartlett.ipynb') are the same file error could be related to this PR? I'm not seeing it elsewhere, but I don't see how that could be affected by this PR. I just merged main, so I don't think it's related to gh-21195.

reading sources... [ 99%] tutorial/stats/discrete_planck
reading sources... [ 99%] tutorial/stats/discrete_poisson
reading sources... [ 99%] tutorial/stats/discrete_randint
reading sources... [ 99%] tutorial/stats/discrete_yulesimon
reading sources... [ 99%] tutorial/stats/discrete_zipf
reading sources... [ 99%] tutorial/stats/discrete_zipfian
reading sources... [ 99%] tutorial/stats/hypothesis_bartlett
/home/circleci/repo/doc/source/tutorial/stats/hypothesis_bartlett.md: Executing notebook using local CWD [mystnb]
/home/circleci/repo/doc/source/tutorial/stats/hypothesis_bartlett.md: Executed notebook in 9.00 seconds [mystnb]

Traceback (most recent call last):
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/cmd/build.py", line 337, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/application.py", line 378, in build
    self.builder.build_update()
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 297, in build_update
    self.build(to_build,
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 318, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 425, in read
    self._read_serial(docnames)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 477, in _read_serial
    self.read_doc(docname)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 536, in read_doc
    publisher.publish()
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/core.py", line 234, in publish
    self.document = self.reader.read(self.source, self.parser,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sphinx/io.py", line 106, in read
    self.parse()
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/readers/__init__.py", line 76, in parse
    self.parser.parse(self.input, document)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_nb/sphinx_.py", line 157, in parse
    mdit_renderer.render(mdit_tokens, mdit_parser.options, mdit_env)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_parser/mdit_to_docutils/base.py", line 243, in render
    self._render_tokens(list(tokens))
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_parser/mdit_to_docutils/base.py", line 222, in _render_tokens
    self.rules[f"render_{child.type}"](child)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_nb/core/render.py", line 117, in render_nb_cell_markdown
    self.render_children(token)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_parser/mdit_to_docutils/base.py", line 396, in render_children
    self.rules[f"render_{child.type}"](child)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_parser/mdit_to_docutils/base.py", line 752, in render_fence
    return self.render_restructuredtext(token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_parser/mdit_to_docutils/base.py", line 1680, in render_restructuredtext
    MockRSTParser().parse(pseudosource, newdoc)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/myst_parser/mocking.py", line 552, in parse
    super().parse(inputstring, document)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/parsers/rst/__init__.py", line 184, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/parsers/rst/states.py", line 169, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/statemachine.py", line 233, in run
    context, next_state, result = self.check_line(
                                  ^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/statemachine.py", line 445, in check_line
    return method(match, context, next_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/parsers/rst/states.py", line 2357, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/parsers/rst/states.py", line 2369, in explicit_construct
    return method(self, expmatch)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/parsers/rst/states.py", line 2106, in directive
    return self.run_directive(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/docutils/parsers/rst/states.py", line 2156, in run_directive
    result = directive_instance.run()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/site-packages/jupyterlite_sphinx/jupyterlite_sphinx.py", line 368, in run
    shutil.copy(notebook, notebooks_dir)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/shutil.py", line 431, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/shutil.py", line 236, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/home/circleci/repo/doc/source/_contents/hypothesis_bartlett.ipynb' and PosixPath('/home/circleci/repo/doc/source/_contents/hypothesis_bartlett.ipynb') are the same file

Exception occurred:
  File "/home/circleci/.pyenv/versions/3.11.9/lib/python3.11/shutil.py", line 236, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/home/circleci/repo/doc/source/_contents/hypothesis_bartlett.ipynb' and PosixPath('/home/circleci/repo/doc/source/_contents/hypothesis_bartlett.ipynb') are the same file
The full traceback has been saved in /tmp/sphinx-err-ndbm02yd.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [Makefile:114: html-build] Error 2
Skipping build

Exited with code exit status 1

j-bowhay · 2024-07-18T18:29:54Z

The same failure showed up in #21209 so looks unrelated

melissawm · 2024-07-18T21:30:59Z

I believe this has to do with a new release of jupyterlite-sphinx. @steppi is looking into it, thanks for the ping!

melissawm · 2024-07-19T13:10:49Z

Docs CI should be functional again, cheers!

[docs only]

j-bowhay

Thanks @mdhaber

mdhaber added 4 commits June 25, 2024 23:54

ENH: stats.zmap: add array API support

885424d

ENH: stats.zscore/gzscore: add array API support

be4e28f

MAINT: stats.zmap: warn/produce NaNs on precision loss

a09c28e

TST: stats.zmap: skip JAX for tests that need item assignment

c8aa2e5

mdhaber added scipy.stats enhancement A new feature or improvement array types Items related to array API support and input array validation (see gh-18286) labels Jun 27, 2024

j-bowhay reviewed Jun 27, 2024

View reviewed changes

mdhaber commented Jun 27, 2024

View reviewed changes

scipy/stats/tests/test_stats.py Outdated Show resolved Hide resolved

MAINT: stats.zmap: adjustments per review

e91e693

mdhaber mentioned this pull request Jun 28, 2024

ENH: stats: add array API-support #20544

Open

j-bowhay reviewed Jun 29, 2024

View reviewed changes

mdhaber added 2 commits June 30, 2024 08:26

MAINT: stats,gzscore: avoid conversion of masked array to regular array

396e12c

MAINT: stats.zmap: remove old implementation

214e957

mdhaber commented Jun 30, 2024

View reviewed changes

scipy/stats/_stats_py.py Outdated Show resolved Hide resolved

Update scipy/stats/_stats_py.py

ce58c81

[lint only]

mdhaber commented Jun 30, 2024

View reviewed changes

mdhaber mentioned this pull request Jul 1, 2024

ENH: stats.alexandergovern: vectorize calculation for n-D arrays #21089

Merged

j-bowhay reviewed Jul 5, 2024

View reviewed changes

scipy/stats/_stats_py.py Show resolved Hide resolved

j-bowhay reviewed Jul 7, 2024

View reviewed changes

MAINT: stats.zscore: preserve array subclasses

e7d83d9

j-bowhay mentioned this pull request Jul 16, 2024

BUG: stats.differential_entropy: incorrect results with integer dtype and method='ebrahimi' #21192

Closed

mdhaber requested a review from j-bowhay July 16, 2024 16:35

j-bowhay reviewed Jul 18, 2024

View reviewed changes

mdhaber added 2 commits July 18, 2024 09:44

Merge remote-tracking branch 'upstream/main' into xp_zmap

dc70d26

STY: stats.zmap: merge _zmap into zmap

a9c91ec

j-bowhay mentioned this pull request Jul 18, 2024

DOC: doc build failing in CI #21212

Closed

Merge remote-tracking branch 'upstream/main' into xp_zmap

9461920

[docs only]

j-bowhay approved these changes Jul 19, 2024

View reviewed changes

j-bowhay merged commit d60b48d into scipy:main Jul 19, 2024

j-bowhay added this to the 1.15.0 milestone Jul 19, 2024

	arrays = [xp.asarray(array) for array in arrays]
	arrays = [_asarray(array, subok=True) for array in arrays]

Uh oh!

ENH: stats.zmap/zscore/gzscore: add array API support #21068

ENH: stats.zmap/zscore/gzscore: add array API support #21068

Uh oh!

Conversation

mdhaber commented Jun 27, 2024

Reference issue

What does this implement/fix?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdhaber commented Jun 28, 2024

Uh oh!

j-bowhay Jun 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdhaber Jun 30, 2024

Choose a reason for hiding this comment

Uh oh!

mdhaber Jun 30, 2024

Choose a reason for hiding this comment

Uh oh!

j-bowhay Jun 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mdhaber Jun 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

j-bowhay left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdhaber commented Jul 7, 2024

Uh oh!

j-bowhay commented Jul 7, 2024

Uh oh!

mdhaber commented Jul 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

j-bowhay left a comment

Choose a reason for hiding this comment

Uh oh!

mdhaber commented Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

j-bowhay commented Jul 18, 2024

Uh oh!

melissawm commented Jul 18, 2024

Uh oh!

melissawm commented Jul 19, 2024

Uh oh!

j-bowhay left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ENH: `stats.zmap`/`zscore`/`gzscore`: add array API support #21068

ENH: `stats.zmap`/`zscore`/`gzscore`: add array API support #21068

j-bowhay Jun 29, 2024 •

edited

Loading

mdhaber Jun 30, 2024 •

edited

Loading

j-bowhay left a comment •

edited

Loading

mdhaber commented Jul 7, 2024 •

edited

Loading

mdhaber commented Jul 18, 2024 •

edited

Loading