DEP: `stats.ttest_ind`: deprecate `random_state` and `permutation` in favor of `method`; require keywords for most args #21921

mdhaber · 2024-11-21T04:59:44Z

Reference issue

What does this implement/fix?

The permutation and random_state arguments of ttest_ind were added in gh-4824 to enable the user to perform a permutation version of the t-test. A standard API for performing resampling versions of tests using a method argument was developed in gh-18227. The SPEC 7 deprecation of the random_state arguments provides some impetus for transitioning from permutation and random_state to method.

This PR makes the following changes to ttest_ind:

adds a method argument that allows the user to configure permutation and Monte Carlo versions of the test
deprecates permutation and random_state arguments in favor of method
deprecates positional use of all arguments of ttest_ind after x and y. The removal of permutation and random_state would also require later arguments (alternative and trim) to become keyword-only, anyway; we might as well make all the optional parameters keyword-only at the same time.

The transition to the new interface is quite simple:

from scipy import stats
... # define variables

# This:
stats.ttest_ind(a, b, 0, True, 'propagate', 9999, rng, 'two-sided', 0)

# becomes
method = stats.PermutationMethod(n_resamples=9999, rng=rng)
stats.ttest_ind(a, b, axis=0, equal_var=True, nan_policy='propagate',
                method=method, trim=0)

… use of permutations/random_state

…f test

mdhaber

Some comments to aid the reviewer.

mdhaber · 2024-11-21T05:01:43Z

scipy/_lib/deprecation.py

+            kwonly_extra_args = set(kwonly_args[:extra_args]) - deprecated_args
+            args_msg = ", ".join(kwonly_extra_args)


This warns about the deprecated positional use of arguments passed by position. We don't want the deprecated_args (which are completely decorated) to be included in that list; warn about those separately.

mdhaber · 2024-11-21T05:03:02Z

scipy/_lib/deprecation.py

+        admonition = f"""
+.. deprecated:: {version}
+    Use of argument(s) ``{kwonly_extra_args}`` by position is deprecated; beginning in 
+    SciPy {version}, these will be keyword-only. """


Ugly indentation; I know, but it makes it easy to keep Sphinx happy.

mdhaber · 2024-11-21T05:05:46Z

scipy/stats/_stats_py.py

+    (In the following, note that the argument `permutations` itself is
+    deprecated, but a nearly identical test may be performed by creating
+    an instance of `scipy.stats.PermutationMethod` with ``n_resamples=permutuations``
+    and passing it as the `method` argument.)


We could just remove all of this temporarily. We will be writing an interactive notebook tutorial for permutation and Monte Carlo versions of the t-test (like https://scipy.github.io/devdocs/tutorial/stats/hypothesis_normaltest.html) shortly, and we will link to that from here.

mdhaber · 2024-11-21T05:08:23Z

scipy/stats/tests/test_resampling.py

-    # here in permutation_test.
-    @pytest.mark.parametrize('permutations', (30, 1e9))
-    @pytest.mark.parametrize('axis', (0, 1, 2))
-    def test_against_permutation_ttest(self, alternative, permutations, axis):


This tested scipy.stats.permutation_test against using the permutations argument of ttest_ind, but there are other, stronger tests now.

mdhaber · 2024-11-21T05:08:43Z

scipy/stats/tests/test_stats.py

@@ -5343,6 +5343,7 @@ def test_ttest_ind_scalar():
    assert np.isnan(p)


+@pytest.mark.filterwarnings("ignore:Arguments...:DeprecationWarning")


We test for the warning about deprecated permutations below.

I suppose there are no tests for deprecated positional use, but I assume the _deprecate_positional_args decorator was already tested, so I'm just testing the new part.

mdhaber · 2024-11-21T05:09:38Z

scipy/stats/tests/test_stats.py

+
+    @pytest.mark.parametrize("alternative", ['less', 'greater', 'two-sided'])
+    @pytest.mark.parametrize("shape", [(12,), (2, 12)])
+    def test_monte_carlo_method(self, alternative, shape):


I tried combining this with the previous test, but this is really more readable.

We could also consider removing the Monte Carlo version of the test entirely during this (complicated) transition period, just to keep things a little simpler.

j-bowhay · 2024-11-22T16:32:19Z

Could you write a post to the forum regarding the deprecation?

j-bowhay · 2024-12-01T22:17:54Z

scipy/stats/_stats_py.py

+    if not isinstance(method, PermutationMethod | MonteCarloMethod | None):
+        message = ("`method` must be an instance of `PermutationMethod`, an instance "
+                   "of `MonteCarloMethod`, or None (default).")
+        raise ValueError(message)


Can we add a test for this

j-bowhay · 2024-12-01T22:18:01Z

scipy/stats/_stats_py.py

+
+    if not is_numpy(xp) and method is not None:
+        message = "Use of resampling methods is compatible only with NumPy arrays."
+        raise NotImplementedError(message)


j-bowhay · 2024-12-01T22:20:57Z

scipy/stats/tests/test_stats.py

-            stats.ttest_ind([1, 2], [2, 3], trim=.2, permutations=2)
+            message = "Arguments {'permutations'} are deprecated, whether..."
+            with pytest.warns(DeprecationWarning, match=message):
+                stats.ttest_ind([1, 2], [2, 3], trim=.2, permutations=2)


Can we test randomatate also raises a warning

j-bowhay

A few comments about tests but otherwise looks good

mdhaber · 2024-12-05T15:08:36Z

scipy/stats/tests/test_stats.py

+        assert_allclose(res.pvalue, ref.pvalue, rtol=6e-2)
+
+    def test_resampling_input_validation(self):
+        rng = np.random.default_rng(328924823)


Will commit with [lint only] if the rest looks good.

Suggested change

rng = np.random.default_rng(328924823)

…thod

j-bowhay

Thanks! Can you make sure this makes it into the release notes

mdhaber · 2024-12-06T15:17:03Z

Done. Thanks @j-bowhay!

mdhaber added 5 commits November 20, 2024 09:09

ENH: stats.ttest_ind: add method arugment

9b579dc

DEP: stats.ttest_ind: deprecate positional use of most arguments; all…

164e1f1

… use of permutations/random_state

DOC: stats.ttest_ind: adjust documentation about deprecated parameters

94fbf82

TST: stats: adjust/remove tests using deprecated features

fa78cd5

MAINT: stats.ttest_ind: pass additional args to permutation version o…

1e476d2

…f test

mdhaber added scipy.stats maintenance Items related to regular maintenance tasks labels Nov 21, 2024

github-actions bot added scipy._lib deprecated Items related to behavior that has been deprecated labels Nov 21, 2024

mdhaber commented Nov 21, 2024

View reviewed changes

mdhaber mentioned this pull request Nov 21, 2024

SPEC 7 Transition Tracker #21833

Closed

50 tasks

lucascolley changed the title ~~DEP: stats.ttest_ind: deprecate random_state and permutation in favor of method; require keywords for most args~~ DEP: stats.ttest_ind: deprecate random_state and permutation in favor of method; require keywords for most args Nov 21, 2024

mdhaber mentioned this pull request Nov 26, 2024

TST: add more thorough SPEC 7 tests #21948

Merged

j-bowhay reviewed Dec 1, 2024

View reviewed changes

TST: stats.ttest_ind: improve tests of errors/warnings

258cc0d

mdhaber commented Dec 5, 2024

View reviewed changes

mdhaber added 3 commits December 5, 2024 08:58

Merge remote-tracking branch 'upstream/main' into ttest_method

77a3d84

Merge branch 'ttest_method' of github.com:mdhaber/scipy into ttest_me…

b5280bb

…thod

MAINT: stats.ttest_ind: avoid passing 'rng' to 'monte_carlo_test'

b0eaff2

j-bowhay approved these changes Dec 6, 2024

View reviewed changes

j-bowhay added this to the 1.15.0 milestone Dec 6, 2024

j-bowhay merged commit abdadb7 into scipy:main Dec 6, 2024
37 checks passed

tylerjereddy added the needs-release-note a maintainer should add a release note written by a reviewer/author to the wiki label Dec 6, 2024

j-bowhay mentioned this pull request Dec 9, 2024

META: deprecation tracker #15765

Open

32 tasks

j-bowhay mentioned this pull request May 31, 2025

DEP: stats.ttest_ind: remove random_state and permutation #23088

Merged

lucascolley removed the needs-release-note a maintainer should add a release note written by a reviewer/author to the wiki label Jun 9, 2025

		kwonly_extra_args = set(kwonly_args[:extra_args]) - deprecated_args
		args_msg = ", ".join(kwonly_extra_args)

		@@ -5343,6 +5343,7 @@ def test_ttest_ind_scalar():
		assert np.isnan(p)


		@pytest.mark.filterwarnings("ignore:Arguments...:DeprecationWarning")

Uh oh!

DEP: stats.ttest_ind: deprecate random_state and permutation in favor of method; require keywords for most args #21921

DEP: stats.ttest_ind: deprecate random_state and permutation in favor of method; require keywords for most args #21921

Uh oh!

Conversation

mdhaber commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference issue

What does this implement/fix?

Uh oh!

mdhaber left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

j-bowhay commented Nov 22, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

j-bowhay left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

j-bowhay left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mdhaber commented Dec 6, 2024

Uh oh!

Uh oh!

DEP: `stats.ttest_ind`: deprecate `random_state` and `permutation` in favor of `method`; require keywords for most args #21921

DEP: `stats.ttest_ind`: deprecate `random_state` and `permutation` in favor of `method`; require keywords for most args #21921

mdhaber commented Nov 21, 2024 •

edited

Loading