Allow *_like array creation functions to respect input array type (eg: numpy/cupy/etc.) #6680

GenevieveBuckley · 2020-09-28T11:18:00Z

This PR provides support for different array types in the *_like array creation functions. Eg: you can pass a cupy array (or cupy-backed dask array) to da.ones_like and get back a cupy-backed dask array of the appropriate shape filled with ones.

Tests added / passed
Passes black dask / flake8 dask

…py/etc.)

…e in *_like creation functions

mrocklin · 2020-09-28T17:11:57Z

cc @pentschev

dask/array/wrap.py

dask/array/creation.py

dask/array/tests/test_wrap.py

mrocklin · 2020-09-30T23:11:22Z

The doctest failure seems genuine (it looks like we test docstrings on Python 3.8)

I'm not sure what's going on in Python 3.6, and if that is the fault of this PR nor not.

GenevieveBuckley · 2020-10-01T00:09:34Z

I have no idea what's going on with the test failures for python 3.6 either. Confusingly, when I make a python 3.6 development environment on my local windows machine, all the tests pass without error. I'm not exactly sure what to try next.

The docstring that's failing is one that is being wrapped from an external library. If it was defined within dask I could just add the four extra spaces, that would be easy. Not totally sure what's the best way for us to handle this either.

daxiongshu · 2020-10-06T18:41:58Z

@GenevieveBuckley I did see the same error when testing in 3.6. Here is how I create a test environment:

conda env create -f continuous_integration/environment-3.6.yaml

Did you do the same?

jsignell · 2020-10-06T19:34:59Z

I don't think the doctest failures are related to the python version. Just in the CI testing matrix the doctests are only run on certain python versions.

Just to check are you running the doctests. I think I don't normally run them locally. I am able to reproduce the failures with

pytest -nauto --doctest-modules --ignore-glob "test_*.py" dask/array

jsignell · 2020-10-06T19:59:50Z

I think it's a numpy issue and opened a PR up there: numpy/numpy#17472. We can probably work around though.

jsignell · 2020-10-06T20:27:10Z

@GenevieveBuckley I hope you don't mind that I pushed a fix. I was hoping that we could merge this today if it passes. @mrocklin to confirm that this is good to go once it passes.

jsignell · 2020-10-06T22:17:42Z

Apparently not a very good fix 🙄

GenevieveBuckley · 2020-10-07T04:12:29Z

Problem 1 - python 3.6 test failures

Yes, it looks like I had created a python 3.6 environment in a different way than conda env create -f continuous_integration/environment-3.6.yaml. When I do that I can reproduce the failures locally, thank you for suggesting I double check @daxiongshu

Things are still weird though: I can reproduce the failing tests using pytest, but I can't launch python/ipython from the same test environment and manually cause the same error using the same example as in the tests. I tried this using test_blockwise_literals just because this was the first failing test.

# This code snippet is equivalent to the first part of the failing test_blockwise_literals() in dask/array/tests/test_array_core.py
from operator import add
import dask.array as da
from dask.array.utils import assertt_eq

x = da.ones((10, 10), chunks(5, 5))
z = da.blockwise(add, "ij", x, "ij", 100, None, dtype=x.dtype)
assert_eq(z, x + 100)  # I expected this to fail because test_blockwise_literals fails in pytest, but it didn't

I've also tried looking at the differences in packages between the passing test environment and the failing test environment, and trying to "narrow the gap" between the two situations by installing some of the extra missing packages one by one into the passing environment. This hasn't helped narrow it down much so far.

Problem 2 - doctest failure

Thank you for trying to fix this @jsignell, I really appreciate it!

I think your fix might have worked? I'm not sure why it says "This check failed" on this CI action, because it looks like the same CI action passed here on my branch. Your fix certainly looks like it should work - maybe the fail message was just a glitch?

We'll still need to fix the first problem before merging this PR. Merging today might be a little ambitious, given that.

GenevieveBuckley · 2020-10-07T07:40:19Z

So it seems I was doing something silly with my test earlier.

Here is a very simple test case that might be useful for tracking down the source of all these CI failures:

import dask.array as da
da.ones((2, 2)).compute()

Using python 3.8 and the dask development version from this branch, we get the expected answer:

array([[1., 1.],
       [1., 1.]])

But with the python 3.6 test environment from continuous_integration/environment3-6.yaml I get this unexpected output, which is what's causing all the downstream test failures:

<function _broadcast_trick_inner at 0x0000021DA3F488C8>

Even more annoyingly, in other python 3.6 environments with this development dask version, I get the correct answer again:

array([[1., 1.],
       [1., 1.]])

GenevieveBuckley · 2020-10-07T07:58:18Z

Here's how to see the weird difference between behaviour between different python 3.6 environments.

Setup environments

Run conda env create -f continuous_integration/environment-3.6.yaml. This command will create test-environment. Make sure you have the development dask version installed into this environment
Create a second python 3.6 environment

conda create -n dask-env-py36 python=3.6 pip
conda activate dask-env-py36
cd Documents/Github/dask
pip install -e .[complete]

Test

import dask
import dask.array as da

print(dask.__version__)
da.ones((2, 2)).compute()
# should output an array of ones, not something like <function _broadcast_trick_inner at 0x0000021DA3F488C8>

Compare environments

I've used this conda compare script to compare the two environments. (You could also use the conda compare command if you have a very recent version of conda.)

CLICK ME

``` >python conda_compare.py test-environment dask-env-py36 conda list -n test-environment conda list -n dask-env-py36 In only test-environment scikit-image 0.17.2 py36hcc50265_1 conda-forge olefile 0.46 py_0 conda-forge intel-openmp 2019.4 245 defaults libxslt 1.1.33 h579f668_1 conda-forge execnet 1.7.1 py_0 conda-forge pytables 3.6.1 py36he284194_2 conda-forge libxml2 2.9.10 h1006b36_2 conda-forge backports.functools_lru_cache1.6.1 py_0 conda-forge python-jose 3.1.0 pyh9f0ad1d_0 conda-forge yarl 1.3.0 py36hfa6e2cd_1000 conda-forge botocore 1.18.13 pyh9f0ad1d_0 conda-forge tiledb-py 0.5.9 py36_vc14h707988f_1 conda-forge yaml 0.2.5 he774522_0 conda-forge attrs 20.2.0 pyh9f0ad1d_0 conda-forge c-ares 1.16.1 he774522_3 conda-forge importlib-metadata 2.0.0 py36h9f0ad1d_0 conda-forge mkl-service 2.3.0 py36hfa6e2cd_0 conda-forge curl 7.71.1 h4b64cdc_8 conda-forge numexpr 2.7.1 py36h003fed8_1 conda-forge h5py 2.10.0 nompi_py36h3b971d6_104conda-forge blosc 1.20.1 ha925a31_0 conda-forge sparse 0.11.2 py_0 conda-forge typing_extensions 3.7.4.2 py_0 conda-forge jedi 0.17.2 py36h9f0ad1d_0 conda-forge numba 0.50.1 py36hcc50265_1 conda-forge parso 0.7.1 pyh9f0ad1d_0 conda-forge pygments 2.7.1 py_0 conda-forge openssl 1.1.1h he774522_0 conda-forge ecdsa 0.13 py_0 conda-forge werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge toml 0.10.1 pyh9f0ad1d_0 conda-forge traitlets 4.3.3 py36h9f0ad1d_1 conda-forge monotonic 1.5 py_0 conda-forge brotli 1.0.9 ha925a31_1 conda-forge pyarrow 0.14.1 py36h803c963_1 conda-forge importlib_metadata 2.0.0 0 conda-forge websocket-client 0.57.0 py36h9f0ad1d_2 conda-forge importlib_resources 1.5.0 py36h9f0ad1d_0 conda-forge pyrsistent 0.17.3 py36h779f372_0 conda-forge libpng 1.6.37 ha81a0f5_2 conda-forge urllib3 1.25.10 py_0 conda-forge httpretty 1.0.2 py_0 conda-forge cycler 0.10.0 py_2 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge xxhash 0.8.0 he774522_1 conda-forge re2 2020.04.01vc14h6538335_0 conda-forge krb5 1.17.1 hc04afaa_3 conda-forge scikit-learn 0.23.1 py36h13569b7_0 conda-forge py 1.9.0 pyh9f0ad1d_0 conda-forge bzip2 1.0.8 he774522_3 conda-forge thrift-cpp 0.12.0 hd042d19_1004 conda-forge itsdangerous 1.1.0 py_0 conda-forge imageio 2.9.0 py_0 conda-forge appdirs 1.4.3 py_1 conda-forge mkl 2019.4 245 defaults jsonpointer 2.0 py_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge cfn-lint 0.37.1 py36h9f0ad1d_0 conda-forge bcolz 1.2.1 py36he350917_1001 conda-forge mock 4.0.2 py36h9f0ad1d_0 conda-forge jpeg 9d he774522_0 conda-forge pytest-forked 1.2.0 pyh9f0ad1d_0 conda-forge boto3 1.15.13 pyh9f0ad1d_0 conda-forge flask 1.1.2 pyh9f0ad1d_0 conda-forge pyopenssl 19.1.0 py_1 conda-forge boto 2.49.0 py_0 conda-forge multidict 4.7.5 py36h779f372_1 conda-forge tbb 2018.0.5 he980bc4_0 conda-forge libblas 3.8.0 14_mkl conda-forge libtiff 4.1.0 h885aae3_6 conda-forge llvmlite 0.33.0 py36hefa7ec1_1 conda-forge scipy 1.5.2 py36h9439919_0 defaults liblapack 3.8.0 14_mkl conda-forge wrapt 1.12.1 py36h68a101e_1 conda-forge networkx 2.5 py_0 conda-forge matplotlib-base 3.3.2 py36h856a30b_0 conda-forge responses 0.12.0 pyh9f0ad1d_0 conda-forge pysocks 1.7.1 py36h9f0ad1d_1 conda-forge coverage 5.3 py36h779f372_0 conda-forge tifffile 2020.6.3 py_0 conda-forge blas 1.0 mkl defaults ipython 7.16.1 py36h95af2a2_0 conda-forge pandas-datareader 0.8.1 py_0 conda-forge mmh3 2.5.1 py36h6538335_0 conda-forge pytest 6.1.1 py36h9f0ad1d_0 conda-forge libprotobuf 3.8.0 h1a1b453_0 conda-forge iniconfig 1.0.1 pyh9f0ad1d_0 conda-forge chest 0.2.3 py_2 conda-forge parquet-cpp 1.5.1 2 conda-forge atomicwrites 1.4.0 pyh9f0ad1d_0 conda-forge imagecodecs-lite 2019.12.3 py36h7725771_1 conda-forge uriparser 0.9.3 he025d50_1 conda-forge brotlipy 0.7.0 py36h779f372_1000 conda-forge msgpack-python 1.0.0 py36h246c5b5_1 conda-forge python-snappy 0.5.4 py36h9a6de72_1 conda-forge colorama 0.4.3 py_0 conda-forge asn1crypto 1.4.0 pyh9f0ad1d_0 conda-forge cookies 2.2.1 py_0 conda-forge jsonschema 3.2.0 py36h9f0ad1d_1 conda-forge gflags 2.2.2 ha925a31_1004 conda-forge requests 2.24.0 pyh9f0ad1d_0 conda-forge future 0.18.2 py36h9f0ad1d_1 conda-forge python_abi 3.6 1_cp36m conda-forge libiconv 1.16 he774522_0 conda-forge idna_ssl 1.1.0 py36_1000 conda-forge async-timeout 3.0.1 py_1000 conda-forge cffi 1.14.3 py36hef61171_0 conda-forge pytest-xdist 2.1.0 py_0 conda-forge tiledb 1.7.7 hffbbd95_0 conda-forge numcodecs 0.7.2 py36h003fed8_0 conda-forge zipp 3.3.0 py_0 conda-forge docker-py 4.3.1 py36h9f0ad1d_0 conda-forge lxml 4.5.2 py36hd9844f4_0 conda-forge icc_rt 2019.0.0 h0cc432a_1 defaults double-conversion 3.1.5 h6538335_2 conda-forge prompt-toolkit 3.0.7 py_0 conda-forge jsonpatch 1.24 py_0 conda-forge win_inet_pton 1.1.0 py36_0 conda-forge jsondiff 1.1.2 py_0 conda-forge fasteners 0.14.1 py_3 conda-forge crick 0.0.3 py36hc8d92b1_1000 conda-forge tk 8.6.10 he774522_0 conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge pyreadline 2.1 py36_1001 conda-forge docker-pycreds 0.4.0 py_0 conda-forge sshpubkeys 3.1.0 py_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge decorator 4.4.2 py_0 conda-forge xz 5.2.5 h62dcd97_1 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge glog 0.4.0 h0174b99_3 conda-forge ipython_genutils 0.2.0 py_1 conda-forge xmltodict 0.12.0 py_0 conda-forge lz4 3.0.2 py36hec809b8_1 conda-forge backports.tempfile 1.0 py_0 conda-forge hdf5 1.10.6 nompi_he0bbb20_101 conda-forge lz4-c 1.8.3 he025d50_1001 conda-forge libcblas 3.8.0 14_mkl conda-forge grpc-cpp 1.25.0 h4d7d3fa_0 conda-forge cryptography 3.1.1 py36hef61171_0 conda-forge arrow-cpp 0.14.1 py36hb9d24f8_1 conda-forge graphviz 2.38.0 h6538335_1011 conda-forge apipkg 1.5 py_0 conda-forge aiohttp 3.6.2 py36hfa6e2cd_0 conda-forge s3fs 0.4.2 py_0 conda-forge joblib 0.17.0 py_0 conda-forge pickleshare 0.7.5 py36h9f0ad1d_1001 conda-forge python-xxhash 2.0.0 py36h779f372_0 conda-forge backports 1.0 py_2 conda-forge kiwisolver 1.2.0 py36h246c5b5_0 conda-forge sqlalchemy 1.3.19 py36h779f372_0 conda-forge pluggy 0.13.1 py36h9f0ad1d_2 conda-forge chardet 3.0.4 py36h9f0ad1d_1007 conda-forge s3transfer 0.3.3 py36h9f0ad1d_1 conda-forge cachey 0.2.1 pyh9f0ad1d_0 conda-forge libcurl 7.71.1 h4b64cdc_8 conda-forge python-graphviz 0.14.1 pyh9f0ad1d_0 conda-forge freetype 2.10.2 hd328e21_0 conda-forge asciitree 0.3.3 py_2 conda-forge zstd 1.4.4 hd8a0e53_2 conda-forge aws-sam-translator 1.27.0 pyh9f0ad1d_0 conda-forge snappy 1.1.8 ha925a31_3 conda-forge libssh2 1.9.0 hb06d900_5 conda-forge pywavelets 1.1.1 py36h7725771_1 conda-forge pywin32 227 py36hfa6e2cd_0 conda-forge zarr 2.4.0 py_0 conda-forge backports.weakref 1.0.post1 py36h9f0ad1d_1001 conda-forge idna 2.8 py36_1000 conda-forge moto 1.3.14 py_0 conda-forge jsonpickle 1.4.1 pyh9f0ad1d_0 conda-forge more-itertools 8.5.0 py_0 conda-forge jmespath 0.10.0 pyh9f0ad1d_0 conda-forge aws-xray-sdk 0.95 py_0 conda-forge codecov 2.1.9 pyh9f0ad1d_0 conda-forge junit-xml 1.9 pyh9f0ad1d_0 conda-forge cytoolz 0.11.0 py36h779f372_0 conda-forge boost-cpp 1.70.0 h2ba7cf6_3 conda-forge

In only dask-env-py36
msgpack 1.0.0 pypi_0 pypi
sqlite 3.33.0 h2a8f88b_0 defaults
typing-extensions 3.7.4.3 pypi_0 pypi

In both, same version:
dask 2.25.0+65.g458a2d4d.dirtydev_0

Packages in both, with differnt versions:
certifi :
test-environment2020.6.20 py36h9f0ad1d_0 conda-forge
dask-env-py36 2020.6.20 py36_0 defaults
contextvars :
test-environment2.4 py_0 conda-forge
dask-env-py36 2.4 pypi_0 pypi
jinja2 :
test-environment2.11.2 pyh9f0ad1d_0 conda-forge
dask-env-py36 2.11.2 pypi_0 pypi
packaging :
test-environment20.4 pyh9f0ad1d_0 conda-forge
dask-env-py36 20.4 pypi_0 pypi
python :
test-environment3.6.11 h6f26aa1_2_cpython conda-forge
dask-env-py36 3.6.12 h5500b2f_2 defaults
python-dateutil :
test-environment2.8.1 py_0 conda-forge
dask-env-py36 2.8.1 pypi_0 pypi
wincertstore :
test-environment0.2 py36_1003 conda-forge
dask-env-py36 0.2 py36h7fe50ca_0 defaults
sortedcontainers :
test-environment2.2.2 pyh9f0ad1d_0 conda-forge
dask-env-py36 2.2.2 pypi_0 pypi
wheel :
test-environment0.35.1 pyh9f0ad1d_0 conda-forge
dask-env-py36 0.35.1 py_0 defaults
numpy :
test-environment1.15.4 py36h8078771_1002 conda-forge
dask-env-py36 1.19.2 pypi_0 pypi
vs2015_runtime :
test-environment14.16.27012h30e32a0_2 conda-forge
dask-env-py36 14.16.27012hf0eaf9b_3 defaults
cloudpickle :
test-environment1.6.0 py_0 conda-forge
dask-env-py36 1.6.0 pypi_0 pypi
toolz :
test-environment0.11.1 py_0 conda-forge
dask-env-py36 0.11.1 pypi_0 pypi
bokeh :
test-environment2.2.1 py36h9f0ad1d_0 conda-forge
dask-env-py36 2.2.1 pypi_0 pypi
heapdict :
test-environment1.0.1 py_0 conda-forge
dask-env-py36 1.0.1 pypi_0 pypi
pillow :
test-environment7.2.0 py36he4e95fe_1 conda-forge
dask-env-py36 7.2.0 pypi_0 pypi
psutil :
test-environment5.7.2 py36h779f372_0 conda-forge
dask-env-py36 5.7.2 pypi_0 pypi
pip :
test-environment20.2.3 py_0 conda-forge
dask-env-py36 20.2.3 py36_0 defaults
pytz :
test-environment2020.1 pyh9f0ad1d_0 conda-forge
dask-env-py36 2020.1 pypi_0 pypi
vc :
test-environment14.1 h869be7e_1 conda-forge
dask-env-py36 14.1 h0510ff6_4 defaults
distributed :
test-environment2.29.0 py36h9f0ad1d_0 conda-forge
dask-env-py36 2.30.0 pypi_0 pypi
pandas :
test-environment0.23.4 py36h830ac7b_1000 conda-forge
dask-env-py36 1.1.3 pypi_0 pypi
zlib :
test-environment1.2.11 h62dcd97_1009 conda-forge
dask-env-py36 1.2.11 h62dcd97_4 defaults
zict :
test-environment2.0.0 py_0 conda-forge
dask-env-py36 2.0.0 pypi_0 pypi
setuptools :
test-environment49.6.0 py36h9f0ad1d_1 conda-forge
dask-env-py36 50.3.0 py36h9490d1a_1 defaults
six :
test-environment1.15.0 pyh9f0ad1d_0 conda-forge
dask-env-py36 1.15.0 pypi_0 pypi
tblib :
test-environment1.6.0 py_0 conda-forge
dask-env-py36 1.7.0 pypi_0 pypi
pyyaml :
test-environment5.3.1 py36h779f372_0 conda-forge
dask-env-py36 5.3.1 pypi_0 pypi
click :
test-environment7.1.2 pyh9f0ad1d_0 conda-forge
dask-env-py36 7.1.2 pypi_0 pypi
fsspec :
test-environment0.8.3 py_0 conda-forge
dask-env-py36 0.8.3 pypi_0 pypi
locket :
test-environment0.2.0 py_2 conda-forge
dask-env-py36 0.2.0 pypi_0 pypi
tornado :
test-environment6.0.4 py36hfa6e2cd_0 conda-forge
dask-env-py36 6.0.4 pypi_0 pypi
pyparsing :
test-environment2.4.7 pyh9f0ad1d_0 conda-forge
dask-env-py36 2.4.7 pypi_0 pypi
markupsafe :
test-environment1.1.1 py36h779f372_1 conda-forge
dask-env-py36 1.1.1 pypi_0 pypi
partd :
test-environment1.1.0 py_0 conda-forge
dask-env-py36 1.1.0 pypi_0 pypi
immutables :
test-environment0.14 py36h779f372_0 conda-forge
dask-env-py36 0.14 pypi_0 pypi

</p>
</details>

daxiongshu · 2020-10-07T12:49:32Z

It seems the main difference is that the test environment has a lot of packages from conda-forge. 🤔

pentschev · 2020-10-07T14:50:04Z

It seems that the Python 3.6 environment is pinning to NumPy 1.15, and we need __array_function__ for *_like functions which is only available in NumPy >= 1.17 (or NumPy 1.16 with NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1). For that reason, we have

dask/dask/array/utils.py

Line 406 in 34fd626

IS_NEP18_ACTIVE = _is_nep18_active()

to only enable tests in such cases, for example those in https://github.com/dask/dask/blob/master/dask/array/tests/test_cupy.py . However, we still need to ensure that code is resilient both with and without __array_function__, for that we have "safe" functions in

dask/dask/array/utils.py

Lines 289 to 335 in 34fd626

    
           def empty_like_safe(a, shape, **kwargs): 
        
               """ 
        
               Return np.empty_like(a, shape=shape, **kwargs) if the shape argument 
        
               is supported (requires NumPy >= 1.17), otherwise falls back to 
        
               using the old behavior, returning np.empty(shape, **kwargs). 
        
               """ 
        
               try: 
        
                   return np.empty_like(a, shape=shape, **kwargs) 
        
               except TypeError: 
        
                   return np.empty(shape, **kwargs) 
        
           def full_like_safe(a, fill_value, shape, **kwargs): 
        
               """ 
        
               Return np.full_like(a, fill_value, shape=shape, **kwargs) if the 
        
               shape argument is supported (requires NumPy >= 1.17), otherwise 
        
               falls back to using the old behavior, returning 
        
               np.full(shape, fill_value, **kwargs). 
        
               """ 
        
               try: 
        
                   return np.full_like(a, fill_value, shape=shape, **kwargs) 
        
               except TypeError: 
        
                   return np.full(shape, fill_value, **kwargs) 
        
           def ones_like_safe(a, shape, **kwargs): 
        
               """ 
        
               Return np.ones_like(a, shape=shape, **kwargs) if the shape argument 
        
               is supported (requires NumPy >= 1.17), otherwise falls back to 
        
               using the old behavior, returning np.ones(shape, **kwargs). 
        
               """ 
        
               try: 
        
                   return np.ones_like(a, shape=shape, **kwargs) 
        
               except TypeError: 
        
                   return np.ones(shape, **kwargs) 
        
           def zeros_like_safe(a, shape, **kwargs): 
        
               """ 
        
               Return np.zeros_like(a, shape=shape, **kwargs) if the shape argument 
        
               is supported (requires NumPy >= 1.17), otherwise falls back to 
        
               using the old behavior, returning np.zeros(shape, **kwargs). 
        
               """ 
        
               try: 
        
                   return np.zeros_like(a, shape=shape, **kwargs) 
        
               except TypeError: 
        
                   return np.zeros(shape, **kwargs)

@GenevieveBuckley could you try replacing https://github.com/dask/dask/pull/6680/files#diff-c083fd0d8ac3c4d1f49e6071bf977a8eR176-R178 with their safe counterparts?

GenevieveBuckley · 2020-10-07T23:00:42Z

Thank you @pentschev I think that will solve our test failures! The CI is running now, but a small subset of previously failing tests now pass locally for me, so I'm feeling good about this.

pentschev

Thanks @GenevieveBuckley for working on this! All looks good to me, I only suggested adding tests for the other functions as well, but I confirmed locally everything works.

dask/array/tests/test_cupy.py

Co-authored-by: Peter Andreas Entschev <peter@entschev.com>

mrocklin · 2020-10-08T16:05:04Z

This is in. Thank you @GenevieveBuckley for the work here and @pentschev , @jsignell , and @daxiongshu for review

daxiongshu · 2020-10-08T19:45:02Z

Thank you all! Especially @GenevieveBuckley for the great work!

mathause · 2020-10-11T16:22:18Z

Just to note that this causes a test failure in xarray (but we probably need to fix this on our end). The following used to work:

import xarray as xr
import dask

da = xr.DataArray(dask.array.array(""))
xr.zeros_like(da, dtype=bool)

basically returning array(False). Now it fails with

        if dtype and meta.dtype != dtype:
>           meta = meta.astype(dtype)
E           ValueError: invalid literal for int() with base 10: ''

xr.zeros_like(da, dtype=bool) is used in isnull to turn bool, int, char arrays to arrays of False like so

https://github.com/pydata/xarray/blob/080caf4246fe2f4d6aa0c5dcb65a99b376fa669b/xarray/core/duck_array_ops.py#L100-L102

mathause · 2020-10-12T19:18:53Z

Can you advise how to make the following work with this change?

import dask.array
da = dask.array.array("")
dask.array.ones_like(da, dtype=bool)

Assign None to meta (da._meta = None)? Is there a more elegant way?

pentschev · 2020-10-12T22:11:21Z

I honestly don't know if there's a better way to handle literals, but I gave it a try at #6731 . Could you check if that fixes the problems you're having in xarray @mathause ?

…sk#6680) This PR provides support for different array types in the *_like array creation functions. Eg: you can pass a cupy array (or cupy-backed dask array) to da.ones_like and get back a cupy-backed dask array of the appropriate shape filled with ones.

GenevieveBuckley added 4 commits September 28, 2020 19:56

Dask *_like creation functions should respect arraytype (eg: numpy/cu…

4a59956

…py/etc.)

Dask tasks have slight name change as a result of respecting arraytyp…

328e212

…e in *_like creation functions

Match _meta in *_like function output arrays

1714679

Handle different output shape in *_like functions

6a28424

daxiongshu mentioned this pull request Sep 28, 2020

Fix algorithms "newton", "gradient_descen" and "proximal_grad" with cupy inputs dask/dask-glm#87

Merged

mrocklin reviewed Sep 29, 2020

View reviewed changes

dask/array/wrap.py Outdated Show resolved Hide resolved

mrocklin reviewed Sep 29, 2020

View reviewed changes

dask/array/creation.py Outdated Show resolved Hide resolved

mrocklin reviewed Sep 29, 2020

View reviewed changes

dask/array/tests/test_wrap.py Outdated Show resolved Hide resolved

GenevieveBuckley added 6 commits September 29, 2020 17:02

Simpler broadcast_trick function

437dfbc

Strip '_like' from dask task names

12e2a71

Handle _like functions _meta in a more elegant way

220cbaf

Add cupy test for da.ones_like()

d743c94

Fix formatting with black

fe9658d

Update reduction test for adjustment in dask task name

f82f85c

GenevieveBuckley marked this pull request as ready for review September 29, 2020 08:18

GenevieveBuckley added 3 commits September 29, 2020 19:32

Special handling for scalars, shape=()

9d32054

Fix failing sparse array test calculating moments

f688aaf

Fix black formatting

0c52865

Patch for numpy.full_like doctest failure

458a2d4

jsignell mentioned this pull request Oct 6, 2020

Upcoming release schedule around high level graphs dask/community#99

Closed

Use *_like_safe functions to avoid python 3.6 test failures

29b6432

Strip '_like_safe' suffix from functions, eg: ones_like_safe, etc.

858c87d

pentschev approved these changes Oct 8, 2020

View reviewed changes

dask/array/tests/test_cupy.py Show resolved Hide resolved

Update dask/array/tests/test_cupy.py

b403a4a

Co-authored-by: Peter Andreas Entschev <peter@entschev.com>

mrocklin merged commit 1213248 into dask:master Oct 8, 2020

mathause mentioned this pull request Oct 11, 2020

dask dev upstream test failiures pydata/xarray#4502

Closed

pentschev mentioned this pull request Oct 12, 2020

Handle literal in meta_from_array #6731

Merged

GenevieveBuckley mentioned this pull request Oct 15, 2020

Added dask implementation of ndimage.affine_transform in new ndinterp directory. dask/dask-image#159

Merged

Uh oh!

Allow *_like array creation functions to respect input array type (eg: numpy/cupy/etc.) #6680

Allow *_like array creation functions to respect input array type (eg: numpy/cupy/etc.) #6680

Uh oh!

Conversation

GenevieveBuckley commented Sep 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrocklin commented Sep 28, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrocklin commented Sep 30, 2020

Uh oh!

GenevieveBuckley commented Oct 1, 2020

Uh oh!

daxiongshu commented Oct 6, 2020

Uh oh!

jsignell commented Oct 6, 2020

Uh oh!

jsignell commented Oct 6, 2020

Uh oh!

jsignell commented Oct 6, 2020

Uh oh!

jsignell commented Oct 6, 2020

Uh oh!

GenevieveBuckley commented Oct 7, 2020

Problem 1 - python 3.6 test failures

Problem 2 - doctest failure

Uh oh!

GenevieveBuckley commented Oct 7, 2020

Uh oh!

GenevieveBuckley commented Oct 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Setup environments

Test

Compare environments

Uh oh!

daxiongshu commented Oct 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pentschev commented Oct 7, 2020

Uh oh!

GenevieveBuckley commented Oct 7, 2020

Uh oh!

pentschev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrocklin commented Oct 8, 2020

Uh oh!

daxiongshu commented Oct 8, 2020

Uh oh!

mathause commented Oct 11, 2020

Uh oh!

mathause commented Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pentschev commented Oct 12, 2020

Uh oh!

Uh oh!

GenevieveBuckley commented Sep 28, 2020 •

edited

Loading

GenevieveBuckley commented Oct 7, 2020 •

edited

Loading

daxiongshu commented Oct 7, 2020 •

edited

Loading

mathause commented Oct 12, 2020 •

edited

Loading