cc @mathause

Thanks! This does seem to fix the issue (I tested locally but did not run the xarray test suite over it). Testing for the error message may be brittle, though (but I also don't know how to do this better).

Testing for the error message may be brittle, though (but I also don't know how to do this better).

I generally agree with that, but it seems that such issues have no better ways to be handled. Due to the lack of a more concise way to do it, we already have similar handling in

With the above said, I'm happy to change the proposed solution for something more resilient if anyone can suggest an alternative.

Agreed with everyone here that this seems brittle but I don't immediately see a better way to handle it.

I just tried the xarray tests with this branch and this fixes 4 of our 5 failures. I don't understand the remaining failure, yet.

What if we just special case when x is a str?

Admittedly I don't have a good handle on the use case here. More context would be very welcome.

I just tried the xarray tests with this branch and this fixes 4 of our 5 failures. I don't understand the remaining failure, yet.

Please keep us posted.

What if we just special case when x is a str?

Who is x in your suggestion?

I found the problem - with this workaround the dtype of da.zeros_like(da.array("bar"), dtype=bool) becomes '<U3':

import numpy as np
import dask.array as da

bar = da.zeros_like(da.array("bar"), dtype=bool)

bar.dtype
# returns dtype('<U3')

bar.compute().dtype
# returns bool

For empty string and float it is correct

da.zeros_like(da.array(""), dtype=bool).dtype
# returns bool
da.zeros_like(da.array([1.2, 3.4]), dtype=bool).dtype
# returns bool

which is why 4 of the 5 tests pass. As for the use case - we try to compare

bar & np.array(False)

which then returns

TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'bitwise_and'>, '__call__', dask.array<zeros, shape=(), dtype=<U3, chunksize=(), chunktype=numpy.ndarray>, array(False)): 'Array', 'ndarray'

Do I understand correctly the meta.astype(dtype) is there because meta is not necessarily a numpy array? So we want to ensure it does not change its type? (Else we could just do meta=np.array([], dtype=dtype)?).

The problem stems from if meta == np.array("") in your code because:

da.array("jjj")._meta == np.array("")
# -> array(False)
da.array(["jjj"])._meta == np.array("")
# -> array([], dtype=bool)
da.array("")._meta == np.array("")
# -> array(True)

Given you already test for str(e) in ["invalid literal", "could not convert string to float"] - is the additional test necessary? If yes you can probably make it more general by doing meta.dtype.kind == "U". Also you could combine the two tests like so

if any(...) and meta.dtype.kind == "U" 

to ensure the error is raised.

What if we just special case when x is a str?

Who is x in your suggestion?

x is the name of the variable meta_from_array takes

Do I understand correctly the meta.astype(dtype) is there because meta is not necessarily a numpy array? So we want to ensure it does not change its type? (Else we could just do meta=np.array([], dtype=dtype)?).

meta was introduced because we need to ensure we know what type the output array will have before computing it, and that may not be a NumPy array. The .astype(dtype) is part of the mechanism to ensure the dtype is also correct, each operation may have different requirements regarding to dtype and that's there to ensure meta will have the correct one. In summary, meta tries to infer the array's output type, its dtype and ndim.

Given you already test for str(e) in ["invalid literal", "could not convert string to float"] - is the additional test necessary?

For the case at hand you're correct the test isn't strictly necessary. However, I've been taking the approach of not generalizing fixes, meaning if this exception exists in a completely unrelated case (e.g., a non-literal array that may happen to raise the same error message) we won't just assume we're handling it correctly when we probably aren't. This helps prevent silent errors that are much harder to be found in the future.

If yes you can probably make it more general by doing meta.dtype.kind == "U".

Yes, I think this is a better approach than comparing against np.array(""), thanks for the suggestion. I'll just pushed a fix, could you try it out and let us know if that resolves all issues?

What if we just special case when x is a str?

Who is x in your suggestion?

x is the name of the variable meta_from_array takes

In the case above x isn't necessarily a str, but anything that happens to be converted into a NumPy literal. So we actually need to check the final meta for it, which I believe we now do successfully with 6026297 .

I addressed your suggestions @mathause , could you check if everything works now?

Now all xarray tests pass (locally) so this should be good to go from my side.

Thanks @mathause for confirming. From my side it's complete too, appreciate it if @TomAugspurger or @jakirkham have chance to review/merge at some point.

Thanks all!