-
Notifications
You must be signed in to change notification settings - Fork 25.2k
Description
It is currently not possible to xfail
on a parametrized test that is marked e.g. as
"TestPSRoIAlign.test_faketensor__test_forward[x_dtype0-True-mps]": {
"comment": "AssertionError: Dtypes torch.int64 and torch.int32 are not equal!",
"status": "xfail"
},
in the failures_dict
. (I thought #111797 was a fix, but it's not enough)
From what I can tell, the problem is that here
pytorch/torch/testing/_internal/optests/generate_tests.py
Lines 748 to 753 in cf5479b
def get_status(self, qualname: str, test_name: str) -> str: | |
if qualname not in self.data: | |
return "xsuccess" | |
dct = self.data[qualname] | |
if test_name not in dct: | |
return "xsuccess" |
test_name
isn't found in dct
(i.e. in the failures_dict) because test_name looks like TestPSRoIAlign.test_faketensor__test_forward
, i.e. it's missing the [x_dtype0-True-mps]
part. That's because this name is generated all the way up in
pytorch/torch/testing/_internal/optests/generate_tests.py
Lines 252 to 254 in cf5479b
for attr in test_methods: | |
for prefix, tester in test_utils.items(): | |
construct_method(attr, prefix, tester) |
where attr
is test_forward
, but of course it's missing any parametrization info because this is only looking at the "static" test class object.
I'm not sure how to fix this without a significant rework of the whole opcheck logic, but perhaps @zou3519 has an idea?
Unfortunately, for now we cannot really use optests on the torchvision test suite. Note that we can't just put TestPSRoIAlign.test_faketensor__test_forward
without the [...]
part in the failure_dict
, because that test actually passes on cpu and cuda, which would lead to an "unexpected success".
cc @zou3519