Skip to content

Optest is unable to xfail on parametrized tests #112251

@NicolasHug

Description

@NicolasHug

It is currently not possible to xfail on a parametrized test that is marked e.g. as

      "TestPSRoIAlign.test_faketensor__test_forward[x_dtype0-True-mps]": {
        "comment": "AssertionError: Dtypes torch.int64 and torch.int32 are not equal!",
        "status": "xfail"
      },

in the failures_dict. (I thought #111797 was a fix, but it's not enough)

From what I can tell, the problem is that here

def get_status(self, qualname: str, test_name: str) -> str:
if qualname not in self.data:
return "xsuccess"
dct = self.data[qualname]
if test_name not in dct:
return "xsuccess"

test_name isn't found in dct (i.e. in the failures_dict) because test_name looks like TestPSRoIAlign.test_faketensor__test_forward, i.e. it's missing the [x_dtype0-True-mps] part. That's because this name is generated all the way up in

for attr in test_methods:
for prefix, tester in test_utils.items():
construct_method(attr, prefix, tester)

where attr is test_forward, but of course it's missing any parametrization info because this is only looking at the "static" test class object.

I'm not sure how to fix this without a significant rework of the whole opcheck logic, but perhaps @zou3519 has an idea?

Unfortunately, for now we cannot really use optests on the torchvision test suite. Note that we can't just put TestPSRoIAlign.test_faketensor__test_forward without the [...] part in the failure_dict, because that test actually passes on cpu and cuda, which would lead to an "unexpected success".

cc @zou3519

Metadata

Metadata

Assignees

No one assigned

    Labels

    module: opcheckRelated to opcheck testing for custom operatorstriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions