This was already mentioned in https://github.com/microsoft/testfx/issues/1285 but we didn't have a sub-issue for it. Opening this for easier tracking through "Breaking !!" label