Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jun 5, 2025

This PR implements validation to show errors when VSTest-specific command-line switches are used with Microsoft Testing Platform (MTP), helping users understand why their VSTest properties are being ignored.

Problem

Users were confused when using VSTest-specific properties with dotnet test while using MTP, because these properties get silently ignored. For example:

  • dotnet test --filter something - filter is ignored
  • dotnet test --logger trx - logger is ignored

This led to users wondering why their test filtering or logging wasn't working as expected.

Solution

Added comprehensive validation that produces an error when VSTest-specific properties are set with MTP:

Key Changes

  1. New opt-out property: TestingPlatformIgnoreVSTestProperties (default: false) allows users to suppress the error when needed.

  2. Validation target: Added _ValidateVSTestProperties target that checks for all VSTest properties mentioned in the issue:

    • VSTestSetting, VSTestListTests, VSTestTestCaseFilter, VSTestTestAdapterPath
    • VSTestLogger, VSTestDiag, VSTestResultsDirectory, VSTestCollect
    • VSTestBlame, VSTestBlameCrash, VSTestBlameHang
  3. Clear error message: Provides helpful guidance including:

    • Lists which VSTest properties are set
    • Explains they will be ignored with MTP
    • Shows how to suppress the error
    • Links to migration documentation
  4. Integration with existing workflow: The validation runs as part of the InvokeTestingPlatform target, ensuring it catches issues early in the build process.

Example Error Message

VSTest-specific properties are set but will be ignored when using Microsoft Testing Platform. 
The following properties are set: VSTestTestCaseFilter; VSTestLogger; VSTestBlame. 
To suppress this error, set TestingPlatformIgnoreVSTestProperties to true. 
For more information about migrating from VSTest to Microsoft Testing Platform, see: https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-intro

Usage Examples

Error case:

dotnet test --filter Category=Unit --logger trx
# Now shows clear error explaining the issue

Opt-out:

dotnet test --filter Category=Unit -p:TestingPlatformIgnoreVSTestProperties=true
# Suppresses the error for migration scenarios

Project file opt-out:

<PropertyGroup>
  <TestingPlatformIgnoreVSTestProperties>true</TestingPlatformIgnoreVSTestProperties>
</PropertyGroup>

Testing

Added comprehensive integration tests covering:

  • Error when VSTest properties are set
  • No error when opt-out property is enabled
  • No error when no VSTest properties are set

Breaking Change Notice

This is a breaking change as it will now show errors for previously "working" (but ignored) VSTest property usage. However, this is intentional to help users identify and fix configuration issues. Users can opt-out during migration using the TestingPlatformIgnoreVSTestProperties property.

Fixes #5697.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 8xbvsblobprodcus382.vsblob.vsassets.io
    • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
  • bcnvsblobprodcus378.vsblob.vsassets.io
    • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
  • http://168.63.129.16:80/machine/
    • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)
  • i1qvsblobprodcus353.vsblob.vsassets.io
    • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
  • l49vsblobprodcus358.vsblob.vsassets.io
    • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
  • mfjvsblobprodcus373.vsblob.vsassets.io
    • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
  • s4uvsblobprodcus326.vsblob.vsassets.io
    • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
  • s8mvsblobprodcus38.vsblob.vsassets.io
    • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)

If you need me to access, download, or install something from one of these locations, you can either:


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits June 5, 2025 09:03
Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] [MTP] Error when VSTest arguments are passed via old dotnet test when they will be ignored [MTP] Error when VSTest arguments are passed via dotnet test when they will be ignored Jun 5, 2025
@Copilot Copilot AI requested a review from Youssef1313 June 5, 2025 09:06
Copilot finished work on behalf of Youssef1313 June 5, 2025 09:06
… feedback

Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
Copilot finished work on behalf of Youssef1313 June 5, 2025 09:10
Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jun 5, 2025

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://168.63.129.16:80/machine/
    • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of Youssef1313 June 5, 2025 09:19
Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
Copilot finished work on behalf of Youssef1313 June 5, 2025 09:28
Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jun 5, 2025

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://168.63.129.16:80/machine/
    • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of Youssef1313 June 5, 2025 09:33
@Youssef1313 Youssef1313 marked this pull request as ready for review June 5, 2025 10:17
@Youssef1313 Youssef1313 requested a review from nohwnd June 5, 2025 10:18
@codecov-commenter
Copy link

codecov-commenter commented Jun 5, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.37%. Comparing base (37692a0) to head (9cc0bd3).
⚠️ Report is 467 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5698      +/-   ##
==========================================
+ Coverage   73.89%   76.37%   +2.47%     
==========================================
  Files         602      602              
  Lines       36850    36850              
==========================================
+ Hits        27231    28144     +913     
+ Misses       9619     8706     -913     
Flag Coverage Δ
Debug 76.37% <ø> (+2.47%) ⬆️
integration 76.37% <ø> (+2.47%) ⬆️
production 76.37% <ø> (+2.47%) ⬆️
unit 76.36% <ø> (+2.47%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 59 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
Copilot finished work on behalf of Youssef1313 June 6, 2025 06:21
@Youssef1313
Copy link
Member

Drafting, this will break DotNetCoreCLI@2. We need to discuss.

@Youssef1313 Youssef1313 marked this pull request as draft June 10, 2025 06:46
Copy link
Contributor Author

Copilot AI commented Jun 10, 2025

Drafting, this will break DotNetCoreCLI@2. We need to discuss.

You're absolutely right - the validation will break existing DotNetCoreCLI@2 tasks that pass VSTest properties. Should I modify the implementation to be less aggressive (e.g., warning instead of error, or different default behavior) or would you prefer to discuss the approach first before I make further changes?

Copilot finished work on behalf of Youssef1313 June 10, 2025 06:48
@Youssef1313
Copy link
Member

Closing in favor of #5722

@Youssef1313 Youssef1313 deleted the copilot/fix-5697 branch July 19, 2025 03:13
@Youssef1313 Youssef1313 restored the copilot/fix-5697 branch August 28, 2025 06:32
@Youssef1313
Copy link
Member

Going to take it as a "warning" in 3.11/1.9

@Youssef1313 Youssef1313 reopened this Aug 28, 2025
</PropertyGroup>
<Warning Code="MTP0001" Condition=" '$(_VSTestPropertiesFound)' != '' "
Text="VSTest-specific properties are set but will be ignored when using Microsoft.Testing.Platform. The following properties are set: $(_VSTestPropertiesFound)."
HelpLink="https://learn.microsoft.com/dotnet/core/testing/unit-testing-with-dotnet-test" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: aka.ms and make sure which doc page we want to link to, and enhance doc if needed.

@Youssef1313 Youssef1313 marked this pull request as ready for review September 3, 2025 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[MTP] Error when VSTest arguments are passed via old dotnet test when they will be ignored
4 participants