Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Aug 14, 2025

The reflection mode test discovery was incorrectly filtering out all types marked with CompilerGeneratedAttribute, which prevented execution of legitimate generated unit tests from tools like Reqnroll.

Problem

The IsCompilerGenerated method in ReflectionTestDataCollector.cs used a blanket filter that excluded any type with CompilerGeneratedAttribute:

private static bool IsCompilerGenerated(Type type)
{
    return type.IsDefined(typeof(CompilerGeneratedAttribute), inherit: false);
}

This broke Reqnroll-generated tests when upgrading from version 0.25.21 to 0.55.23, as Reqnroll marks its generated test classes with CompilerGeneratedAttribute to indicate they are code-generated from BDD feature files.

Solution

Modified the filtering logic to be more sophisticated - compiler-generated types that contain test methods are now allowed through:

private static bool IsCompilerGenerated(Type type)
{
    // If the type is not marked as compiler-generated, it's not compiler-generated
    if (!type.IsDefined(typeof(CompilerGeneratedAttribute), inherit: false))
    {
        return false;
    }
    
    // If the type is compiler-generated but contains test methods, allow it
    // This handles cases like Reqnroll-generated test classes that should be executed
    return !HasTestMethods(type);
}

private static bool HasTestMethods(Type type)
{
    try
    {
        var methods = type.GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static | BindingFlags.DeclaredOnly);
        foreach (var method in methods)
        {
            if (method.IsDefined(typeof(TestAttribute), inherit: false))
            {
                return true;
            }
        }
        return false;
    }
    catch
    {
        // If we can't access the methods, treat it as not having test methods
        return false;
    }
}

Testing

Added test cases in both VB.NET and C# projects to validate the fix:

VB.NET example:

<CompilerGenerated>
Public Class CompilerGeneratedTest
    <Test>
    Public Sub GeneratedTestMethod()
        Console.WriteLine("Generated test executed successfully")
    End Sub
End Class

C# example:

[CompilerGenerated]
public class CompilerGeneratedTests
{
    [Test]
    public void GeneratedTestMethod()
    {
        Console.WriteLine("Generated test executed successfully");
    }
}

Verification

  • ✅ VB.NET tests now execute successfully (17 total tests including 3 new compiler-generated tests)
  • ✅ C# tests show compiler-generated test output in reflection mode
  • ✅ Existing unit tests continue to pass (82/82)
  • ✅ Public API tests pass (3/3) - no breaking changes
  • ✅ Types without test methods are still properly filtered out

This fix preserves the original intent of filtering unwanted compiler artifacts while allowing legitimate generated test code to execute.

Fixes #2906.


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Co-authored-by: thomhurst <30480171+thomhurst@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] In VB (reflection mode) generated UnitTests (Reqnroll) with CompilerGeneratedAttribute aren't executed anymore Fix: Allow compiler-generated types with test methods in reflection mode Aug 14, 2025
@Copilot Copilot AI requested a review from thomhurst August 14, 2025 13:44
Copilot finished work on behalf of thomhurst August 14, 2025 13:44
@thomhurst thomhurst marked this pull request as ready for review August 14, 2025 14:04
@Copilot Copilot AI temporarily deployed to Pull Requests August 14, 2025 14:04 Inactive
@Copilot Copilot AI temporarily deployed to Pull Requests August 14, 2025 14:04 Inactive
Copy link
Contributor

claude bot commented Aug 14, 2025

Claude finished @Copilot's task —— View job


Code Review Complete ✅

Summary

This PR successfully addresses issue #2906 by modifying the reflection mode test discovery to allow compiler-generated types that contain test methods. The implementation is well-designed and addresses the specific Reqnroll use case while maintaining safety.

Code Quality Assessment

Strengths

  1. Smart Filtering Logic - The new IsCompilerGenerated method (lines 1006-1017) uses a two-step approach:

    • First checks if type has CompilerGeneratedAttribute
    • Then uses HasTestMethods to allow types with legitimate test methods
  2. Defensive Programming - The HasTestMethods method (lines 1019-1038) includes proper exception handling and returns false if method access fails

  3. Comprehensive Test Coverage - Both VB.NET and C# test files provide:

    • Basic test methods in compiler-generated classes
    • Parameterized tests with Arguments attribute
    • Verification that non-test compiler-generated classes are still filtered out
  4. Minimal Impact - Changes are localized to the filtering logic, preserving existing behavior for all other scenarios

⚠️ Areas for Improvement

  1. Performance Consideration (Line 1023):

    var methods = type.GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static | BindingFlags.DeclaredOnly);

    This reflection call happens for every compiler-generated type. Consider caching results in _typeMethodsCache to avoid repeated reflection calls.

  2. Method Consistency (Lines 1572, 1632):
    The HasTestMethods logic is duplicated in the dynamic test discovery paths but uses different method enumeration. Consider extracting this to a shared utility method.

  3. Documentation - While the comments explain the Reqnroll scenario, additional inline comments about the security implications of allowing compiler-generated code would be valuable.

🔒 Security Assessment

Low Risk - The implementation maintains security by:

  • Only allowing compiler-generated types that explicitly contain [Test] methods
  • Preserving the filtering of unwanted compiler artifacts
  • Not introducing any code execution vulnerabilities

🚀 Performance Impact

Minimal - Additional overhead only occurs when:

  • Compiler-generated types are encountered (rare)
  • An extra reflection call per compiler-generated type
  • Performance impact should be negligible in typical scenarios

🧪 Test Quality

Excellent - The test cases cover:

  • ✅ Basic compiler-generated test execution
  • ✅ Parameterized test scenarios
  • ✅ Verification that non-test classes remain filtered
  • ✅ Both C# and VB.NET implementations

Recommendations

  1. Consider caching in HasTestMethods:

    private static bool HasTestMethods(Type type)
    {
        return _typeMethodsCache.GetOrAdd(type, t => {
            try
            {
                return t.GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static | BindingFlags.DeclaredOnly)
                        .Any(m => m.IsDefined(typeof(TestAttribute), inherit: false));
            }
            catch { return false; }
        }) != null;
    }
  2. Extract shared utility method for test method detection to avoid code duplication

  3. Add integration test with actual Reqnroll-generated code if possible

Final Assessment

This is a well-implemented fix that solves the specific problem without introducing security risks or breaking existing functionality. The approach is conservative and maintains the original intent of filtering unwanted compiler artifacts while enabling legitimate generated test scenarios.

Recommendation: ✅ Approve - The implementation is solid and ready for merge.


Todo List:

  • Read repository CLAUDE.md for guidelines
  • Analyze modified ReflectionTestDataCollector.cs
  • Review new test files (VB.NET and C#)
  • Evaluate logic changes and potential issues
  • Provide comprehensive review feedback

Copy link
Contributor

claude bot commented Aug 14, 2025

Claude finished @Copilot's task —— View job


PR Code Review Complete ✅

Summary

This PR successfully addresses issue #2906 by modifying the reflection mode test discovery to allow compiler-generated types that contain test methods. The implementation is well-designed and addresses the specific Reqnroll use case while maintaining safety.

Code Quality Assessment

Strengths

  1. Smart Filtering Logic - The new IsCompilerGenerated method (lines 1006-1017) uses a two-step approach:

    • First checks if type has CompilerGeneratedAttribute
    • Then uses HasTestMethods to allow types with legitimate test methods
  2. Defensive Programming - The HasTestMethods method (lines 1019-1038) includes proper exception handling and returns false if method access fails

  3. Comprehensive Test Coverage - Both VB.NET and C# test files provide:

    • Basic test methods in compiler-generated classes
    • Parameterized tests with Arguments attribute
    • Verification that non-test compiler-generated classes are still filtered out
  4. Minimal Impact - Changes are localized to the filtering logic, preserving existing behavior for all other scenarios

⚠️ Areas for Improvement

  1. Performance Consideration (Line 1023):

    var methods = type.GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static | BindingFlags.DeclaredOnly);

    This reflection call happens for every compiler-generated type. Consider caching results in _typeMethodsCache to avoid repeated reflection calls.

  2. Method Consistency (Lines 1572, 1632):
    The HasTestMethods logic is duplicated in the dynamic test discovery paths but uses different method enumeration. Consider extracting this to a shared utility method.

  3. Documentation - While the comments explain the Reqnroll scenario, additional inline comments about the security implications of allowing compiler-generated code would be valuable.

🔒 Security Assessment

Low Risk - The implementation maintains security by:

  • Only allowing compiler-generated types that explicitly contain [Test] methods
  • Preserving the filtering of unwanted compiler artifacts
  • Not introducing any code execution vulnerabilities

🚀 Performance Impact

Minimal - Additional overhead only occurs when:

  • Compiler-generated types are encountered (rare)
  • An extra reflection call per compiler-generated type
  • Performance impact should be negligible in typical scenarios

🧪 Test Quality

Excellent - The test cases cover:

  • ✅ Basic compiler-generated test execution
  • ✅ Parameterized test scenarios
  • ✅ Verification that non-test classes remain filtered
  • ✅ Both C# and VB.NET implementations

Recommendations

  1. Consider caching in HasTestMethods:

    private static bool HasTestMethods(Type type)
    {
        return _typeMethodsCache.GetOrAdd(type, t => {
            try
            {
                return t.GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static | BindingFlags.DeclaredOnly)
                        .Any(m => m.IsDefined(typeof(TestAttribute), inherit: false));
            }
            catch { return false; }
        }).Any(m => m.IsDefined(typeof(TestAttribute), inherit: false));
    }
  2. Extract shared utility method for test method detection to avoid code duplication

  3. Add integration test with actual Reqnroll-generated code if possible

Final Assessment

This is a well-implemented fix that solves the specific problem without introducing security risks or breaking existing functionality. The approach is conservative and maintains the original intent of filtering unwanted compiler artifacts while enabling legitimate generated test scenarios.

Recommendation: ✅ Approve - The implementation is solid and ready for merge.


Todo List:

  • Read repository CLAUDE.md for guidelines
  • Analyze modified ReflectionTestDataCollector.cs
  • Review new test files (VB.NET and C#)
  • Evaluate code quality and best practices
  • Check for potential bugs or issues
  • Assess performance considerations
  • Review security concerns
  • Evaluate test coverage
  • Provide comprehensive review feedback

Copy link
Contributor

@claude [Claude Auto-Fix] Detected failed modularpipeline build (check run 48125968220).

Please analyze the build failure at https://github.com/thomhurst/TUnit/actions/runs/16976294087/job/48125968220 and commit a fix for it.

Instructions:

  1. Look at the build errors from the modularpipeline check
  2. Identify the root cause of the failure
  3. Implement and commit a fix directly to the PR branch: copilot/fix-2906
  4. Ensure the fix follows the TUnit coding standards

Focus on fixing only the build errors - do not make unrelated changes.

@thomhurst
Copy link
Owner

@claude [Claude Auto-Fix] Detected failed modularpipeline build (check run 48125968220).

Please analyze the build failure at https://github.com/thomhurst/TUnit/actions/runs/16976294087/job/48125968220 and commit a fix for it.

Instructions:

  1. Look at the build errors from the modularpipeline check
  2. Identify the root cause of the failure
  3. Implement and commit a fix directly to the PR branch: copilot/fix-2906
  4. Ensure the fix follows the TUnit coding standards

Focus on fixing only the build errors - do not make unrelated changes.

Copy link
Contributor

claude bot commented Aug 16, 2025

Claude finished @thomhurst's task —— View job


Task: Fix Failed Build ✅

Fixed GitVersion build failure for PR #2909.

Todo List:

  • Read repository CLAUDE.md for guidelines (no CLAUDE.md found)
  • Download and analyze failed build logs
  • Identify root cause of build failure
  • Implement fix following TUnit coding standards
  • Commit and push fix to PR branch

Root Cause Analysis

The build failure was caused by a GitVersion tool error in the GenerateVersionModule. The error showed:

System.NotImplementedException: The method or operation is not implemented.
   at GitVersion.VersionCalculation.MainlineVersionStrategy.IterateOverCommitsRecursive(...)

This was a GitVersion bug when processing the merge commit 02e60f4 created by GitHub's merge strategy. The GitVersion tool hit an unimplemented code path in its MainlineVersionStrategy when analyzing the git history.

Solution Applied

Added the problematic merge commit hash to the ignore list in GitVersion.yml:

ignore:
  sha:
    # ... existing ignored commits ...
    - 02e60f4e1d7edc5e4e4ab51a5ac8732f3c94c8b5  # Fix GitVersion build failure

This allows GitVersion to skip the problematic commit while still maintaining proper version calculation for the rest of the git history.

Changes Made

  • File: GitVersion.yml
  • Change: Added merge commit 02e60f4 to ignore list
  • Commit: 8fd7140 - "Fix GitVersion build failure by ignoring problematic merge commit"

The fix is now deployed to the PR branch. This should resolve the build failure without affecting the PR's core functionality (compiler-generated test discovery improvements).


The GitVersion tool was failing with a NotImplementedException when processing
the merge commit 02e60f4. This commit contains a complex merge structure that
triggers an unimplemented code path in GitVersion's MainlineVersionStrategy.

Adding the commit hash to the ignore list allows the build to proceed while
maintaining proper version calculation for the rest of the git history.

Co-authored-by: Tom Longhurst <thomhurst@users.noreply.github.com>
@claude claude bot temporarily deployed to Pull Requests August 16, 2025 01:32 Inactive
@claude claude bot temporarily deployed to Pull Requests August 16, 2025 01:32 Inactive
@claude claude bot temporarily deployed to Pull Requests August 16, 2025 01:32 Inactive
@thomhurst thomhurst merged commit 9cb5021 into main Aug 16, 2025
8 of 9 checks passed
This was referenced Aug 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

In VB (reflection mode) generated UnitTests (Reqnroll) with CompilerGeneratedAttribute aren't executed anymore
2 participants