A Julia package for selective test execution using pattern matching. TestRunner allows you to run specific tests from a test file based on testset names, test expressions, or line numbers, while ensuring all necessary dependencies are executed.

• Pattern-based test selection: Run only the tests that match your specified patterns
• Multiple pattern types: String matching, regex patterns, expression, and line number patterns.
• Fast execution: Test code is interpreted only at top-level; function calls within tests are compiled normally while avoiding execution of unrelated test code
• JSON output: Machine-readable test results with diagnostics for integration with editors and CI systems

• Julia 1.12 or higher

pkg> add https://github.com/aviatesk/TestRunner.jl

TestRunner can also be installed as a standalone Julia application:

pkg> app add https://github.com/aviatesk/TestRunner.jl

This will install the testrunner executable. Note that you need to manually make ~/.julia/bin available on the PATH environment for the executable to be accessible. See https://pkgdocs.julialang.org/dev/apps/ for the details.

julia> using TestRunner

julia> runtest("demo.jl", ["basic tests"]) # Run tests matching a specific testset name

julia> runtest("demo.jl", ["basic tests", "struct tests"]) # Run multiple testsets

julia> runtest("demo.jl", [:(@test startswith(inner_func2(), "inner"))]) # Run standalone test case

Or quivalently via the command line app:

$ testrunner demo.jl "basic tests"

$ testrunner demo.jl "basic tests" "struct tests"

$ testrunner demo.jl '(:(@test startswith(inner_func2(), "inner")))'

runtest(filename::AbstractString, patterns, lines=(); topmodule::Module=Main)

Run tests from a file that match the given patterns and/or are on the specified lines.

Arguments:

• filename::AbstractString: Path to the test file
• patterns: Patterns to match. Can be strings, regexes, expressions, integers (line numbers), or ranges (line ranges)
• filter_lines=nothing: Optional line numbers to filter pattern matches. When provided, only pattern matches that overlap with these lines will be executed. This is particularly useful for IDE integration where clicking on a specific test should run only that test, even when multiple tests may match the same pattern
• topmodule::Module=Main: Module context for execution (default: Main)

Returns:

• Test results from the selectively executed tests

runtests(entryfilename::AbstractString, patterns; filter_lines=nothing, topmodule::Module=Main)

The package also provides a runtests function for advanced use cases like selectively running package test cases from test/runtests.jl, where you need to specify different patterns for different files in a test suite that includes multiple files via include statements. See its docstring for detailed usage.

Since runtest and runtests execute Test.jl code using JuliaInterpreter, they cannot directly utilize the test failure handling provided by Test.jl.

Therefore, this package implements a custom @testset type called TestRunnerTestSet. When runtest[s] is executed within @testset TestRunnerTestSet, you get equivalent test failure handling to Test.jl. For programmatic usage, it can be used specifically in the following way:

using Test, TestRunner
@testset TestRunnerTestSet runtest("mytest.jl", ["my tests"])

It is recommended to always use runtest[s] together with @testset TestRunnerTestSet as shown above.

TestRunnerTestSet is automatically used in the testrunner app.

See the Test Failure/Error Handling section for detailed comparisons showing how different types of failures are reported.

Match testsets by exact name:

# Match a testset by name
runtest("demo.jl", ["struct tests"])  # matches any testset whose name is "struct tests"

Match testsets using regular expressions:

runtest("demo.jl", [r"foo"])  # matches any testset containing "foo"

Match arbitrary Julia expressions using MacroTools patterns:

runtest("demo.jl", [:(@test startswith(s_, prefix_))])  # matches e.g. `@test startswith(s, "Julia")`
runtest("demo.jl", [:(@test a_ > b_)])                  # matches e.g. `@test x > 0`

Directly specify line numbers or ranges to execute:

# Run code on specific lines
runtest("demo.jl", [10, 20, 30])

# Run code in a line range
runtest("demo.jl", [10:15])

# Combine with other patterns
runtest("demo.jl", ["basic tests", 42, 50:55])

TestRunner can be installed as a CLI executable (see the installation section):

# Run specific testsets by name
testrunner mypkg/runtests.jl.jl "basic tests" "advanced tests"

# Run tests on specific lines
testrunner mypkg/runtests.jl.jl L10
testrunner mypkg/runtests.jl.jl L10:20

# Run tests matching expression patterns
testrunner mypkg/runtests.jl.jl ':(@test foo(x_) == y_)'

# Run tests matching regex patterns
testrunner mypkg/runtests.jl.jl r"^test.*basic"

# Run all tests in a file
testrunner mypkg/runtests.jl.jl

# Combine patterns with filter lines
testrunner mypkg/runtests.jl.jl "my tests" --filter-lines=10,15,20:25

# Use verbose output
testrunner -v mypkg/runtests.jl.jl L55:57

# Use a specific project environment
testrunner --project=/path/to/project mypkg/runtests.jl.jl "my tests"

# Show help
testrunner --help

# Output results in JSON format
testrunner --json mypkg/runtests.jl "my tests"

Pattern formats:

• L10 - Run tests on line 10
• L10:20 - Run tests on lines 10-20
• :(expr) - Match expression pattern
• r"^test.*" - Match testset names with regex
• "my tests" - Match testset by exact name (default)

Options:

• --project[=<dir>] - Set project/environment (same format and meaning as Julia's --project flag)
• --filter-lines=1,5,10:20 or -f=1,5,10:20 - Filter to specific lines
• --verbose or -v - Show verbose output
• --json - Output results in JSON format for machine-readable test results

Given this demo.jl file:

demo.jl

using Test

struct MyStruct
    value::Int
end

function process(s::MyStruct)
    return s.value * 2
end

@testset "basic tests" begin
    @test 1 + 1 == 2
    @test 2 * 2 == 4
    @test_broken 1 + 1 == 3  # This is expected to fail
end

@testset "struct tests" begin
    s = MyStruct(5)
    @test process(s) == 10
    @test s.value == 5
end

# Standalone test
@test process(MyStruct(3)) == 6

@testset "nested tests" begin
    outer_func() = "outer"
    @test outer_func() == "outer"

    @testset "inner tests 1" begin
        inner_func1() = "inner1"
        @test inner_func1() == "inner1"
        @test length(inner_func1()) == 6
    end

    @testset "inner tests 2" begin
        inner_func2() = "inner2"
        @test inner_func2() == "inner2"
        @test startswith(inner_func2(), "inner")
    end
end

@testset "calculator tests" begin
    add(a, b) = a + b
    mul(a, b) = a * b

    @test add(2, 3) == 5    # line 55
    @test mul(3, 4) == 12   # line 56
    @test add(10, 20) == 30 # line 57
end

# More standalone tests
@test 100 - 50 == 50  # line 61
@test sqrt(16) == 4    # line 62

@testset "Test failure" begin
    @test sin(0) == π
end

@testset "Exception inside of `@test`" begin
    @test sin(Inf) == π
    @test sin(0) == 0
    @test cos(Inf) == π
end

@testset "Exception outside of `@test`" begin
    v = sin(Inf)
    @test v == π
    @test @isdefined v # not executed
end

julia> using TestRunner

Run @testset "basic tests":

julia> @testset "Basic tests runner" verbose=true runtest("demo.jl", ["basic tests"]);
Test Summary:      | Pass  Broken  Total  Time
Basic tests runner |    2       1      3  0.0s
  basic tests      |    2       1      3  0.0s

@testset can be selected with regex:

julia> @testset "Regex runner" verbose=true runtest("demo.jl", [r".*tests"]);
Test Summary:      | Pass  Broken  Total  Time
Regex runner       |   12       1     13  0.0s
  basic tests      |    2       1      3  0.0s
  struct tests     |    2              2  0.0s
  nested tests     |    5              5  0.0s
  calculator tests |    3              3  0.0s

Run individual @test cases that use the process function:

julia> @testset "Standalone runner" verbose=true runtest("demo.jl", [:(@test process(s_) == n_)]);
Test Summary:     | Pass  Total  Time
Standalone runner |    2      2  0.0s

Nested @testset can be selected:

julia> @testset "Nested runner" verbose=true runtest("demo.jl", ["inner tests 1"]);
Test Summary:   | Pass  Total  Time
Nested runner   |    2      2  0.0s
  inner tests 1 |    2      2  0.0s

Individual @test cases can be selectively matched using pattern expressions:

julia> @testset "Pattern in nested" verbose=true runtest("demo.jl", [:(@test startswith(s_, prefix_))]);
Test Summary:     | Pass  Total  Time
Pattern in nested |    1      1  0.0s

We can run tests by directly specifying line numbers:

julia> @testset "Single line" verbose=true runtest("demo.jl", [56]); # Run only the test on line 56
Test Summary: | Pass  Total  Time
Single line   |    1      1  0.0s

julia> @testset "Line range" verbose=true runtest("demo.jl", [55:57]); # Run tests in lines 55-57
Test Summary: | Pass  Total  Time
Line range    |    3      3  0.0s

julia> @testset "Mixed patterns" verbose=true runtest("demo.jl", ["calculator tests", 61]); # Combine named testsets with line numbers
Test Summary:       | Pass  Total  Time
Mixed patterns      |    4      4  0.0s
  calculator tests  |    3      3  0.0s

When tests fail or encounter errors, TestRunner provides enhanced debugging capabilities through its custom TestRunnerTestSet type. This section explains how different types of test failures are handled and reported. Let's compare how different types of test failures are reported with and without TestRunnerTestSet:

For simple assertion failures, there's no difference between the two approaches. Both provide the full interpreter stacktrace.

With Test.DefaultTestSet:

julia> @testset verbose=true runtest("demo.jl", ["Test failure"]);
Test failure: Test Failed at demo.jl:68
  Expression: sin(0) == π
   Evaluated: 0.0 == π

Stacktrace:
  [1] evaluate_call!(interp::TestRunner.TRInterpreter, frame::JuliaInterpreter.Frame, fargs::Vector{Any}, ::Bool)
    @ TestRunner ~/julia/packages/TestRunner/src/TestRunner.jl:515
  ...

With TestRunnerTestSet:

julia> @testset TestRunnerTestSet verbose=true runtest("demo.jl", ["Test failure"]);
Test failure: Test Failed at demo.jl:68
  Expression: sin(0) == π
   Evaluated: 0.0 == π

Stacktrace:
  [1] evaluate_call!(interp::TestRunner.TRInterpreter, frame::JuliaInterpreter.Frame, fargs::Vector{Any}, ::Bool)
    @ TestRunner ~/julia/packages/TestRunner/src/TestRunner.jl:515
  ...

When an exception occurs within a @test expression, full exception information is only available with TestRunnerTestSet.

With Test.DefaultTestSet:

julia> @testset verbose=true runtest("demo.jl", ["Exception inside of `@test`"]);
Exception inside of `@test`: Error During Test at demo.jl:72
  Test threw exception
  Expression: sin(Inf) == π

Exception inside of `@test`: Error During Test at demo.jl:74
  Test threw exception
  Expression: cos(Inf) == π

With TestRunnerTestSet:

julia> @testset TestRunnerTestSet verbose=true runtest("demo.jl", ["Exception inside of `@test`"]);
Exception inside of `@test`: Error During Test at demo.jl:72
  Test threw exception
  Expression: sin(Inf) == π
  DomainError with Inf:
  sin(x) is only defined for finite x.
  Stacktrace:
    [1] sin_domain_error(x::Float64)
      @ Base.Math ./special/trig.jl:28
    [2] sin(x::Float64)
      @ Base.Math ./special/trig.jl:39
    ...

Exception inside of `@test`: Error During Test at demo.jl:74
  Test threw exception
  Expression: cos(Inf) == π
  DomainError with Inf:
  cos(x) is only defined for finite x.
  Stacktrace:
    [1] cos_domain_error(x::Float64)
      @ Base.Math ./special/trig.jl:97
    [2] cos(x::Float64)
      @ Base.Math ./special/trig.jl:108
    ...

When an exception occurs outside of a @test macro (preventing subsequent tests from running), full exception information is available only with TestRunnerTestSet:

With Test.DefaultTestSet:

julia> @testset verbose=true runtest("demo.jl", ["Exception outside of `@test`"]);
Exception outside of `@test`: Error During Test at demo.jl:77
  Got exception outside of a @test

With TestRunnerTestSet:

julia> @testset TestRunnerTestSet verbose=true runtest("demo.jl", ["Exception outside of `@test`"]);
Exception outside of `@test`: Error During Test at demo.jl:77
  Got exception outside of a @test
  DomainError with Inf:
  sin(x) is only defined for finite x.
  Stacktrace:
    [1] sin_domain_error(x::Float64)
      @ Base.Math ./special/trig.jl:28
    [2] sin(x::Float64)
      @ Base.Math ./special/trig.jl:39
    ...

TestRunner leverages JuliaInterpreter and LoweredCodeUtils to selectively execute test code:

1. Pattern Matching on AST: Uses JuliaSyntax to parse code and MacroTools to match patterns against the syntax tree
2. Line-based Selection: Maps matched AST nodes to source line numbers, which serve as the bridge to lowered code
3. Selective Interpretation: Only top-level code is interpreted; function calls within tests are compiled and run at normal speed
4. Conservative Dependency Execution: Executes all top-level code except @test and @testset expressions to ensure tests don't fail due to missing dependencies

The key insight is that in reasonably-organized test code, the conservative dependency execution would only run the function and type definitions necessary for tests, without actual test code executed. Since test execution bottlenecks are often in the test cases themselves (not in defining functions), TestRunner allows efficient execution of test cases in interest by skipping execution of unrelated tests while still ensuring all code dependencies are available.

1. Source Provenance: Pattern matching occurs at the surface AST level and results are converted to line numbers. However, lowered code representation lacks proper source provenance (especially for macro expansions), causing surface-level pattern match information to be incorrectly mapped to lowered code.

   Example (from limitation1.jl):

   @testset "limitation1" begin
       limitation1() = nothing
       @test isnothing(limitation1()) #=want to run only this=#; @test isnothing(identity(limitation1())) #=but this runs too=#
   end

   When trying to run only the first test:

   julia> @testset "Limitation1 runner" verbose=true runtest("limitations/limitation1.jl", [:(@test isnothing(limitation1()))]);
   Test Summary:      | Pass  Total  Time
   Limitation1 runner |    2      2  0.0s

   Both tests are executed (2 tests pass) because they are on the same line. The line-based selection mechanism cannot distinguish between multiple expressions on the same line.

   Workaround (see workaround1.jl): Place each @test on a separate line:

   @testset "workaround1" begin
       workaround1() = nothing
       @test isnothing(workaround1())             # want to run only this
       @test isnothing(identity(workaround1()))   # now this test is skipped
   end

   With this workaround:

   julia> @testset "Workaround1 runner" verbose=true runtest("limitations/workaround1.jl", [:(@test isnothing(workaround1()))]);
   Test Summary:      | Pass  Total  Time
   Workaround1 runner |    1      1  0.0s

   Now only the matched test is executed (1 test passes).

   This limitation will be resolved with JuliaLowering.jl integration, which will eliminate the need for crude line number conversion from surface AST pattern matches.

2. Conservative Dependency Execution: Due to the conservative approach, ALL top-level code except @test and @testset expressions is executed. This means individual @test cases within function calls cannot be selectively executed.

   Example (from limitation2.jl):

   using Test
   
   function limitation2()
       @test String(nameof(Test)) == "Test"
   end
   
   limitation2()  # This executes during dependency execution
   
   @testset "selected test" limitation2()  # This also executes when selected

   When trying to match the @testset pattern, both the direct function call and the testset run:

   julia> @testset "limitation2 demo" verbose=true runtest("limitations/limitation2.jl", ["selected test"]);
   Test Summary:     | Pass  Total  Time
   limitation2 demo  |    2      2  0.3s
     selected test   |    1      1  0.0s

   The test inside limitation2() executes twice: once from the top-level limitation2() call (which executes as part of conservative dependency execution) and once from the matched @testset "selected test".

   Workaround (see workaround2.jl): Wrap test execution code in @testset blocks instead of functions, or avoid top-level function calls that contain tests.

3. Tests Within Function Bodies: Individual @test cases within function bodies cannot be selectively executed using expression patterns.

   Example (from limitation3.jl):

   function test_arithmetic()
       @test 1 + 1 == 2   # line 10: Can't match this with :(@test 1 + 1 == 2)
       @test 2 * 2 == 4   # line 11: Can't match this with :(@test 2 * 2 == 4)
   end

   When trying to match a specific @test pattern:

   julia> @testset "limitation3 demo" verbose=true runtest("limitations/limitation3.jl", [:(@test 1 + 1 == 2)]);
   Test Summary:    | Total  Time
   limitation3 demo |     0  0.0s

   No tests execute because the pattern matching doesn't look inside function bodies. This happens because:

   • Pattern matching occurs at the AST level where the @test is inside a function definition
   • Function bodies are not executed during pattern matching
   • Only top-level @test expressions or those within @testset blocks can be matched

   Workarounds (see workaround3.jl):

   1. Move tests into @testset blocks:

   @testset "arithmetic tests" begin
       @test 1 + 1 == 2   # This CAN be matched with :(@test 1 + 1 == 2)
       @test 2 * 2 == 4
   end

   2. Call test functions inside @testset blocks to make them selectable:

   @testset "function calls" begin
       test_arithmetic()  # Now these tests execute when this testset is selected
   end

TestRunner is built on top of:

• JuliaInterpreter.jl and LoweredCodeUtils.jl for selective code execution
• JuliaSyntax.jl for parsing
• MacroTools.jl for pattern matching