Skip to content

Neko-Box-Coder/ssTest

Repository files navigation

πŸ§ͺ ssTest

An easy to use, flexible, single header, C++11 testing framework.

A testing framework is literally just a fancy way of printing asserts results. The last thing you want is to spend so much time trying to figure out how to import and use a testing framework.

πŸ“¦οΈ Installation

Just include ssTest.hpp into your project To disable color output, either #ssTEST_NO_COLOR_OUTPUT 1 before the include or set it in cmake option

πŸƒ Quick Start

Here is a quick example of what ssTest looks like in a simple example. Many other features are covered in πŸ“– Documentations

int AddOne(int input) { return input + 1; }

int main()
{
    int testVar = 0;
    
    //Name of test group is optional
    ssTEST_INIT_TEST_GROUP("ssTest Quick Start");
    
    ssTEST_COMMON_SETUP
    {
        testVar = 1;
    };
    
    ssTEST("testVar Should Be Initialized By Common Setup")
    {
        ssTEST_OUTPUT_ASSERT_TRUE(testVar == 1);
    };
    
    ssTEST("Setup And Execution Example")
    {
        //ssTEST_OUTPUT_SETUP is optional
        ssTEST_OUTPUT_SETUP
        (
            int testVar2 = 10;
        );
        
        //ssTEST_OUTPUT_EXECUTION is optional
        ssTEST_OUTPUT_EXECUTION
        (
            testVar = AddOne(testVar2);
        );
        
        ssTEST_OUTPUT_ASSERT_EQ(testVar, testVar2 + 1);
    };

    ssTEST_END_TEST_GROUP();
}

πŸ“– Documentations

πŸ˜Άβ€πŸŒ«οΈ Concepts

In ssTest, asserts (like ssTEST_OUTPUT_ASSERT_TRUE() or ssTEST_OUTPUT_ASSERT_EQ()) are grouped into test (ssTEST()), tests are grouped into test group (ssTEST_INIT_TEST_GROUP()).

You can also name each assert, test and test group however you want.

Test groups are also nestable so you can have a hierarchy of tests.

So you can have a simple structure like this (Expandable)
- Test Group: My App
    - Test: Feature A Scenario 1
        - Assert: Action 1
        - Assert: Action 2
    - Test: Feature A Scenario 2
        - Assert: Action 1
        - Assert: Action 2
Or a more complex structure like this (Expandable)
- Test Group: My App
    - Test: Feature A
        - Test Group: Feature A
            - Test: Feature A Scenario 1
                - Assert: Action 1
                - Assert: Action 2
            - Test: Feature A Scenario 2
                - Assert: Action 1
                - Assert: Action 2
    
    - Test: Feature B
        - Test Group: Feature B
            - Test: Feature B Scenario 1
                - Assert: Action 1
                - Assert: Action 2
            - Test: Feature B Scenario 2
                - Assert: Action 1
                - Assert: Action 2

Each test in a test group can optionally share a common setup (ssTEST_COMMON_SETUP()) and a common cleanup function (ssTEST_COMMON_CLEANUP()).

πŸ“‚ Test Group

Each test group must be inside a function that returns an int.

If that test group is successful, it will return 0. Otherwise, it will return non 0.

It is common to put a single test group in a single .cpp file inside the int main() function but you can do any other way you see fit.

Initializing A Test Group (Required)

  • ssTEST_INIT_TEST_GROUP(); : Initialize a test group with the name of the current file
  • ssTEST_INIT_TEST_GROUP("testGroupName");: Initialize a test group with testGroupName.

Common Setup And Cleanup

These should be called before declaring the first test (ssTEST()) in the test group

  • ssTEST_COMMON_SETUP(){...};: A common setup function that is called before each test in the test group
  • ssTEST_COMMON_CLEANUP(){...};: A common cleanup function that is called after each test in the test group
  • ssTEST_DISABLE_COMMON_SETUP_CLEANUP_BETWEEN_TESTS();: Disable calling the common setup and cleanup functions between tests. But still be called at the beginning and end of the test group.

Ending A Test Group (Required)

  • ssTEST_END_TEST_GROUP();: Finishes the declarations of all the tests, executes them and outputs the results.

A short example of a test group

int main()
{
    int testVar = 0;
    int cleanUpCount = 0;
    
    ssTEST_INIT_TEST_GROUP();
    
    ssTEST_COMMON_SETUP{ testVar = 1;};
    ssTEST_COMMON_CLEANUP{ cleanUpCount++; };
    //ssTEST_DISABLE_COMMON_SETUP_CLEANUP_BETWEEN_TESTS();
    
    ssTEST("A Normal Test")
    {
        ssTEST_OUTPUT_ASSERT_EQ(testVar, 1);
        testVar = 5;
    };
    
    ssTEST("Another Normal Test")
    {
        ssTEST_OUTPUT_ASSERT_EQ(testVar, 1);
        ssTEST_OUTPUT_ASSERT_EQ(cleanUpCount, 1);
        
        //If ssTEST_DISABLE_COMMON_SETUP_CLEANUP_BETWEEN_TESTS() is called, 
        //  testVar would be 5 and cleanUpCount would be 0
    };

    ssTEST_END_TEST_GROUP();
}

Nesting Test Groups (Advanced)

Just like a normal test group, you just need to call ssTEST_INIT_TEST_GROUP() inside a function that is called from the outer test group.

However, the (optional) extra step you need to do to make it format well is to pass the indentation string from the outer test group to the inner test group.

These should be called before declaring the first test (ssTEST()) in the test group

  • ssTEST_GET_NESTED_TEST_GROUP_INDENT();: Gets the indentation string for nested test group
  • ssTEST_SET_TEST_GROUP_INDENT("indent");: Sets the indentation level using the string indent

Ax example would be

int AssertGroupA(std::string outerIndent)
{
    ssTEST_INIT_TEST_GROUP("Test Group A");
    ssTEST_SET_TEST_GROUP_INDENT(outerIndent);
    
    ssTEST("A Normal Test In Group A")
    {
        ssTEST_OUTPUT_ASSERT_TRUE(true);
    };

    ssTEST_END_TEST_GROUP();
}

int main()
{
    ssTEST_INIT_TEST_GROUP("Root Test Group");
    ssTEST("A Normal Test")
    {
        ssTEST_OUTPUT_ASSERT_TRUE(true);
    };

    ssTEST("Calling A Nested Test Group")
    {
        ssTEST_OUTPUT_ASSERT_EQ( AssertGroupA( ssTEST_GET_NESTED_TEST_GROUP_INDENT() ), 0 );
    };

    ssTEST_END_TEST_GROUP();
}

Disable Output For Certain Actions Inside A Test (Advanced)

These should be called before declaring the first test (ssTEST()) in the test group

  • ssTEST_DISABLE_OUTPUT_SETUP();: Disable outputting to the console for the test setup action
  • ssTEST_DISABLE_OUTPUT_EXECUTION();: Disable outputting to the console for the test execution action
  • ssTEST_DISABLE_OUTPUT_ASSERT();: Disable outputting to the console for the test assert action. If the assert fails however, it will still output associated line that failed the assert.

πŸ§ͺ Test

  • ssTEST("testName"){statement_1; statement_2; ...};: Creates a test with testName
  • ssTEST_ONLY_THIS("testName"){statement_1; statement_2; ...};: Creates a test with testName and only runs this test, great when debugging a single test.
  • ssTEST_SKIP("testName"){statement_1; statement_2; ...};: Creates a test with testName and skips it

Example:

int main()
{
    ssTEST_INIT_TEST_GROUP();
    
    ssTEST("A Normal Test")
    {
        ssTEST_OUTPUT_ASSERT_TRUE(true);
    };
    
    ssTEST_ONLY_THIS("Will Only Run This Test")
    {
        ssTEST_OUTPUT_ASSERT_TRUE(true);
    };
    
    ssTEST_SKIP("Will Skip This Test")
    {
        ssTEST_OUTPUT_ASSERT_TRUE(true);
    };
    
    ssTEST_END_TEST_GROUP();
}

πŸ‘Š Actions And Assertions Inside A Test

Required Assertions

All assertion macros accept an optional "info" parameter as the last argument for additional context.

  • ssTEST_OUTPUT_ASSERT_TRUE(expression [, "info"]);: Assert that the expression evaluates to true.
  • ssTEST_OUTPUT_ASSERT_EQ(actual, expected [, "info"]);: Assert that actual equals expected.
  • ssTEST_OUTPUT_ASSERT_NOT_EQ(actual, expected [, "info"]);: Assert that actual does not equal expected.
  • ssTEST_OUTPUT_ASSERT_LT(actual, expected [, "info"]);: Assert that actual is less than expected.
  • ssTEST_OUTPUT_ASSERT_LT_EQ(actual, expected [, "info"]);: Assert that actual is less than or equal to expected.
  • ssTEST_OUTPUT_ASSERT_GT(actual, expected [, "info"]);: Assert that actual is greater than expected.
  • ssTEST_OUTPUT_ASSERT_GT_EQ(actual, expected [, "info"]);: Assert that actual is greater than or equal to expected.

Optional Assertions

  • ssTEST_MARK_OPTIONAL_ASSERT_START(); and ssTEST_MARK_OPTIONAL_ASSERT_END();: Mark the start and end of optional assertions. Any assertion between these markers will not fail the test if they fail.
    ssTEST_MARK_OPTIONAL_ASSERT_START();
    ssTEST_OUTPUT_ASSERT_TRUE(condition);
    ssTEST_OUTPUT_ASSERT_EQ(value1, value2);
    ssTEST_MARK_OPTIONAL_ASSERT_END();

Test Setup and Execution

  • ssTEST_OUTPUT_SETUP(statement_1; statement_2; ...);: Executes and output the setup statements.
  • ssTEST_OUTPUT_EXECUTION(statement_1; statement_2; ...);: Executes and output the execution statements.

Skipping Actions

  • ssTEST_MARK_SKIP_ASSERT_START(); and ssTEST_MARK_SKIP_ASSERT_END();: Mark the start and end of skipped assertions. Any assertion between these markers will be skipped.
    ssTEST_MARK_SKIP_ASSERT_START();
    ssTEST_OUTPUT_ASSERT_TRUE(condition);
    ssTEST_OUTPUT_ASSERT_EQ(value1, value2);
    ssTEST_MARK_SKIP_ASSERT_END();
  • ssTEST_OUTPUT_SKIP_SETUP(statement_1; statement_2; ...);: Skips the test setup action.
  • ssTEST_OUTPUT_SKIP_EXECUTION(statement_1; statement_2; ...);: Skips the test execution.

Outputting Values (Up to 5 values) When An Assertion Failed

  • ssTEST_OUTPUT_VALUES_WHEN_FAILED(comma, separated, values,...);: Output the values of the variables when the assertion failed.

Outputting Values In Test Format

  • ssTEST_OUTPUT(expression);: Output message which expands to std::cout << expression << std::endl

Calling Common Setup And Cleanup Manually (Advanced)

  • ssTEST_CALL_COMMON_SETUP();: Calls the common setup function manually
  • ssTEST_CALL_COMMON_CLEANUP();: Calls the common cleanup function manually

πŸ“ Complete Example

Let's say I have a C memory allocator:

//An Example Class To Test
typedef struct
{
    int PageSize;
    int PageCount;
    short* AllocateID_Table;
    int NextAllocateID;
    char* Memory;
} MyMemoryAllocator;

MyMemoryAllocator MyMemoryAllocator_Create(size_t pageSize, size_t pageCount);
bool MyMemoryAllocator_Destroy(MyMemoryAllocator* allocator);
void* MyMemoryAllocator_Allocate(MyMemoryAllocator* allocator, size_t size);
bool MyMemoryAllocator_Free(MyMemoryAllocator* allocator, void* memory);

My test file could look like this:

int main()
{
    MyMemoryAllocator testAllocator;
    
    ssTEST_INIT_TEST_GROUP();

    ssTEST_COMMON_SETUP
    {
        testAllocator = MyMemoryAllocator_Create(64, 12);
    };

    ssTEST_COMMON_CLEANUP
    {
        MyMemoryAllocator_Destroy(&testAllocator);
    };

    ssTEST("MyMemoryAllocator_Create Should Not Crash When Requesting Zero Memory")
    {
        //We want to create our own memory allocator and request empty memory
        ssTEST_CALL_COMMON_CLEANUP();
        
        ssTEST_OUTPUT_EXECUTION
        (
            testAllocator = MyMemoryAllocator_Create(64, 0);
        );
        
        ssTEST_OUTPUT_ASSERT_EQ(testAllocator.PageSize, 64);
        ssTEST_OUTPUT_ASSERT_EQ(testAllocator.PageCount, 0);
    };
    
    ssTEST("MyMemoryAllocator_CreateShared Should Create Shared Memory Allocator")
    {
        ssTEST_OUTPUT_SETUP
        (
            const int pageSize = 64;
            const int pageCount = 4;
            MyMemoryAllocator sharedAllocator;
            void* sharedMemory = MyMemoryAllocator_Allocate(&testAllocator, 
                                                            pageSize * pageCount + 
                                                            pageCount * sizeof(short));
        );
        
        ssTEST_OUTPUT_EXECUTION
        (
            sharedAllocator = MyMemoryAllocator_CreateShared(sharedMemory, pageSize, pageCount);
        );
        
        ssTEST_OUTPUT_ASSERT_EQ(sharedAllocator.AllocateID_Table, (short*)sharedMemory);
        ssTEST_OUTPUT_ASSERT_EQ(sharedAllocator.Memory, (char*)sharedMemory + pageCount * sizeof(short));
        ssTEST_OUTPUT_ASSERT_EQ(sharedAllocator.PageSize, pageSize);
        ssTEST_OUTPUT_ASSERT_EQ(sharedAllocator.PageCount, pageCount);
        
        MyMemoryAllocator_Destroy(&sharedAllocator);
    };

    ssTEST("MyMemoryAllocator_Allocate Should Allocate Memory When Enough Memory Is Available")
    {
        ssTEST_OUTPUT_SETUP
        (
            const int pageSize = 64;
            char* memory;
            int pageCount = 4;
        );
        
        for(int i = 0; i < 2; i++)
        {
            ssTEST_OUTPUT_EXECUTION
            (
                memory = (char*)MyMemoryAllocator_Allocate(&testAllocator, pageSize * pageCount);
            );
        }
        
        ssTEST_OUTPUT_EXECUTION
        (
            *memory = 17;
            *(memory + pageSize * pageCount - 1) = 17;
        );
        
        //This also works without a message as well
        ssTEST_OUTPUT_ASSERT_NOT_EQ(memory, nullptr, "Memory Result");
        
        //Asserting by passing two values (to compare equality) will also print the values
        ssTEST_OUTPUT_ASSERT_EQ(*memory, 17, "Reading to beginning of memory");
        
        //You can also specify the operator to use for comparison
        ssTEST_OUTPUT_ASSERT_EQ(*(memory + pageSize * pageCount - 1), 17, "Reading to end of memory");
    };
    
    ssTEST("MyMemoryAllocator_Allocate Should Return NULL When Not Enough Memory Is Available")
    {
        ssTEST_OUTPUT_SETUP
        (
            const int pageSize = 64;
        );
        
        //Let's say this is not implemented or we know it is crashing, we can skip the assert 
        //or even skip the whole test with ssTEST_SKIP
        ssTEST_MARK_SKIP_ASSERT_START();
        
        ssTEST_OUTPUT_ASSERT_EQ((char*)MyMemoryAllocator_Allocate(&testAllocator, pageSize * 250),
                               nullptr,
                               "Allocation Result");
        
        ssTEST_MARK_SKIP_ASSERT_END();
    };
    
    ssTEST("MyMemoryAllocator_Allocate Should Return NULL When Zero Size Is Passed In")
    {
        ssTEST_OUTPUT_SETUP
        (
            char* memory;
        );
        (void)memory;
        
        ssTEST_OUTPUT_EXECUTION
        (
            memory = (char*)MyMemoryAllocator_Allocate(&testAllocator, 0);
        );
        
        //Let's say there's a bug in the allocator which will probably fail the assert, 
        //but we still want to know the result. An optional assert can be used in this case.
        ssTEST_MARK_OPTIONAL_ASSERT_START();
        
        ssTEST_OUTPUT_ASSERT_EQ(memory, nullptr, "Allocation Result");
        
        ssTEST_MARK_OPTIONAL_ASSERT_END();
    };

    //etc...

    ssTEST_END_TEST_GROUP();
}

For full example, see Src/ssTestExample.cpp

🎯 Command Line Arguments

ssTest can be controlled through command line arguments. To enable this, pass the program arguments to ssTEST_PARSE_ARGS:

int main(int argc, char* argv[])
{
    ssTEST_INIT_TEST_GROUP();
    ssTEST_PARSE_ARGS(argc, argv);
    
    ssTEST("My Test")
    {
        //...
    };
    
    ssTEST_END_TEST_GROUP();
}

Available options:

--help                   Show help message
--no-setup-output        Disable setup output
--no-execution-output    Disable execution output
--no-assert-output       Disable assert output
--test-only <name>       Only run the specified test
--skip-test <name>       Skip the specified test
--list-only              Only list available tests
--min-output             Only show the minimum output
--assert-output          Only show assertion output

Note: ssTEST_PARSE_ARGS must be called after ssTEST_INIT_TEST_GROUP but before defining any tests.

Here are some screenshots of the output of the tests

πŸ“Š Tests And Assert Results

When all the tests have finished running, you can quickly find the ones that failed by searching for certain keywords from different types of result output.

Assertions

  • Required Assertions:
    • Assertion Passed (O) when passed
    • Assertion Failed (X) when failed
    • Error Caught (X) ... when an error is caught
  • Optional Assertions:
    • Assertion Passed (+) when passed
    • Assertion Failed (-) when failed
    • Error Caught (-) ... when an error is caught
  • Skipped Assertions:
    • Assertion Skipped (/) when skipped

Test

  • ...passed all tests (OOO) when all assertions passed
  • ...failed some tests (XXX) when some assertions failed
  • ...Error Caught (XXX)... when an error is caught

Notice each result has a symbol inside the parentheses, which makes it easier to identify the result.

You can easily capture the symbol inside the parentheses to quickly output the related failures in CI. For example, if you want the context of all failed asserts, you can quickly do

cat testResults.txt | grep -B15 "(X)"

Or in powershell

Select-String -path .\testResult.txt -Pattern "\(X\)" -Context 15,0

About

πŸ§ͺ A simple fancy test framework

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published