index

You can just write a unit test framework in C++

As a Rust developer, where unit testing semantics are essentially bundled into the language with cargo, the internals of writing and debugging tests is normally an afterthought. In C++, not so much the case. There are a number of frameworks to write tests, including but not limited to Boost.Test, Catch2, and doctest. Why? I suppose because there are different approaches to each aspect of a unit testing framework, but I believe this is mostly due to how the ecosystem developed over the years. Boost.Test maintains a backwards compatible API at the detriment of supporting a huge swath of macros, making it unintuitive to select the modern choice. Catch2 and doctest were not burdened with this backward compatibility concern, and innovated on how users can write tests, including useful debug strings and expression capturing.

So why would we rewrite what Catch2 and doctest already do? Mostly because it isn’t a tremendous amount of code, but a custom framework allows for trivial control of console outputs, CLI flags, debug strings for failing expressions, and anything else that might be domain or project specific. A unit test suite with great flexibility is achievable in a modest amount of code, which is what this blog will demonstrate.

Before looking at any code, the goals of the unit test framework will help guide the design decisions. In my opinion, these are the priorities of a unit tester:

  • Provide simple and intuitive macros. Limit the number of macros as much as possible to reduce technical debt for other developers. Writing tests should be easy.
  • Failure outputs should be as useful as possible. Provided type names should be printed to the console, as well as the domain specific representation of the type.
  • Adding fixtures to a unit test and using them in the test should be easy.
  • Console logs should be concise and predictable.
  • The runner should integrate well with the rest of the system (CI, domain-specific types)

Disclaimer that this blog reads procedurally. This is to fully enumerate how the internals of the implementation work and is not a discussion on testing philosophy.

Component one: the runner

A unit test runner is just a binary like any other, and this binary is simple from a high level. First, the tests are registered in a global registry. This normally comes in the form of adding function pointers to a vector. As we will see later, the body of these functions are exactly the tests that the user defines. Next, after the tests are registered, we start popping these tests off the vector and run them. At the end, we simply report the result of each test. Easy. Let’s see some example code.

First, we define a test case, which is what we will collect at the beginning of the program.

struct TestCase {
    const char* suite;
    const char* name;
    void (*fn)();
};

We could get more fancy, but for my implementation, this was good enough. Just some strings and the function pointer.

When we need to get the elements from the global registry, we may use the following function.

inline std::vector<TestCase>& registry()
{
    static std::vector<TestCase> tests;
    return tests;
}

This guarantees that the tests vector will be initialized before access, when we attempt to emplace test cases later.

Now, in the main function, we can simply iterate over this vector. For instance, here is how to log the total number of tests that match the user filter.

for (const auto& tc : registry()) {
    if (matches_filter(tc, opts.filter)) ++total_matching;
}
log(LogLevel::Error, "Running %d test cases...\n", total_matching);

Component two: registering tests

Here is where some of the black magic comes into play. We will see here how a user can write a simple macro to define a unit test, and how that test gets added to our vector of tests under the hood. In this section, we will also see how we can handle test fixtures here as well. We can start with the end goal. We would like for a user to define tests like so.

TEST_CASE(mustfail)
{
    auto a{1};
    auto b{2};
    CHECK(a == b);
}

Therefore, we need the TEST_CASE macro to accept a name, and allow the user to define the test within { ... }. The second part can be tricky. We want to declare the function first, then pass it’s reference onto our registry.

As a first step, I made this convenience structure that allows a declaration to push a function pointer to our registry. This is because, at the namespace scope, we can only use declarations, not bare expressions. For example, we can’t just randomly call a function bar(); in a namespace. We can circumvent this rule by declaring a Registrar structure every time we want to push a test.

struct Registrar {
    Registrar(const char* name, void (*fn)())
    {
        registry().emplace_back(TestCase{current_test_suite(), name, fn});
    }
};

Now for the hard part, defining the TEST_CASE macro. I will start with the full body, then break down each line.

#define TEST_CASE(name)                          
    static void BITCOIN_TEST_CAT(btc_test_fn, __LINE__)();
    static ::framework::Registrar BITCOIN_TEST_CAT(btc_test_reg, __LINE__){#name, &BITCOIN_TEST_CAT(btc_test_fn, __LINE__)};
    static void BITCOIN_TEST_CAT(btc_test_fn, __LINE__)()
static void BITCOIN_TEST_CAT(btc_test_fn, __LINE__)();

BITCOIN_TEST_CAT is a simple wrapper that concatenates two streams of tokens. In this case, BITCOIN_TEST_CAT takes the string btc_test_fn and adds the line number of where the test is defined. This forces a unique name for each function in a file. We start by forward declaring this function, which will define the test body.

static ::framework::Registrar BITCOIN_TEST_CAT(btc_test_reg, __LINE__){#name, &BITCOIN_TEST_CAT(btc_test_fn, __LINE__)};

Next, we add name of the function and the reference to the registry before the body is defined. This allows the user to write { ... } naturally without having to introduce additional semantics like a closing macro at the end of the test case.

static void BITCOIN_TEST_CAT(btc_test_fn, __LINE__)()

Finally, we can let the user define the function by starting the definition with the function name we just made. Suddenly, the user is writing a function that our registry is aware of, nice.

Now for the harder part: registering a test with fixtures. Again, I’ll post the full macro.

#define FIXTURE_TEST_CASE(name, Fixture)                     
    namespace {                                              \
    namespace BITCOIN_TEST_CAT(btc_test_fixture, __LINE__) { \
    struct Impl : Fixture {                                  \
        void btc_test_run();                                 \
    };                                                       \
    static void runner()                                     \
    {                                                        \
        Impl impl;                                           \
        impl.btc_test_run();                                 \
    }                                                        \
    static ::framework::Registrar reg{#name, &runner};       \
    }                                                        \
    }                                                        \
    void BITCOIN_TEST_CAT(btc_test_fixture, __LINE__)::Impl::btc_test_run()

Having a look at the first two lines, we have the following.

namespace {
namespace BITCOIN_TEST_CAT(btc_test_fixture, __LINE__) {

These nested namespaces resolve an issue that is not present in the TEST_CASE macro, which declares the function as static. The outer namespace gives everything in the body internal linkage, meaning, type names do not collide across different translation units. The second namespace provides per-test isolation, so multiple tests can define the Impl structure. Now onto Impl itself.

struct Impl : Fixture {                                  
    void btc_test_run();                                 
};

We are balancing two goals here. Firstly, we need to have a callable function that we can register with our registry, but we also need the members of Fixture available to the user. We can accomplish this by having Impl inherit from Fixture and declare its own function. Notice it is not defined yet. Next, let’s actually register a function.

static void runner()                                    
{                                                        
    Impl impl;                                           
    impl.btc_test_run();                                 
}                                                        
static ::framework::Registrar reg{#name, &runner};

Great, now we’ve pushed a function to the test vector that we can run. Looking inside runner, it does exactly what we need. Namely, it inherits and executes the constructor of Fixture, which is exactly what we’d like, then calls the btc_test_run function which has yet to be defined. As a last step, we will let the user define that exact function.

void BITCOIN_TEST_CAT(btc_test_fixture, __LINE__)::Impl::btc_test_run()

Voila. The user can now define this function body, and all the members of Fixture will be available, neato.

Component three: expression capture and evaluation

Now, we need to macros that evaluate expressions and add debug output on failure. Revisiting the simple test example, we would like to write something like the following.

TEST_CASE(mustfail)
{
    auto a{1};
    auto b{2};
    CHECK(a == b);
}

We will turn to the CHECK(a == b) line in this section. If we naively compared a and b within the macro, we would know the result as true or false, but we would not be able to report the values of a and b for debugging, as the == would be evaluated. This introduces us to the next set of tricks. In C++, the evaluation of operators follows a defined order, known as operator precedence. To learn the values of the expression before == is executed, we can override an operator that has a higher precedence than ==. There are a few of these, particularly the relational comparison or streaming operators (<= or <<) will be evaluated before ==. Let’s start by defining a structure that will override one of these.

struct Decomposer {
    template <typename T>
    CapturedExpression<T> operator<=(const T& lhs) const
    {
        return {lhs};
    }
};

With some templating, we can take any T and ensure it will be taken before the == evaluates via the opeartor<= that Decomposer defines. Notice the return type of operator<= is CapturedExpression. We will define this structure next.

template <typename T>
struct CapturedExpression {
    const T& lhs;
    ...

Here is the start of the structure. At it’s core, it simply holds whatever T that the Decomposer provided. The primary purpose of this structure is to define all of the comparison operators. We will do so in yet another macro.

#define DECOMPOSE_OP(op)
    template <typename U>
    Result operator op(const U& rhs) const
	{
        btc_test_result = static_cast<bool>(lhs op rhs);
        return btc_test_result ? Result::ok() : Result::failed(stringify(lhs) + " " #op " " + stringify(rhs)); 
    }

    DECOMPOSE_OP(==)
    DECOMPOSE_OP(!=)
    DECOMPOSE_OP(<)
    DECOMPOSE_OP(<=)
    DECOMPOSE_OP(>)
    DECOMPOSE_OP(>=)
#undef DECOMPOSE_OP

We now take the right hand side of the expression and static_cast the result of the comparison to a bool. Then, depending on the result, we either return an ok or a failed result. Notice that the failure type takes a string of the two values and the comparator used. Great! Now the Result type holds a string with the expression that failed, along with the values that caused the failure. Note that we omitted the actual implementation of the stringify function here, but it is essentially a wrapper around << with additional cases. This is where implementation specific logic might enter the framework, like custom printing for domain-specific types. For completeness, here is the definition of the Result type.

struct Result {
    std::optional<std::string> failed_expression;

    static Result ok()
    {
        return {std::nullopt};
    }

    static Result failed(std::string&& expression)
    {
        return {std::move(expression)};
    }

    bool is_ok() const noexcept
    {
        return !failed_expression.has_value();
    }
};

With that, we may now define the CHECK macro.

#define CHECK(expr)
    do {
        ::framework::Result btc_test_res_ = ::framework::Decomposer{} <= expr;
        ::framework::record_check(btc_test_res_, "CHECK", #expr, __FILE__, __LINE__)
    } while (false)

Leveraging the Decomposer, we pass expr through the <= operator, which ensures the left hand side gets converted to a CapturedExpression, which is then converted to a framework::Result after considering the right hand side of the expression. This result is then passed to some record_check function, which can do whatever we please, like print the expression or record the number of failures.

Final Output

The example that has been used so far is aptly named mustfail. We can have a quick look at what all this trickery has afforded us. Repeating the example a final time:

TEST_CASE(mustfail)
{
    auto a{1};
    auto b{2};
    CHECK(a == b);
}

We see a very obviously does not equal b. Our CapturedExpression holds the value 1 as the lhs, then, after evaluating == with 2 and discovering it is false, we return a failed result and print the failed expression thereafter.

[FAIL]: test/mustfail.cpp:30: CHECK(a == b)
1 == 2

Downsides of this approach

There is a small but looming tradeoff with this approach. The Result type is returned after evaluating the comparison operators, but the Result itself does not implement the &&, ||, &, | operators. This means we cannot chain Result within a check macro. If we think about the implementation of such operators, we would be overloading the semantics of && and ||, which would require the early return properties to hold. What I mean here is true || cond_2 will never actually evaluate cond_2. This is unfortunate for developers that like to group conditions, but Catch2, doctest, and recent versions of Boost.Test have all determined this tradeoff is acceptable in their releases. It is possible to preserve the behavior of && and || at the expense of losing debug information, but the entire point of this black magic was for debugging.

Closing

I started a similar project in an effort to remove Boost.Test from Bitcoin Core. The motivation began in an effort to simplify the depends system and remove a dependency from the build, but I think what came out on the other end was a nice framework. I greatly enjoy the concept of having a handful of macros that developers can write tests with, and appreciated the opportunity to learn such insane C++ voodoo. To see the full thing, check out the pull request here.