Framework Performance

Overview

Benchmarking was conducted using CMake via the this script. There are 3 benchmark scenarios:

• Cost of Including the Header
• Cost of an Assertion Macro
• Runtime Speed with a Large Number of Assertions

Compilers used:

• Windows: Microsoft Visual Studio Community 2017 - Version 15.8.1+28010.2003
• Windows: gcc 8.1.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
• Linux: gcc 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2)
• Linux: clang 4.0.0-1 (tags/RELEASE_400/rc1) Target: x86_64-pc-linux-gnu

Environment used (Intel i7 3770k, 16GB RAM):

• Windows 7 - on SSD
• Ubuntu 17.04 in VirtualBox VM - on HDD

doctest version: 2.2.0 (released on December 02, 2018) Catch version: 2.3.0 (released on July 22, 2018)

Compile-Time Benchmarks

Cost of Including the Header

This benchmark is only relevant to single-header and header-only frameworks – such as doctest and Catch.

The script generates 201 source files: 200 of them create a function in the form of int f135() { return 135; }, and all 200 such dummy functions are forward-declared in main.cpp. Their results are accumulated to return from the main() function. This ensures all source files are built and the linker does not remove or optimize any content.

• Baseline - Time taken to build source files single-threaded with msbuild/make
• + Implement - Only in main.cpp, the header is included with a preceding #define to implement the test runner:

  #define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
  #include "doctest.h"

• + header everywhere - The framework header is also included in all other source files
• + disabled - Remove all testing-related code from the binary

doctest	baseline	+ implement	+ header everywhere	+ disabled
MSVC Debug	4.89	6.21	8.33	6.39
MSVC Release	4.38	6.39	8.71	6.02
MinGW GCC Debug	8.12	10.86	14.73	10.17
MinGW GCC Release	8.21	11.11	15.03	10.71
Linux GCC Debug	4.20	6.23	9.81	6.24
Linux GCC Release	4.29	6.93	11.05	6.76
Linux Clang Debug	8.70	10.02	14.43	11.13
Linux Clang Release	9.30	11.68	16.20	11.58

Catch	baseline	+ implement	+ header everywhere	+ disabled
MSVC Debug	4.82	7.83	88.85	88.72
MSVC Release	4.38	9.97	87.17	88.35
MinGW GCC Debug	8.00	57.28	137.28	132.73
MinGW GCC Release	8.38	22.94	97.17	97.22
Linux GCC Debug	4.42	15.57	97.94	97.18
Linux GCC Release	4.50	19.59	99.48	100.75
Linux Clang Debug	8.76	15.60	107.99	110.61
Linux Clang Release	9.32	25.75	118.67	117.11

Conclusion

doctest

• Instantiating the test runner in one source file takes approximately 1–3 seconds (implement - baseline)
• Including "doctest.h" in one source file costs 11 ms – 23 ms ((header_everywhere - implement) / 200)
• Including the library everywhere with all features disabled costs approximately 2 seconds (disabled - baseline) for 200 files

Catch

• Instantiating the test runner in one source file takes approximately 3–50 seconds (implement - baseline)
• Including catch.hpp in one source file costs 380 ms – 470 ms ((header_everywhere - implement) / 200)
• Disabling the library with the CATCH_CONFIG_DISABLE configuration option has no impact on header inclusion cost

---

Therefore, if "doctest.h" costs 11 ms on MSVC and "catch.hpp" costs 400 ms – the doctest header is >> 36<< times lighter (for MSVC)!

The results are shown in seconds and are by no means intended to criticize Catch – the doctest framework would not exist without it.

The doctest header has minimal compile-time overhead because it forward-declares all content and does not pull any headers into source files (except the one implementing the test runner). This is a key design decision.

Cost of an Assertion Macro

The script generates 11 .cpp files: 10 files create 50 test cases each, with 100 assertions per test case (in the form of CHECK(a==b) where a and b are the same int variable) – totaling 50k assertions! The test framework is implemented in main.cpp.

• Baseline - Time for a single-threaded build with the header included everywhere (no test cases or assertions)
• CHECK(a==b) - Adds CHECK() assertions that decompose expressions using template mechanisms

doctest specific:

• +fast 1 - Adds DOCTEST_CONFIG_SUPER_FAST_ASSERTS to speed up compilation of normal CHECK(a==b) assertions
• CHECK_EQ(a,b) - Uses CHECK_EQ(a,b) instead of expression decomposition
• +fast 2 - Adds DOCTEST_CONFIG_SUPER_FAST_ASSERTS to speed up compilation of binary CHECK_EQ(a,b) assertions
• +disabled - Disables all test cases and assertion macros with DOCTEST_CONFIG_DISABLE

Catch specific:

• +fast - Adds CATCH_CONFIG_FAST_COMPILE to speed up compilation of normal CHECK(a==b) assertions
• +disabled - Disables all test cases and assertion macros with CATCH_CONFIG_DISABLE

doctest	baseline	CHECK(a==b)	+fast 1	CHECK_EQ(a,b)	+fast 2	+disabled
MSVC Debug	2.69	27.37	10.37	17.17	4.82	1.91
MSVC Release	3.15	58.73	20.73	26.07	6.43	1.83
MinGW GCC Debug	3.78	97.29	43.05	59.86	11.88	1.67
MinGW GCC Release	4.09	286.70	95.42	156.73	18.16	2.03
Linux GCC Debug	2.39	91.36	41.92	52.26	10.16	1.32
Linux GCC Release	3.29	257.40	97.46	128.84	19.38	1.79
Linux Clang Debug	2.40	85.52	43.53	51.24	8.32	1.62
Linux Clang Release	3.40	160.65	79.34	81.52	11.90	1.82

Here are the results for Catch (only supports normal CHECK(a==b) assertions):

Catch	baseline	CHECK(a==b)	+fast	+disabled
MSVC Debug	8.20	31.22	25.54	8.22
MSVC Release	10.13	448.68	168.67	10.20
MinGW GCC Debug	53.54	152.38	131.85	49.07
MinGW GCC Release	19.26	590.16	466.69	18.99
Linux GCC Debug	15.05	117.30	95.33	14.79
Linux GCC Release	18.77	608.94	482.73	18.96
Linux Clang Debug	12.27	94.39	77.33	12.11
Linux Clang Release	20.75	545.84	506.02	20.15

Conclusion

doctest:

• Regex-decomposed CHECK(a==b) assertions are 0–8 times faster than Catch
• CHECK_EQ(a,b) assertions (no expression decomposition) are approximately 31–63% faster than CHECK(a==b)
• The DOCTEST_CONFIG_SUPER_FAST_ASSERTS identifier speeds up normal assertions by 57–68%
• The DOCTEST_CONFIG_SUPER_FAST_ASSERTS identifier speeds up binary assertions by an additional 84–91%
• The DOCTEST_CONFIG_DISABLE identifier makes assertions disappear entirely (even faster than baseline, as most implementation code is removed)

Catch:

• CATCH_CONFIG_FAST_COMPILE speeds up assertion compilation by 10–30% (73% in one case)
• CATCH_CONFIG_DISABLE provides the same significant benefit for assertions as DOCTEST_CONFIG_DISABLE – but does not reduce header inclusion cost

Runtime Benchmarks

Runtime benchmarks use a single test case with a loop of 10 million iterations: either a single normal assertion (with expression decomposition) or an assertion plus logging of the loop iterator i:

for(int i = 0; i < 10000000; ++i)
    CHECK(i == i);

or

for(int i = 0; i < 10000000; ++i) {
    INFO(i);
    CHECK(i == i);
}

Note: All assertions pass – the goal is to optimize for the common case (many passing test cases, few failures).

doctest	assert	+ info		Catch	assert	+ info
MSVC Debug	4.00	11.41		MSVC Debug	5.60	213.91
MSVC Release	0.40	1.47		MSVC Release	0.76	7.60
MinGW GCC Debug	1.05	2.93		MinGW GCC Debug	1.17	9.54
MinGW GCC Release	0.34	1.27		MinGW GCC Release	0.36	4.28
Linux GCC Debug	1.24	2.34		Linux GCC Debug	1.44	9.69
Linux GCC Release	0.29	0.52		Linux GCC Release	0.29	3.60
Linux Clang Debug	1.15	2.38		Linux Clang Debug	1.21	9.91
Linux Clang Release	0.28	0.50		Linux Clang Release	0.32	3.27

Conclusion

doctest assertions are approximately 20% faster than Catch's, and several times faster (over 18 times faster in one specific compiler scenario) when logging variables and context.

---

Bar charts were generated by pasting table data into this Google Spreadsheet.

For a non-synthetic benchmark, see this blog post by Baptiste Wicht, who tested assertion compile times in version 1.1 with his expression template library!

When reading the article: note that if a process segment takes 50% of the total time and is sped up 10,000 times, the entire process will only be about 50% faster overall.