Strange output

[robert-andrzejuk]

The below code generates strange output.
Multiple times the Failed tests are listed in an incremental manner.
I would expect each check to be listed once.

Code:

#define BOOST_UT_DISABLE_MODULE
#include "ut.hpp"

#define EXPECT_THAT(expr) expect(that % expr) << "\"" #expr "\"\n"

int main() {
    using namespace boost::ut;
    using namespace boost::ut::literals;

    "basic"_test = [] {
        int a1 = 1, a2 = 1, a3 = 1, a4 = 1, a5 = 1, a6 = 1;
        int b1 = 0, b2 = 0, b3 = 0, b4 = 0, b5 = 0, b6 = 0;

        EXPECT_THAT(a1 == b1);
        EXPECT_THAT(a2 == b2);
        EXPECT_THAT(a3 == b3);
        EXPECT_THAT(a4 == b4) << fatal;
        EXPECT_THAT(a5 == b5);
        EXPECT_THAT(a6 == b6);
    };
}

Output:

Running test "basic"... FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:17 - test condition:  [1 == 0]

 "a1 == b1"

Running test "basic"... FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:17 - test condition:  [1 == 0] "a1 == b1"
FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:18 - test condition:  [1 == 0]

 "a2 == b2"

Running test "basic"... FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:17 - test condition:  [1 == 0] "a1 == b1"
FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:18 - test condition:  [1 == 0] "a2 == b2"
FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:19 - test condition:  [1 == 0]

 "a3 == b3"

Running test "basic"... FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:17 - test condition:  [1 == 0] "a1 == b1"
FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:18 - test condition:  [1 == 0] "a2 == b2"
FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:19 - test condition:  [1 == 0] "a3 == b3"
FAILED
in: C:\Github\beman\scope\tests\beman\scope\scope.test.cpp:20 - test condition:  [1 == 0]

 "a4 == b4"

===============================================================================
Suite global
tests:   1 | 5 failed
asserts: 5 | 0 passed | 5 failed

[JeffGarland]

I'm not sure what the interaction issue is here but maybe that it's == instead of % you'd need in the macro -- regardless you should be able to write this without the macro and get decent location information anyway.

    expect(a1 == b1);
    expect(fatal(a4 == b4)) << "fatal errror a4 not equal b4";

[parihaaraka]

same problem

int main(int argc, char *argv[])
{
    ...
    cfg<override> = {.tag = {connected ? "connected" : "skip"}};
    tag("connected") / "integrational"_test = []() {
        should("eval") = []{
            sync_tnt_request([](tnt::connection &cn)  // functor to be called from another thread
            {
                ...
                // set callback to be called from main thread
                set_handler(cn.last_request_id(), [](const mp_map_reader &header, const mp_map_reader &body) {
                    // violation prints multiple times:
                    expect(ut::nothrow([&](){ throw_if_error(header, body); }));   //  <- tests.cpp:207
                    ...
                });
            });
            ...
        };
    };

    return EXIT_SUCCESS;
}

Breakpoint on error print line (within throw_if_error()) only triggers once but the output is:

$ ./cpp2tnt_tests

 "Execute access to universe '' is denied for user 'guest'" 
 
Running test "integrational"... 
  Running test "eval"... 
 "Execute access to universe '' is denied for user 'guest'" 
 FAILED
in: .../tests/tests.cpp:207 - test condition:  [nothrow]

  Running test "eval"... FAILED

Running test "integrational"... 
  Running test "eval"... 
 "Execute access to universe '' is denied for user 'guest'" 
 FAILED
in: .../tests/tests.cpp:207 - test condition:  [nothrow]Unexpected exception with message:
key not found
31<d9 38 45 78 65 63 75  74 65 20 61 63 63 65 73 
...
===============================================================================
Suite global
tests:   4 | 2 failed
asserts: 33 | 2 passed | 2 failed

[maxtruxa]

Minimal repro on Godbolt.

To reproduce locally:

cat - >demo.cpp <<'EOF'
#include <boost/ut.hpp>
int main() {
    using namespace boost::ut;
    "demo"_test = [] {
        expect(1 == 2);
        expect(1 == 3);
    };
}
EOF
g++ -std=c++23 -I/path/to/boost-ext-ut/include demo.cpp -o demo
./demo

Result:

Running test "demo"... FAILED
in: demo.cpp:5 - test condition:  [false]


Running test "demo"... FAILED
in: demo.cpp:5 - test condition:  [false]FAILED
in: demo.cpp:6 - test condition:  [false]


===============================================================================
Suite global
tests:   1 | 2 failed
asserts: 2 | 0 passed | 2 failed

Note how even the final test counter is bugged (2 out of 1 tests failed).

Expected:

Running test "demo"... FAILED
in: demo.cpp:5 - test condition:  [false]
in: demo.cpp:6 - test condition:  [false]


===============================================================================
Suite global
tests:   1 | 1 failed
asserts: 2 | 0 passed | 2 failed

[ILer32]

Hi @robert-andrzejuk, @JeffGarland, @parihaaraka, @maxtruxa,
I recently addressed some issues for this project.
I think some issues you mentioned before were also solved.
Could you help to check it?

Example 1 (modified from the example provided by @robert-andrzejuk),

#define BOOST_UT_DISABLE_MODULE
#include <boost/ut.hpp>
#define EXPECT_THAT(expr) expect(that % expr) << "\"" #expr "\""
int main() {
    using namespace boost::ut;
    using namespace boost::ut::literals;

    "basic"_test = [] {
        int a1 = 1, a2 = 1, a3 = 1, a4 = 1, a5 = 1, a6 = 1;
        int b1 = 0, b2 = 0, b3 = 0, b4 = 0, b5 = 0, b6 = 0;

        EXPECT_THAT(a1 == b1);
        EXPECT_THAT(a2 == b2);
        EXPECT_THAT(a3 == b3);
        EXPECT_THAT(a4 == b4) << fatal;
        EXPECT_THAT(a5 == b5);
        EXPECT_THAT(a6 == b6);
        };
}

Result 1

UT starts =====================================================================
Running test "basic"...
FAILED in:...\example.cpp:12 - test condition: [1 == 0] "a1 == b1"
FAILED in:...\example.cpp:13 - test condition: [1 == 0] "a2 == b2"
FAILED in:...\example.cpp:14 - test condition: [1 == 0] "a3 == b3"
FAILED in:...\example.cpp:15 - test condition: [1 == 0] "a4 == b4"
=> terminated for the fatal issue
===============================================================================
Suite global
tests:   1 | 1 failed
asserts: 4 | 0 passed | 4 failed
Completed =====================================================================

Example 2 (modified from the example provided by @maxtruxa),

#include <boost/ut.hpp>
int main() {
    using namespace boost::ut;
    "demo"_test = [] {
        expect(1 != 1_i);
        expect(2 != 2_i);
    };
}

Result 2

UT starts =====================================================================
Running test "demo"...
FAILED in: ...\example.cpp:5 - test condition: [1 != 1]
FAILED in: ...\example.cpp:6 - test condition: [2 != 2]
===============================================================================
Suite global
tests:   1 | 1 failed
asserts: 2 | 0 passed | 2 failed
Completed =====================================================================

Example 3 (modified from the example provided by @JeffGarland),

#include <boost/ut.hpp>
int main() {
    using namespace boost::ut;
	expect(1_i != 1) << "log messages...." << fatal; // fatal assertion with log messages
    //expect(fatal(1_i != 1));                       // fatal assertion without log messages
    expect(2_i != 2) << "should not run the row";    // not executed
}

Result 3

UT starts =====================================================================
FAILED in: ...\example.cpp:4 - test condition: [1 != 1] log messages....
=> terminated for the fatal issue
===============================================================================
Suite global
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed
Completed =====================================================================

[maxtruxa]

LGTM