[BsAtHome]

Several issues are addressed in this PR:

• Update parallel run to be able to stop with -s on a failed test. The remaining parallel jobs will run until done after which parallel exits.
• Test availability of parallel and bwrap. Running with -j requires both parallel and bwrap. These commands are now tested to be available.
• Fix various typos, comment style problems and some whitespace issues.
• Fix and update usage message to represent the current state of affairs.

@hdiethelm you may want to have a look at this too ;-)

[hdiethelm]

Looks good! Thanks for the fixes.

I found a setting in my spellchecker, the errors where limited to 100. Next time, it should be better and it even found an issue... ;-)

There are a few other places that could be optimized, which I figured to do later or not at all due to they are not that important vs. the speed benefit you get right now. Same for the -s option, I figured if you can run all the tests in ~1m30, -s it is not important any more and by scripts/runtests -j 32 2>/dev/null, you see only the errors.

Parallel tests swap the output compared to single threaded. The reason behind is that parallel gathers stdout and stderr separately and prints it sequentially:

scripts/runtests -j 1 tests/ccomp/lathe-comp
*** /home/hannes/linuxcnc-src/tests/ccomp/lathe-comp: FAIL: test run exited with 1
Running test: /home/hannes/linuxcnc-src/tests/ccomp/lathe-comp
Runtest parallel: 1 tests run, 0 successful, 1 failed, 0 skipped, 0 shmem errors
Failed:
/home/hannes/linuxcnc-src/tests/ccomp/lathe-comp/test.sh

vs

scripts/runtests tests/ccomp/lathe-comp
Running test: tests/ccomp/lathe-comp
*** tests/ccomp/lathe-comp: FAIL: test run exited with 1
Runtest: 1 tests run, 0 successful, 1 failed, 0 skipped, 0 shmem errors
Failed: 
tests/ccomp/lathe-comp

Just piping all trough stdout is also not nice, due to in this case scripts/runtests -j 32 2>/dev/null will not hide the Running tests messages any more which would make runtests_flaky unusable. A "quiet" option would make this possible.

Also runtests show the errors on stdout and the information on stderr. Looks not standard to me.

[BsAtHome]

There are a few other places that could be optimized

You are right, there are some rough edges to smooth. But the basics hold up.

Parallel tests swap the output compared to single threaded. The reason behind is that parallel gathers stdout and stderr separately and prints it sequentially:

Running parallel always comes at a price. Same for make. However, the default is that parallel keeps the output of one process together by default (--group is the default) and doesn't mix it randomly (why you need -O in make). Other ordering is just the nature of running things concurrently.

When you see problems, then you'd normally run the specific test separately if it is not a race related error.

Also runtests show the errors on stdout and the information on stderr. Looks not standard to me.

Then you'd need to collect the output before outputting it to the "right" FD.

[hdiethelm]

BTW:
3 tests are slowing down the parallel runs considerably, that's why more than ~32 jobs doesn't help much:

Running test: /home/hannes/linuxcnc-src/tests/mdi-while-queuebuster-waitflag

real	0m59.614s
user	0m4.746s
sys	0m1.778s
Running test: /home/hannes/linuxcnc-src/tests/abort/g64

real	1m12.782s
user	0m0.498s
sys	0m0.152s
Running test: /home/hannes/linuxcnc-src/tests/workplane-retain

real	1m9.495s
user	0m0.648s
sys	0m0.230s

With these tests: 1m31.553s
If I just delete them: 0m35.956s

I did not yet look in to them, but improving these could cut the runtime again by a factor of 3.

[hdiethelm]

There are a few other places that could be optimized

You are right, there are some rough edges to smooth. But the basics hold up.

Parallel tests swap the output compared to single threaded. The reason behind is that parallel gathers stdout and stderr separately and prints it sequentially:

Running parallel always comes at a price. Same for make. However, the default is that parallel keeps the output of one process together by default (--group is the default) and doesn't mix it randomly (why you need -O in make). Other ordering is just the nature of running things concurrently.

When you see problems, then you'd normally run the specific test separately if it is not a race related error.

Agreed, this was also my opinion. There would be some solutions but not worth the effort and adding to much code adds also effort for maintenance.

Also runtests show the errors on stdout and the information on stderr. Looks not standard to me.

Then you'd need to collect the output before outputting it to the "right" FD.

Indeed, would mean a lot of change, so I decided to just not touch it.

[BsAtHome]

@grandixximo one of these tests is yours. Can you look at speeding it up a bit 8-)

BTW I also noted that the ui-smoke tests are slow. Even when started early in the series. It occurred to me that the tests need to be started in reverse time consumed order (slowest first) for the parallel jobs to run fastest with fewest jobs.

[hdiethelm]

It also came to my mind to create a run time list which can be updated by a command and then the tests can be run in reverse order and additionally create a "bad test list".

And then fake the system time, so rtapi_app does no sleep at all or runs at 100x speed. :-D

But let's not overdo it. ;-) 35s vs. 13m9.683s initially would be already 22x faster... Before, it was quite annoying to run the tests and now it is faster than the build.

[grandixximo]

@BsAtHome Mostly waiting, not testing. Fixed in #4770:

• Every linuxcnc shutdown waited its full 2 s: Cleanup waits for halcmd to be the last component, but __servo-thread stays and milltask never hal_exit()s iocontrol.0.
• abort/g64 and workplane-retain (mine) called wait_complete() after AUTO_RUN, a 5 s timeout per aborted case.

workplane-retain 70 => 24 s, abort/g64 72 => 46 s, mdi-while-queuebuster-waitflag 65 => 24 s.

Ordering, simulated from measured times: slowest first saves 8% at -j 4 up to 26% at -j 16, nothing at -j 32 where abort/g64 alone sets the floor. Starting the dirs with an .ini first gets most of that without a runtime list. I can send it after this PR, if wanted.

[hdiethelm]

I found a race created by me. The race was only visible when viewing the results file, the number of SUCCES words is correct, however, they where on the wrong line.

Feel free to pull this commit in your PR: hdiethelm@6162f5d

I can also create a PR later.

[BsAtHome]

I found a race created by me... I can also create a PR later.

Added it to this PR. Have a look.

[hdiethelm]

Looks good!