Fuzz: Wire up ClusterFuzzLite CI fuzzing

[LucaCappelletti94]

Step 2 of #2519, modelled on diesel-rs/diesel#5212.

Adds the ClusterFuzzLite configuration and four workflows for the existing fuzz_parse_sql target.

Every pull request fuzzes for 10 minutes in code-change mode, pushes to main refresh a runner-cached build, and a daily batch at 06:00 UTC plus a corpus prune at 05:00 keep the shared corpus on the clusterfuzzlite branch of this repository using the default GITHUB_TOKEN.

Given this is the first time we actually start to fuzz the code automatically and the various PRs, I expect some blood bath of bugs will ensue, but that is sort of the point. We will add more fuzz harnesses over time.

[LucaCappelletti94]

A bit unexpected, but no crash on the first fuzz makes sense given how weak is the current harness. The next PRs will bring better harnesses and a decent seed body.

[alamb]

Thanks @LucaCappelletti94 -- I left some comments. Let me know what you think

[alamb]

I am not sure about the value of running fuzzing on main pushes as well -- my rationale is that if the fuzz test hits an issue on main then there will be a red x recorded on that commit and we will perhaps be mislead into thinking that a bug was introduced by that commit (or at least will have to rule it out)

Also it seems strange to me to make the amount of effort devoted to fuzz testing a function of the number of PRs merged to main (which is what this will do)

I personally suggest just running the fuzz testing in batch mode to begin with. If you want more fuzz time, then just crank it up there

[LucaCappelletti94]

This particular workflow is solely focused on ensuring the fuzzer harnesses keep building. It does not run the harnesses themselves.

It additionally works alongside the PR fuzzing job as it is the harness built that is used for the differential fuzzing.

[alamb]

likewise here I think PR fuzzing will just generate more confusion to contributors than it is worth -- if the fuzz test fails on their PR they probably will either

1. Rerun it
2. Get confused and think it is related to something in their PR

I think both of these are confusing outcomes and recommend against it

[LucaCappelletti94]

My hope was that, as the fuzzer stabilizes over time, the bugs that get raised by a PR fuzzing run are exactly bugs of the PR, and this should simplify a lot the job of reviewers.

I do agree that at this time most bugs raised by the fuzzer are pre-existing things that need handling before it can be deployed on PRs as at this time yes they would be confusing.

[LucaCappelletti94]

I am a tad new to ClusterFuzzLite specifically, so I double checked. In theory, IF I configured this correctly (and I primarely copied the documentation examples) if a bug exists on main and survives in the PR, it is reported as pre-existing, and if it exists in the PR and not in the main it is noted as new bug. Only the latter case should be marked as a blocking error.

[alamb]

I am not familiar with corpus pruning -- maybe a definition to the link would be helpful
Also I don't understand why this is a separate job from fuzz testing -- why not make the daily schedule a single action (perhaps with two jobs)?

[LucaCappelletti94]

TLDR of it, the corpus entries over time become overlapping/redundant, and we want to keep the minimum set of entries that seed the maximum coverage. This minimization procedure trims down the corpus by assigning to each entry a fingerprint of the coverage it produces, and then keeps sufficiently distinct ones. It is generally done less frequently than fuzzing itself.

I will add a definition and link the documentation.

[alamb]

I expect some blood bath of bugs will ensue, but that is sort of the point.

Yes I expect this too -- but I would recommend shielding other contributors from the pain (and yourself from having to explain it to them)

[LucaCappelletti94]

I will set up analogous workflows in a test repo to verify whether the PR workflow behaves as I described, just in case I either (a) copied the wrong documentation examples / configured them poorly (b) misunderstood anything of the documentation.

I believe it works as advertised but given it is a relatively new tool for me (I generally just fuzz locally) I want to make extra sure.

If it goes red only when the bug is new, that would be exactly what we want and should put the issue of having random red CIs in PR at rest, I suppose.

[alamb]

If it goes red only when the bug is new, that would be exactly what we want and should put the issue of having random red CIs in PR at rest, I suppose.

I think the gate should also be about false positives (not just true negatives)

So in theory I would hope our existing process / new unit tests would cover regressions and then the fuzzing is a net new set of tests

[LucaCappelletti94]

I think the gate should also be about false positives (not just true negatives)

What do you mean by false positives?

[alamb]

I think the gate should also be about false positives (not just true negatives)

What do you mean by false positives?

I mean the chance of the fuzz test failing on a PR that did not cause whatever problem the fuzzer found.

I think if we run non deterministic tests on PRs we are going to inject some noise into the process -- I would recommend against it if at all possible (at least until we have some confidence that the fuzz test is mostly successful -- e.g. run it for a week or more every day and not find anything new)

[LucaCappelletti94]

I think the gate should also be about false positives (not just true negatives)

What do you mean by false positives?

I mean the chance of the fuzz test failing on a PR that did not cause whatever problem the fuzzer found.

I think if we run non deterministic tests on PRs we are going to inject some noise into the process -- I would recommend against it if at all possible (at least until we have some confidence that the fuzz test is mostly successful -- e.g. run it for a week or more every day and not find anything new)

I am not sure how that would occur, suppose the PR compiles to f and the main is compiled to g. For a given fuzzer input x, if f(s) crashes and g(x) does not, there is no non-determinism. The only non-deterministic aspect is that a failing input x might not be found in the limited 10 minutes window.

The scenario you describe where f(x) crashes and g(x) does not but it should can only happen if I configured ClusterFuzzLite poorly, if I understood it correctly.

To be clear, I perfectly agree that if the CI starts to randomly give out bugs to PRs that have nothing to do with that bug it would be something extremely problematic and to be absolutely avoided. If it happens to be the case, I would suggest to only keep the periodic fuzz job.

That being said, again if I understood correctly its documentation, that should not be the case as it would make ClusterFuzzLite principal use case be nigh useless.

[alamb]

The scenario you describe where f(x) crashes and g(x) does not but it should can only happen if I configured ClusterFuzzLite poorly, if I understood it correctly.

Maybe it is different in ClusterFuzz but with most fuzzers, if you run them twice they test two different sets of queries.

So in that case you don't actually compare f(x) and g(x) -- instead you are basically testing f(x) and g(x') where the queries tested in x' are different than x because they are randomly generated

Perhaps I am misunderstanding 🤔

[LucaCappelletti94]

Basically here if I understood how it works, you have:

1. the .github/workflows/cflite_build.yml builds the main fuzzer harness, our g
2. a PR comes along, and triggers the .github/workflows/cflite_pr.yml workflow, and builds the PR branch fuzzer harness, our f
3. When, as in Fuzz: Add curated seed corpus #2530 there is no g compatible to f, g collapses to be an infallible function
4. The fuzzer starts generating inputs, feeds f(x), if f(x) crashes it checks g(x). If the latter crashes, the bug is old, otherwise it is new.

So for any given fuzzer iteration, f, g, and x are given by that iteration of the fuzzer h, then it will produce x' and iterate onward. The fuzzer used in ClusterFuzz is nothing special, it is libfuzzer. There is nothing particular to the fuzzer itself.

[alamb]

Sounds like a solid plan -- Let's give it a try!

[alamb]

Thank you for the lesson @LucaCappelletti94 🙏