# Fuzz testing
Halide has a set of fuzz-testing harnesses that can be used to find those
tricky to find, edge cases and bugs that would otherwise not be caught
by a regular unit-testing suite. At the moment these fuzz-tests are housed
in the `test/fuzz` directory. The fuzz testing suite use the common,
[libfuzzer](https://www.llvm.org/docs/LibFuzzer.html) interface for fuzz-tests.

## Building fuzz tests
Fuzz testing requires specific instrumentation across the entire build;
to do this we make use of a fuzzing-specific-toolchain/preset. e.g.

```
cmake -B build --preset linux-x64-fuzzer -DLLVM_ROOT=/path/to/llvminstall
cmake --build ./build -j$(nproc)
```

Note that the LLVM install that you use must be built with
`-D LLVM_ENABLE_RUNTIMES="compiler-rt"` set if you want to build the fuzzer
tests (failing to do so will fail at configure time); not all prebuilt LLVM
installs include this, so you may need to build LLVM from source to run the
fuzz tests locally.

## Using the fuzz-harnesses
Fuzz-testing harnesses are a little different to a more traditional unit-test
and don't have a definitive end of test. In other words, a fuzz test will
run:
- for an infinite amount of time (the default),
- for a user specified maximum amount of time,
- until the fuzzer finds a bug and crashes,
- you manually kill the process e.g. (ctrl-C).

Once you have built the fuzz testing suite using the commands listed above you
can list the fuzz testing harnesses using the command:

```
ls ./build/test/fuzz/fuzz_*
```

To run a fuzzer simply run the fuzz-testing harness with no arguments. e.g.

`./build/test/fuzz/fuzz_simplify`

By default this will run the fuzz test on a single core and discard whatever.
temporary corpus is created.

To reuse a given corpus (recommended) create a new directory to store the
corpus generated by your fuzz testing harness and pass that directory into
your fuzzer e.g.

```
mkdir fuzz_simplify_corpus -p
./build/test/fuzz/fuzz_simplify fuzz_simplify_corpus
```

This will save the state of the fuzzer between runs, this way any progress
that your fuzzer makes improving code-coverage will remain persistent on
your disk.

Up until this point the fuzzer has only been running on a single core. To
speed things up a little, let's run the fuzzer in parallel across all
available cores on our machine.

```
./build/test/fuzz/fuzz_simplify fuzz_simplify_corpus -fork=$(nproc)
```

## Reproducing crashes
An important part of fuzz testing is reproducing the crashing input. To
handle this, a libfuzzer-based fuzz harness will create a crash file
whenever the fuzzer exits unexpectedly. This will look something like:

`crash-<some_random_hash>`

To reproduce a crash we simply rerun our fuzz harness with our crash
file as the first argument.

`./build/test/fuzz/fuzz_simplify crash-<some_random_hash>`

So long as your fuzz harness and library are deterministic this should
reproduce the original crash.

## Adding new fuzz tests
A bare-bones fuzzer will look something like the following:
```cpp
#include <stdint.h>
#include <stddef.h>
#include <my_library.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
  // Randomly throw data at our function and hope it doesn't crash.
  foo(data, size);
  return 0;
}
```

This assumes that our function foo takes in a buffer and the size of said
buffer. But in many cases we would like to make use of more structured data.
e.g. a string or a vector of integers etc. Thankfully libfuzzer provides
a handy helper to convert a raw buffer into common structured data types,
the [FuzzedDataProvider class](https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/fuzzer/FuzzedDataProvider.h).
For examples on how to use this class see `test/fuzz/simplify.cpp`.

## Other useful materials
- [The official libfuzzer docs](https://www.llvm.org/docs/LibFuzzer.html)
- [The libfuzzer tutorial](https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md)