Uninstrumented crowbar binaries trigger weird error in `afl-fuzz`

[NathanReb]

@pascutto and I recently ran into a weird issue while trying to fuzz https://github.com/mirage/index.
We tried fuzzing an uninstrumented binary by mistake and got a Fork server handshake failed error from afl-fuzz instead of the usual No instrumentation detected one which made it a bit hard to realize our mistake.

I tried reproducing this on simpler examples from https://github.com/NathanReb/ocaml-afl-examples and it seems to indicate that this happens when using crowbar but not otherwise.

To reproduce you can clone the repo and run the following commands from non afl opam switch:

$ dune build @simple-parser/fuzz
    afl-fuzz alias simple-parser/fuzz/fuzz (exit 1)
(cd _build/default/simple-parser/fuzz && /usr/bin/afl-fuzz -i inputs -o findings -- ./fuzz_me.exe @@)
afl-fuzz 2.52b by <lcamtuf@google.com>

...

[-] Looks like the target binary is not instrumented! The fuzzer depends on
    compile-time instrumentation to isolate interesting test cases while
    mutating the input data. For more information, and for tips on how to
    instrument binaries, please see /usr/share/doc/afl-doc/docs/README.

    When source code is not available, you may be able to leverage QEMU
    mode support. Consult the README for tips on how to enable this.
    (It is also possible to use afl-fuzz as a traditional, "dumb" fuzzer.
    For that, you can use the -n option - but expect much worse results.)

[-] PROGRAM ABORT : No instrumentation detected
         Location : check_binary(), afl-fuzz.c:6920

and

$ dune build @awesome-list/fuzz
    afl-fuzz alias awesome-list/fuzz/fuzz (exit 1)
(cd _build/default/awesome-list/fuzz && /usr/bin/afl-fuzz -i inputs -o findings -- ./fuzz_me.exe @@)
afl-fuzz 2.52b by <lcamtuf@google.com>

...

[-] Hmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. There are two probable explanations:

    - The current memory limit (50.0 MB) is too restrictive, causing an OOM
      fault in the dynamic linker. This can be fixed with the -m option. A
      simple way to confirm the diagnosis may be:

      ( ulimit -Sv $[49 << 10]; /path/to/fuzzed_app )

      Tip: you can use http://jwilk.net/software/recidivm to quickly
      estimate the required amount of virtual memory for the binary.

    - Less likely, there is a horrible bug in the fuzzer. If other options
      fail, poke <lcamtuf@coredump.cx> for troubleshooting tips.

[-] PROGRAM ABORT : Fork server handshake failed
         Location : init_forkserver(), afl-fuzz.c:2253

As you can see, the first example is just a simple binary trying to parse an int from the input. It doesn't use crowbar and we get the expected No instrumentation detected error.

The second one on the other hand uses crowbar and leads to the Fork server handsake failed.

Do you have any idea why afl-fuzz isn't able to detect that the binary isn't instrumented?

[stedolan]

This is annoying and I'm not sure how to improve the error message.

Crowbar uses afl-fuzz in "persistent" mode, which means that it reuses the same process rather than re-forking for every testcase, since this is much faster. It does this by linking against the stedolan/ocaml-afl-persistent library, which uses a special primitive in the OCaml runtime to reset instrumentation.

afl-fuzz detects whether a binary is instrumented by grepping it for a magic string. This magic string is used in the runtime in afl.c. Since Crowbar uses ocaml-afl-persistent, and ocaml-afl-persistent uses a function from afl.c (the special primitive above), then afl.c gets linked into all programs using crowbar, regardless of whether they're instrumented. So, afl-fuzz sees the magic string in an uninstrumented binary and gets confused.

[NathanReb]

Ok I see, thanks for the explanation, that makes total sense now.

Do you think there is a chance that we could split afl.c in a way that we can use the primitive needed for resetting instrumentation without linking against the part that introduce the magic string and get that merged upstream in the compiler?

That's a relatively long-term fix though. Would you accept an addition to the README that warn against that limitation and suggest people to double check they compiled their to-be-fuzzed crowbar binary with a +afl compiler variant if they get that error? I think that's the best we can do in the meantime!

[stedolan]

Yes, we could split afl.c. I think the easiest is to add a new file afl-init.c that defines caml_setup_afl and has the magic string, but as you say this is a relatively long-term fix. The README addition sounds like a good idea in the meantime.

[Lupus]

I've ran into this problem and googled this issue, which helped me to resolve it. My binary turned out to be not instrumented, I've added compiler flags only to library that I was testing, but not the binary. Quite confusing experience overall.