OUnit/Alcotest-style interface?

[yallop]

(After writing this up, I noticed something similar-sounding in the TODO, which is probably a good sign. I'm sending this anyway, to add support for the idea, to flesh out the design a bit, and to open up discussion.)

At present the interface to tests is quite stateful: there's a function for adding tests to a global list

val add_test : ?name:string -> ('f, test_result) gens -> 'f -> unit

and the tests are implicitly invoked by a function installed with at_exit.

This interface makes it a little difficult to integrate crowbar into larger test frameworks.

It would be helpful to have an ounit/alcotest-style interface, where the user constructs test suites, perhaps hierarchically, and where invoking the tests is the user's responsibility.

For example, here's a basic less-stateful interface that sticks quite closely to the spirit of the current design:

type test
val test : ?name:string -> ('f, test_result) gens -> 'f -> test
val run_main : test list -> unit

With an interface like this one might write:

let tests = [
  Crowbar.test "some property"
   test_some_property [int; string] @@ fun i s ->
     (* ... *)
  ;

  Crowbar.test "another property"
   test_another_property [list1 string] @@ fun ss ->
     (* ... *)
  ;
]
    
let () = run_main tests

Crucially, since this interface separates the operations of defining tests and registering/invoking them, it's easier to select tests (or even decide whether or not to invoke the crowbar-based testsuite at all) based on command-line options or other input.

Is there some reason why it'd be tricky to add such an interface? I wonder if the current approach with atexit is just for convenience, or if there's some deeper technical reason why it's better to do things that way.

[stedolan]

Yes, I had always intended something like that. It would definitely be useful.

However, the current design supports a particularly handy mode of use, which I'd like to preserve in subsequent interfaces. Given a stateful interface like add_test, you can compile many different test files into a single binary which runs all the tests, without having to list the complete set anywhere.

In the add_test style, the single binary has a single test suite that contains many different tests, rather than a series of different test suites run one after the other. This makes it easy to fuzz the resulting test suite, with afl-fuzz being able to decide which test to spend effort on, rather than doing separate fuzzing jobs for each one.

[talex5]

Another problem with the at_exit system is that it interacts badly with https://github.com/aantron/bisect_ppx, which also uses at_exit to dump the coverage reports. The result is that the coverage gets written before the tests are run.