Skip to content

Update Readme #336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
atouchet wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Update Readme #336

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 89 additions & 98 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,17 @@

QuickCheck is a way to do property based testing using randomly generated
input. This crate comes with the ability to randomly generate and shrink
integers, floats, tuples, booleans, lists, strings, options and results.
All QuickCheck needs is a property function—it will then randomly generate
inputs to that function and call the property for each set of inputs. If the
property fails (whether by a runtime error like index out-of-bounds or by not
satisfying your property), the inputs are "shrunk" to find a smaller
counter-example.

The shrinking strategies for lists and numbers use a binary search to cover
the input space quickly. (It should be the same strategy used in
[Koen Claessen's QuickCheck for
Haskell](https://hackage.haskell.org/package/QuickCheck).)

[![Build status](https://github.com/BurntSushi/quickcheck/workflows/ci/badge.svg)](https://github.com/BurntSushi/quickcheck/actions)
integers, floats, tuples, booleans, lists, strings, options and results. All
QuickCheck needs is a property function—it will then randomly generate inputs
to that function and call the property for each set of inputs. If the property
fails (whether by a runtime error like index out-of-bounds or by not satisfying
your property), the inputs are "shrunk" to find a smaller counter-example.

The shrinking strategies for lists and numbers use a binary search to cover the
input space quickly. (It should be the same strategy used in
[Koen Claessen's QuickCheck for Haskell](https://hackage.haskell.org/package/QuickCheck).)

[![Build status](https://github.com/BurntSushi/quickcheck/actions/workflows/ci.yml/badge.svg)](https://github.com/BurntSushi/quickcheck/actions)
[![crates.io](https://img.shields.io/crates/v/quickcheck.svg)](https://crates.io/crates/quickcheck)

Dual-licensed under MIT or the [UNLICENSE](https://unlicense.org/).
Expand Down Expand Up @@ -55,8 +53,8 @@ old versions of Rust.

## The `#[quickcheck]` attribute

To make it easier to write QuickCheck tests, the `#[quickcheck]` attribute
will convert a property function into a `#[test]` function.
To make it easier to write QuickCheck tests, the `#[quickcheck]` attribute will
convert a property function into a `#[test]` function.

To use the `#[quickcheck]` attribute, you must import the `quickcheck` macro
from the `quickcheck_macros` crate:
Expand Down Expand Up @@ -99,7 +97,7 @@ development dependency instead:
quickcheck = "1"
```

If you want to use the `#[quickcheck]` attribute, then add `quickcheck_macros`
If you want to use the `#[quickcheck]` attribute, then add `quickcheck_macros`:

```toml
[dev-dependencies]
Expand All @@ -108,9 +106,8 @@ quickcheck_macros = "1"
```

N.B. When using `quickcheck` (either directly or via the attributes),
`RUST_LOG=quickcheck` enables `info!` so that it shows useful output
(like the number of tests passed). This is **not** needed to show
witnesses for failures.
`RUST_LOG=quickcheck` enables `info!` so that it shows useful output (like the
number of tests passed). This is **not** needed to show witnesses for failures.

Crate features:

Expand Down Expand Up @@ -148,8 +145,8 @@ semver compatible releases.
## Alternative Rust crates for property testing

The [`proptest`](https://docs.rs/proptest) crate is inspired by the
[Hypothesis](https://hypothesis.works/) framework for Python.
You can read a comparison between `proptest` and `quickcheck`
[Hypothesis](https://hypothesis.works/) framework for Python. You can read a
comparison between `proptest` and `quickcheck`
[here](https://github.com/proptest-rs/proptest/blob/main/proptest/README.md#differences-between-quickcheck-and-proptest)
and
[here](https://github.com/proptest-rs/proptest/issues/15#issuecomment-348382287).
Expand All @@ -160,8 +157,7 @@ might be worth a try!
## Alternatives for fuzzing

Please see the
[Rust Fuzz Book](https://rust-fuzz.github.io/book/introduction.html)
and the
[Rust Fuzz Book](https://rust-fuzz.github.io/book/introduction.html) and the
[`arbitrary`](https://crates.io/crates/arbitrary) crate.

## Discarding test results (or, properties are polymorphic!)
Expand Down Expand Up @@ -222,8 +218,8 @@ any type (that also satisfies `Testable`). So a function with type `fn(usize)
satisfies `Testable`.

So to discard a test, we need to return something other than `bool`. What if we
just returned a `TestResult` directly? That should work, but we'll need to
make sure `TestResult` satisfies `Testable`:
just returned a `TestResult` directly? That should work, but we'll need to make
sure `TestResult` satisfies `Testable`:

```rust
impl Testable for TestResult {
Expand Down Expand Up @@ -251,8 +247,7 @@ quickcheck(prop as fn(Vec<isize>) -> TestResult);

So now our property returns a `TestResult`, which allows us to encode a bit
more information. There are a few more
[convenience functions defined for the `TestResult`
type](https://docs.rs/quickcheck/*/quickcheck/struct.TestResult.html).
[convenience functions defined for the `TestResult` type](https://docs.rs/quickcheck/*/quickcheck/struct.TestResult.html).
For example, we can't just return a `bool`, so we convert a `bool` value to a
`TestResult`.

Expand All @@ -263,15 +258,15 @@ N.B. Since discarding a test means it neither passes nor fails, `quickcheck`
will try to replace the discarded test with a fresh one. However, if your
condition is seldom met, it's possible that `quickcheck` will have to settle
for running fewer tests than usual. By default, if `quickcheck` can't find
`100` valid tests after trying `10,000` times, then it will give up.
These parameters may be changed using
`100` valid tests after trying `10,000` times, then it will give up. These
parameters may be changed using
[`QuickCheck::tests`](https://docs.rs/quickcheck/*/quickcheck/struct.QuickCheck.html#method.tests)
and [`QuickCheck::max_tests`](https://docs.rs/quickcheck/*/quickcheck/struct.QuickCheck.html#method.max_tests),
or by setting the `QUICKCHECK_TESTS` and `QUICKCHECK_MAX_TESTS`
environment variables.
There is also `QUICKCHECK_MIN_TESTS_PASSED` which sets the minimum number of
valid tests that need pass (defaults to `0`) in order for it to be considered a
success.
and
[`QuickCheck::max_tests`](https://docs.rs/quickcheck/*/quickcheck/struct.QuickCheck.html#method.max_tests),
or by setting the `QUICKCHECK_TESTS` and `QUICKCHECK_MAX_TESTS` environment
variables. There is also `QUICKCHECK_MIN_TESTS_PASSED` which sets the minimum
number of valid tests that need pass (defaults to `0`) in order for it to be
considered a success.

## Shrinking

Expand Down Expand Up @@ -314,26 +309,24 @@ guaranteed to get this counter-example every time:

Which is going to be much easier to debug.

## More Thorough Checking
## More thorough checking

Quickcheck uses random input to test, so it won't
always find bugs that could be uncovered with a particular
property. You can improve your odds of finding these latent
bugs by spending more CPU cycles asking quickcheck to find
them for you. There are a few different ways to do this, and
which one you choose is mostly a matter of taste.
QuickCheck uses random input to test, so it won't always find bugs that could
be uncovered with a particular property. You can improve your odds of finding
these latent bugs by spending more CPU cycles asking QuickCheck to find them
for you. There are a few different ways to do this, and which one you choose is
mostly a matter of taste.

If you are finding yourself doing this sort of thing a
lot, you might also be interested in trying out
[`cargo fuzz`](https://github.com/rust-fuzz/cargo-fuzz),
which runs in a loop by default.
If you are finding yourself doing this sort of thing a lot, you might also be
interested in trying out
[`cargo fuzz`](https://github.com/rust-fuzz/cargo-fuzz), which runs in a loop
by default.

### Running in a Loop

One approach is to run your quickcheck properties in a loop that
just keeps going until you tell it to stop or it finds a bug.
For example, you could use a bash script such as the following
one.
One approach is to run your QuickCheck properties in a loop that just keeps
going until you tell it to stop or it finds a bug. For example, you could use a
Bash script such as the following one:

```bash
#!/usr/bin/bash
Expand All @@ -347,37 +340,34 @@ do
done
```

One thing to note is that this script passes the `qc_` filter to
`cargo test`. This assumes that you've prefixed all your quickcheck
properties with `qc_`. You could leave off the filter, but then
you would be running all your deterministic tests as well, which
would take time away from quickcheck!
One thing to note is that this script passes the `qc_` filter to `cargo test`.
This assumes that you've prefixed all your QuickCheck properties with `qc_`.
You could leave off the filter, but then you would be running all your
deterministic tests as well, which would take time away from QuickCheck!

Checking the return code and exiting is also important. Without that
test, you won't ever notice when a failure happens.
Checking the return code and exiting is also important. Without that test, you
won't ever notice when a failure happens.

### Cranking the Number of Tests

Another approach is to just ask quickcheck to run properties more
times. You can do this either via the
Another approach is to just ask QuickCheck to run properties more times. You
can do this either via the
[tests()](https://docs.rs/quickcheck/*/quickcheck/struct.QuickCheck.html#method.tests)
method, or via the `QUICKCHECK_TESTS` environment variable.
This will cause quickcheck to run for a much longer time. Unlike,
the loop approach this will take a bounded amount of time, which
makes it more suitable for something like a release cycle that
wants to really hammer your software.
method, or via the `QUICKCHECK_TESTS` environment variable. This will cause
QuickCheck to run for a much longer time. Unlike the loop approach this will
take a bounded amount of time, which makes it more suitable for something like
a release cycle that wants to really hammer your software.

### Making Arbitrary Smarter

This approach entails spending more time generating interesting
inputs in your implementations of Arbitrary. The idea is to
focus on the corner cases. This approach can be tricky because
programmers are not usually great at intuiting corner cases,
and the whole idea of property checking is to take that burden
off the programmer. Despite the theoretical discomfort, this
approach can turn out to be practical.
This approach entails spending more time generating interesting inputs in your
implementations of Arbitrary. The idea is to focus on the corner cases. This
approach can be tricky because programmers are not usually great at intuiting
corner cases, and the whole idea of property checking is to take that burden
off the programmer. Despite the theoretical discomfort, this approach can turn
out to be practical.

## Generating Structs
## Generating structs

It is very simple to generate structs in QuickCheck. Consider the following
example, where the struct `Point` is defined:
Expand All @@ -389,8 +379,8 @@ struct Point {
}
```

In order to generate a random `Point` instance, you need to implement
the trait `Arbitrary` for the struct `Point`:
In order to generate a random `Point` instance, you need to implement the trait
`Arbitrary` for the struct `Point`:

```rust
use quickcheck::{Arbitrary, Gen};
Expand All @@ -407,14 +397,15 @@ impl Arbitrary for Point {

## Case study: The Sieve of Eratosthenes

The [Sieve of Eratosthenes](https://en.wikipedia.org/wiki/Sieve_of_Eratosthenes)
is a simple and elegant way to find all primes less than or equal to `N`.
Briefly, the algorithm works by allocating an array with `N` slots containing
booleans. Slots marked with `false` correspond to prime numbers (or numbers
not known to be prime while building the sieve) and slots marked with `true`
are known to not be prime. For each `n`, all of its multiples in this array
are marked as true. When all `n` have been checked, the numbers marked `false`
are returned as the primes.
The
[Sieve of Eratosthenes](https://en.wikipedia.org/wiki/Sieve_of_Eratosthenes) is
a simple and elegant way to find all primes less than or equal to `N`. Briefly,
the algorithm works by allocating an array with `N` slots containing booleans.
Slots marked with `false` correspond to prime numbers (or numbers not known to
be prime while building the sieve) and slots marked with `true` are known to
not be prime. For each `n`, all of its multiples in this array are marked as
true. When all `n` have been checked, the numbers marked `false` are returned
as the primes.

As you might imagine, there's a lot of potential for off-by-one errors, which
makes it ideal for randomized testing. So let's take a look at my
Expand Down Expand Up @@ -450,8 +441,8 @@ sieve(5) => [2, 3, 5]
sieve(8) => [2, 3, 5, 7, 8] # !!!
```

Something has gone wrong! But where? The bug is rather subtle, but it's an
easy one to make. It's OK if you can't spot it, because we're going to use
Something has gone wrong! But where? The bug is rather subtle, but it's an easy
one to make. It's OK if you can't spot it, because we're going to use
QuickCheck to help us track it down.

Even before looking at some example outputs, it's good to try and come up with
Expand Down Expand Up @@ -515,11 +506,11 @@ A `2*p..n` range will never yield `4` when `n = 4`. When we change this to
`2*p..n+1`, all tests pass.

In addition, if our bug happened to result in an index out-of-bounds error,
then `quickcheck` can handle it just like any other failure—including
shrinking on failures caused by runtime errors.
then `quickcheck` can handle it just like any other failure—including shrinking
on failures caused by runtime errors.

But hold on... we're not done yet. Right now, our property tests that all
the numbers returned by `sieve` are prime but it doesn't test if the list is
But hold on... we're not done yet. Right now, our property tests that all the
numbers returned by `sieve` are prime but it doesn't test if the list is
complete. It does not ensure that all the primes between `0` and `n` are found.

Here's a property that is more comprehensive:
Expand All @@ -530,8 +521,8 @@ fn prop_prime_iff_in_the_sieve(n: usize) -> bool {
}
```

It tests that for each number between 0 and n, inclusive, the naive primality test
yields the same result as the sieve.
It tests that for each number between 0 and n, inclusive, the naive primality
test yields the same result as the sieve.

Now, if we run it:

Expand All @@ -542,7 +533,7 @@ fn main() {
}
```

we see that it fails immediately for value n = 2.
We see that it fails immediately for value n = 2.

```text
[quickcheck] TEST FAILED. Arguments: (2)
Expand All @@ -557,13 +548,13 @@ passing.
I think I've captured the key features, but there are still things missing:

- Only functions with 8 or fewer parameters can be quickchecked. This
limitation can be lifted to some `N`, but requires an implementation for each
`n` of the `Testable` trait.
limitation can be lifted to some `N`, but requires an implementation for each
`n` of the `Testable` trait.
- Functions that fail because of a stack overflow are not caught by QuickCheck.
Therefore, such failures will not have a witness attached
to them. (I'd like to fix this, but I don't know how.)
Therefore, such failures will not have a witness attached to them. (I'd like
to fix this, but I don't know how.)
- `Coarbitrary` does not exist in any form in this package. It's unlikely that
it ever will.
it ever will.
- `Arbitrary` is not implemented for closures. See
[issue #56](https://github.com/BurntSushi/quickcheck/issues/56)
for more details on why.
[issue #56](https://github.com/BurntSushi/quickcheck/issues/56) for more
details on why.