Efficacy Sampling

Colin Green, author of the SharpNEAT library, wrote a nice piece on the use of efficacy sampling to compare two versions of a software to discern improvement (or deterioration) due to changes from one to the other. His motivation rings true for me as well

The primary motivation for the development of efficacy sampling is the ongoing development of SharpNEAT; in particular as a means of evaluating proposed modifications and enhancements, but also as a high level test when making more significant changes to the software. That is, efficacy sampling can be used to ensure that a newer code base continues to work at least as well as previous versions of the software, and that no inadvertent subtle defects have been introduced.

As I add features to and make other changes to the evo library, this same tool will ensure I appreciate the impact of whatever decisions I make. To that end I added a package, efficacy, to the evo project and updated the examples to utilise the Sampler helper which records the final state of the experiment at the end of each batch run. Inside efficacy there is a application that reads in the result files and can produce histograms.

For example, here I use these new tools to show the improvement in xor by switching to the ReLU activation function from SteependSigmoid. I begin by running two batches of the experiment with different configurations, being sure to save the output to separate files:

cd $GOPATH/src/gitub.com/klokare/evo/neat/examples/xor
go run *.go --runs 1000 --config xor.json --output $HOME/Downloads/steepened-sigmoid.json
go run *.go --runs 1000 --config xor-relu.json --output $HOME/Downloads/relu.json

1000 runs of the SteepenedSigmoid version took 6 minutes on my tiny Digital Ocean droplet. The ReLU version, as you will see below, went faster, taking only 3 minutes and 18 seconds.

Once the result files are created, use the histogram application to generate the images:

cd $GOPATH/src/gitub.com/klokare/evo/efficacy/cmd/histogram
go run *.go --field seconds --output $HOME/Downloads/xor-seconds.png $HOME/Downloads/steepened-sigmoid.json $HOME/Downloads/relu.json
go run *.go --field encoded --method mean --output $HOME/Downloads/xor-complexity.png $HOME/Downloads/steepened-sigmoid.json $HOME/Downloads/relu.json

NOTE: normally one would explore the fitness differences between the two versions but the xor experiment finds a solution in very nearly every case so comparing fitness does not provide any additional insight.

You can see from both charts that the ReLU version not only executes, on average, in less time but also produces smaller networks. The purple part of the columns is the space shared by both series, shown separately here:

Some notes:

• The histogram application makes use of the plot from the kind folks at gonum.
• If your results contain cases where some are solved and some are not, use the --solved flag to exclude the failures in the comparison