If doexp crashes, all experiments are killed.

[krzentner]

This is somewhat convenient, since Ctrl-C will stop all running experiments. However, unintentional doexp crashes could be disastrous. Ideally doexp would double check before Ctrl-C closes it (like jupyter does), and have some mechanism to reliably avoid or recover from crashes.

I've considered splitting doexp into one script that syncs exps.py with a sqlite database, and another that starts up scripts using the contents of that database, to prevent sys.exit or infinite loops in exps.py taking down all the experiments.