Progressive Neural Net based on DQN

Code for a simplified version of paper Progressive Neural Networks, on top of the DQN architecture. Trains a DQN for a new game based on several frozen DQN models using the progressive neural network training architecture. This is a simplified version of Progressive Netowork where transfer is based on the following formula and just using the fully connected layers.

We have used this implementation for our proposal on a nested architecture which combines Progressive Neural Network and Actor-Mimic to provide better transfer learning for training an Atari agent. More details may be found in our technical report.

Installation

1. Install torch (http://torch.ch/docs/getting-started.html#_).
2. Run ./install_dependencies.sh to install xitari and alewrap.
3. Find Atari 2600 ROMs and put them in roms/ directory.

Experiments

NOTE: training the full pipeline from DQNs to transfer using Progressive architecure may take about two weeks with a fast GPU.

Training DQN

You may use the code available here to train the DQN and generate t7 torch files. Then, put the t7 files in experts/ directory.

Training Progressive network

To train progressive network, run following script:

$ cd scripts
$ ./run_progressive [1-based gpuid] ["'SourceGame_1','SourceGame_2','SourceGame_3'"] ["'TargetGame'"] [OutputFile]

Here, the first parameter is the GPU id, the second parameter is a list of source games (that is, the ones with frozen DQN networks), the third parameter is the Target game which we want to train a DQN for that, and the last parameter is the output file name. An example is:

$ ./run_progressive 0 "'pong','breakout'" "'tennis'" tennis_progressive

Acknowledgments

This implementation is an adapatation of Actor-Mimic, which provides code for the Actor-Mimic deep multitask and transfer reinforcement learning.

References

1. Human-level control through deep reinforcement learning, Nature 2015
2. Progressive Neural Networks, arXiv preprint 2016
3. Actor-Mimic deep multitask and transfer reinforcement learning, ICLR 2016
4. A Nested Architecture to Improve Transfer Learning in Deep Reinforcement Learning, Technical Report 2017