Common Feature Learning

Official implementation of Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning (IJCAI 2019) in pytorch.

Results

Teacher Performance

Teacher Model	Dataset	num_classes	Acc
ResNet18	CUB200	200	0.7411
ResNet34	StanfordDogs	120	0.8663

Student Performance (CUB200+StanfordDogs)

Target Model	KD	CFL
ResNet34	0.7684	0.7721
ResNet50	0.7965	0.7997
DenseNet121	0.7769	0.7815

see logs for more information

Accuracy Curve

TSNE Visualization of 20 Classes

Feature Space: space constructed with intermediate outputs.
Common Space: common feature space in CFL Blocks.

Some Feature spaces are None because of different feature dimensions (e.g. 2048 for ResNet50 but 512 for ResNet34)

Target Model	Common Space	Feature Space
ResNet34
ResNet50		None
DenseNet121		None

Quick Start

1. Download Datasets

python download_data.py

2. Get Trained Teacher Models

ResNet18 & ResNet34, 242.9 MB
Google Drive
BaiDu Yun

3. Train

python amal.py --model resnet34 --gpu_id 0 --lr 1e-4 --cfl_lr 5e-4
python kd.py --model resnet34 --gpu_id 0 --lr 1e-4

or

bash run_all.sh

4. Draw Accuracy Curve

cd logs/
python draw_acc_curve.py

5. TSNE

TSNE results will be saved at tsne_results/MODEL_NAME/

# ResNet34
python tsne_common_space.py --ckpt checkpoints/amal_resnet34_best.pth --t1_ckpt checkpoints/cub200_resnet18_best.pth --t2_ckpt checkpoints/dogs_resnet34_best.pth --gpu_id 0

# ResNet50
python tsne_common_space.py --ckpt checkpoints/amal_resnet50_best.pth --t1_ckpt checkpoints/cub200_resnet18_best.pth --t2_ckpt checkpoints/dogs_resnet34_best.pth --gpu_id 0

# DenseNet121
python tsne_common_space.py --ckpt checkpoints/amal_densenet121_best.pth --t1_ckpt checkpoints/cub200_resnet18_best.pth --t2_ckpt checkpoints/dogs_resnet34_best.pth --gpu_id 0

Citation

@inproceedings{luo2019knowledge,
  title={Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning},
  author={Luo, Sihui and Wang, Xinchao and Fang, Gongfan and Hu, Yao and Tao, Dapeng and Song, Mingli},
  booktitle={Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)},
  year={2019},
}