Balanced Incomplete Block Design (BIBD) is the core technique in this project.

bibd_visualization.ipynb

Depthwise (channel) sparsification

Three models:

• MLP
• MLP with BIBD
• Random sparse MLP

You can check the live Jupyter Notebook here: mlp_bibd_experiments.ipynb

The experiments are performed under combinations of the following three settings, which are orthogonal to each other.

• Linear layers

  • No sparsification
  • BIBD
  • Random sparsification

• Convolutional layers

  Depthwise sparsification (channel)

  • No sparsification
  • BIBD
  • Random sparsification
  • Grouped convolution

• No vertical partitioning
• Vertical partitioning

• No early exiting
• Early exiting

Ensemble learning techniques are applied to improve inference accuracy.

• Ensemble learning
• Without ensemble learning

• Training loss, test accuracy per epoch
• Training time, inference time
• # of parameters, FLOPs (inference stage)

See setup.md

iddl_experiments.pdf

Run the experiment

$ conda activate iddl_env
$ cd mlp
$ ./run_mlp_experiments.sh

• Log file: ./mlp/log/mlp_experiments_${BEGIN_TIME}.log
• PID file: ./mlp/log/mlp_experiments_${BEGIN_TIME}_pid.txt
• Dumped file: ./mlp/mlp_experiments_${BEGIN_TIME}.p can be unpicked as an instance of Experiment class

Plot the results

$ conda activate iddl_env
$ cd mlp
$ python plot_mlp_experiments.py

• Figures: ./mlp/mlp_experiments.png, ./mlp/mlp_experiments.eps, ./mlp/mlp_experiments_sparse_only.png, ./mlp/mlp_experiments_sparse_only.eps
• Log: ./mlp/log/mlp_experiments_plot_{BEGIN_TIME}.log

Results

Run the experiments

$ conda activate iddl_env
$ cd resnet
$ bash run_resnet_experiments.sh

• Log file: ./resnet/log/resnet_experiments_${BEGIN}.log
• ./resnet/accuracy_array_{BEGIN}.pkl: The pickled file of the array of test accuracy of type numpy.array.
• ./resnet/model_name_array_{BEGIN}.pkl: The pickled file of the array of model names of type numpy.array.

Plot the results

$ conda activate iddl_env
$ cd resnet
$ python plot_resnet_experiments.py

• ./resnet/fig_resnet_experiments.eps, ./resnet/fig_resnet_experiments.png
• ./resnet/fig_resnet_experiments_sparse_only.eps, ./resnet/fig_resnet_experiments_sparse_only.png: Only include the legends of sparse models: B-ResNet and R-ResNet.

Results

Implemented using grouped convolution.

Early exiting.

Run the experiments

$ cd resnet
$ conda activate iddl_env
$ bash ./run_ensemble_experiments.sh 1
$ bash ./run_ensemble_experiments.sh 2

• Log file: ./resnet/log/ensemble_experiments_${INDEX}_${BEGIN}.log
• PID file: log/ensemble_experiments_${INDEX}_${BEGIN}.pid
• Dumped file: ensemble_result_${INDEX}_${BEGIN}.pkl

Plot the results

$ cd resnet
$ conda activate iddl_env
$ python plot_ensemble_experiments.py

• ./resnet/fig_ensemble_experiments.png, ./resnet/fig_ensemble_experiments.eps
• ./resnet/fig_ensemble_experiments_improved.png, ./resnet/fig_ensemble_experiments_improved.eps

Please follow the instructions if you want to contribute to this repository: contribution_guide.md

• Google Python Style Guide