This repository is a collection of commonly used Keras examples adapted to be used for profiling the performance of different Keras backends, mainly Apache MXNet (You could change the backend to TensorFlow). These performance profiling data may not be fully accurate. However, this should help in giving a relative insights into various performance metrics - training time/speed, accuracy, memory consumption, with different Keras backends.
You can use these examples to profile on CPU or GPU machine. You can train and profile with multiple GPUs for MXNet and Tensorflow Backends.
Install the following NVIDIA libraries on your GPU machine:
- Install CUDA 8.0 following the NVIDIA's installation guide.
- Install cuDNN 5 for CUDA 8.0 following the NVIDIA's installation guide. You need to register with NVIDIA for downloading the cuDNN library.
Note: Make sure to add CUDA install path to LD_LIBRARY_PATH.
For Example, if you have downloaded CUDA debian package (cuda-repo-ubuntu1604_8.0.61-1_amd64.deb) and cuDNN 5.1 library (cudnn-8.0-linux-x64-v5.1.tgz), below are set of commands you run to setup CUDA and cuDNN.
# Setup CUDA 8.0.
$ sudo apt-get update
$ sudo apt-get install build-essential
$ sudo apt-get install linux-headers-$(uname -r)
# Assuming you have downloaded CUDA deb package from https://developer.nvidia.com/cuda-downloads
$ sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
$ sudo apt-get update
$ sudo apt-get install cuda
$ export CUDA_HOME=/usr/local/cuda-8.0
$ PATH=${CUDA_HOME}/bin:${PATH}
$ export PATH
$ export LD_LIBRARY_PATH=/usr/local/cuda/lib64/:$LD_LIBRARY_PATH
# Setup cuDNN 5.1 for CUDA 8.0.
# Assuming you have registered with NVIDA and downloaded cuDNN 5.1 for CUDA 8 from https://developer.nvidia.com/cudnn
$ tar -xvzf cudnn-8.0-linux-x64-v5.1.tgz
$ sudo cp cuda/include/cudnn.h /usr/local/cuda/include/
$ sudo cp cuda/lib64/* /usr/local/cuda/lib64/You can verify your CUDA setup with following commands.
$ nvcc --version
$ nvidia-smiInstall Prerequisites with following commands.
# Install git
$ sudo apt-get update
$ sudo apt-get install -y build-essential git
# Install pip
$ sudo apt-get install -y wget python
$ wget https://bootstrap.pypa.io/get-pip.py && sudo python get-pip.py$ sudo pip install numpy
# Dependencies required by Keras
$ sudo pip install nose
$ sudo pip install nose-parameterized
# Dependencies required for profiling
$ sudo pip install memory_profilerInstall latest MXNet from source. Below instructions derived from MXNet install guide.
# BLAS library
$ sudo apt-get install -y libopenblas-dev
# OpenCV
$ sudo apt-get install -y libopencv-dev
# Download MXNet sources and build MXNet core shared library
$ git clone --recursive https://github.com/dmlc/mxnet
$ cd mxnet
$ make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1
# Install MXNet Python bindings
$ sudo apt-get install -y python-dev python-setuptools python-numpy
$ cd python
$ sudo python setup.py install$ sudo pip install tensorflow-gpuReference - TensorFlow install guide.
DMLC's fork of Keras has MXNet backend. For more details about known issues, not supported functionalities see here
$ git clone https://github.com/dmlc/keras.git ~/keras --recursive
$ cd ~/keras
$ sudo python setup.py installFirst, you clone this repository to your test machine.
$ git clone --recursive https://github.com/sandeep-krishnamurthy/keras-mxnet-benchmarksYou use 3 environment variables - KERAS_BACKEND, MXNET_KERAS_TEST_MACHINE and GPU_NUM to control the backend and machine set up for running the profiling.
Below are allowed values for each of these environment variables.
- KERAS_BACKEND - "mxnet" or "tensorflow".
- MXNET_KERAS_TEST_MACHINE = "CPU" or "GPU".
- GPU_NUM = Integer. Number of GPUs to be used for profiling. Ex: GPU_NUM = 8, will use 8 GPUs.
To run RESTNET50 profiling with MXNet backend using 4 GPUs, you run the following commands.
First, set the appropriate environment variables.
# Change "mxnet" to "tensorflow" for profiling with TensorFlow backend.
$ export KERAS_BACKEND=mxnet
$ export MXNET_KERAS_TEST_MACHINE=GPU
$ export GPU_NUM=4Run the profiling python script. You will the profiling results on the terminal.
$ python keras-mxnet-benchmarks/keras1.2/test_cifar_resnet.py- Resnet50 Keras code is borrowed from - raghakot/keras-resnet
- Most of the examples are borrowed from - fchollet/keras/examples