Resources and code from my Summer 2017 REU with Dr. Anthony Maida
My reseach was focused on creating a Deep Learning neural network that can predict object movement from video input. The model created uses a LSTM model that utilizes convoloution to process images, called a convLSTM. The purpose of this research was to recreate a model similar to that of PredNet. It uses an unsupervised convLSTM model that trains off a variety of video data, but is much to complicated for this summer. I helped create a simpler convLSTM model that can detect 2D movement of a black sphere on a white background. This model is basic, but can be expanded by the addition of more featuremaps and slight code reshaping.
During this summer REU, I worked alongside Dr. Maida and his lab group. We had weekly lab presentations where we would showcase our progress. I started off with reading the papers linked below and Google's Deep Learning course on Udacity. The convLSTM model created was primarily dictated by Dr. Maida, but went through individual updates as the research group progressed.
- Created a Blender python script to automate video creation.
- Created a script to load the created animations into ndarrays of size [frame_count, 256, 256].
- Created a method to load the each animation ndarray into a .pickle serialized file for use in the convLSTM.
- Optimized the learning rate by adding an exponentially decaying learning as opposed to a static rate.
- Created a new optimizer block to improve loss from ~800 to ~130 utilizing tf.clip_by_global_norm() to handle exploding or vanishing gradients.
- Added weight visualizations by splitting the multi channel tensors along their first axis and added code to visualize the weight matrixes in TensorBoard.
- Created a new model that uses the convLSTM model as a class, able to feed multiple animations and return the new trained weights (loss increased, but predictions became more accurate).
- Various bug fixes during development.
- Download the contained .blend file
- Make sure Blender is installed (from here)
- Double click the .blend file, or open it in Blender with File -> Open..
- Edit the animation output location (line 49) (Default: /tmp/anims)
- Navigate to the bottom of the python code window
- Find the button labeled Run Script above the console.
- Click the Run Script button
- Wait some time, this process may take a while. Blender will freeze but will continue working.
This python script will take the animations created, push them into 3-D arrays of size (frame_count,image_length,image_width), and create a pickle file for each animation.
- Place this file in the folder containing the animation folders (Default: /tmp/anims/load.py)