DeepIceLearning - A software package to use Deep Neural Networks for Reconstructions and Classifications in IceCube

This software package is designed to make Deep Learning applications in IceCube as easy as possible. It contains a set of scripts to generate training datasets from i3 files, train Deep Neural Networks and apply them to data.

The main functionalities are provided by essentially three scripts and their corresponding submit files in the /submit_scripts folder. Addtional configurations for the training and dataset defintions can be done using the config files in /config. Neural networks are defined using the Keras functional API (see examples in /networks/). In addition to the Keras model definition python dictionaries in the network file are used in order to define the output and input features as well as transformations applied on them.

This file converts a set of i3files into training data. Training data files consist of 3d input tensors in the format 10x10x60 for IceCube and 3x5x60 for Deep core. The values to be stored in the tensor are defined in a config file (see /configs/create_dataset_default.cfg), classical examples are time at which x% of the charge are collected, the overall number of hits, total collected charge, charge after x ns, .. Also a set of reco_vals can be defined, i.e. quantities that are later used as target in the training of the network or for analyzing the network's performance.

In order create a MC dataset run something like

python create_dataset_dagman.py --dataset_config file_to_dataset_config_file

or add the --data option if you are using real data

The corresponding submit file for HTCondor can be used similarly

python create_dataset_dagman.py --dataset_config config_file_for_the_dataset --files_per_job num_i3files_per_job

This is the main script for training a network. As command line arguments it needs a config file that sets the hyperparameters and filepath for the training and a network definition file. Most of the network definition is done using standart Keras. Additionall transformations to the input and output data, as well as loss function settings can be defined as well. Compare the examples in ./networks/

An example to run the training is

bash nn_env.sh --main_config path_to_config_file --input all --model path_to_model_file

or when sending the job to a (multi-gpu) cluster. Run from within the submit_scripts folder

python submit_training.py --main_config path_to_config_file --input all --model path_to_model_file

Once a network is trained one might want to apply it to a set of MC data. For this just run something like

bash apply_env.sh --folder path_to_trained_network_folder --test_data path_to_the_test_data

If you want to apply it to 'real' event data use instead

bash apply_env.sh --folder path_to_trained_network_folder --data data_file(s)