Revel is a tool for verified training in reinforcement learning. Revel offers guaranteed safety at training time using an iteratively updated shield to bound the behavior of the learner. See our NeurIPS paper for more details.

Note: all of this is written using Python 3, and we will assume in these instructions that pip and python refer to pip3 and python3 respectively.

Revel relies on several python packages which can be installed from the included requirements file:

pip install -r requirements.txt

Additionally, Revel requires Eigen and Apron. Please refer to their webpages to install these. Note that setup.py will need to be modified by replacing 'path/to/eigen' with the path to the Eigen headers after you have installed them. The relevant Apron headers and objects will need to be somewhere where your C++ compiler can find them. You will need a C++17 compliant compiler.

Once all of the dependencies are installed you can build Revel:

python setup.py install

At this point you should be ready to run the benchmarks.

The benchmarks used to evaluate Revel are available in the benchmarks folder. To execute revel on a particular benchmark:

python benchmarks/<benchmark>.py --retrain_nn --retrain_shield --nn_test --shield_test --max_episodes 1000 --episode_len 100 --safe_training --shields 5 --penalty_ratio 1.0

Running without the safe_training, shields, and penalty_ratio arguments will result in a model learned using standard DDPG without safe training. This command will generate a large log file containing the rewards and safety violations at each episode as well as information about the performance and synthesis of new shields. It will also contain a sample trajectory of the final policy. We recommend dumping the output of this command into a log file to be examined later.

For most benchmarks the above command is exactly what we used to generate results. For mountain-car.py, you should instead use episode_len 200. For obstacle.py, mid_obstacle.py, and car-racing.py you should use episode_len 200 and max_episodes 2000.

Additional hyperparameters may be changed in the args object within each benchmark file. The current settings are the ones used in the reported results.

Our CPO experiments were run using the OpenAI implementation of CPO available here.

Note that Apron has been known to have memory errors at times. In our experiments we use a fork of Apron where we have fixed a few issues, but we don't provide a link to that fork in order to maintain anonymity. I believe those issues have also been fixed in the main Apron repository now, but if the code crashes with segfaults that may be why.