About this project: This was a research project for MIT's 6.883 (Meta Learning, Fall 2020). We aim to present a novel curriculum learning framework to improve performance of learning-based combinatorial optimization models. This is an ongoing project -- our preliminary results suggest that the approach we are taking allows for the creation of test-aware curricula.

Credits to mveres01/pytorch-drl4vrp for the implementation of the Nazari paper (Nazari, Mohammadreza, et al. "Deep Reinforcement Learning for Solving the Vehicle Routing Problem." arXiv preprint arXiv:1802.04240 (2018)), and to Openmind for compute resources.

Currently, only the Traveling Salesman Problem is supported. See the src/tasks/ for details. The Vehicle Routing Problem (VRP) is the same as the mveres01 implementation, while we have implemented infrastructure wrapping for curriculum learining in TSP

• Python 3.7.5
• pytorch=1.7.0+cu101
• matplotlib=3.3.1
• ortools=8.0.8283

Run by calling python src/main.py with the appropriate flags. Documentation for flags is contained in src/main.py file. Tasks and complexity can be changed through the "task" and "nodes" flag: python src/main.py --task=vrp --nodes=10.

Note: Everything below this line is documentation from mveres01's original repository.

• Uses a GRU instead of LSTM for the decoder network
• Critic takes the raw static and dynamic input states and predicts a reward
• Use demand scaling (MAX_DEMAND / MAX_VEHICLE_CAPACITY), and give the depot for the VRP a negative value proportionate to the missing capacity (Unsure if used or not)

Left: TSP with 20 cities

Right: TSP with 50 cities

Left: VRP with 10 cities + load 20

Right: VRP with 20 cities + load 30

The following masking scheme is used for the TSP:

1. If a salesman has visited a city, it is not allowed to re-visit it.

The VRP deals with dynamic elements (load 'L', demand 'D') that change everytime the vehicle / salesman visits a city. Each city is randomly generated with random demand in the range [1, 9]. The salesman has an initial capacity that changes with the complexity of the problem (e.g. number of nodes)

The following masking scheme is used for the VRP:

1. If there is no demand remaining at any city, end the tour. Note this means that the vehicle must return to the depot to complete
2. The vehicle can visit any city, as long as it is able to fully satisfy demand (easy to modify for partial trips if needed)
3. The vehicle may not visit the depot more then once in a row (to speed up training)
4. A vehicle may only visit the depot twice or more in a row if it has completed its route and waiting for other vehicles to finish (e.g. training in a minibatch setting)

In this project the following dynamic updates are used:

1. If a vehicle visits a city, its load changes according to: Load = Load - Demand_i, and the demand at the city changes according to: Demand_i = (Demand_i - load)+
2. Returning to the vehicle refills the vehicles load. The depot is given a "negative" demand that increases proportional to the amount of load missing from the vehicle

This repo only implements the "Greedy" approach during test time, which selects the city with the highest probability. Tour length comparing this project to the corresponding paper is reported below. Differences in tour length may likely be optimized further through hyperparameter search, which has not been conducted here.

On a Tesla P-100 GPU, the following training times are observed. Results were obtained by taking the the total time for the first 100 training iterations (with respective batch sizes), and converting into the appopriate time unit. Note that for the VRP in particular, as models are relatively untrained during this time, this may be slightly inaccurate results and YMMV.

Thanks to https://github.com/pemami4911/neural-combinatorial-rl-pytorch for insight on bug with random number generator on GPU