Project for the deep learning course. The mechanics are inspired by the AlphaZero AI from DeepMind.

You can play against DeepFour here: www.juleshuisman.com

All parts needed to play games

simulation/game.py

Class used to represent a game. It is able to check if a game was won, and can encode the board state to feed it to the neural network. AlphaZero encodes the current player in a third dimension, however this made DeepFour behave differently depending on the players turn. I adjusted it to only two dimensions, where the first dimension is the (hot) encoded stones of the current player.

simulation/mcts.py

Function to perform the monte-carlo tree search. A value function is added to terminal states to speed up training of the network. Instead of simply using {-1, 1, 0} the value is scaled to how many moves were played in the game (value = (1.18 - (9 * leaf.depth / 350))), this makes DeepFour prefer fast wins over slow wins and slow losses over fast losses.

def dirichlet_noise(priors, noise_eps, dirichlet_alpha):
    return (1 - noise_eps) * priors + noise_eps * np.random.dirichlet([dirichlet_alpha] * len(priors))

Adding dirichlet noise proved to be import to prevent DeepFour from getting stuck in a feedback loop between MCTS and the neural network. By adding noise to the predicted priors of the neural network you make sure the tree search has some randomness which helps it from avoiding feedback loops.

search_depth

After a lot of experimentation I found the search depth of the MCTS was a really important parameter. The tree search improves upon the initial policy prediction of the neural network and the neural network predictions are based on the results of the MCTS. If the search depth is too shallow the MCTS cannot improve the neural network predictions which means the whole system deteriorates. I ended up with a search depth of 700.

simulation/nodes.py

These nodes represent game states and perform the MCTS functions (select, expand, backprop). An important aspect is that the value of nodes are being alternated when backpropagating (What is good for green is bad for red).

deepfour.py

The actual neural network. The structure of the network is as follows:

Overall the network is a (way) more shallow version of the AlphaGo Zero network. But as connect four is a simpler game this makes sense. The kernels were changed to 4x4 to capture 16 stones at once.

A L2 regulation value of 0.0001 is introduced to prevent the model from overfitting on the training data.

The three main processes of DeepFour

process/play.py

This process runs the self-play of DeepFour. This is run in parallel to speed up the game generation. For a MCTS search depth of 700, 3 workers on 16 threads was optimal. However, because of the search depth the game generation was still quite slow. This is because the MCTS needs to request individual board states from the network, which means this process cannot be sped up by using a GPU. AlphaZero uses distributed MCTS which allows batched requests of board states, which drastically speeds up game generation.

Each game is stored on disk for the training process, each board state is also mirrored to enrich the training data.

For me it took around 8 hours per generation (For a total of 6 generations).

process/optimize.py

This process trains the neural network. There is a memory of all previous moves (for up to 500.000) games. From this memory we take 300 random batches of 512 moves to train the network on. The network is optimized using SGD using a learning rate of 0.001 and a momentum of 0.9. I removed the learning rate schedule as we only reach 6 generations.

process/evaluate.py

To make sure the new networks are up to par, we let them dual with the previous best network. We play 30 games, if the challenger network wins more than 55% of the games it becomes the new best model.

run.py

Entrypoint to run all the different processes. All processes are logged using MLFlow.

python run.py --process self python run.py --process opt python run.py --process eval

All settings can be found in settings.json

https://web.stanford.edu/~surag/posts/alphazero.html

https://github.com/plkmo/AlphaZero_Connect4

https://towardsdatascience.com/from-scratch-implementation-of-alphazero-for-connect4-f73d4554002a

https://medium.com/applied-data-science/how-to-build-your-own-alphazero-ai-using-python-and-keras-7f664945c188

https://medium.com/@jonathan_hui/alphago-zero-a-game-changer-14ef6e45eba5

https://medium.com/oracledevs/lessons-from-alphazero-part-3-parameter-tweaking-4dceb78ed1e5

https://github.com/Zeta36/connect4-alpha-zero

http://tim.hibal.org/blog/alpha-zero-how-and-why-it-works/