Live Demo

In the online video game Dota 2, each match is a battle between 2 teams of 5 players. However, many players do not actually play this game as a team; instead they're drafted along with other individuals by the game's matchmaking system.

Video games have the unique problem that every time someone interacts with their service, there is a inherent chance for the user to lose. How do developers keep players engaged with their game? Traditionally, players are matched based on their skill level for balanced teams so even losses are fun experiences as opposed to frutrating ones. Under this system, the team for one match is not carried over to the other. This is a missed opportunity.

My project aims to reimagine matches as social interactions. Every match means a player has won with 4 other people and lost with 4 other people. These relationships can be represented as a property graph. Players can be matched based on their shared win rate with players they have played with before. This will help them build their own story when they engage with the game since they'll eventually learn the other player's playstyle, fostering teamwork while also keeping them engaged with the game.

1. Pull JSON match data from S3 and produce each JSON message to Kafka with a Python producer
2. Kafka streams program transforms the data into a format useable by Neo4j ie. deriving player pairs
3. Neo4j reads the resulting Kafka topic and updates its database
4. My front-end website queries my database to matchmake teams using Python and Dash

I setup Kafka using https://kafka.apache.org/quickstart. I put specific instructions in the kafka folder's README.

Location: kafka folder

Purpose: Downloads from S3 in chunks using GET requests and returns parseable JSON messages.

Info: Uses python 3 and depends on boto3, json, and confluent-kafka libraries. Stream.py is the part that connects to S3 and drops irrelevant fields while producer.py produces the message to Kafka. The main method is send_data.py, which connects both of these classes.

Run Instructions: You can run it by using the included Dockerfile. You'll need a config.py file with info like aws credentials and Kafka broker IP's; a template is included here.

cd dataengproject/kafka
docker build -t send_data .
docker run send_data

Or if you have the dependencies:

python3 send_data.py

Location: scala folder

Purpose: Consumes match data from the Kafka cluster and turns it into player pairs for each interaction like WON_WITH or HEALED, producing these into different topics.

Info: Uses scala 2.12 and sbt to compile.

Run Instructions: You can run it by using the included Dockerfile. Make sure to change the IP's for the bootstrap servers to the Kafka cluster by changing the property BOOTSTRAP in transform.scala.

cd dataengproject/scala
docker build -t transform .
docker run transform

Or if you want to run it locally:

cd dataengproject/scala
sudo sbt ~run

Location: neo4j folder

Purpose: Gets data from topics (node and relationship data) and stores it into the Neo4j database.

Info: Uses the Neo4j Kafka plugin (not Kafka Connect). This plugin works with Neo4j 3.5, and the plugin version I used was 3.5.5.

Run Instructions: You can run Neo4j using the run.sh script, which runs a Docker container in the foreground. It has the bolt protocol on port 7687 and the browser on port 7474. Install the plugin according to these instructions. You also need to configure the config file. There is a sample included here. When starting the neo4j database, I login to the browser and enter the contraints to make indexes.

cd dataengproject/neo4j
./run.sh

Location: redis folder

Purpose: Stores general best teams for some accounts (simulating online players), in order to cache online player queries and avoid recursing through many offline player nodes in Neo4j.

Info: Uses the redis python library and py2neo library.

Run Instructions: Run redis with the docker file for redis and expose the port 6379. Then run store.py to start storing player info. Use clear.py to wipe the database.

sudo docker run -d -p 6379:6379 redis
python store.py

Location website folder

Purpose Front-end for matchmaking selected account id's

Info: Uses Dash to connect to Neo4j. Relies on the py2neo library and dash.

Run Instructions: You can run this using the instructions here. The commands are in the run.sh

cd dataengproject/website
sudo ./run.sh