Light weight netlify-like server-side CI build system using EC2 instance

• No docker.
• Some IAM policies(DescribeInstances, StartInstances, StopInstances) are needed.
• At least, two ec2-instances are needed.
• One MySQL like database is needed.

MIT

create PR (on GitHub) -> start EC2 instance automatically -> happy!

We made some workflow using GitHub and GitHub flow:

For implement this flow, we need some system that automatically create staging environment as soon as pull request is made. Unfortunately, we had issues:

• Database Scheme. The developer's database is ready to test the functions, but the database of staging environment is not ready to test.
• We couldn't use docker for some tests.

This application is the solution of these issues. The system configuration is here.

Create EC2 instances. And allow port 5250 for the one of them. The one is called "Controller Instance". The others are called "Staging Instance".

Create IAM policy that allowed following actions:

"ec2:DescribeInstances" "ec2:StartInstances" "ec2:StopInstances"

The resources of policy are instances you created. And attach the policy to user/role, then save the ACCESS KEY and SECRET KEY.

Install git for the whole instances. note: if you use amazon linux2,

$ sudo yum install git

Setup your application to staging servers.

Install python3.8 for the whole instances. note: if you use amazon linux2,

$ sudo amazon-linux-extras install python3.8
$ sudo yum install python38-devel
$ sudo yum install gcc
$ sudo yum install gcc-c++

Install mysql-client for the whole instances. note: if you use amazon linux2,

$ sudo yum install mysql

Run or install some MySQL like database.

note: if you want to install mysql-server,

$ sudo yum localinstall https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm
$ sudo yum install mysql-community-server
$ sudo service mysqld.service restart
$ sudo cat /var/log/mysqld.log
(watch temporary password)
$ mysql -u root -p
(input temporary password)
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'ragp)!WJ(3UC';
('ragp)!WJ(3UC' is the password... you may change it as you like)
CREATE DATABASE pullre CHARACTER SET utf8mb4;
\q
$ sudo vi /etc/my.cnf
(add folowings at the last of file)
default-authentication-plugin=mysql_native_password

Clone pullre-kun.

Install the requirements.

Controller instance has basic authentication. You should create the token(hash) of password, and save it to app.ini.

Create app.ini file. The sample is app.ini.default. And deploy it for the whole instances.

Add following line to crontab of controller server.

* * * * * cd /home/ec2-user/pullre-kun; python3.8 update_pull.py

Add following line to crontabs of staging servers.

* * * * * cd /home/ec2-user/pullre-kun; python3.8 client.py

Execute following command at controller server.

$ cd ~/pullre-kun
$ python3.8 init.py

Execute following command at controller server.

$ cd ~/pullre-kun
$ nohup python3.8 app.py &

Access https://(your-domain)/server/list, then you would see the whole servers. Then you click register button of staging servers. And access https://(your-domain)/master/server, and update the db_schema of each record.

Access https://(your-domain)/master/git_hub_users, and register the users. "login" is github user login, db_schema is the original schema of clone.

manual alembic(old version)

Create alembic.ini file. The sample is alembic.ini.default. Usually, you edit database connection string only.

Execute following command at controller server.

$ export PYTHONPATH=/home/ec2-user/pullre-kun
$ cd /home/ec2-user/pullre-kun/alembic
$ alembic revision --autogenerate -m update
$ alembic upgrade head