Sempo Admin Dashboard and crypto financial inclusion infrastructure with USSD, Android and NFC Payments.

Postgres

We use postgres for regular (non-blockchain) data persistance.

If you plan on using the quick setup script, be sure to install the PSQL terminal application as well.

Sempo defaults to using postgres credentials of username: postgres, password: password. We recommend you create a new user with a fresh password and CREATEDB permissions. For example: CREATE USER dev_sempo WITH PASSWORD 'superSecret' CREATEDB;.

Set these to the environment variables PGUSER and PGPASSWORD.

Redis Redis is used for passing tasks to an asynchronous worker queue

Local Test Blockchain

You can use your preferred implementation of the Ethereum Blockchain to test things locally. Our setup scripts use the v6.4.1 Ganache-CLI - other versions of Ganache may not perform as expected.

npm install -g ganache-cli@6.4.1

Cairo and Pango (Optional) PDF generation requires Cairo and Pango. On OSX:

brew install cairo && brew install pango

Python

Download and install python 3.8 and its respective pip and virtualenv. Then:

from devtools directory:

python3 -m venv venv
source venv/bin/activate
./install_python_requirements.sh

Front-End

Our frontend uses react.

from app directory:

npm install

To build and watch for changes:

npm run dev

The platform uses three kinds of config files:

• deployment specific config: things that aren't sensitive, and change on a per deployment basis
• deployment specific secrets: things that ARE sensitive, and change on a per deployment basis
• common secrets: things that ARE sensitive, and can be the same between all deployments

There's already a reasonable set of local configs in config_files/local_config.ini To create some suitable secrets quickly use the following (note that if you're using the quick setup script below, this is done for you):

from config_files directory

python generate_secrets.py

(Requires PSQL to run) If you're using a custom postgres user, remember to set the environment variables PGUSER and PGPASSWORD first!

For quick setup, use the quick setup script in devtools

From devtools directory

./quick_setup_script.sh [activation path for your python env]

For example: ./quick_setup_script.sh /venvs/sempo/bin/activate

The script will:

• Reset your local Sempo state
• Generate new secrets
• Launch Ganache and Redis
• Create an adminstrator account with email admin@acme.org and password C0rrectH0rse
• Create a reserve token and bonded token

When the script has finished running*, you can start your own app and worker processes (see next section) and continue on. *This can be a little hard to identify because ganache continues to run, but a good indicator is if Bringing Ganache to foreground is echo'd in the console

From app directory:

python ./run.py

Transaction on the blockchain are made using asynchronous workers that consume a celery task-queue.

From eth_worker directory:

celery worker -A celery_app --loglevel=INFO --concurrency=8 --pool=gevent -Q processor,celery,low-priority,high-priority

If you wish to forego installing ganache and redis, you can enable a simulator mode. What this does is bypass the ethworker and any queued jobs, and instead returns dummy responses to any functions relying on eth_worker. _Be warned, this will make your database fall out of sync with any ganache instance you have set up so use this with care, but it is very useful in eliminating dependencies when working on any features in the API or frontend. It also allows you to run contract_setup_script.py without additional dependencies.

To enable simulator mode, open /config_files/local_config.ini and add the line enable_simulator_mode = true under the [APP] heading.

Transaction on the blockchain are made using asynchronous workers that consume a celery task-queue. If you are using ganache, the following command will launch ganache in a compatible manner.

ganache-cli -l 80000000 -i 42 --db './ganacheDB' \
--account=[YOUR PRIVATE KEY],10000000000000000000000000"

Here:

• the -l 80000000 argument increases the gas-limit, which is required for token-exchange deployments.
• the --db argument persists your data to a local folder. This can be a little flaky, so if you get any strange errors, try deleting the contents of the ganacheDB directory and retrying.
• the --account argument creates an account with LOTS of test ether, to stop you running out too quickly. Note that you'll need to provide the 'master wallet' private key that's found within your config files.

The bash script gacnache_cli.sh can be used to run this quickly. Be sure to set the MASTER_WALLET_PK environment variable beforehand.

If it's not already, you will also need to set your local config to match:

[ETHEREUM]
http_provider           = http://127.0.0.1:8545

In order for the workers to run, you'll need to launch redis and celery. On production we run multiple workers to handle different tasks, but locally you can just launch one worker to run everything.

First launch the redis server, which acts as a message-broker for celery tasks. In terminal run:

redis-server

If you get some message about Creating Server TCP listening socket *:6379: bind: Address already in use, this is probably just because redis is already running (did you use the quick setup script). You can probably carry on!

Start celery:

From eth_worker directory

celery worker -A celery_app --loglevel=INFO --concurrency=8 --pool=gevent -Q processor,celery,low-priority,high-priority

You can also add a runtime configuration with the script path set as the path to your virtual env [path-to-your-python-env]/bin/celery.

Set the parameters to the run line above.

Lastly, if you want to run full blockchain tests, set the environment variable FORCE_BLOCKCHAIN_TESTS to True

Migrate differs slightly for the main app (uses flask-migrate version of alembic) versus the ethereum worker (uses pure alembic).

For more commands, see Alembic documentation: https://alembic.sqlalchemy.org/en/latest/

For main app:

First, setup your database sempo_blockchain_local, using the username and password from the local config file.

Next, to update your database to the latest migration file:

cd app
python manage.py db upgrade

To create a new migration:

Make the modifications to the model file to reflect the database changes.

python manage.py db migrate

Remember to commit the file!

Sometimes, branches split and you will have multiple heads:

python manage.py db merge heads

For ethereum worker:

First, setup your database eth_worker, using the username and password from the local config file.

Next, to update your database to the latest migration file:

cd eth_worker
alembic upgrade head

To create a migrations file (remember to commit the file!):

alembic revision --autogenerate

• Pull the below repo and follow steps https://github.com/enjeyw/SempoVendorMobile

(if you have installed the prod vendor app, ensure you clear data and uninstall before installing from dev)

To run and test the USSD interface locally there are two options.

• Run ./devtools/ussd.py with the necessary environment variables
• Setup ngrok and use Africa's Talking or another USSD simulator

Ensure your test_config.ini and test_secrets.ini files are up to date. Test secrets can be generated using the previous python script, and supplying test as the filename:

cd config files
python generate_secrets.py -n test

Create the test databases:

create database eth_worker_test;
create database sempo_blockchain_test;

Ensure redis-server is running (this is not ideal but necessary atm).

Then run python invoke_tests.py, or if that doesn't work, set it up as a runnable in PyCharm: Run -> Edit Configurations -> Add New Configuration (Python) -> set script path as SempoBlockchain/invoke_tests.py

Platform APIs are documented using Postman. API documentation:

• Live Docs
• JSON Collections and Environments (stored in /postman in root directory)

We use Mozilla Sops for sharing low-risk dev keys and secrets.

If you're working on a sempo-specfic deployment, contact nick or tristan to gain decryption access. You will be given an AWS account - set up your credentials locally using AWS CLI, setting the profile name to sempo inside the credentials file.

Once you've been authorised, you can access config settings by first creating a directory called config_files in the root folder, and running _SOPS_load.py inside .sempoconfig (note that you have to be in the .sempoconfig directory. This will extract all keys into the config_files folder. To encrypt them again for sharing on git, run _SOPS_save.py.

Not that SOPS doesn't handle merge conflicts currently - if you try and merge an encrypted file, it will break in a bad way!

Instead, if you need to merge in two config files, you need to save the old config, load the new one and merge them by hand.

To run with docker first you must generate local secrets that will be read by the config.py script that is copied into the app, worker and pgbouncer

cd config files
python generate_secrets.py -n local_docker
python generate_secrets.py -n docker_test
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up

You may find it useful to add an alias to your .bashrc (or .zshrc etc) that aliases the docker-compose command

alias sempo='cd /path/to/repo && docker-compose -f ./docker-compose.yml -f ./docker-compose.dev.yml'
alias sempoci='cd /path/to/repo && docker-compose -f ./docker-compose.yml -f ./docker-compose.ci.yml'