BPPRC database repository for GitHub actions

The pesticidal protein database is part of the Bacterial Pesticidal Protein Resource Center (BPPRC), which is under development. This database is intended to replace and extend the current Bacillus thuringiensis nomenclature site.

The database currently contains proteins listed in the Bt nomenclature site but with new mnemonics to reflect assignment of proteins to different homology groups. New bacteria-derived proteins with pesticidal properties are to be added.

In addition to the database, the BPPRC will contain links to additional information about these proteins, as well as applications to allow for analysis and comparison between proteins.

The development team is composed of:

Suresh Pannerselvam¹ , Neil Crickmore ² , Colin Berry ³, Thomas Connor³, Ruchir Mishra¹ and Bryony C. Bonning^{1   1} Department of Entomology and Nematology, University of Florida, USA ² School of Life Sciences, University of Sussex, UK ³ School of Biosciences, Cardiff University, UK

This is the source code of BBPRC 2021 website developed in Python/Django. To run the website locally, you need to install Django and a list of other Python packages which are listed in the requirements.txt file. In addition, you can add an .env file to provide SECRET_KEY and other required softwares path (needle/blastp etc). You can see this example.

SECRET_KEY='un!=5p(zdcsel07a7awa&svy-w3g4&v3c0&i&ffxnyz*+q3y-m'
DEVELOPMENT=False
AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
CRISPY_TEMPLATE_PACK='bootstrap4'
RECAPTCHA_PRIVATE_KEY=""
NEEDLE_PATH='/usr/local/bin/'
BLAST_PATH='/usr/local/bin/'
CLUSTAL_PATH='/usr/local/bin/'
DATABASE_NAME=''
DATABASE_USER=''
DATABASE_PASSWORD=''
DATABASE_HOST='localhost'
DATABASE_PORT='5432'
DATABASE_ENGINE='django.db.backends.postgresql_psycopg2'
CSRF_TRUSTED_ORIGINS= ['camtech-bpp.ifas.ufl.edu','camtech-bpp.test.ifas.ufl.edu', 'ifs-ent-camtech2.ifas.ufl.edu']
EMAIL_BACKEND='django.core.mail.backends.smtp.EmailBackend'
EMAIL_USE_TLS='True'
EMAIL_HOST='smtp.gmail.com'
EMAIL_PORT='587'
EMAIL_HOST_USER=''
EMAIL_HOST_PASSWORD=''
DEFAULT_FROM_EMAIL='<>'

If you have git and pip installed, use this:

pip install virtualenv virtualenv env source env/bin/activate

git clone https://github.com/Amrithasuresh/BPPRC.git
cd bpprc
pip install -r requirements.txt
python manage.py migrate
python manage.py runserver

Then copy the following URL in your browser.

http://127.0.0.1:8000/

1. Celery is an open-source task queue or job queue which is based on the distributed message passing. It supports scheduling too. Here we use redis the message-broker for celery.

• How to install Celery?
  pip install celery
  

• How to run Celery from the base project directory? Here, it will be ~/database/ Django base directory.
  celery worker -A BPPRC –loglevel=info
  

• How to install the Redis server?
  wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz
cd redis-stable
make
  

• How to start Redis server?
  sudo service redis start
  

• How to check the status of Redis server?
  sudo service redis status
  

2. PostgreSQL is a powerful, open-source object-relational database system.

• How to install PostgreSQL?
  sudo apt-get install python-dev
sudo apt-get install postgresql-server-dev-9.1
sudo apt-get install python-psycopg2 - Or sudo pip install psycopg2
sudo apt-get install postgresql pgadmin3
  

• Change the default password
  sudo su
su postgres -c psql postgres
ALTER USER postgres WITH PASSWORD 'YourPassWordHere';
\q
  

• Create the database
  sudo su
su postgres -c psql postgres
CREATE DATABASE dbname;
CREATE USER djangouser WITH ENCRYPTED PASSWORD 'myPasswordHere';
GRANT ALL PRIVILEGES ON DATABASE dbname TO djangouser;
  

• Settings.py file
  DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'dbname',
'USER': 'postgres',
'PASSWORD': 'postgres',
'HOST': '',
'PORT': '',
}
}
  

3. Create virtual environment
   pip install virtualenv
virtualenv venv
source venv/bin/activate
   

• Dependencies (requirements.txt file is inside the Django base directory)
  pip install -r requirements.txt
  

4. Nginx is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. Nginx can be deployed to server dynamic HTTP content on the network using WSGI application servers (an example) and it can serve as a software load balancer. Nginx will face the outside world. It will serve media files (images, CSS, etc) directly from the file system. However, it can't talk directly to Django applications; it needs something (uWSGI) that will run the application, feed it requests from the web, and return responses. Source

The outside world (HTTP client) <-> Nginx <-> The socket <-> uWSGI <-> Python app

• Install Nginx
  sudo apt-get install nginx
  

• Nginx configuration file Location
  /etc/nginx/conf.d/virtual.conf
  

• Nginx virtual.conf file contents
  

  server {
listen 80;
server_name {your domain_name};
error_log /srv/www/database/logs/error.log;
access_log /srv/www/database/logs/access.log;
charset utf-8;
  

  location /static/ {
alias /srv/www/database/static/;
}
  

  location /media/ {
alias /srv/www/database/media/;
}
  

  location / {
add_header 'Cache-Control' 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
expires off;
#uwsgi_pass unix:/opt/uwsgi/sock/database.sock;
proxy_connect_timeout 300s;
proxy_read_timeout 300s;
proxy_pass http://127.0.0.1:8000;
include uwsgi_params;
}
  

  #root /srv/www/database_test/database/templates/extra;
#index site-down.html;
  

  }
  

• Nginx log file location
  /var/log/nginx/error.log
  

5. uWSGI is a popular web server that implements the WSGI (pronounced wiz-gee) standard. WSGI (Web Server Gateway Interface) is a software specification, uWSGI is a web server. It’s pretty common to pair Django and uWSGI since they both talk WSGI. The uWSGI server is a full featured HTTP server that is quite capable of running production web apps. However, it’s not as performant as nginx at serving static content, so it’s pretty common to see nginx sitting in front a uWSGI server. Here’s where some poor naming choices make things even more confusing. So we know WSGI is a software spec, uWSGI is a server, so what the hell is uwsgi? When it’s spelled using all lowercase letters, it refers to a binary protocol for connecting the uWSGI server to other applications. Source
   

• Installation
  sudo apt-get install build-essential python
sudo apt-get install python-dev
pip install uwsgi (in the virtual environment)
  

• uwsgi emperor configuration location and contents
  `/etc/uwsgi/emperor.ini
  

  [uwsgi]
emperor = /etc/uwsgi/vassals
uid = uwsgi
gid = uwsgi
logto = /etc/uwsgi/log/uwsgilog
  

• uwsgi vassals configuration location and contents
  /etc/uwsgi/vassals/demo.ini
  

  [uwsgi]
http = :8000
workers = 1
processes = 1
socket = /opt/uwsgi/sock/database.sock
chdir = /srv/www/database/
pythonpath = /srv/www/database/database/
home = /opt/python-virtual-env/
logto = /srv/www/database/logs/uwsgi.log
module = BPPRC.wsgi
uid = uwsgi
chmod-socket = 666
chown-socket = uwsgi
harakiri = 300
  

• uwsgi log file location
  /etc/uwsgi/log/uwsgilog
  

6. Install bioinformatics software’s
   

• How to install Needle?. It should be able to run in the terminal. Provide a full path in both production as well as local. Otherwise, in production it will be headache. This applies to all the binary external softwares.
  Download EMBOSS-6.x.x.tar.gz
Gunzip EMBOSS-6.x.x.tar.gz
tar xvf EMBOSS-6.x.x.tar.gz
  

  Compile
  cd EMBOSS-6.x.x
./configure
make
  

• Clustal Omega. Download and Install.
  Installation Instructions
  

7. How do I manage the production and local settings?
   

We have separate .env files both in the production and local settings <br />

• Production .env file location and contents
  /srv/www/database/.env
  

  SECRET_KEY='………'
EMAIL_BACKEND='django_ses.SESBackend'
  

  #Amazon key
AWS_ACCESS_KEY_ID='……..'
AWS_SECRET_ACCESS_KEY='…’
AWS_SES_REGION_NAME='us-east-1'
AWS_SES_REGION_ENDPOINT='email-smtp.us-east-1.amazonaws.com'
  

  CRISPY_TEMPLATE_PACK='bootstrap4'
CSRF_COOKIE_SECURE=True
  

  # google recaptcha
RECAPTCHA_PUBLIC_KEY="6Lc-HfMUAAAAALHi0-vkno4ntkJvLW3rAF-d5UXT"
  

  # used betweeen server and reCAPTCHA
RECAPTCHA_PRIVATE_KEY="6Lc-HfMUAAAAAI2H-DuGJKPETsB_ep3EQNKkdesC"
  

  NEEDLE_PATH='/opt/EMBOSS-6.6.0/emboss/'
BLAST_PATH='/usr/local/bin/'
CLUSTAL_PATH='/usr/local/bin/'
DATABASE_TYPE='production'
  

• Local .env file location and contents
  ~/database_test/
  

  SECRET_KEY=’’
DEVELOPMENT=True
  

  AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
  

  CRISPY_TEMPLATE_PACK='bootstrap4'
CSRF_COOKIE_SECURE=True
  

  # used betweeen server and reCAPTCHA
RECAPTCHA_PRIVATE_KEY="6Lc-HfMUAAAAAI2H-DuGJKPETsB_ep3EQNKkdesC"
  

  NEEDLE_PATH='/usr/local/bin/'
BLAST_PATH=''
CLUSTAL_PATH='/sw/bin/'
  

  DATABASE_TYPE='sqlite3'
  

• We add/change these lines to access the path in the settings.py file
  

  SECRET_KEY = os.environ.get("SECRET_KEY")
  

  if os.environ.get('DEVELOPMENT'):
DEBUG = True
else:
DEBUG = False
  

  if os.environ.get('DATABASE_TYPE') == 'sqlite3':
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'YOURDATABASENAME',
'USER': 'YOURNAME',
'PASSWORD': 'YOURPASSWORD',
'HOST': 'localhost',
'PORT': '5432',
}
}
  

  NEEDLE_PATH = os.environ.get('NEEDLE_PATH', '')
BLAST_PATH = os.environ.get('BLAST_PATH', '')
CLUSTAL_PATH = os.environ.get('CLUSTAL_PATH', '')
CRISPY_TEMPLATE_PACK = os.environ.get('CRISPY_TEMPLATE_PACK')
  

  # AWS
EMAIL_BACKEND = os.environ.get('EMAIL_BACKEND')
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_SES_REGION_NAME = os.environ.get('AWS_SES_REGION_NAME')
AWS_SES_REGION_ENDPOINT = os.environ.get('AWS_SES_REGION_ENDPOINT')
  

  # google reCAPTCHA
RECAPTCHA_PUBLIC_KEY = os.environ.get('RECAPTCHA_PUBLIC_KEY')
RECAPTCHA_PRIVATE_KEY = os.environ.get('RECAPTCHA_PRIVATE_KEY')
  

8. How do you manage the local repository and the server repository?
   

I have setup a private repository for GitHub. Push the changes from Atom editor to the GitHub. And, pull the changes from the hosted server.
git pull origin database

9. Tmux usage – Terminal multiplexer. It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.
   

   sudo apt-get install tmux
   

To start a tmux session <br />
tmux in terminal <br />

Opening a multiple tab <br />
` Ctrl + B and then C` <br />

To move among these tables hit the following keys <br />
`Ctrl + B and then n to go the next tab on the right` <br />
` Ctrl + B and then p to go to the previous tab on the left` <br />
` Ctrl + B and then {number} to go the tab with number equal to 0, 1` <br />

To detach the tmux session <br />
`Ctrl + B and then d` <br />

To close a tab <br />

Ctrl + B and then x and then hit y and Enter to confirm closing of this tab

10. Remove temporary files through cron jobs
    cat /opt/script/tmp-cleanup.sh
#!/bin/bash
Find /tmp -type f -mmin +14440 -exec rm -f {} \;