Flask applications for accessing eRSA reporting databases.

The package only works with PostgreSQL because it uses:

• UUID
• INET
• MACADDR

0. Modify prepare_db.sql.demo and save as prepare_db.sql then run sudo -u postgres psql -f prepare_db.sql
1. Create config.py file
2. Activate env
3. Create tables from either command:

bin/ersa_reporting-prep ersa_reporting.PACKAGE
# or
export APP_SETTINGS=config.py
bin/unified-apis-prep PACKAGE

The package can be served by, for example, nginx (proxy) + gunicorn.

If deploy to a CentOS 7 cloud instance, script can be used to prepare the instance with nginx and gunicorn if one application is being deployed. For multiple application deployment, run service_generator.sh and then gconf_generator.sh. To finish off, in /usr/lib/ersa_reporting, create a config-PACKAGE.py file described in this document and example.

Access and error logs of gunicorn of an application are saved under /var/log/gunicorn/.

An application's log is named as application.log, e.g. 'hnas.log' in /usr/lib/ersa_reporting. Where this log is saved can be configured in config-PACKAGE.py with key LOG_DIR.

Example of gunicorn configuration file generated by gconf_generator.sh:

timeout = 7200
proc_name = PACKAGE
workers = 2
PDIRfile = "/run/gunicorn/PACKAGE.PDIR"
raw_env = ["APP_SETTINGS=config-PACKAGE.py"]
accesslog = "/var/log/gunicorn/PACKAGE_access.log"
errorlog = "/var/log/gunicorn/PACKAGE_error.log"
loglevel = "info"

The package can be served by, for example, nginx (proxy) + gunicorn.

If deploy to a CentOS 7 cloud instance, script can be used to set up nginx and gunicorn.

The log of an application is currently hard-coded to be saved in /var/log/gunicorn/ which assumes gunicorn is configured to save logs to there.

An application's log is named as ersa_reporting.application.log, e.g. 'ersa_reporting.hnas.log'.

This package contains multiple applications. Main package ersa_reporting expects config.py to have relevant settings:

• SQLALCHEMY_TRACK_MODIFICATIONS
• SQLALCHEMY_DATABASE_URI
• ERSA_REPORTING_PACKAGE
• ERSA_AUTH_TOKEN

The packge needs to talk to a database so there is a configuration file for gunicorn to use which is listed below in the example of hnas.conf. It also can be generated by a script like gconf_example. Each web api application served by gunicorn can be configured by a conf file like this:

raw_env = ["ERSA_REPORTING_PACKAGE=hnas", "ERSA_DEBUG=True",
           "ERSA_DATABASE_URI=postgresql://user:pass@host/db",
           "ERSA_AUTH_TOKEN=DEBUG_TOKEN"]
timeout = 7200
proc_name = "hnas"
workers = 2
pidfile = "/run/gunicorn/hnas.pid"
accesslog = "/var/log/gunicorn/hnas_access.log"
errorlog = "/var/log/gunicorn/hnas_error.log"
loglevel = "info"

Applications in unified get their configurations from the file set in environment variable APP_SETTINGS. See example for common settings of an API application. For other applications, they can have different keys.

A basic config.py should have these keys:

• SQLALCHEMY_TRACK_MODIFICATIONS
• SQLALCHEMY_DATABASE_URI
• ERSA_REPORTING_PACKAGE
• ERSA_AUTH_TOKEN (optional for development)
• DEBUG = True (optional, make Flask give more error messages)

To interact with the package:

import time
import os
os.environ['APP_SETTINGS'] = 'config-xfs.py'
from unified.models.xfs import Snapshot, Filesystem, Usage, Owner, Host
start = time.time()
x = Snapshot.summarise(1452930321, 1452676846)
end = time.time()
print(end - start)

To serve an application for testing:

# run dev instance by Flask
export APP_SETTINGS=config-xfs.py
export FLASK_APP=unified/apis/xfs.py
flask run -h 0.0.0.0 --reload

# run dev instance by Gunicorn
gunicorn -e APP_SETTINGS=config-xfs.py --access-logfile - -b 0.0.0.0:5000 unified.apis.xfs:app

The script needs boto package installed.

This script is for generating bills directly. It has a few custom Flask commands. For generating Nova bills, it needs credential for accessing Nectar Keystone (not from reporting database). As with other API applications, the config.py should have all keys in those applications, and at least these two keys:

NECTAR_USER = "username"
NECTAR_USER_PASS = "password"

Default config is config.json. You can use other by calling with --conf. The structure of config.json looks like this:

{
  "Nova": {
    "APP": "comm-config.py",
    "workers": 3,
    "crm": { "url": "crm" }
  },
  "Hpc": {
    "APP": "hpc-config.py",
    "crm": { "url": "crm" }
  }
}

There are two ways to define a time period for usage calculation:

1. by year and month

python calculator.py -y 2016 -m 8 nova

2. or by start and end:

python calculator.py -s 20160801 -e 20160831 nova

Tests are run through unittest package. They need a running API application to test against and auth_token to connect to it. auth_token can be the development server token set in config file set by APP_SETTINGS by ERSA_AUTH_TOKEN or a valid token in other set ups.

python -m unittest usage/tests/test_calculator.py
python -m unittest usage.tests.test_calculator.TestUsages.test_filename_of_usage_save
python -m unittest unified.models.tests.test_xfs
python -m unittest unified.tests.test_xfs