Application for geo-spatial data collection and processing.

• Python 3.2+

Note: To use geo-features for django (GDAL, GEOS, PROJ), you need to have set env variable: BUILD_WITH_GEO_LIBRARIES=1 see: https://help.heroku.com/ONEQG0L6/how-do-i-enable-geo-libraries-e-g-geodjango-for-use-with-python

• docker (https://www.docker.com/community-edition#/download)
  • Linux post-install: enable Docker for non-root user
• docker-compose (https://docs.docker.com/compose/install/)

both installed on localhost.

• run

git clone git@github.com:gis4dis/poster.git
cd poster

# copy exapmle.env as .env - it should have reasonable defaults.
cp example.env .env

• clone any version of your choice of Luminol:

cd src/
git clone git@github.com:gis4dis/luminol.git
cd -

• If you have PostgreSQL (or something else) running at local port 5432, stop it.

• run

# Run the whole cluster on localhost { Django + Celery + Redis + Postgis + (Mongo) }.
# The first run will also runs `docker-compose build` automatically, which builds the "web" docker container and downloads other containers.
docker-compose up

# Make django migrations (creates DB structures).
./dcmanage.sh migrate
# Windows: docker-compose run --rm poster-web python manage.py migrate

# Create Django superuser
./dcmanage.sh createsuperuser
# Windows: docker-compose run --rm poster-web python manage.py createsuperuser

• Login with superuser at http://localhost:8000/admin

• You can also import and aggregate some data

# import some meteorological data
./dcmanage.sh ala_import

# aggregate it
./dcmanage.sh aggregate_observations

You can check imported data at http://localhost:8000/admin/ala/observation/. Notice that aggregation is computed in the background. You can check the state at http://localhost:5555/tasks.

(Re)builds the "web" docker container.

This will (re)build the web docker image with new requirement.txt file

For more information see Dockerfile.

This command removes created temporary volumes that were attached to docker containers.

This command removes created named volumes, e.g. the database.

When the docker containers are not properly shut, the attached volume(s) can in some cases stay detached but created and running docker-compose up will end with error, complaining about "same volume names".

Taken from man tee page:

|tee - read from standard input and write to standard output and files

This command will send output (logs) of all docker containers to:

• terminal
• file docker.log

You can use this log file to further examine and filter logs with your favorite tool. (grep, less, ...) The logfile will get erased on every new run (feature).

This will open docker.log file in less (-R flag preserves the terminal colors). Then you can you in-build filter function to obtain only the logs you are interested in.

This will display last 20 log messages from poster_celery_worker service. You can find the service names in docker-compose.yml file as well as in the log prepended on each line.

If you omit the --tail # flag it will display all logs from all previous runs. Use it with caution.

Replace for proper variable, this will also prompt for DB password

pg_dump -h <HOST> -p <PORT> -U <USER> -W --format=custom --no-acl --no-owner <DB_NAME> > (date -I)_poster.dump

psql postgres://postgres:postgres@localhost:5432/postgres < /path/to/dump

Buildpacks are used when building slug for deployment. Below are some examples.

https://github.com/heroku/heroku-buildpack-apt
https://github.com/mojodna/heroku-buildpack-gdal.git
https://github.com/cyberdelia/heroku-geo-buildpack.git
https://github.com/heroku/heroku-buildpack-python.git#009d0ddb
https://github.com/heroku/heroku-buildpack-python.git
https://github.com/weibeld/heroku-buildpack-run.git

• used by https://github.com/heroku/heroku-buildpack-apt

Contains list of "to-be-installed" apt packages for the build.

• used by https://github.com/weibeld/heroku-buildpack-run.git

Contains shell script that will be run on build.

Shell wrapper for docker-compose commands that shoud be used to run "manage.py" commands but on development docker environment

Basic docker-compose file used to spin up development environment.

Example environment variable settings for the application.

Standard Django manage.py script

• used by https://github.com/heroku/heroku-buildpack-python.git

Processes defined to be run on container deployment to server

Python dependencies file.

• used by https://github.com/heroku/heroku-buildpack-python.git

Python runtime version dependency.

• used by https://github.com/heroku/heroku-buildpack-python.git

You need to have docker and docker-compose installed. They are bundled together on windows. Follow basic tutorial to setup Docker, Docker-compose and (Virtualbox) https://docs.docker.com/docker-for-windows/install/

After cloning repository you need to create custom .env file. Reasonable defaults are in example.env so just start with copying this file.

Change the JUPYTER_TOKEN environment variable to a reasonably strong passphrase to avoid serious security threat.

Then because of windows handling all stuff differently then normal OS, you need to specify path in the docker-compose-windows.yml. So edit line 15 - //c/Users/<path to code>:/code and change it to the path you have the source code. eg. - //c/Users/Carl/code:/code

Please note the forward slashes, // starting notation and the fact, that on Windows, you can only mount folders under C:/Users or you need to setup custom sharing in docker on Windows.

During the first run the initial configuration is done on the background. This results in longer startup time. Please be patient.

Start docker containers with docker-compose. Because windows have different settings for networking and stuff, there is second file that is used for configuration called docker-compose-windows.yml

Run basic command to spin up all required containers in first docker terminal:

docker-compose -f docker-compose-windows.yml up

This terminal needs to be running all the time. It displays log of all containers, which can be handy while debugging stuff. It takes some time to spin up all containers on the first run, so again, please be patient.

Then you can run second docker terminal to make maintenance of the application like creating database tables or creating local admin user so you can control the system.

Run these command in SECOND terminal

docker-compose -f docker-compose-windows.yml run --rm poster-web python manage.py migrate
docker-compose -f docker-compose-windows.yml run --rm poster-web python manage.py createsuperuser

In some cases (during the first run mostly) it happens that web container will spin up faster than the database (Postgres) container. This results in error in the FIRST terminal. Don't stop that console - you just need to restart web container

So - To restart the web process container run in SECOND terminal:

docker-compose -f docker-compose-windows.yml restart poster-web

You can also see log changing in the FIRST terminal.

You can stop the docker containers by pressing CTRL+C in the FIRST terminal and waiting. If you close that terminal it is possible that docker will keep running. In this case you can use down command to stop all defined running containers.

docker-compose -f docker-compose-windows.yml down

Stop running docker containers and remove all volumes (!!! This will clear the database !!! use with caution !)

docker-compose -f docker-compose-windows.yml down -v

Stop docker machine on Windows - otherwise it will require to Force-stop stuff while restarting the Windows machine.

docker-machine stop

Arch linux (and probably some others) with linux kernel version 4.15.x+ (confirmed on 4.16.5) has an issue with running some older versions of containers (redis:3.0.0 in our case, also found mentions of CentOS 6, with CentOS 7 working).

• The fix is described in (https://bbs.archlinux.org/viewtopic.php?pid=1774110#p1774110), vsyscall=emulate needs to be added as a kernel parameter.