This Loris build is intended for Ubuntu 14.0, Loris 2.0.0-beta1 (https://github.com/pulibrary/loris) and Kakadu 7.4. These versions can be changed, see development.cfg
, development_docker.cfg
and Dockerfile
.
Please note: setup.py
, transforms.py
and webapp.py
from the loris source have hard coded paths and kakadu library names. These files have been brought in as templates in the conf/
directory (Loris version dependent) and will be deployed with the correct paths and filenames.
If you are updating the loris source from 2.0.0-beta1 (you can do this by entering a new tag name inside the development[_docker].cfg
file) you may need to update these files with their new versions. However, the buildout parameter tags ${...}
will need to be replaced.
Core application source is held in /home/bodl-loris-svc/sites/bodl-loris-svc/src
Eggs are held in /home/bodl-loris-svc/.buildout/eggs
Loris conf is held in /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/etc/
Loris WSGI is held in /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/www
Virtualenv Python is held in /home/bodl-loris-svc/python
Caches and logs and images are held in /home/bodl-loris-svc/sites/bodl-loris-svc/var
(never cleared in event of buildout re-run)
Components of application stack (such as webserver) are held in /home/bodl-loris-svc/sites/bodl-loris-svc/parts
Apache start script is held in /home/bodl-loris-svc/sites/bodl-loris-svc/bin
kdu_compress is compiled and moved to /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/bin
as given in /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/etc/loris.conf
kdulib_vXX.so is compiled and moved to /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/lib
as given in /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/etc/loris.conf
NB: The CI does not download, compile and install kakadu. This causes some of the validation tests to fail.
The Dockerfile will run the _docker.cfg
version of development.cfg. This just ensures that users are named properly (the 'env' recipe does not work inside containers) and that the localhost is pointed to all IPs (as this cannot be dictated or predicted when creating a container).
Docker https://registry.hub.docker.com/u/bdlss/buildout.loris/
If any of the 21 IIIF validation tests fail, Docker will exit with a non-zero result. This means the Docker build will fail and read "Error".
More about IIIF validation can be found here: http://iiif-test.stanford.edu/
Pytest is executed in the docker run.
This runs all test scripts using the filename format of test_<something>.py
in the tests/
folder.
This is done automatically in the docker CI. However, you can do this manually via the website:
http://iiif-test.stanford.edu/
Or you can download the validator and run it on your server (once you have started the application), as follows:
cd /home/bodl-loris-svc/sites/bodl-loris-svc/parts
wget --no-check-certificate https://pypi.python.org/packages/source/i/iiif-validator/iiif-validator-0.9.1.tar.gz
tar zxfv iiif-validator-0.9.1.tar.gz
su - <sudo user>
sudo apt-get -y install libmagic-dev libxml2-dev libxslt-dev
su - bodl-loris-svc
cd /home/bodl-loris-svc/sites/bodl-loris-svc
. bin/activate
pip install bottle
pip install python-magic
pip install lxml
pip install Pillow
cd /home/bodl-loris-svc/sites/bodl-loris-svc/parts/iiif-validator-0.9.1/
./iiif-validate.py -s <internal IP address>:8080 -p 'loris' -i 67352ccc-d1b0-11e1-89ae-279075081939.jp2 --version=2.0 -v
To deploy Loris on a server, follow these instructions. Whenever this GIT account is updated, Docker will run a test deployment at https://registry.hub.docker.com/u/bdlss/buildout.loris/
. Please see Continuous Integration section above for more details.
If intending the docker CI server to download and compile kakadu, edit Dockerfile curl commands to contain databank username and password when downloading kakadu library. However, currently this will not work.
For a full working instance of Loris on your server, follow the buildout instructions below (which includes download and compiling of Kakadu).
sudo useradd bodl-loris-svc
sudo passwd bodl-loris-svc
sudo mkdir -p /home/bodl-loris-svc/.ssh
cd /home
sudo chown -R bodl-loris-svc:bodl-loris-svc bodl-loris-svc/
sudo chsh -s /bin/bash bodl-loris-svc
su - bodl-loris-svc
ssh-keygen -t rsa
Copy and paste your key into gitlab by choosing My Profile (the grey person graphic link in the top right hand corner) then Add Public Key.
cat ~/.ssh/id_rsa.pub
su - <sudo user>
sudo apt-get install git
git config --global user.email "my@address.com"
git config --global user.name "name in quotes"
su - bodl-loris-svc
mkdir -p ~/sites/bodl-loris-svc
cd ~/sites/bodl-loris-svc
git clone https://github.com/BDLSS/buildout.loris.git ./
For PIL/Pillow to run with JPEG2000 capability we need to install the OpenJpeg libraries before python-imaging.
http://shortrecipes.blogspot.co.uk/2014/06/python-34-and-pillow-24-with-jpeg2000.html
su - <sudo user>
sudo apt-get install -y -q wget cmake make
su - bodl-loris-svc
mkdir -p /home/bodl-loris-svc/Downloads
cd /home/bodl-loris-svc/Downloads
wget http://downloads.sourceforge.net/project/openjpeg.mirror/2.0.1/openjpeg-2.0.1.tar.gz
tar xzvf openjpeg-2.0.1.tar.gz
cd openjpeg-2.0.1/
cmake .
make
su - <sudo user>
cd /home/bodl-loris-svc/Downloads/openjpeg-2.0.1
sudo make install
export LD_LIBRARY_PATH=/usr/local/lib
su - <sudo user>
sudo apt-get install $(cat /home/bodl-loris-svc/sites/bodl-loris-svc/ubuntu_requirements)
su - bodl-loris-svc
If you haven't already (by installing the OpenJPEG libraries for Ubuntu 14.0) create the downloads directory:
mkdir -p /home/bodl-loris-svc/Downloads
su - bodl-loris-svc
cd ~/Downloads
wget http://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz --no-check-certificate
tar zxfv Python-2.7.6.tgz
cd Python-2.7.6
./configure --prefix=$HOME/python/2.7.6 --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath=/home/bodl-loris-svc/python/2.7.6/lib"
make
make install
cd ..
wget https://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz
tar zxfv distribute-0.6.49.tar.gz
~/python/2.7.6/bin/python distribute-0.6.49/distribute_setup.py
~/python/2.7.6/bin/easy_install pip
~/python/2.7.6/bin/pip install virtualenv
mkdir /home/bodl-loris-svc/.buildout
cd /home/bodl-loris-svc/.buildout
mkdir eggs
mkdir downloads
mkdir extends
echo "[buildout]
eggs-directory = /home/bodl-loris-svc/.buildout/eggs
download-cache = /home/bodl-loris-svc/.buildout/downloads
extends-cache = /home/bodl-loris-svc/.buildout/extends" >> ~/.buildout/default.cfg
edit development or production.cfg:
cd ~/sites/bodl-loris-svc
vi development.cfg
Edit the following section:
[hosts]
internalIP = <your server internal IP address>
externalIP = <your server external IP address>
To install Mirador viewer alter the end section of base.cfg
. Uncomment the viewer you want.
[sources]
# Mooviewer
www = git https://github.com/ruven/iipmooviewer egg=false
# Mirador
#www = git https://github.com/IIIF/mirador.git egg=false
After running the buildout you will need to install a couple of other things and then install the Mirador source (see section 'Installing Mirador' below).
For BDLSS users, you can retrieve the source from databank (you will need a user account for databank):
cd ~/Downloads
curl --user <username>:<password> -o Kakadu_v72.zip https://databank.ora.ox.ac.uk/dmt/datasets/Kakadu/Kakadu_v72.zip
unzip -d kakadu Kakadu_v72.zip
Otherwise you will need to scp
, wget
or curl
your licensed Kakadu source into the ~/Downloads
directory as a folder called 'kakadu'.
Buildout will compile the source and distribute the libraries and applications required (namely the shared object library and kdu_expand).
Add _docker to development.cfg if running in docker environment.
cd ~/sites/bodl-loris-svc
~/python/2.7.6/bin/virtualenv .
. bin/activate
pip install zc.buildout
pip install distribute
buildout init
buildout -c development.cfg
If you are running version 2.0.0-beta1
, please rename the loris.conf file as follows:
mv /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/etc/loris.conf /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/etc/loris2.conf
Then run setup.
cd /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris
python setup.py install
You should see the following information on your screen.
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
Installation was successful. Here's where things are:
* Loris configuration: /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/etc/loris
* Cache cleaner cron: /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/bin/loris-cache_clean.sh
* kdu_expand: /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/bin/kdu_expand
* Kakadu libraries: /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/lib/libkdu_v72R.so
* Logs: /home/bodl-loris-svc/sites/bodl-loris-svc/var/log/loris
* Image cache (opaque): /home/bodl-loris-svc/sites/bodl-loris-svc/var/cache/loris/img
* Image cache (symlinks that look like IIIF URIs): /home/bodl-loris-svc/sites/bodl-loris-svc/var/cache/loris/links
* Info cache: /home/bodl-loris-svc/sites/bodl-loris-svc/var/cache/loris/info
* www/WSGI application directory: /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/www
* Temporary directory: /home/bodl-loris-svc/sites/bodl-loris-svc/tmp
su - <sudo user>
sudo ln -s /usr/include/freetype2 freetype
sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/
sudo ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/
sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/
sudo ln -s /usr/lib/`uname -i`-linux-gnu/liblcms.so /usr/lib/
sudo ln -s /usr/lib/`uname -i`-linux-gnu/libtiff.so /usr/lib/
sudo ln -s /usr/lib/`uname -i`-linux-gnu/libopenjp2.so /usr/lib/
Copy the test images into your image root:
su - bodl-loris-svc
cp -R /home/bodl-loris-svc/sites/bodl-loris-svc/src/loris/tests/img/* /home/bodl-loris-svc/sites/bodl-loris-svc/var/images
For Ubuntu 12.0:
sudo apt-get update
sudo apt-get install -y python-software-properties
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs
For Ubuntu 14.0:
sudo apt-get install nodejs nodejs-legacy npm
Install grunt and install mirador with npm:
sudo npm install -g grunt-cli
cd /home/bodl-loris-svc/sites/bodl-loris-svc/src/www
sudo npm install
Browse as below to load the viewer.
su - <sudo user>
sudo /home/bodl-loris-svc/sites/bodl-loris-svc/bin/lorisctl start
Browse the following links to test:
http://{your_server, e.g. localhost:8080}/loris/67352ccc-d1b0-11e1-89ae-279075081939.jp2/full/full/0/default.jpg
http://{your_server, e.g. localhost:8080}/loris/67352ccc-d1b0-11e1-89ae-279075081939.jp2/info.json
To run a IIIF validation test see the IIIF Validation section in the introduction (above).
su - <sudo user>
sudo crontab /home/bodl-loris-svc/sites/bodl-loris-svc/bin/cron.txt
su - bodl-loris-svc
The following script can be run manually as sudo.
su - <sudo user>
/home/bodl-loris-svc/sites/bodl-loris-svc/bin/lorisctl [start|stop|restart]
It will stop/start/restart Loris. It runs under a @reboot directive in the sudo crontab to ensure the service comes back up in the event of a server shutdown/restart. It logs progress in var/log/reboot.log
.
@reboot /home/bodl-loris-svc/sites/bodl-loris-svc/bin/lorisctl start > /home/bodl-loris-svc/sites/bodl-loris-svc/var/log/reboot.log 2>&1