= Introduction =
This is the full MiG project code released at the MiGrid project at
SourceForge:

https://sourceforge.net/projects/migrid/

We previously used Google Code so history is avilable at:
 
http://code.google.com/p/migrid/

MiG is Free Software and it is developed by the MiG Project lead by
Brian Vinter (brian DOT vinter AT nbi DOT ku DOT dk).

Please refer to the COPYING file in this directory for further information
about the GPL v2 license under which MiG is distributed.


= Getting Started =
Please refer to the information available at the aforementioned URL
especially the wiki pages including:

https://sourceforge.net/p/migrid/wiki/GettingStarted/


= Requirements =
A MiG server basically requires an Apache web server, the OpenSSH client
tools and a Python interpreter with a few external modules.
-Apache 2.x (http://httpd.apache.org/)
-Apache SSL module (http://httpd.apache.org/docs/current/mod/mod_ssl.html)
-Apache proxy module (http://httpd.apache.org/docs/current/mod/mod_proxy.html)
-Apache Rewrite module (http://httpd.apache.org/docs/current/mod/mod_rewrite.html)
-OpenSSH clients (https://www.openssh.org/)
-Python 2.6 or later but not 3.x (https://www.python.org/)

Optional file synchronization, WSGI interface, OpenID login, instant
messaging service, efficient file access services, event handler
service, JSONRPC access, background data transfer service, efficient
sftp, spell checking, interactive computing, VGrid Wiki / SCM / tracker, 
Jupyter, password strength testing and PDF generation features rely on
the following additional software:
-Seafile server, community edition (https://www.seafile.com/en/download/)
-Apache WSGI module (https://code.google.com/p/modwsgi/)
-Apache OpenID auth module (http://findingscience.com/mod_auth_openid/)
-Python OpenID module (https://github.com/openid/python-openid/)
-Python irclib module (https://pypi.python.org/pypi/python-irclib/)
-Python Paramiko module (https://pypi.python.org/pypi/paramiko/)
-Python FTPD library (https://pypi.python.org/pypi/pyftpdlib/)
-Python OpenSSL module (https://pypi.python.org/pypi/pyOpenSSL/)
-Python WSGI WebDAV module (http://wsgidav.readthedocs.org/)
-Python watchdog module (https://pypi.python.org/pypi/watchdog/)
-Python scandir module (https://pypi.python.org/pypi/scandir/)
-Python jsonrpclib module (https://pypi.python.org/pypi/jsonrpclib/)
-Python requests module (https://pypi.python.org/pypi/requests/)
-Python cracklib module (https://pypi.org/project/cracklib/)
-Python PDFKit module (https://github.com/Martin-Rehr/python-pdfkit/)
-Python xvfb module (https://pypi.python.org/pypi/xvfbwrapper/)
-Python OTP (https://pypi.org/project/pyotp/)
-Python pyyaml (https://pypi.org/pyyaml/)
-Python nbformat (https://pypi.org/project/nbformat/)
-Python more-itertools (https://pypi.org/project/more-itertools/5.0.0/)
-Python nbconvert (https://pypi.org/project/nbconvert/)
-Python papermill (https://pypi.org/project/papermill/)
-Python notebook_parameterizer (https://pypi.org/project/notebook_parameterizer)
-LFTP (http://lftp.yar.ru/)
-RSync (https://rsync.samba.org/)
-OpenSSH server (https://www.openssh.org/)
-PAM: Pluggable Authentication Modules (http://www.linux-pam.org/)
-Python Enchant module (https://pypi.python.org/pypi/pyenchant/)
-Mercurial (http://mercurial.selenic.com/)
-Trac (http://trac.edgewall.org/)
-Trac plugins (e.g. http://trac-hacks.org/)
-Jupyter (https://jupyter.org/)
-Docker (https://www.docker.com/)
-Fail2Ban (http://www.fail2ban.org/) w. IPSet (http://ipset.netfilter.org/)

On Debian/Ubuntu servers the corresponding basic packages can be
installed with:
sudo apt install apache2 openssh-client python python-pip python-setuptools \
     python-dev build-essential
and most of the optional dependencies similarly with:
sudo apt install libapache2-mod-wsgi libapache2-mod-auth-openid  \
     python-irclib python-paramiko python-enchant python-jsonrpclib \
     python-requests python-cracklib openssh-server libpam0g-dev mercurial \
     trac trac-mercurial lftp rsync fail2ban ipset

On RedHat/CentOS servers the basic packages can be installed with:
sudo yum install epel-release
sudo yum install httpd mod_ssl openssh-clients python python-pip \
     python-setuptools python-devel gcc

and most of the optional dependencies similarly with:
sudo yum install mod_wsgi mod_auth_openid python-paramiko python-enchant \
     python-jsonrpclib python-requests cracklib-python openssh-server \
     pam-devel mercurial lftp rsync fail2ban ipset

while the remaining ones need to be installed from EPEL
sudo yum install trac trac-mercurial-plugin
or alternatively directly from pypi:
sudo pip install trac tracmercurial

Apache comes with a number of modules natively included, so it is usually not
necessary to explicitly install e.g. mod-proxy and mod-rewrite, only the modules
explicitly packaged separately.

We highly recommend installing the optional python wsgidav and openid modules
directly from upstream, because packaged versions are generally outdated
even on pypi. The packaged openid module lacks some security fixes and
the wsgidav one lacks proper chrooting support and a fix for upload/write
access for OSX clients.
They can easily be installed with pip:
sudo pip install https://github.com/openid/python-openid/archive/master.zip
sudo pip install wsgidav

Additional packaged Trac extensions can be installed with:
sudo aptitude install trac-customfieldadmin trac-graphviz \
     trac-mastertickets trac-wikiprint trac-wikirename trac-wysiwyg

and the unpackaged ones can be grabbed from trac-hacks.org with pip
and easy_install:
sudo easy_install https://trac-hacks.org/svn/wikicssplugin
sudo easy_install https://trac-hacks.org/svn/fullblogplugin
sudo easy_install https://trac-hacks.org/svn/discussionplugin
sudo easy_install https://trac-hacks.org/svn/tracpasteplugin
sudo easy_install https://trac-hacks.org/svn/downloadsplugin
sudo pip install TracStats

where 0.11 in the URLs may need to be changed to fit your particular version
of Trac. Please note that the source code stats in the TracStats plugin
do not currently work for Mercurial repositories!
Additional plugins are available from http://trac-hacks.org/

The downloads plugin currently needs patching to work. Please refer to
the notes in the [downloads] section of the generated MiG trac.ini file.

Please note that there may be subtle internal plugin dependencies and
conflicts that e.g. can cause problems if plugins are not loaded in the
right order. We have seen database upgrade problems if FullBlog,
Watchlist and Discussion are enabled but not loaded in an order where
Watchlist is loaded in between the other two.
Similar issues appeared when we enabled the Pastebin and Downloads
plugins in one step. It was necessary to either patch
tracdownloads/db/db1.py to ignore errors for existing tables or to
enable one plugin and upgrade all Trac environments before enabling the
other plugin and upgrading again.
Thus you may have to experiment with the installed plugins in a
conservative way.

With the inclusion of Trac we no longer rely on the MoinMoin software
for stand-alone wikis.

The optional grid_ftps daemon requires the pyftpdlib module in a recent
version and the pyopenssl module for SSL/TLS support, so it is easiest
to install it with pip:
sudo pip install pyOpenSSL pyftpdlib

The optional grid_webdavs daemon requires the wsgidav module in a recent
version (1.3.0+), so it is easiest to install it with pip as mentioned above.

The optional grid_openid daemon requires the openid module in a recent
version, so it is easiest to install it with pip as mentioned above.

The optional grid_events daemon requires the watchdog and scandir modules
 which may be installed with:
sudo pip install watchdog scandir
it is likely that the inotify default settings are too low to handle any
serious number of vgrid shares so you may have to additionally tune the sysctl
settings e.g. by adding the following to your /etc/sysctl.conf :
# For grid_events daemon
# It may be necessary to increase the number of watched files
fs.inotify.max_user_watches=1048576
fs.inotify.max_user_instances=1024

The optional grid_transfers daemon requires lftp and rsync clients to
handle the background data transfers. Most Linux distributions come with
versions that can be used right out of the box. However, the lftp 4.4.8
in CentOS/RHEL 7 appears to have an annoying bug, so that it loops
forever instead of just failing if the username/password is
incorrect. From the changelog it sounds like it was fixed in version
4.4.12, and neither the 4.4.13 version available in Ubuntu LTS (14.04)
nor the 4.6 version on Debian stable exhibit this issue. Luckily the
upstream project provides RPM's directly installable with yum as in:
sudo yum install http://lftp.yar.ru/ftp/binaries/lftp-4.6.5-1.x86_64.rpm
The only downside is that it then requires manual updates until the
distro catches up.

The optional interactive computing environment relies on one or more
separate Jupyter hosts. The easiest way of setting that up is to use the
existing jupyter swarm-spawner with docker packs to launch all user 
sessions in isolated containers. This part is still work-in-progress so
please get in touch for details. Only the python-requests module is
needed for the actual MiG server-side integration.

The optional PDF generator environment relies on python pdfkit and 
xvfbwrapper which may be installed with:
sudo pip install xvfbwrapper
sudo pip install pip install git+https://github.com/Martin-Rehr/python-pdfkit.git
xvfbwrapper require xorg-x11-server-Xvfb and pdfkit require
wkhtmltopdf. If links are _NOT_ working in the generated PDF's then 
try to install the latest stable version of wkhtmltopdf from:
https://wkhtmltopdf.org/downloads.html

If the generated PDFs are gibberish then xorg-x11-fonts are most likely
missing on the system.

The optional PyOTP module used for 2-factor authentication is not
necessarily recent in distributions but then readily available with pip:
sudo pip install pyotp

The optional country code validation in certificate and OpenID account 
request backends requires the iso3166 module which may be installed with:
sudo pip install iso3166

The optional pygdb module used for debugging is available with pip:
sudo pip install pygdb

The optional workflows module relies on the python special modules 
nbformat, nbconvert to validate and produce correct Jupyter Notebook formatted files.
Nbformat specifically requires that the more_itertools module is both installed beforehand
and that it is pinned to version 5.0.0 to ensure Python2.7 support.
In addition the workflows module creates yaml parameter files via the pyyaml module.
To schedule and execute workflow tasks, the execution nodes are required to 
provide the papermill module via the PAPERMILL environment variable,
the notebook_parameterizer module via the NOTEBOOK_PARAMETERIZER environment variable,
and the sshfs command via the SSHFS_MOUNT environment variable.

= Installing MiG =
If you want to run your own MiG server for your own grid or to develop
MiG you should download and unpack the source code (including this
file) on a UNIX compatible computer as described below.

The MiG core services are provided by the MiG daemons from the mig/server
directory and they can simply be run directly from the unpacked source
code directory when a suitable server configuration is added.

For the web interfaces to work you will need to run an apache server as
described in the mig/install directory. Grid job handout relies on OpenSSH
client commands like ssh and scp.
MiG does not include the actual Apache web server or OpenSSH clients, so
you will need to install those using either packages provided by your
distribution or install it from source.

MiG is tested on Debian/Ubuntu and Redhat Linux using Apache 1.3 or 2.X
with mod-ssl respectively but other distribution and apache combinations
should also work.
MiG relies on apache's mod-ssl for automatic certificate validation and
access control. Furthermore quite a bit of rewrite rules are used for
access and convenience so the mod-rewrite apache module is required
too.

You can read more about the apache configuration in
the provided mig/install/README.Debian file.

This server documentation expects the MiG code to run as a separate 'mig'
user on the UNIX system, but this is not a requirement. Just modify your
apache and MiG configurations appropriately if you want to run MiG as a
different user or with other paths. 
It is important to configure apache so that the MiG web interfaces can
read and write the files created by the MiG daemons and vice versa.
This may require extra care if the MiG installation and apache runs as
different system users. If you use the default setup you do not need to
worry about this.

As root you can create an ordinary user, mig, for running the MiG server:
# su -
# useradd -m -U mig 

Login as the new user:
# su - mig

To avoid other processes from tampering it is a good idea to set either
the permissions on the entire mig user home very restrictively:
# chmod 700 ~mig
or at least set the umask tight enough to avoid unauthorized access to
the MiG server files. 
If you run MiG with different apache and mig users, you will most likely
need to provide both users write access to the mig user home, though.

Download and unpack the MiG source or make a checkout from svn as described on:

https://sourceforge.net/p/migrid/code/HEAD/tree/trunk/

At this point it may be comfortable to copy some of the basic
account configuration files from mig/install/mig-user to ~/ but this is
not mandatory.

Now you are ready to actually configure your installation.
The easiest way to do that is to use the configuration generator in 
mig/install/generateconfs.py to create configurations that match your
setup.
For the default settings it could just be done as:
cd mig/install/
./generateconfs.py

If your setup uses custom paths or settings just provide them on the
commandline like the command help indicates:
~/mig/install > ./generateconfs.py -h
Usage:
./generateconfs.py [OPTIONS]
Where supported options include -h/--help for this help or the conf
settings:
--source=SOURCE
--destination=DESTINATION
--destination_suffix=DESTINATION_SUFFIX
--base_fqdn=BASE_FQDN
--public_fqdn=PUBLIC_FQDN
--mig_cert_fqdn=MIG_CERT_FQDN
--ext_cert_fqdn=EXT_CERT_FQDN
--mig_oid_fqdn=MIG_OID_FQDN
--ext_oid_fqdn=EXT_OID_FQDN
--sid_fqdn=SID_FQDN
--io_fqdn=IO_FQDN
--jupyter_hosts=JUPYTER_HOSTS
--jupyter_base_url=JUPYTER_BASE_URL
--user=USER
--group=GROUP
--apache_version=APACHE_VERSION
--apache_etc=APACHE_ETC
--apache_run=APACHE_RUN
--apache_lock=APACHE_LOCK
--apache_log=APACHE_LOG
--openssh_version=OPENSSH_VERSION
--mig_code=MIG_CODE
--mig_state=MIG_STATE
--mig_certs=MIG_CERTS
--enable_sftp=ENABLE_SFTP
--enable_sftp_subsys=ENABLE_SFTP_SUBSYS
--enable_davs=ENABLE_DAVS
--enable_ftps=ENABLE_FTPS
--enable_wsgi=ENABLE_WSGI
--wsgi_procs=WSGI_PROCS
--enable_jobs=ENABLE_JOBS
--enable_resources=ENABLE_RESOURCES
--enable_events=ENABLE_EVENTS
--enable_sharelinks=ENABLE_SHARELINKS
--enable_transfers=ENABLE_TRANSFERS
--enable_freeze=ENABLE_FREEZE
--enable_sandboxes=ENABLE_SANDBOXES
--enable_vmachines=ENABLE_VMACHINES
--enable_preview=ENABLE_PREVIEW
--enable_jupyter=ENABLE_JUPYTER
--enable_gdp=ENABLE_GDP
--enable_hsts=ENABLE_HSTS
--enable_vhost_certs=ENABLE_VHOST_CERTS
--enable_verify_certs=ENABLE_VERIFY_CERTS
--enable_seafile=ENABLE_SEAFILE
--enable_duplicati=ENABLE_DUPLICATI
--enable_crontab=ENABLE_CRONTAB
--enable_imnotify=ENABLE_IMNOTIFY
--enable_dev_accounts=ENABLE_DEV_ACCOUNTS
--enable_twofactor=ENABLE_TWOFACTOR
--enable_cracklib=ENABLE_CRACKLIB
--enable_openid=ENABLE_OPENID
--mig_oid_provider=MIG_OID_PROVIDER
--ext_oid_provider=EXT_OID_PROVIDER
--dhparams_path=DHPARAMS_PATH
--daemon_keycert=DAEMON_KEYCERT
--daemon_pubkey=DAEMON_PUBKEY
--daemon_show_address=DAEMON_SHOW_ADDRESS
--alias_field=ALIAS_FIELD
--signup_methods=SIGNUP_METHODS
--login_methods=LOGIN_METHODS
--hg_path=HG_PATH
--hgweb_scripts=HGWEB_SCRIPTS
--trac_admin_path=TRAC_ADMIN_PATH
--trac_ini_path=TRAC_INI_PATH
--public_port=PUBLIC_PORT
--mig_cert_port=MIG_CERT_PORT
--ext_cert_port=EXT_CERT_PORT
--mig_oid_port=MIG_OID_PORT
--ext_oid_port=EXT_OID_PORT
--sid_port=SID_PORT
--user_clause=USER_CLAUSE
--group_clause=GROUP_CLAUSE
--listen_clause=LISTEN_CLAUSE
--serveralias_clause=SERVERALIAS_CLAUSE
--distro=DISTRO
--landing_page=LANDING_PAGE
--skin=SKIN

For one of our servers running MiG as the 'mig' user with the code
checked out directly in the home directory and Debian apache 2.4 without OpenID:
./generateconfs.py --source=. --destination=generated-confs \
                   --base_fqdn=migrid.org \
                   --public_fqdn=www.migrid.org \
                   --mig_cert_fqdn=dk-cert.migrid.org \
                   --ext_cert_fqdn= \
                   --mig_oid_fqdn=dk-ext.migrid.org \
                   --ext_oid_fqdn=dk-oid.migrid.org \
                   --sid_fqdn=dk-sid.migrid.org \
                   --io_fqdn=dk-io.migrid.org \
                   --user=mig --group=mig \
                   --apache_version=2.4 \
                   --apache_etc=/etc/apache2 \
                   --apache_run=/var/run/apache2 \
                   --apache_lock=/var/lock/apache2 \
                   --apache_log=/var/log/apache2 \
                   --openssh_version=7.2 \
                   --mig_code=/home/mig/mig \
                   --mig_state=/home/mig/state \
                   --mig_certs=/etc/apache2/MiG-certificates \
                   --hg_path=/usr/bin/hg \
                   --hgweb_scripts=/usr/share/doc/mercurial-common/examples \
                   --trac_admin_path=/usr/bin/trac-admin \
                   --trac_ini_path=/home/mig/mig/server/trac.ini \
                   --public_port=80 --mig_cert_port=443 --mig_oid_port=443 \
                   --ext_oid_port=443 --sid_port=443 \
                   --enable_openid=False --enable_wsgi=True \
                   --enable_sftp=False --enable_sftp_subsys=True \
                   --enable_sandboxes=True --enable_vmachines=True \
                   --user_clause=User --group_clause=Group \
                   --listen_clause='#Listen' \
                   --serveralias_clause='ServerAlias' \
                   --signup_methods="migcert" \
                   --login_methods="migcert" \
                   --skin=migrid-basic \
                   --short_title=MiG
                    
or the same with HSTS, WSGI (default web), Seafile, optimized SFTP,
vhost-specific certificates from LetsEncrypt and OpenID with optional
2FA support:
./generateconfs.py --source=. --destination=generated-confs \
                   --destination_suffix="_svn$(svnversion -n ~/)" \
                   --base_fqdn=migrid.org \
                   --public_fqdn=www.migrid.org \
                   --mig_cert_fqdn=dk-cert.migrid.org \
                   --ext_cert_fqdn= \
                   --mig_oid_fqdn=dk-ext.migrid.org \
                   --ext_oid_fqdn=dk-oid.migrid.org \
                   --sid_fqdn=dk-sid.migrid.org \
                   --io_fqdn=dk-io.migrid.org \
                   --seafile_fqdn=dk-sid.migrid.org \
                   --user=mig --group=mig \
                   --apache_version=2.4 \
                   --apache_etc=/etc/apache2 \
                   --apache_run=/var/run/apache2 \
                   --apache_lock=/var/lock/apache2 \
                   --apache_log=/var/log/apache2 \
                   --openssh_version=7.2 \
                   --mig_code=/home/mig/mig \
                   --mig_state=/home/mig/state \
                   --mig_certs=/etc/apache2/MiG-certificates \
                   --hg_path=/usr/bin/hg \
                   --hgweb_scripts=/usr/share/doc/mercurial-common/examples \
                   --trac_admin_path=/usr/bin/trac-admin \
                   --trac_ini_path=/home/mig/mig/server/trac.ini \
                   --public_port=80 --ext_cert_port=443 --mig_oid_port=443 \
                   --ext_oid_port=443 --sid_port=443 \
                   --mig_oid_provider=https://dk-ext.migrid.org/openid/ \
                   --ext_oid_provider=https://openid.ku.dk/ \
                   --enable_openid=True --enable_wsgi=True \
                   --enable_sftp=False --enable_sftp_subsys=True \
                   --enable_davs=True --enable_ftps=True \
                   --enable_duplicati=False --enable_seafile=True \
                   --enable_sandboxes=True --enable_vmachines=False \
                   --enable_crontab=True --enable_jobs=True \
                   --enable_resources=True --enable_notify=True \
                   --enable_events=True --enable_imnotify=True \
                   --enable_twofactor=True --enable_cracklib=True \
                   --enable_hsts=True \
                   --enable_vhost_certs=True --enable_verify_certs=True \
                   --user_clause=User --group_clause=Group \
                   --listen_clause='#Listen' \
                   --serveralias_clause='ServerAlias' --alias_field=email \
                   --dhparams_path=~/certs/dhparams.pem \
                   --daemon_keycert=~/certs/combined.pem \
                   --daemon_pubkey=~/certs/combined.pub \
                   --daemon_pubkey_from_dns=False \
                   --signup_methods="extoid migoid migcert" \
                   --login_methods="extoid migoid migcert" \
                   --landing_page='/wsgi-bin/dashboard.py' \
                   --skin=migrid-basic --short_title=MiG \
                   --wsgi_procs=25 --secscan_addr=130.225.213.72

and a storage-only setup with CentOS 7.x, apache 2.4, WSGI (default web), 
optimized SFTP, Seafile integration and OpenID login:
./generateconfs.py --source=. --destination=generated-confs \
                   --destination_suffix="_svn$(svnversion -n ~/)" \
                   --base_fqdn=erda.dk \
                   --public_fqdn=www.erda.dk \
                   --public_alias_fqdn=erda.ku.dk \
                   --mig_cert_fqdn= \
                   --ext_cert_fqdn=cert.erda.dk \
                   --mig_oid_fqdn=ext.erda.dk \
                   --ext_oid_fqdn=erda.dk \
                   --sid_fqdn=sid.erda.dk \
                   --io_fqdn=io.erda.dk \
                   --seafile_fqdn=sid.erda.dk \
                   --user=mig --group=mig \
                   --apache_version=2.4 \
                   --apache_etc=/etc/httpd \
                   --apache_run=/var/run/httpd \
                   --apache_lock=/var/lock/subsys/httpd \
                   --apache_log=/var/log/httpd \
                   --openssh_version=5.3 \
                   --mig_code=/home/mig/mig \
                   --mig_state=/home/mig/state \
                   --mig_certs=/etc/httpd/MiG-certificates \
                   --hg_path=/usr/bin/hg \
                   --hgweb_scripts=/usr/share/doc/mercurial-2.6.2 \
                   --trac_admin_path='' --trac_ini_path='' \
                   --public_port=80 --ext_cert_port=443 --mig_oid_port=443 \
                   --ext_oid_port=443 --sid_port=443 \
                   --mig_oid_provider=https://ext.erda.dk/openid/ \
                   --ext_oid_provider=https://openid.ku.dk/ \
                   --enable_openid=True --enable_wsgi=True \
                   --enable_sftp=False --enable_sftp_subsys=True \
                   --enable_davs=True --enable_ftps=True \
                   --enable_duplicati=True --enable_seafile=True \
                   --enable_sandboxes=False --enable_vmachines=False \
                   --enable_crontab=True --enable_jobs=False \
                   --enable_resources=False --enable_events=True \
                   --enable_freeze=True --enable_hsts=True \
                   --enable_vhost_certs=True --enable_verify_certs=True \
                   --user_clause=User --group_clause=Group \
                   --listen_clause='#Listen' \
                   --serveralias_clause='#ServerAlias' --alias_field=email \
                   --dhparams_path=~/certs/dhparams.pem \
                   --daemon_keycert=~/certs/combined.pem \
                   --daemon_pubkey=~/certs/combined.pub \
                   --daemon_pubkey_from_dns=True \
                   --signup_methods="extoid migoid extcert" \
                   --login_methods="extoid migoid extcert" \
                   --distro=centos --landing_page='/wsgi-bin/fileman.py' \
                   --skin=erda-ucph-science --short_title=ERDA \
                   --vgrid_label=Workgroup --wsgi_procs=25
                   
and the same with added Jupyter integration and optional 2FA support:
./generateconfs.py --source=. --destination=generated-confs \
                   --destination_suffix="_svn$(svnversion -n ~/)" \
                   --base_fqdn=erda.dk \
                   --public_fqdn=www.erda.dk \
                   --public_alias_fqdn=erda.ku.dk \
                   --mig_cert_fqdn= \
                   --ext_cert_fqdn=cert.erda.dk \
                   --mig_oid_fqdn=ext.erda.dk \
                   --ext_oid_fqdn=erda.dk \
                   --sid_fqdn=sid.erda.dk \
                   --io_fqdn=io.erda.dk \
                   --seafile_fqdn=sid.erda.dk \
                   --user=mig --group=mig \
                   --apache_version=2.4 \
                   --apache_etc=/etc/httpd \
                   --apache_run=/var/run/httpd \
                   --apache_lock=/var/lock/subsys/httpd \
                   --apache_log=/var/log/httpd \
                   --openssh_version=5.3 \
                   --mig_code=/home/mig/mig \
                   --mig_state=/home/mig/state \
                   --mig_certs=/etc/httpd/MiG-certificates \
                   --hg_path=/usr/bin/hg \
                   --hgweb_scripts=/usr/share/doc/mercurial-2.6.2 \
                   --trac_admin_path='' --trac_ini_path='' \
                   --public_port=80 --ext_cert_port=443 --mig_oid_port=443 \
                   --ext_oid_port=443 --sid_port=443 \
                   --mig_oid_provider=https://ext.erda.dk/openid/ \
                   --ext_oid_provider=https://openid.ku.dk/ \
                   --enable_openid=True --enable_wsgi=True \
                   --enable_sftp=False --enable_sftp_subsys=True \
                   --enable_davs=True --enable_ftps=True \
                   --enable_duplicati=True --enable_seafile=True \
                   --seafile_fqdn=seafile.erda.dk \
                   --enable_sandboxes=False --enable_vmachines=False \
                   --enable_crontab=True --enable_jobs=False \
                   --enable_resources=False --enable_events=True \
                   --enable_freeze=True --enable_twofactor=True \
                   --enable_cracklib=True --enable_hsts=True \
                   --enable_vhost_certs=True --enable_verify_certs=True \
                   --enable_notify=True --enable_jupyter=True \
                   --jupyter_services='DAG.http://dag002.science DAG.http://dag003.science DAG.http://dag004.science DAG.http://dag005.science DAG.http://dag006.science DAG.http://dag007.science MODI.http://130.225.104.214' \
                   --jupyter_services_desc="{'DAG': '/home/mig/state/wwwpublic/dag_desc.html', 'MODI': '/home/mig/state/wwwpublic/modi_desc.html'}" \
                   --user_clause=User --group_clause=Group \
                   --listen_clause='#Listen' \
                   --serveralias_clause='#ServerAlias' --alias_field=email \
                   --dhparams_path=~/certs/dhparams.pem \
                   --daemon_keycert=~/certs/combined.pem \
                   --daemon_pubkey=~/certs/combined.pub \
                   --daemon_pubkey_from_dns=True \
                   --signup_methods="extoid migoid extcert" \
                   --login_methods="extoid migoid extcert" \
                   --distro=centos --landing_page='/wsgi-bin/fileman.py' \
                   --skin=erda-ucph-science --short_title=ERDA \
                   --vgrid_label=Workgroup --apache_worker_procs=2048 \
                   --davs_port=8020 --openid_port=8001 \
                   --wsgi_procs=50 --sftp_subsys_auth_procs=25

and a similar setup with CentOS 7.x, apache 2.4, WSGI (default web), 
optimized SFTP, job execution, Jupyter integration, previews and OpenID login
with optional 2-FA support and legacy sftp clients:
./generateconfs.py --source=. --destination=generated-confs \
                   --destination_suffix="_svn$(svnversion -n ~/)" \
                   --base_fqdn=idmc.dk \
                   --public_fqdn=www.idmc.dk \
                   --mig_cert_fqdn= \
                   --ext_cert_fqdn=cert.idmc.dk \
                   --mig_oid_fqdn=ext.idmc.dk \
                   --ext_oid_fqdn=oid.idmc.dk \
                   --sid_fqdn=sid.idmc.dk \
                   --io_fqdn=io.idmc.dk \
                   --user=mig --group=mig \
                   --apache_version=2.4 \
                   --apache_etc=/etc/httpd \
                   --apache_run=/var/run/httpd \
                   --apache_lock=/var/lock/subsys/httpd \
                   --apache_log=/var/log/httpd \
                   --openssh_version=5.3 \
                   --mig_code=/home/mig/mig \
                   --mig_state=/home/mig/state \
                   --mig_certs=/etc/httpd/MiG-certificates \
                   --hg_path=/usr/bin/hg \
                   --hgweb_scripts=/usr/share/doc/mercurial-2.6.2 \
                   --trac_admin_path='' --trac_ini_path='' \
                   --public_port=80 --ext_cert_port=443 --mig_oid_port=443 \
                   --ext_oid_port=443 --sid_port=443 \
                   --mig_oid_provider=https://ext.idmc.dk/openid/ \
                   --ext_oid_provider=https://openid.ku.dk/ \
                   --enable_openid=True --enable_wsgi=True \
                   --enable_sftp=False --enable_sftp_subsys=True \
                   --enable_davs=True --enable_ftps=True \
                   --enable_sharelinks=True --enable_transfers=True \
                   --enable_duplicati=False --enable_seafile=False \
                   --enable_sandboxes=False --enable_vmachines=False \
                   --enable_crontab=True --enable_jobs=True \
                   --enable_resources=True --enable_events=True \
                   --enable_freeze=False --enable_imnotify=False \
                   --enable_twofactor=True --enable_cracklib=True \
                   --enable_notify=True --enable_preview=True \
                   --enable_hsts=True \
                   --enable_vhost_certs=True --enable_verify_certs=True \
                   --enable_jupyter=True \
                   --jupyter_services='DAG.http://dag002.science DAG.http://dag003.science DAG.http://dag004.science DAG.http://dag005.science DAG.http://dag006.science DAG.http://dag007.science MODI.http://130.225.104.214' \
                   --jupyter_services_desc="{'DAG': '/home/mig/state/wwwpublic/dag_desc.html', 'MODI': '/home/mig/state/wwwpublic/modi_desc.html'}" \
                   --user_clause=User --group_clause=Group \
                   --listen_clause='#Listen' \
                   --serveralias_clause='#ServerAlias' --alias_field=email \
                   --dhparams_path=~/certs/dhparams.pem \
                   --daemon_keycert=~/certs/combined.pem \
                   --daemon_pubkey=~/certs/combined.pub \
                   --daemon_pubkey_from_dns=False \
                   --daemon_show_address=io.idmc.dk \
                   --signup_methods="extoid migoid extcert" \
                   --login_methods="extoid migoid extcert" \
                   --distro=centos --landing_page='/wsgi-bin/fileman.py' \
                   --skin=idmc-basic --short_title=IDMC \
                   --vgrid_label=Workgroup --apache_worker_procs=512 \
                   --wsgi_procs=25 --sftp_subsys_auth_procs=25 \
                   --davs_port=8020 --openid_port=8001 \
                   --secscan_addr=130.225.213.72

Finally a storage-only with CentOS 7.x, apache 2.4, WSGI (default web), 
optimized SFTP, strict access control and extensive logging to comply with the 
General Data Protection Regulation (GDPR) imposed by EU:
https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
./generateconfs.py --source=. --destination=generated-confs \
                   --destination_suffix="_svn$(svnversion -n ~/)" \
                   --base_fqdn=sif.erda.dk \
                   --public_fqdn=sif-www.erda.dk \
                   --mig_cert_fqdn= \
                   --ext_cert_fqdn= \
                   --mig_oid_fqdn=sif-ext.erda.dk \
                   --ext_oid_fqdn=sif.erda.dk \
                   --sid_fqdn=sif-sid.erda.dk \
                   --io_fqdn=sif-io.erda.dk \
                   --user=mig --group=mig \
                   --apache_version=2.4 \
                   --apache_etc=/etc/httpd \
                   --apache_run=/var/run/httpd \
                   --apache_lock=/var/lock/subsys/httpd \
                   --apache_log=/var/log/httpd \
                   --openssh_version=7.4 \
                   --mig_code=/home/mig/mig \
                   --mig_state=/home/mig/state \
                   --mig_certs=/etc/httpd/MiG-certificates \
                   --hg_path='' \
                   --hgweb_scripts='' \
                   --trac_admin_path='' --trac_ini_path='' \
                   --public_port=80 --ext_cert_port=443 --mig_oid_port=443 \
                   --ext_oid_port=443 --sid_port=443 \
                   --mig_oid_provider=https://sif-ext.erda.dk/openid/ \
                   --ext_oid_provider=https://openid.ku.dk/ \
                   --enable_openid=False --enable_wsgi=True \
                   --enable_sftp=True --enable_sftp_subsys=False \
                   --enable_davs=True --enable_ftps=False \
                   --enable_sharelinks=False --enable_transfers=False \
                   --enable_duplicati=False --enable_seafile=False \
                   --enable_sandboxes=False --enable_vmachines=False \
                   --enable_crontab=False --enable_jobs=False \
                   --enable_resources=False --enable_events=False \
                   --enable_freeze=False --enable_preview=False \
                   --enable_gdp=True \
                   --enable_notify=True --enable_twofactor=True \
                   --enable_twofactor_strict_address=True \
                   --enable_cracklib=True --enable_hsts=True \
                   --enable_vhost_certs=True --enable_verify_certs=True \
                   --user_clause=User --group_clause=Group \
                   --listen_clause='#Listen' \
                   --serveralias_clause='#ServerAlias' --alias_field=email \
                   --dhparams_path=~/certs/dhparams.pem \
                   --daemon_keycert=~/certs/combined.pem \
                   --daemon_pubkey=~/certs/combined.pub \
                   --daemon_pubkey_from_dns=True \
                   --signup_methods="extoid" --login_methods="extoid" \
                   --password_policy=HIGH \
                   --distro=centos --landing_page='/wsgi-bin/gdpman.py' \
                   --skin=erda-ucph-science --short_title=SIF \
                   --wsgi_procs=25

Most of the arguments should be relatively straight forward, but you
need to provide the MIG_CERTS path where your apache server key and
certificates are available along with optional MiG x509 server
certificates (used for MiG server to server communication).
The actual keys and certificates can be added later, so you can just
choose a suitable directory path at first.

The hg and trac path pairs are optional and can be set to the 
empty string if mercurial/trac is not available or if VGrid wikis,
SCMs and trackers should simply not be enabled. If you want VGrid
trackers including mercurial integration, but don't want the direct
VGrid SCM links, you can set the trac_X and hg_X options but leave out
the scm entry in the ordered list of vgrid_links in the SITE
section. The same procedure applies for visibility of the other VGrid
components.

Similarly the mercurial package provides all required components for 
VGrid SCMs on Debian/Ubuntu. The same applies for the trac + trac-mercurial
packages.
Paramiko is required for the optional grid_sftp daemon to work and the
python-paramiko package provides all required components for it on
recent Debian/Ubuntu. You want to use 1.15+ because it added significant
performance improvements. Just pull it in from pip if your distribution
only provides an older version.
In case you want to run a high-performance SFTP frontend you can combine
OpenSSH with the same paramiko SFTP backend through the use of the
sftp_subsys.py module. You need to compile and configure the PAM and NSS
modules in mig/pam-mig and mig/libnss-mig as described in the README
files there. Benchmarks show that it performs and scales far better to
multiple clients.
Python FTP server library (pyftpdlib) in a recent version (1.x) is required
for the optional grid_ftps daemon to work and the python-pyftpdlib
package provides all required components for it on recent
Debian/Ubuntu. If no recent version is available, it can still easily be
installed with pip instead.
WsgiDAV is required for the optional grid_webdavs daemon to work and a
recent version is needed for full OSX client support. Thus it is
recommended to install directly from github or with pip.
All optional file server services like sftp, ftps and davs rely on
one of the python pbkdf2 modules for password auth support and it is
distributed with the MiG code base.

The four CLAUSE arguments can be used to comment out the explicit
setting of user, group, serveralias and ports in the apache conf by
providing a '# User', '# Group', '# ServerAlias' and '# Listen'. This is
mostly relevant if using apache2 with WSGI.

The generator will inform you about the steps to install your
configuration files in the right locations.


== Running a MiG Server ==
Before you run the MiG daemons you need to have a working configuration for
your daemons in mig/server/MiGserver.conf or another location you can
specify in the MIG_CONF environment variable. Please note that if you
want to use this environment variable, it must be available to *all* MiG
components to work. 
You can use the generator as mentioned above or manually modify e.g. the
localhost example configuration in MiGserver-localhost.conf.
At any time you can verify the validity of your configuration with the
checkconf.py script in the same directory.

Once set up you can use the provided init script to manage all services.
It is also possible to run the services individually as explained below.

The central daemon is grid_script.py which takes care of all job
management on the server. If you want to include grid monitor web pages you
should additionally run the grid_monitor.py daemon. The optional job
notifications and ssh multiplexing daemons are available as grid_imnotify.py
/ grid_imnotify_stdout.py and grid_sshmux.py in the same location. In case
you don't know what they do, you can most likely safely ignore them and
just run the grid_script.py daemon.

All the daemons can be launched from inside the mig/server directory:
cd /path/to/unpacked/mig/source/mig/server
python grid_script.py

Alternatively they can be launched from other locations as long as the
configuration path is provided in the environment: 
export MIG_CONF="/path/to/MiGserver.conf"
python /path/to/unpacked/mig/source/mig/server/grid_script.py

Each daemon will keep running until you actively stop it, so you need
individual shell sessions for each daemon.

For testing purposes this interactive execution is fine, but in more
permanent setups you will probably want to run the MiG daemons as true
daemons so that you can disconnect from the server and leave them
running. The easiest solution to that problem is to run the daemons
inside a GNU Screen session:
screen -S MiG
cd /path/to/unpacked/mig/source/mig/server
python grid_script.py
[ctrl-a d to disconnect]

Then you can disconnect from the server and resume the session any time
later by reattaching the screen session:
screen -S MiG -R

Please refer to 'man screen' or other screen documentation for further
details.


== Adding users ==
You need a either an OpenID login or a MiG certificate+key to fully
interact with any MiG server as a user. If you don't want to set up your
on Certificate Authority (CA) you can use e.g. our certificates.
Please use the certificate request link from http://www.migrid.org if
you haven't got a certificate yet.

If you run your own CA you can simply use the certificate request
mechanism included in MiG (https://server/cgi-sid/reqcert.py) to combine
certificate and user creation. Certificate requests will automatically
result in an email with full certificate and MiG user creation
instructions to the configured MiG admins. 

Otherwise you can use the external certificate sign up request mechanism
included in MiG (https://server/cgi-sid/extcert.py) or simply run the
MiG user creation commands directly as described below.

To manually add a user to your MiG server you need to look up the certificate
fields and run the createuser script:

cd ~/mig/server
./createuser.py

You will be prompted for user details one by one before the user is
added to the local MiG user database. Any user added to this database
can access your MiG server and manage his/her MiG jobs and files. The
user must present a MiG certificate with the exact same Distinguished Name
to get access, however. If you do not use the MiG CA or another CA with
the same Distinguished Name format
(/C=.*/ST=.*/O=.*/CN=.*/emailAddress=.*) you have to supply the -i DN
option for the user to work.

Example: adding myself as a user on a MiG server:

# ./createuser.py
Please enter the details for the new user:
Full Name: Jonas Bardino
Organization: NBI
State: 
2-letter Country Code: DK
Email: bardino@nbi.ku.dk
Comment: This is my own MiG user
Password: 
using user dict: {'comment': 'This is my own MiG user', 'country': 'DK',
'state': '', 'full_name': 'Jonas Bardino', 'organization': 'NBI',
'password': '*****==', 'email': 'bardino@nbi.ku.dk'}
logging to: server.log ; level: info
Creating dirs and files for new user: Jonas Bardino
User name without spaces: Jonas_Bardino

User Jonas Bardino was successfully added to user DB!
DB entry and dirs for Jonas Bardino were created or updated

My MiG certificate with Full Name Jonas Bardino and so on will now give
me access to this development server.

You can find the field details using openssl or by viewing the
certificate imported in a browser.


== Adding resources ==
When you have added yourself as a user on your MiG server, you can open
your personal Resources page on the corresponding web interface and add
resources of all kinds there. Please refer to the wiki pages online for
explanations on each kind of resource and some examples of setups.


== Stopping a MiG server ==
All the daemons can be stopped with ctrl-c and most also support a
SHUTDOWN message through the named input pipe defined in the
configuration file:
echo SHUTDOWN >> /path/to/server.stdin

To completely stop MiG you need to stop all the MiG daemons and the
apache server.


= Uninstalling MiG =
The default server configuration template keeps all MiG files installed
under the single directory where the MiG source code is unpacked, so
uninstalling is simply a matter of deleting that directory. If you
change your server configuration to save e.g. state files outside this
directory you will have to manually clean up those directories as well
to completely uninstall MiG.