An open-data framework for the public data of the Austrian Parliament

Check out the more complete documentation over at offenesparlament.readthedocs.org.

• Vagrant
• Vagrant Hostmanager Plugin
• Virtualbox
• Ansible (preferrably via pip install ansible)

1. Clone the github repository (duh)
2. Navigate into the project dir cd OffenesParlament
3. Setup and run the vagrant VM vagrant up. All requirements will be installed automatically inside the VM which may take a few minutes the first time.
4. The script might ask you for your password as it will add offenesparlament.vm pointing to this VM to your hosts-file. It also automatically creates a django superuser admin with password admin.
5. Log in to the running VM with vagrant ssh
6. For the initial scraping instructions see below
7. Run the server inside the VM (0.0.0.0 lets the server respond to requests from outside the VM - ie your physical machine where you probably run your browser)

cd offenesparlament
python manage.py runserver 0.0.0.0:8000

8. If you work on client files that have to be compiled (CSS, JS) you have to run grunt as well. ATM we have the tasks dev and reloading. dev watches and regenerates files when their sources change. (Remember that sources also change when you do a git pull and generated client files aren't commited to git) And reloading does that and uses Browsersync to reload your browser when files change.

cd /vagrant
grunt dev

9. To exit and shutdown the VM run

exit
vagrant halt

Vagrant provisioners can be used to update the dev setup and to flush the database:

'bootstrap' installs the necessary dependencies:

vagrant provision --provision-with bootstrap

'reset_db' clears and re-creates the postgres database and runs migrations:

vagrant provision --provision-with reset_db

Documentation is available via Sphinx. To generate cd to the docs directory and run:

make html

The documentation will then be available at docs/build/html/index.html.

There is also an online version available over at offenesparlament.readthedocs.org.

In case you need to reset the database (delete all migrations, flush the db content, recreate all objects etc.), run these commands in the django project folder 'offenesparlament':

bin/clear_db.sh

It's possible to view the current database model residing in the op_scraper app by calling:

bin/graph_models.sh

A png-image will be generated as ignore/models.png.

There are currently five available scrapers, which should initially run in this order:

1. llp (legislative periods)
2. persons (for instance Rudolf Anschober)
3. administrations (also persons, but those that are/were in a a Regierung)
4. pre_laws (for instance Buchhaltungsagenturgesetz, Änderung (513/ME))
5. laws_initiatives (for instance ÖBIB-Gesetz 2015 (458 d.B.))

To run a scraper, use the following command:

python manage.py scrape crawl <scraper_name>

for instance:

python manage.py scrape crawl persons

The law_initiatives scraper also has an additional parameter to define, which legislative period to scan; per default, it scrapes the periods from XX to XV. This can be overriden this way:

python manage.py scrape crawl -a llp=21 laws_initiatives

to only scrape that period. Careful though: scraping of periods before the 20th legislative period is not possible as of yet (since there are no machine-readable documents available).

Furthermore, all of the scrapers try to utilize the parlament website's timestamp to skip objects that haven't changed since the were last scraped. To suppress this behaviour for a complete upgrade, add a ignore_timestamp parameter like so:

python manage.py scrape crawl -a ignore_timestamp=True laws_initiatives

For now, reindexing (or updating the index, for that matter), is only done manually. To have all data indexed, just run:

python manage.py rebuild_index

for a full rebuild (wipes the indices first), or::

python manage.py update_index

to perform a simple update. For this to succeed, make sure ElasticSearch is up and running.

We have a mailing list now, sign up here:

https://lists.metalab.at/mailman/listinfo/offenesparlament_at