A webapp to retrieve historical information about messages on the fedmsg bus. It is a JSON api for the datanommer message store.

https://apps.fedoraproject.org/datagrepper/

• virtualenvwrapper
• Postgresql

Install postgresql and virtualenvwrapper:

$ sudo yum install -y postgresql-server python-virtualenvwrapper

Use a virtualenv:

$ mkvirtualenv datagrepper
$ workon datagrepper

Install dependencies:

$ pip install -r requirements.txt
$ pip install psycopg2

Set up the fedmsg consumer for the job runner:

$ python setup.py develop

In normal operations, the datanommer consumer daemon will be running somewhere and continuously stuff each new fedmsg message that it sees into a postgres DB. If you're just sitting down to hack on datagrepper, that won't be your situation so you'll need a dump of the database.

Install postgres (and fedmsg, while we're at it):

$ sudo yum install -y postgresql-server fedmsg
$ sudo postgresql-setup initdb

Make sure postgres is set to allow connections over tcp/ip using password authentication. Edit the /var/lib/pgsql/data/pg_hba.conf. You might find a line like this:

host    all             all             127.0.0.1/32            ident

Instead of that line, you need one that looks like this:

host    all             all             127.0.0.1/32            md5

Become yourself again (not the postgres user) and start up postgres:

$ sudo systemctl restart postgresql.service

Become the postgres user (again) and run the psql command. Use that psql shell to setup the DB, the user, and privileges:

$ sudo su - postgres
$ psql
# create database datanommer;
# create user datanommer with password 'bunbunbun';
# grant all privileges on database datanommer to datanommer;
# \q

Back in the bash shell (but still as the postgres user), grab a DB dump and restore it:

$ wget http://ralph.fedorapeople.org/datanommer-2014-01-31.dump.xz
$ xzcat datanommer-2014-01-31.dump.xz | psql datanommer

You have to configure your development datagrepper instance to talk to postgres (by default, it looks for a sqlite database). Edit fedmsg.d/example-datagrepper.py and give it these contents:

config = {
    'datanommer.enabled': False,
    'datanommer.sqlalchemy.url': 'postgresql+psycopg2://datanommer:bunbunbun@localhost:5432/datanommer',
    'fedmsg.consumers.datagrepper-runner.enabled': True,
}

As your normal old user self, run the development server:

$ workon datagrepper
$ python runserver.py

In a browser, visit http://localhost:5000 to see the docs.

You can quick test that you can get data by running:

$ sudo yum install -y httpie
$ http get localhost:5000/raw/ delta==1000000 rows_per_page==1

Without starting fedmsg-hub, the job runner won't actually run jobs:

$ workon datagrepper
$ fedmsg-hub