[WIP] This project is still under development

A task distribution system based on rabbitmq message queue. Assigned tasks are published to subscribed endpoints (worker).

Separate exchanges are declared for every configured task type.

The minimal http rest api creates on receiving of a customer request a new task and send the task to the endpoints by the amqp publish/subscribe pattern. On receiving the task by the endpoints:

• create a task-response in the database
• perform the task and
• update the task-response after finishing the task

The customer can get the tasks status and responses by request the api.

Raw overview:

                    POST apiurl                           AMQP                AMQP
create task request +-----------------> create task,      +-------> task type +-----------> all worker asynchronous:
with data as json                       store task in db,           exchange  |               create taskResponse,
paylod                                  enqueue task                          +----------->   perform task,
                                                                                              update task status

                    GET apiurl
                    +-----------------> get task response data

Get the source code by:

git clone git@github.com:MatthiasWiesner/tasker.git

Or download and unzip:

wget https://github.com/MatthiasWiesner/tasker/archive/master.zip

You need to have installed Vagrant and some virtual provider (e.g. VirtualBox)

In the folder run:

export MYSQL_PASSWD=<tasker_user_passwd>; vagrant up

All requirement should be installed. The mysql tasker user has to be created before build tasker. Therefor you need to export the password as an environment variable (must be the same as in the tasker manager.db.uri configuration).

Get into the vagrant box by:

vagrant ssh
cd /vagrant/

An executable script bin/buildout should be created and run. This script install all necessary libraries. Further, some executable scripts are created:

• bin/initialize: run this script to initialize (or reset) the database tables and rabbitmq vhost
• bin/supervisord: run this script to start the supervisor daemon. The supervisord daemon will start and control the api and the manager according to the configuration in buildout.cfg (supervisord section)
• bin/supervisorctl: supervisor control (for more information see supervisord.org)
• bin/api: run this script to start the api (for debug running)
• bin/manager: run this script to start the task manager (for debug running)

You can test the tasker by:

create a task:

curl -X POST -H 'content-type: application/json' -d@testdata.json http://localhost:8888/worker
{"status": "SENT", "endpoints": ["console1", "console2"], "start_time": "2015-05-06 01:14:24", "identifier": "e7bd91d0-16ad-11e2-892e-0800200c9a66", "type": "worker", "id": 1, "payload": {"foo": "bar"}}

After 20 seconds you should get the complete answer from 2 workers:

curl http://localhost:8888/worker/e7bd91d0-16ad-11e2-892e-0800200c9a66
{"task": {"status": "DONE", "endpoints": ["console1", "console2"], "start_time": "2015-05-06 01:14:24", "identifier": "e7bd91d0-16ad-11e2-892e-0800200c9a66", "type": "worker", "id": 1, "payload": {"foo": "bar"}}, "taskResponseList": [{"status": 1, "endpoint": "console1", "task_id": 1, "start_time": "2015-05-06 01:14:24", "payload": {"yiya": "yippie", "yeah": 1234123}, "errnr": 0, "last_modified_time": "2015-05-06 01:14:44", "id": 1, "errmsg": null}, {"status": 1, "endpoint": "console2", "task_id": 1, "start_time": "2015-05-06 01:14:24", "payload": {"yiya": "yippie", "yeah": 1234123}, "errnr": 0, "last_modified_time": "2015-05-06 01:14:44", "id": 2, "errmsg": null}]}

For the tests default values are used but to configure tasker you can edit the configuration yaml file according to the run mode. The configuration file is located in config/<run_mode>.config.yaml.

In this file are the settings:

• task type: For each task type a separate exchange and api handler will be created. Furthermore, for each task type can have multiple workers subscribe on to the exchange. Set this as a list of strings
• api.port: set the port
• api.wait_for_endpoints_timeout: time in which all endpoints should received the task
• manager.db.verbose: verbose flag (unused)
• manager.db.uri: mysql uri
• manager.message_queue.*: several rabbitmq preferences
• rabbitmq.*: rabbitmq settings to initialize/reset the rabbitmq user/vhost (superuser permissions required!).

The requests api url is http://<host>:<port>/<task_type>

A task can be created by POST requesting the restful api. The requests content-type has to be "application/json". The requests body has to be a json object (example data):

	{
	   "identifier":"e7bd91d0-16ad-11e2-892e-0800200c9a66",
	   "payload":{
		  "foo":"bar"
	   },
	   "endpoints":[
		  "console1",
		  "console2"
	   ]
	}

• identifier: is a mandatory customers task identifier
• payload: are optional customers data to perform the task
• endpoints: are a optional list of known endpoints they perform the task. If the list is given, then it is checked whether all endpoints have been received the task. If an endpoint has not been given the task, then the task is marked as incomplete. At the current status, there is no retry mechanism.

The response contains the created task as json object.

The tasks status can be retrieved with a GET request http://<host>:<port>/<task_type>/<task_identifier>. The responses content-type is "application/json", the body contains the task and the endpoints responses (after perform the task) as a json object.

On the endpoint the handler must be implemented according to the customer needs. To get this implemented, create a module with the task-types name in the manager.handler package (e.g. worker.py). The module must implement the handle_task function.

• Your web based administration software requires to do a long time job (> webservers timeout).
• you have to do an application update on a greater list of server