Remote access from one system to models and functions of other one using Celery machinery.

Relies on three outstanding python projects:

• Celery
• Django Rest Framework
• Django

Client and server are designed to:

• filter models with Django ORM lookups, Q-objects and excludes;
• change model state (create, update, update or create, delete);
• change model state in bulk mode (more than one object per request);
• atomic get-set model state with bulk mode support;
• call function;
• client does not require Django;

Default configuration of django-celery-rpc must be overridden in settings.py by CELERY_RPC_CONFIG. The CELERY_RPC_CONFIG is a dict which must contains at least two keys: BROKER_URL and CELERY_RESULT_BACKEND. Any Celery config params also permitted (see Configuration and defaults)

setting.py:

# minimal required configuration
CELERY_RPC_CONFIG = {
	'BROKER_URL': amqp://10.9.200.1/,
	'CELERY_RESULT_BACKEND': 'redis://10.9.200.2/0',
}

setting.py:

# alternate request queue and routing key
CELERY_RPC_CONFIG = {
	'BROKER_URL': amqp://10.9.200.1/,
	'CELERY_RESULT_BACKEND': amqp://10.9.200.1/',
	'CELERY_DEFAULT_QUEUE': 'celery_rpc.requests.alter_queue',
	'CELERY_DEFAULT_ROUTING_KEY': 'celery_rpc.alter_routing_key'
}

setting.py:

# this settings will be used in clients by default
CELERY_RPC_CONFIG = {
	'BROKER_URL': amqp://10.9.200.1/,
	'CELERY_RESULT_BACKEND': 'redis://10.9.200.2/0',
}

# 'eggs' alternative configuration will be explicitly passed to the client constructor
CELERY_RPC_EGGS_CLIENT = {
	# BROKER_URL will be used by default from section above
	'CELERY_RESULT_BACKEND': amqp://10.9.200.1/',
	'CELERY_DEFAULT_QUEUE': 'celery_rpc.requests.alter_queue',
	'CELERY_DEFAULT_ROUTING_KEY': 'celery_rpc.alter_routing_key'
}

*Note:

1. client and server must share the same BROKER_URL, RESULT_BACKEND, DEFAULT_EXCHANGE, DEFAULT_QUEUE, DEFAULT_ROUTING_KEY
2. different server must serve different request queues with different routing keys or must work with different exchanges*

example.py

from celery_rpc.client import Client
from django.conf import settings

# create client with default settings
span_client = Client()

# create client for `eggs` server
eggs_client = Client(CELERY_RPC_EGGS_CLIENT)

Filtering

span_client.filter('app.models:MyModel', kwargs=dict(filter={'a__exact':'a'}))

Filtering with Q object

from django.db.models import Q
span_client.filter('app.models:MyModel', kwargs=dict(filters_Q=(Q(a='1') | Q(b='1')))

Also, we can use both Q and lookups

span_client.filter('app.models:MyModel', kwargs=dict(filter={'c__exact':'c'}, filters_Q=(Q(a='1') | Q(b='1')))

Exclude supported

span_client.filter('app.models:MyModel', kwargs=dict(exclude={'c__exact':'c'}, exclude_Q=(Q(a='1') | Q(b='1')))

You can mix filters and exclude, Q-object with lookups. Try it yourself. ;)

Full list of available kwargs:

filters - dict of terms compatible with django lookup fields
offset - offset from which return a results
limit - max number of results
fields - list of serializer fields, which will be returned
exclude - lookups for excluding matched models
order_by - order of results (list, tuple or string),
    minus ('-') set reverse order, default = []
filters_Q - django Q-object for filtering models
exclude_Q - django Q-object for excluding matched models

List of all MyModel objects with high priority

span_client.filter('app.models:MyModel', high_priority=True)

celery worker -A celery_rpc.app

Server with support task consuming prioritization

celery multi start 2 -A celery_rpc.app -Q:1 celery_rpc.requests.high_priority

Note, you must replace 'celery_rpc.request' with actual value of config param CELERY_DEFAULT_QUEUE

Command will start two instances. First instance will consume from high priority queue only. Second instance will serve both queues.

For daemonization see Running the worker as a daemon

python django-celery-rpc/celery_rpc/runtests/runtests.py

OVERRIDE_BASE_TASKS = {
    'ModelTask': 'package.module.MyModelTask',
    'ModelChangeTask': 'package.module.MyModelChangeTask',
    'FunctionTask': 'package.module.MyFunctionTask'
}

Supported class names: ModelTask, ModelChangeTask, FunctionTask

• Set default non-generic model serializer.
• Test support for RPC result backend from Celery.
• Token auth and permissions support (like DRF).
• Resource map and strict mode.

Thanks for all who contributing to this project:

• https://github.com/tumbler
• https://github.com/voron3x
• https://github.com/dtrst
• https://github.com/anatoliy-larin
• https://github.com/bourivouh
• https://github.com/tumb1er