KQ (Kafka Queue) is a lightweight Python library which provides a simple API to process jobs asynchronously in the background. It uses Apache Kafka and is designed primarily for ease of use.

• Apache Kafka 0.9+
• Python 2.7+ 3.4+ or 3.5+

First, ensure that your Kafka instance is up and running:

# This command is just an example
~$ ./kafka-server-start.sh -daemon server.properties

Let's say you want to run the following function asynchronously:

import time

def my_func(foo, bar, baz=None):
    """This is a blocking function."""
    time.sleep(10)
    return foo, bar, baz

Start a KQ worker:

~$ kq worker --verbose
[INFO] Starting Worker(topic=default) ...

Enqueue the function call as a job:

# Import the blocking function
from my_module import my_func

# Initialize a queue
from kq import Queue
q = Queue()

# Enqueue the function call
q.enqueue(my_func, 1, 2, baz=3)

Sit back and watch the worker process it in the background:

~$ kq worker --verbose
[INFO] Starting Worker(topic=default) ...
[INFO] Processing Record(topic=default, partition=5, offset=3) ...
[INFO] Running Job 1b92xle0: my_module.my_func(1, 2, baz=3) ...
[INFO] Job 1b92xle0 returned: (1, 2, 3)

Enqueue a job to be processed in-order with other jobs with a particular key:

# Import the blocking function
from my_module import my_func

# Initialize a queue
from kq import Queue
q = Queue()

# Enqueue the function call as Job
import uuid, time
from kq import Job
job = Job(
    str(uuid.uuid4()),
    timestamp=int(time.time()),
    func=my_func,
    args=(1, 2),
    kwargs={'baz': 3},
    key="task_category_1"
)
q.enqueue(job)

Check out the full documentation for more details!

To install a stable version from PyPI (recommended):

~$ pip install kq

To install the latest version directly from GitHub:

~$ pip install -e git+git@github.com:joowani/kq.git@master#egg=kq

You may need to use sudo depending on your environment setup.

This project was inspired by RQ and built on top of kafka-python.