Pyople provides simple yet flexible way to create an distribute tasks and sub-tasks in a cluster using ZeroMQ and Python. Only adding a decorator above a function the task will be executed in parallel.
Clustering shouldn't be too much complicated. Not in Python.
In the file: :
@multiprocess
def sum(a, b):
return a + b
@multiprocess
def itersum(a, b):
return list_mp([sum(i, b) for i in a])
In the terminal: :
>>> a = [1, 1, 2 ,3 ,5 ,8]
>>> b = 1
>>> itersum(a, b)
Management of tasks distribution, creation of sub-tasks in parallel and results recollection is transparent to the user. Each task follows a classic pipeline scheme: one fan starts the task, the worker runs it and a collector obtains the result.
The source code is hosted on GitHub at: http://github.com/pyople
Binary installers for the latest released version are available at the Python package index:
http://pypi.python.org/pypi/pyople/
And via easy_install
or pip
:
easy_install pyople
pip install pyople
BSD