Dispatch your trivially parallizable jobs with sharedmem.
To cite sharedmem use the DOI below
Now also supports Python 3.
- sharedmem.empty creates numpy arrays shared by child processes.
- sharedmem.MapReduce dispatches work to child processes, allowing work functions defined in nested scopes.
- sharedmem.MapReduce.ordered and sharedmem.MapReduce.critical implements the equivelant concepts as OpenMP ordered and OpenMP critical sections.
- Exceptions are properly handled, including unpicklable exceptions. Unexpected death of child processes (Slaves) is handled in a graceful manner.
Functions and variables are inherited from a fork
syscall and the copy-on-write mechanism, except sharedmem variables which are writable from both child processes or the main process. Pickability of objects is not a concern.
Usual limitations of fork
do apply. sharedmem.MapReduce is easier to use than multiprocessing.Pool, at the cost of not supporting Windows.
For documentation, please refer to http://rainwoodman.github.io/sharedmem .
Here we provide two simple examples to illustrate the usage: