Basic experiment to benchmark ray against python multiprocessing
conda create -n ray-vs-mp -c conda-forge numpy matplotlib tqdm scikit-image memory_profiler
pip install ray[all]
Total memory
mprof run --include-children main.py
mprof plot
Process by process memory
mprof run --multiprocessing main.py
mprof plot
- on the head
ray start --head --port=6379
- on the children
copy out from ray start in the head
- on the code
ray.init(address="auto")
- run normally