High Performance Analytics Toolkit (HPAT) scales analytics/ML codes in Python to bare-metal cluster/cloud performance automatically. It compiles a subset of Python (Pandas/Numpy) to efficient parallel binaries with MPI, requiring only minimal code changes. HPAT is orders of magnitude faster than alternatives like Apache Spark.

HPAT's documentation can be found here.

HPAT can be installed in Anaconda environment easily (Linux/Mac):

conda create -n HPAT python=3.6
source activate HPAT
conda install numpy scipy pandas
conda install pyarrow=0.8.* mpich -c conda-forge
conda install hpat -c ehsantn

Windows installaton requires Intel MPI to be installed separately instead of mpich. The rest is the same:

conda create -n HPAT python=3.6
activate HPAT
conda install numpy scipy pandas
conda install pyarrow=0.7.* -c conda-forge
conda install hpat -c ehsantn

An HPAT docker image is also available for running containers. For example:

docker run -it ehsantn/hpat bash

Here is a Pi calculation example in HPAT:

import hpat
import numpy as np
import time

@hpat.jit
def calc_pi(n):
    t1 = time.time()
    x = 2 * np.random.ranf(n) - 1
    y = 2 * np.random.ranf(n) - 1
    pi = 4 * np.sum(x**2 + y**2 < 1) / n
    print("Execution time:", time.time()-t1, "\nresult:", pi)
    return pi

calc_pi(200000000)

Save this in a file named pi.py and run (on 8 cores):

mpiexec -n 8 python pi.py

This should demonstrate about 100x speedup compared to regular Python version without @hpat.jit and mpiexec.

These academic papers describe the underlying methods in HPAT:

• HPAT paper at ICS'17
• HPAT at HotOS'17
• HiFrames on arxiv