ScalaFunctional exists to make functional programming with collections easy and intuitive in Python. It implements the Scala collections and Apache Spark RDD APIs in Python to provide access to a rich and declarative way of defining data pipelines.

Below is a comparison of using Python builtins, list comprehensions, and ScalaFunctional

l = [1, 2, -1, -2]

# Python style
reduce(lambda x, y: x * y, map(lambda x: 2 * x, filter(lambda x: x > 0, l)))

# Python list comprehension
reduce(lambda x, y: x * y, [2 * x for x in l if x > 0])

# ScalaFunctional style
from functional import seq
seq(l).filter(lambda x: x > 0).map(lambda x: 2 * x).reduce(lambda x, y: x * y)

ScalaFunctional also makes other tasks much easier than using only python's built in utilities. Below are examples of several tasks such as word count and merging data from two streams.

# ScalaFunctional word count
l = seq("I dont want to believe I want to know".split(" "))
l.map(lambda word: (word, 1)).reduce_by_key(lambda x, y: x + y)
# [('dont', 1), ('I', 2), ('to', 2), ('know', 1), ('want', 2), ('believe', 1)]

# List of key value pairs, where the key is an ID and we want values joined
names = [(1, 'spark'), (2, 'hadoop'), (3, 'django')]
languages = [(1, 'scala'), (2, 'java'), (3, 'python')]
joined = seq(names).inner_join(languages).to_dict()
# {1: ('spark', 'scala'), 2: ('hadoop', 'java'), 3: ('django', 'python')}

More examples and motivation.

ScalaFunctional is available on pypi and can be installed by running:

# Install from command line
$ pip install scalafunctional

Or by

git clone git@github.com:EntilZha/ScalaFunctional.git
python setup.py install

Then import the package using: from functional import seq

Full documentation can be found at scalafunctional.readthedocs.org.

ScalaFunctional has three types of functions:

1. Streams read data for use by the collections API. In 0.3.1 the only stream function is seq, however in 0.4.0 this is getting expanded to read data from text, csv, json, and jsonl files.
2. Transformations: These mutate data from streams with functions such as map, flat_map, and filter
3. Actions: These cause a series of transformations to evaluate to a concrete value. For example, to_list, reduce, and to_dict are examples of actions.

To summarize, suppose we have: seq(1, 2, 3).map(lambda x: x * 2).reduce(lambda x, y: x + y), seq is the stream, map is the transformation, and reduce is the action.

The primary entrypoint to using ScalaFunctional is through functional.seq. seq can take any iterable as input and returns a functional.Sequence which exposes the collections API described in the table below. seq can be called in various ways demonstrated below:

# Passing a list
seq([1, 1, 2, 3]).to_set()
# [1, 2, 3]

# Passing direct arguments
seq(1, 1, 2, 3).map(lambda x: x).to_list()
# [1, 1, 2, 3]

# Passing a single value
seq(1).map(lambda x: -x).to_list()
# [-1]

Below is the complete list of functions which can be called on the object created by seq otherwise known as a functional.Sequence.

Another python package named fn is also helpful. It can be installed via pip install fn and can remove the need for direct lambdas.

from functional import seq
from fn import _

seq(1, 2, 3).map(_ * 2).reduce(_ + _)
# 12

Implement new ways to have ScalaFunctional ingest data and write it out. Principally this means implementing reading from files (csv, json, etc) and writing back to them natively. This is being implemented in functional.streams if you would like to check progress

• Parallel execution engine for users wanting to run operations in parallel
• Decide if package is stable enough to prepare a 1.0 release

Any contributions or bug reports are welcome. Thus far, there is a 100% acceptance rate for pull requests and contributors have offered valuable feedback and critique on code. It is great to hear from users of the package, especially what it is used for, what works well, and what could be improved.