This repository contains Python code shared across the lab. It is public to allow functions to be used in published code.

It is important to note that all dependencies are optional. Being a collaborative toolbox, this package contains a relatively large number of functions for many different uses, extending or patching the functionality of a large number of sometimes heavy packages. For example, there are very few situations where it would make sense to simultaneously install PyTorch, TensorFlow and JaX, but helper functions for any of these could be included in this toolbox.

Thus, to avoid forcing users to install unneeded dependencies, modules are organized into different contexts. When installing the toolbox, one specifies which contexts are needed as optional or 'extra' dependencies. The 'all' option is provided to install all dependencies, which is especially useful for running the test suite.

Each context is associated to one or more modules. To be able to import a particular module, simply install a context associated with it.

These are the currently defined contexts, and their associated modules:

• config: config
  • config-holoviews: config/holoviews.py
• 'iotools': iotools.py
• 'pymc3': pymc3.py, pymc_typing.py, + everything from 'theano', 'typing', 'parameters'
• 'theano': theano.py, cgshim.py, optimizers.py, + everything from 'typing'
• 'torch': torch.py, + everything from 'iotools', 'typing'
• 'tqdm': tqdm.py
• 'typing': typing.py, units.py, pydantic.py, serialize.py, transform.py
• 'utils': utils.py, meta.py
• 'parameters': parameters
• 'all': everything

If you only need mackelab_toolbox as a dependency, the following will suffice

pip install --upgrade pip wheel
pip install "mackelab-toolbox[…] @ git+https://github.com/mackelab/mackelab-toolbox#egg=mackelab-toolbox"

For example, you need functions from modules associated to the 'typing' and 'utils' contexts, you would do the following:

pip install --upgrade pip wheel
pip install "mackelab-toolbox[typing,utils] @ git+https://github.com/mackelab/mackelab-toolbox#egg=mackelab-toolbox"

Alternatively, you can clone the repository first and then install as usual:

pip install --upgrade pip wheel
git clone https://github.com/mackelab/mackelab-toolbox.git
pip install mackelab-toolbox[typing,utils]

As usual, it is recommended to install within a virtual environment.

Upgrading pip and wheel is not usually required, but in some cases can avoid problems. It also prevents pip from warning that it is out of date.

If the toolbox functionality you need is still in development, you will likely find that cloning locally and performing an editable install (with pip install -e) is easier to keep up to date. See Contributing below.

Caveat for the theano context: The theano_shim package is not yet on pip, and therefore must be installed from the repo before installing mackelab-toolbox. This is only required if you install the theano context.

Due to its nature, required features in this package will often be developed as they are needed in some other project. Thus it is not always possible to simply install the version from PyPI. If you want your project to install mackelab_toolbox from GitHub, you can add the following to its requirements files:

-e git+https://github.com/mackelab/mackelab-toolbox@master#egg=mackelab-toolbox[typing,utils]

This allows you to specify the branch (here master) and the required contexts (here typing and utils).

See demo folder for notebooks illustrating functions included in this repository.

(TODO) We should use JupyterBook to convert demo & source notebooks into searchable documentation.

The recommended import abbreviation is mtb:

import mackelab_toolbox as mtb
import mackelab_toolbox.iotools
from mackelab_toolbox.typing import Array

mtb.iotools.save(...)

def f(x: Array):
  ...

If you cloned the repository and installed with -e, then a single git pull in the locally cloned folder will suffice for updating.

Otherwise, use

pip install --update "mackelab-toolbox[…] @ git+https://github.com/mackelab/mackelab-toolbox#egg=mackelab-toolbox"

If you intend to make modifications to mackelab-toolbox, clone to the location where you keep your working files, and install from there. Use a development install (-e) to avoid having to reinstall each time you make a modification:

git clone https://github.com/mackelab/mackelab-toolbox.git
pip install -e mackelab-toolbox[all]

As above, use options to indicate which contexts you need.

• When adding functions, consider using JupyText to include rich inline documentation in the source file.

  Tip: You can use if __name__ == "__main__" guards to include examples directly in source files.

• If possible, also include a notebook in demos illustrating how/when to use the function.
• Report issues.

You will need pytest to run the test suite:

pip install pytest pytest-forked

Then run tests with

pytest --forked

The pytest-forked package (which provides the --forked option) are required due to how the typing context implements dynamic types. This may change in the future.

At present, it is difficult to run tests without having all dependencies installed. This is something we may improve in the future.

• By its nature, this toolbox package will contain modules of varying levels of maturity. It is not clear what the best way to indicate maturity level would be – probably an indication at the the function would be needed.
  • At present (Apr 2021), all modules are somewhere between an 'alpha' and a 'beta' stage: they have all been use-tested, but may still see structural modifications. In particular:
    • Much of the functionality in plot.py is probably redundant with Seaborn.
    • The requirement to “freeze” types from typing.py is a confusing hack and it would be great if we could do without.