Skip to content

A module optimizing the task graph in Dask's threaded scheduler in order to faster I/O operations.

Notifications You must be signed in to change notification settings

big-data-lab-team/dask_io

Repository files navigation

DASK IO

Build Status Coverage Status

A module optimizing the task graph in Dask's threaded scheduler in order to faster I/O operations.

Third-party libraries

Use requirements.txt or requirements_conda.txt to install the dependencies.

For a conda environment:

conda create --name <env> --file requirements_conda.txt

For a pip environment:

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

Troubleshooting

If there are missing dependencies when creating the conda environment

    1. Create a new conda environement conda create --name <envname>
    1. activate the environment conda activate <envname>
    1. 'cd' inside the dask_io directory (where the requirements_conda.txt file is)
    1. install the dependencies that are not missing while read requirement; do conda install --yes $requirement; done < requirements_conda.txt

Note for developers

To create the requirements_conda file:

conda list -e > requirements_conda.txt

To create the ``requirements.txt" file from conda environment:

pip freeze > requirements.txt

Remove mkl dependencies from ``requirements.txt" to get rid of compatibility issues.

About

A module optimizing the task graph in Dask's threaded scheduler in order to faster I/O operations.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published