This repository contains the tools needed to build the Open-CE project.

The open-ce tool allows a user to build collections of conda recipes described within a collection of feedstocks. It also provides tools for validating potential conda environments and running tests for feedstocks.

• conda >= 3.8.3
  • The conda tool can either be installed through Anaconda or Miniconda.
• conda-build >= 3.20
  • Once conda is installed, conda-build can be installed with the command: conda install conda-build
• networkx >= 2.5
• python >= 3.6
• docker >= 1.13 or podman >= 2.0.5
  • docker or podman required only when building within a container (see below).

Currently CUDA 10.2 and 11.0 are supported by the recipes in Open-CE. Please see doc/README.cuda_support.md for details on setting up a proper build enviornment for CUDA support.

Open-CE expects the CUDA_HOME environment variable to be set to the location of the CUDA installation. Note that not all recipes work when CUDA_HOME references a non-standard CUDA installation location. Reference the cuda README for more information.

When building packages that use CUDA, a tar package of TensorRT for the intended CUDA version will need to be downloaded ahead of time. The downloaded file should be placed in a new local directory called local_files. The cuda README has more information.

To get the Open-CE build tools, one can either install them via conda from the Open-CE channel, install them via pip from github or clone the source code from github as below -

# Conda install from the open-ce channel
conda install -c open-ce open-ce-builder

OR

# Pip install from the main branch
pip install git+https://github.com/open-ce/open-ce-builder.git@main

OR

# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce-builder.git
cd open-ce-builder
pip install -e .

To build an entire integrated and functional conda channel using Open-CE, start by installing the needed tools in the Requirements section above. The open-ce build env command can then be used to build a collection of Open-CE packages. An Open-CE environment file needs to be passed in as input. A selection of environment files are provided within the open-ce repo for different frameworks such as TensorFlow and PyTorch. The output from running open-ce build env will be a local conda channel (by default called condabuild) and one or more conda environment file(s) in the output folder depending on the selected build configuration. For more details on open-ce build env, please see doc/README.open_ce_build.md.

The following commands will use the opence-env.yaml Open-CE environment file to build all of the Open-CE packages for Python 3.7 (the default), including CUDA builds and cpu-only builds (also the default). The commands should be run from within the same directory that contains local_files.

# Clone Open-CE from GitHub
git clone https://github.com/open-ce/open-ce.git
# Build packages
open-ce build env ./open-ce/envs/opence-env.yaml

The open-ce tool will also automatically look for environment files within the open-ce repo's env directory if an environment file isn't found locally.

The following commands will build the opence-env.yaml environment file:

# Build packages
open-ce build env opence-env

A specific version of an environment file from the open-ce repo can be built using the --git_tag flag.

The following commands will build version 1.1.4 of the open-ce environment file provided within the open-ce repo:

# Build packages
open-ce build env --git_tag open-ce-v1.1.4 opence-env

The following commands will use the opence-env.yaml Open-CE environment file from a specific Open-CE release to build all of the Open-CE packages for Python 3.6, 3.7 and 3.8, including only CUDA builds. The commands should be run from within the same directory that contains local_files.

# Build packages
open-ce build env --python_versions 3.6,3.7,3.8 --build_types cuda opence-env

Passing the --container_build argument to the open-ce build env command will create a container image and perform the actual build inside of a container based on that image. This will provide a "clean" environment for the builds and make builds more system independent. It is recommended to build with this option as opposed to running on a bare metal machine. For more information on the --container_build option, please see doc/README.open_ce_build.md.

The open-ce build feedstock command can be used to build a single feedstock (which could produce one or more conda packages). The output from running open-ce build feedstock will be a local conda channel (by default called condabuild). For more details on open-ce build feedstock, please see doc/README.open_ce_build.md.

The following commands will build all of the packages within a feedstock named MY_FEEDSTOCK.

# Clone Open-CE Environments from GitHub
git clone https://github.com/open-ce/open-ce.git
# Clone MY_FEEDSTOCK from GitHub
git clone https://github.com/open-ce/MY_FEEDSTOCK-feedstock.git
# Build packages
cd MY_FEEDSTOCK-feedstock
open-ce build feedstock --conda_build_config ../open-ce/envs/conda_build_config.yaml

After performing a build, a local conda channel will be created. By default, this will be within a folder called condabuild (it can be changed using the --output_folder argument). After the build, packages can be installed within a conda environment from this local channel. If the packages are built using open-ce build env script, then a conda environment file will also be generated which can be used to generate a conda environment with the built packages installed in it. See conda's documentation for more information on conda environments.

The following command will install a package named PACKAGE from the local conda channel condabuild into the currently active conda environment.

conda install -c ./condabuild PACKAGE

The following command can be used to create a conda environment using a conda environment file.

conda env create -f <conda_environment_file>

After performing the build using the open-ce build env tool, the open-ce test tool can be used to either test a package or a collection of packages. For more details on open-ce test, please see doc/README.open_ce_test.md.

After performing the build using open-ce build env, the open-ce build image command can be used to create a runtime container image containing the newly created conda channel, as well as a conda environment with the newly build Open-CE packages. For more details on open-ce build image, please see doc/README.open_ce_build.md.

For contribution information, please see the CONTRIBUTING.md page.