Python wrapper for submitting jobs via bsub with the option to do so in a container environment.
make install-full
This creates a funlib.run config file ~/.funlib.run that contains default parameters that can be overwritten for each specific run:
num_gpus = 1
memory = 25600
working_directory = .
singularity = ""
host = ""
queue = "normal"
environment = ""
batch = False
mount_dirs = ""
There are three useful ways to use funlib.run:
- Direct usage via command line arguments (overwrites config file defaults):
python run.py -p "python train.py" -c 5 -g 1 -q normal -s path-to-singularity-image
python run_singularity.py -p "python mknet.py" -s path-to-singularity-image- Indirect call via another script:
from funlib.run import run, run_singularity
run(command="python train.py",
num_cpus=5,
num_gpus=1,
queue="normal",
execute=True)
run_singularity(command="python mknet.py",
singularity_image="path_to_image",
execute=True)- Command creation and subsequent call:
from funlib.run import run, run_singularity
from subprocess import check_call
run_command = run(command="python train.py",
num_cpus=5,
num_gpus=1,
queue="normal",
execute=False)
check_call(run_command,
shell=True)
run_singularity_command = run_singularity(command="python mknet.py",
singularity_image="path_to_image",
execute=False)
check_call(run_singularity_command,
shell=True)When used with daisy.call do not expand the cmd to a string via setting expand=False:
cmd = run(command=base_command,
queue=queue,
num_gpus=1,
num_cpus=num_cpus,
singularity_image=singularity_container,
mount_dirs=mount_dirs,
execute=False,
expand=False)
daisy.call(cmd, log_out=log_out, log_err=log_err)