Skip to content

AndreaCensi/compmake

Repository files navigation

CircleCI

Compmake

Compmake is an unobstrusive Python library that provides Make--like facilities to a Python application, including:

  • Minimal effort job management and parallelization (multiple CPU on a single host, cluster computing using SGE, and experimental support for cloud computing using Multyvac).
  • Caching of temporary results: you can interrupt your program and restart it without losing what was already computed.
  • Failure tolerance: if a job fails, other jobs that do not depend on it continue to be executed.
  • A console for inspecting failures and partial completion, with familiar commands such as make, clean, etc.

Please see the manual at:

http://purl.org/censi/compmake-manual

Issues

Please report any problem using Github's issue tracker at

http://github.com/AndreaCensi/compmake/issues

Acknowledgements

Your tax dollars at work! Compmake's development was supported by the National Science Foundation in the National Robotics Initiative program under grant #1405259.

Screencasts

Some animations of the job graph

Robustness to job failure

Simple dynamic jobs

Dynamic jobs and recursive parallel executions

Changelog

  • v3.5 - Added why command --- compact error visualization. Removed color effects which might not be suitable for all console types.
  • v3.4.1 - Bug fix; experimental multyvac_sync
  • v3.4.0 - Changed the way that dynamic job IDs are generated to avoid a race condition in a corner case.
  • v3.3.7 - Bug fix.
  • v3.3.6 - Experimental Multyvac backend.

About

Compmake is a non-obtrusive module that provides Makefile-like facilities (including parallel processing) for batch Python applications.

Resources

License

Stars

Watchers

Forks

Packages

No packages published