This repository is dedicated to store different training material that we have presented at different events. It contains presentations, as well several 'simple' dispel4py workflows For more complex dispel4py workflows, you can visit the dispel4py workflows GitHub repository
dispel4py is a free and open-source Python library for describing abstract stream-based workflows for distributed data-intensive applications. It enables users to focus on their scientific methods, avoiding distracting details and retaining flexibility over the computing infrastructure they use. It delivers mappings to diverse computing infrastructures, including cloud technologies, HPC architectures and specialised data-intensive machines, to move seamlessly into production with large-scale data loads. The dispel4py system maps workflows dynamically onto multiple enactment systems, such as MPI, STORM and Multiprocessing, without users having to modify their workflows.
Visit the dispel4py GitHub repository, which contains the instructions for installing it.
This reporitory contains:
-
dispel4py-tutorial: Two presentations Basic and Advanced for understanding how to work with dispel4py. This directory also contains two dispel4py workflows:
-
My first dispel4py workflow python / notebook for checking that dispel4py installation works correctly. It also gives an introduction about how to write dispel4py PEs, how to connect them together, and how to execute a dispel4py workflow. Definitly is the first workflow that you should try.
-
The second dispel4py workflow, 'EvenOddworklow' presented as a python / notebook, gives you more insides of dispel4py, since it has more advance features than the previous one. For more complex workflows, you should go to dispel4py_workflow_collection directory.
-
-
dispel4py_simple_workflow_collection: This directory contains a set of dispel4py 'benchmark' workflows (Mycompression_exercise, My_First_dispel4py_Workflow, Testing_dispel4py, WordCount). Instructions and descriptions of the benchmarks workflow are included in the python scripts and notebooks. This directory also have two simplified real applications: Sentiment Twitter Analysis (analysis_sentiment) and Internal Galaxies Extintion (int_ext_graph). Instructions for executing them are included (ReadmeTwitter.txt and ReadmeAstroWF.txt) here.
More explanation about the real applications can be found at the eScience2015 Slides. Furthermore, we have created a new dispel4py workflow GitHub repository for storing their complete version.
- Conferences presentations: EGU-2015 Slides, eScience2015 Slides and AGU2015 Slides. More information can be foud at: https://www.computer.org/csdl/proceedings/e-science/2015/9325/00/9325a454-abs.html.