Skip to content

A light-weight, config-driven, data pipeline framework in python

License

Notifications You must be signed in to change notification settings

bcajes/pipeable

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Pipeable

Light-weight, config-driven, data pipeline framework

Travis_

Example usage

example_pipeline_config.yaml:

pipes:
  - example.pipes.ForwardReverse
  - example.pipes.Title

example.pipes:

class ForwardReverse(object):
    def process_item(self, str_item, step_cnt_ctx):
        yield str_item
        yield str_item[::-1]
        step_cnt_ctx.count += 1
class Title(object):
    def process_item(self, str_item, step_cnt_ctx):
        yield str_item.title()
        step_cnt_ctx.count +=1

example main:

from pipeable.components import YamlPipelineConfigurator

class step_cnt_ctx(object):
    #example context object that is shared between pipes
    count = 0

pipe_line = YamlPipelineConfigurator.createPipeline(example_pipeline_config, step_cnt_ctx())
result = list(pipe_line.run(['foo', 'bar', 'baz']))
assert ['Foo', 'Oof', 'Bar', 'Rab', 'Baz', 'Zab'] == result
assert pipe_line.context.count == 9

Tested on python 2.7, 3.2, 3.3

About

A light-weight, config-driven, data pipeline framework in python

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages