Python Data Streams
Install from pip:
pip install tributary
or from source
python setup.py install
Tributary offers several kinds of streams:
These are synchronous, reactive data streams, built using asynchronous python generators. They are designed to mimic complex event processors in terms of event ordering.
These are functional streams, built by currying python functions (callbacks).
These are lazily-evaluated python streams, where outputs are propogated only as inputs change.
- Python Function/Generator/Async Function/Async Generator
- Random
- File
- Kafka
- Websocket
- Http
- SocketIO
- File
- Kafka
- Http
- Websocket
- SocketIO
- Delay - Streaming wrapper to delay a stream
- Apply - Streaming wrapper to apply a function to an input stream
- Window - Streaming wrapper to collect a window of values
- Unroll - Streaming wrapper to unroll an iterable stream
- UnrollDataFrame - Streaming wrapper to unroll a dataframe into a stream
- Merge - Streaming wrapper to merge 2 inputs into a single output
- ListMerge - Streaming wrapper to merge 2 input lists into a single output list
- DictMerge - Streaming wrapper to merge 2 input dicts into a single output dict. Preference is given to the second input (e.g. if keys overlap)
- Reduce - Streaming wrapper to merge any number of inputs
- Noop
- Negate
- Invert
- Add
- Sub
- Mult
- Div
- RDiv
- Mod
- Pow
- Sum
- Average
- Not
- And
- Or
- Equal
- NotEqual
- Less
- LessOrEqual
- Greater
- GreaterOrEqual
- Log
- Sin
- Cos
- Tan
- Arcsin
- Arccos
- Arctan
- Sqrt
- Abs
- Exp
- Erf
- Int
- Float
- Bool
- Str
- Len
- RollingCount - Node to count inputs
- RollingMin - Node to take rolling min of inputs
- RollingMax - Node to take rolling max of inputs
- RollingSum - Node to take rolling sum inputs
- RollingAverage - Node to take the running average
- Lazy->Streaming
You can visualize the graph with Graphviz. All streaming and lazy nodes support a graphviz
method.
Streaming and lazy nodes also support ipydagred3 for live update monitoring.
Here green indicates executing, yellow indicates stalled for backpressure, and red indicates that StreamEnd
has been propogated (e.g. stream has ended).
Here green indicates executing, and red indicates that the node is dirty. Note the the determination if a node is dirty is also done lazily (we can check with isDirty
whcih will update the node's graph state.