Skip to content

isabella232/pycollector

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

                           _  _              _
 _ __   _   _   ___  ___  | || |  ___   ___ | |_  ___   _ __
| '_ \ | | | | / __|/ _ \ | || | / _ \ / __|| __|/ _ \ | '__|
| |_) || |_| || (__| (_) || || ||  __/| (__ | |_| (_) || |
| .__/  \__, | \___|\___/ |_||_| \___| \___| \__|\___/ |_|
|_|     |___/                                   by Intelie

                           Collecting data should be simple.

--------
| Goal |
--------

Moving data from A to B is a well known task in computing.
Our aim here is to make it easier to 'develop' and 'configure' a piece of
software to collect data.


-------------------
| Main components |
-------------------

There are two main components of any collecting process:

- Readers
  Piece of code that takes the responsability of going to the source of the
  data and prepare it to be written somewhere else.

- Writers
  Piece of code that takes a read message and delivers to the destination.

A pair reader/writer is called collecting unit.
Readers and writers may make any transformation to the original data.


----------------
| Architecture |
----------------

Collecting unit architecture:

 1. ask source_______                           ________   5. write destination
    <-------|        |                         |        |------------------>
            | Reader |                         | Writer |
    ------->|________|                    +--->|________|
   2. answer    |                         |
                |            __________   | 4. get stored msgs
                +---------->|          |--+
                 3.store    | internal |
                            |  queue   |
                            |__________|

It is possible to configure as many collecting units as needed.


--------------
| How to use |
--------------

There are two ways to use it.
 - configuring
   you may configure a predefined reader/writer for you needs.

 - development
   you may adapt an existing reader/writer or code a new one.


---------------
| Configuring |
---------------

A reader/writer always have a 'type' property.
This 'type' will define how the data will be read by a reader or written by a writer.

Moreover, readers and writers have properties in common, regardless of their types:

* 'blockable' 
    if 'blockable : True' is set to a reader, the message will wait for a slot in queue.
    if 'blockable : True' is set to a writer, it will try to rewrite the message.
    default value: True

* 'period' 
    if 'period : N' is set to a reader, the method read() will be called periodically,
        with period N (in seconds)
    if 'period : N' is set to a writer, the method write(msg) will be called periodically,
        with period N (in seconds)
    default value: None, default is to be async, read()/write() are called just once.

... (TODO: complete list)


----------------
| Current step |
----------------

Improving core;
Increasing test coverage.

About

No description, website, or topics provided.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 99.4%
  • Other 0.6%