Python Object Sharer
Reinier Heeres, 2013-2014



Overview

This object sharer is different from other forms by being a peer-to-peer sharer
with equal treatment of client and server. Once connections are made the
communication is completely bi-directional, so not simply following a
request-reply pattern. This allows, for example, the exchange of signal
messages as in regular interactive application programming.

The object sharer allows to use remote python objects as if they were local.
The proxies are generated based on introspection of the remote object, with
the following rules:
- Only functions and variables that do not start with a '_' are exposed.
(except for special functions __getitem__  and __setitem__)
- The reply from functions decorated with '@cache_result' will be cached.

The underlying communication protocol works over plain sockets; objects are
pickled. There are two object types that are treated in a special way:
- Shared objects will be sent as a 'reference' and proxies are created on the
receiving side. If the receiving side is not connected to the 'server' that
contains the object it will automatically do so.
- Numpy arrays are not pickled, but sent in binary form to speed things up.

Clients/servers are identified by a uid at the objectsharer level (i.e. their
RootObject). When shareable objects are exchanged, both the server uid and the
server address are passed around, as the remote client should know how to
connect to the relevant server. At the backend level, socket objects play an
important role as well, as they are used to actually send data around.



Resolving objects

- Each object is identified by a uid
- When registering an object using register() a name can be specified which
can be used as an alias when calling get_object().
- get_object_from() can be used to look for an object (by uid or alias) on
a specific client. It will by default use a local cache for proxies.
- find_object() looks for an object with by uid or alias on any connected
client. (By calling get_object_from() on each client, if not found in cache).



Underlying protocol details

Message structure

- Magic, 'OS'
- Full packet size, 4 bytes
- Number of parts, 1 byte
- For each part:
  - size, 4 bytes
    - data

- The first message part is the pickled python object describing the message.
It should be a tuple and the first item identifies the message type:

    Call, 'c': further items:
        <callid>:
        <object uid>: binary uid
        <function name>:
        <args>: if this contains numpy arrays their content will be available
                in an extra message part.
        <kwargs>: keyword arguments, if this contains numpy arrays content
                  will be available in extra message parts.

    Signal, 's': further items:
        <object uid>: binary uid
        <signal name>:
        <args>: if this contains numpy arrays their content will be available
                in an extra message part.
        <kwargs>: if this contains numpy arrays their content will be available
                in an extra message part.

    Return, 'r': further items:
        <callid>:
        <reply>: if this contains numpy arrays their content will be available
                 in an extra message part.

Extra message parts contain binary data of, for example, numpy arrays and are
provided in the following order:
- Tuples are traversed depth-first.
- Dictionaries are traversed through items(), which should result in same-
ordered traversion for the same elements.


Hand-shaking
- the 'client' initiating the connection sends a 'hello_from' message
- the 'server' replies a 'hello_reply' message, and requests the root object
right after.
- the 'client' requests the root object



Socket backend details
- Sockets are polled using the poll() call, and the main_loop in backend.py
calls recv_from() once data appears to be available. This places the newly
received data in a receive buffer and returns all the complete messages that
were received.
- send_data_to() is called when data is to be sent. It is placed in a buffer
and flush_send_queue() is called, which tries to send as much data as possible.