Skip to content

xiaogaozi/PySyncObj

 
 

Repository files navigation

PySyncObj

[ ![Build Status] travis-image ] travis [ ![Coverage Status] coverage-image ] coverage [ ![Release] release-image ] releases [ License license-image ] license [ gitter gitter-image ] gitter

PySyncObj is a python library that provides ability to sync your data between multiple servers.

  • It use raft protocol for leader election and log replication.
  • It supports log compaction. It use fork for copy-on-write while serializing data on disk.
  • It supports dynamic membership changes (cluster reconfiguration).
  • It supports in-memory and on-disk serialization. You can use in-memory mode for small data and on-disk for big one.
  • It has encryption - you can set password and use it in external network.
  • It supports python2 and python3 on linux, macos and windows. No dependencies required (only optional one, eg. cryptography).
  • Configurable event loop - it can works in separate thread with it's own event loop - or you can call onTick function inside your own one.
  • Convenient interface - you can easily transform arbitrary class into a replicated one (see example below).

Content

  1. Install
  2. Usage
  3. Performance

Install

PySyncObj itself:

pip install pysyncobj

Cryptography for encryption (optional):

pip install cryptography

Usage

Consider you have a class that implements counter:

class MyCounter(object):
	def __init__(self):
		self.__counter = 0

	def incCounter(self):
		self.__counter += 1

	def getCounter(self):
		return self.__counter

So, to transform your class into a replicated one:

  • Inherit it from SyncObj
  • Initialize SyncObj with a self address and a list of partner addresses. Eg. if you have serverA, serverB and serverC and want to use 4321 port, you should use self address serverA:4321 with partners [serverB:4321, serverC:4321] for your application, running at serverA; self address serverB:4321 with partners [serverA:4321, serverC:4321] for your application at serverB; self address serverC:4321 with partners [serverA:4321, serverB:4321] for app at serverC.
  • Mark all your methods that modifies your class fields with @replicated decorator. So your final class will looks like:
class MyCounter(SyncObj):
	def __init__(self):
		super(MyCounter, self).__init__('serverA:4321', ['serverB:4321', 'serverC:4321'])
		self.__counter = 0

	@replicated
	def incCounter(self):
		self.__counter += 1

	def getCounter(self):
		return self.__counter

And thats all! Now you can call incCounter on serverA, and check counter value on serverB - they will be synchronized. You can look at examples and syncobj_ut.py for more use-cases.

Performance

15K rps on 3 nodes; 14K rps on 7 nodes; 22K rps on 10 byte requests; 5K rps on 20Kb requests;

About

A library for replicating your python class between multiple servers, based on raft protocol

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 99.3%
  • Shell 0.7%