Skip to content

batiste/database-experiment

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A python database that partition itself

This project is a experiment that implement a very simple json data store that can partition the data accross several servers in a totally automatic way.

The idea is to have a pool of servers. You can add servers to your pool and some data will be automaticaly migrated from the old servers to the new one to equalize the load.

This experiment currently uses 256 clusters of data. Every json data is assigned to a cluster using the first 2 characters of the index (a md5 hexadecimal hash).

  • An algorithm is used to determine to which server everyone of these clusters should go.
  • Every server get the same amount of of clusters.
  • Everytime a server is added to the pool the clusters are redistributed.
  • The redistribution is designed to move the smallest amount of clusters and to move the same amount from every server.
  • After a new host joined the pool, the clusters are automaticaly moved.

Todo

  • Save pool configuration on disk.
  • Implement a client library.
  • Tests.
  • Save documents in cluster directories.
  • ...

About

A python database that partition itself

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages