forked from pgq/skytools-legacy
Generic Queue and Replication for PostgreSQL
klando/skytools
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
= SkyTools - tools for PostgreSQL = This is a package of tools in use in Skype for replication and failover. It also includes a generic queuing mechanism PgQ and utility library for Python scripts, as well as a script for setting up and managing WAL based standby servers. == Overview == It contains following tools: === PgQ === PgQ is a queuing system written in PL/pgsql, Python and C code. It is based on snapshot based event handling ideas from Slony-I, written for general usage. PgQ provides an efficient, transactionnal, queueing system with multi-nodes support (including work sharing and splitting, failover and switchover, for queues and for consumers). Rules: - There can be several queues in database. - There can be several producers than can insert into any queue. - There can be several consumers on one queue. - There can be several subconsumers on a consumer. PgQ is splited into 3 layers: Producers, Ticker and Consumers. *Producers* and *Consumers* respectively push and read events into a queue. Producers just need to call PostgreSQL stored procedures (like a trigger on a table or a PostgreSQL call from the application). And consumers are frequently written in Python, the prefered language as it has a powerful SKytools Framework but it is not limitative and any language able to run PostgreSQL stored procedures can be used. *Ticker* is a daemon which splits the queues in batch of events and handle the maintenance of the system. The Ticker is provided with the Skytools. Documentation: - PgQ ticker daemon (pgqd) usage: link:pgqd.html[] - PgQ admin tool (qadm) usage: link:qadmin.html[] - PgQ SQL API overview: link:pgq-sql.html[] - PgQ SQL reference: http://skytools.projects.postgresql.org/skytools-3.0/pgq/ === Londiste === Replication tool written in Python, using PgQ as event transport. Features: - Tables can be added one-by-one into set. - Initial COPY for one table does not block event replay for other tables. - Can compare tables on both sides. Documentation: - Londiste script usage: doc/londiste.cmdline.txt (also available as `man 1 londiste`) - Londiste configuration: doc/londiste.config.txt (also available as `man 5 londiste`) - Londiste reference: doc/londiste.ref.txt === walmgr === This script will setup WAL archiving, does initial backup and runtime WAL archive and restore. It can also be used for up-to-last-second partial file copying, so less than whole file is lost in case of loss of master database server. == Source tree contents == doc/ Documentation in asciidoc format. Source for both html and man pages. python/ Python modules and primary executables - walmgr, londiste, qadmin, pgqadm. python/pgq/ Python framework for PgQ. python/londiste/ Londiste replication. python/skytools/ Low-level utilities for writing database scripts in Python. sql/ Database modules. sql/pgq/ Table definitions and functions for PgQ queueing. sql/pgq_node/ Framework for cascaded consuming. sql/pgq_coop/ Functions for cooperative consuming. sql/londiste/ Table definitions and functions for Londiste replication. sql/ticker/ PgQ ticker written in C. scripts/ Python scripts with lesser priority. lib/ libusual C libary, for pgqd. debian/ Debian packaging. This is for creating private packages, official Debian packages uses it's own packagin code. misc/ Random scripts used for building. == Upgrade from 2.1 == Assuming PgQ + Londiste setup. This will upgrade PgQ to 3.0 and install Londiste 3 in parallel with Londiste 2. 1. Install Postgres modules. They are backwards compatible with 2.1. 2. Stop `pgqadm.py ticker` processes. 3. Apply ./upgrade/final/v3.0_pgq_core.sql 4. Apply installed pgq_node.sql 5. Apply installed londiste.sql - this will throw error on CREATE SCHEMA, but should otherwise apply fine. 6. Start pgqd.
About
Generic Queue and Replication for PostgreSQL
Resources
Stars
Watchers
Forks
Packages 0
No packages published
Languages
- Python 76.3%
- C 15.8%
- Shell 5.3%
- Rust 2.1%
- D 0.5%