Backups are good for you. Most people learn this the hard way (including me). Nowadays my Linux laptop automatically creates a full system snapshot every four hours by pushing changed files to an rsync daemon running on the server in my home network and creating a snapshot afterwards using the cp -al command (the article Easy Automated Snapshot-Style Backups with Linux and Rsync explains the basic technique). The server has a second disk attached which asynchronously copies from the main disk so that a single disk failure doesn't wipe all of my backups (the "time delayed replication" aspect has also proven to be very useful).

Okay, cool, now I have backups of everything, up to date and going back in time! But I'm running through disk space like crazy... A proper deduplicating filesystem would be awesome but I'm running crappy consumer grade hardware and e.g. ZFS has not been a good experience in the past. So I'm going to have to delete backups...

Deleting backups is never nice, but an easy and proper rotation scheme can help a lot. I wanted to keep things manageable so I wrote a Python script to do it for me. Over the years I actually wrote several variants. Because I kept copy/pasting these scripts around I decided to bring the main features together in a properly documented Python package and upload it to the Python Package Index.

The rotate-backups package is currently tested on cPython 2.6, 2.7, 3.4, 3.5 and PyPy (2.7). It's tested on Linux and Mac OS X and may work on other unixes but definitely won't work on Windows right now.

Dry run mode

Use it. I'm serious. If you don't and rotate-backups eats more backups than intended you have no right to complain ;-)

Flexible rotation

Rotation with any combination of hourly, daily, weekly, monthly and yearly retention periods.

Fuzzy timestamp matching in filenames

The modification times of the files and/or directories are not relevant. If you speak Python regular expressions, here is how the fuzzy matching works:

# Required components.
(?P<year>\d{4}) \D?
(?P<month>\d{2}) \D?
(?P<day>\d{2}) \D?
(
   # Optional components.
   (?P<hour>\d{2}) \D?
   (?P<minute>\d{2}) \D?
   (?P<second>\d{2})?
)?

All actions are logged

Log messages are saved to the system log (e.g. /var/log/syslog) so you can retrace what happened when something seems to have gone wrong.

The rotate-backups package is available on PyPI which means installation should be as simple as:

$ pip install rotate-backups

There's actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).

There are two ways to use the rotate-backups package: As the command line program rotate-backups and as a Python API. For details about the Python API please refer to the API documentation available on Read the Docs. The command line interface is described below.

Usage: rotate-backups [OPTIONS] [DIRECTORY, ..]

Easy rotation of backups based on the Python package by the same name.

To use this program you specify a rotation scheme via (a combination of) the --hourly, --daily, --weekly, --monthly and/or --yearly options and the directory (or directories) containing backups to rotate as one or more positional arguments.

You can rotate backups on a remote system over SSH by prefixing a DIRECTORY with an SSH alias and separating the two with a colon (similar to how rsync accepts remote locations).

Instead of specifying directories and a rotation scheme on the command line you can also add them to a configuration file. For more details refer to the online documentation (see also the --config option).

Please use the --dry-run option to test the effect of the specified rotation scheme before letting this program loose on your precious backups! If you don't test the results using the dry run mode and this program eats more backups than intended you have no right to complain ;-).

Supported options:

Instead of specifying directories and rotation schemes on the command line you can also add them to a configuration file.

By default two locations are checked for a configuration file, these are ~/.rotate-backups.ini and /etc/rotate-backups.ini. The first of these that exists is loaded. You can load a configuration file in a nonstandard location using the command line option --config.

Configuration files use the familiar INI syntax. Each section defines a directory that contains backups to be rotated. The options in each section define the rotation scheme and other options. Here's an example based on how I use rotate-backups to rotate the backups of the Linux installations that I make regular backups of:

# /etc/rotate-backups.ini:
# Configuration file for the rotate-backups program that specifies
# directories containing backups to be rotated according to specific
# rotation schemes.

[/backups/laptop]
hourly = 24
daily = 7
weekly = 4
monthly = 12
yearly = always
ionice = idle

[/backups/server]
daily = 7 * 2
weekly = 4 * 2
monthly = 12 * 4
yearly = always
ionice = idle

[/backups/mopidy]
daily = 7
weekly = 4
monthly = 2
ionice = idle

[/backups/xbmc]
daily = 7
weekly = 4
monthly = 2
ionice = idle

As you can see in the retention periods of the directory /backups/server in the example above you are allowed to use expressions that evaluate to a number (instead of having to write out the literal number).

Here's an example of a configuration for two remote directories:

# SSH as a regular user and use `sudo' to elevate privileges.
[server:/backups/laptop]
use-sudo = yes
hourly = 24
daily = 7
weekly = 4
monthly = 12
yearly = always
ionice = idle

# SSH as the root user (avoids sudo passwords).
[server:/backups/server]
ssh-user = root
hourly = 24
daily = 7
weekly = 4
monthly = 12
yearly = always
ionice = idle

As this example shows you have the option to connect as the root user or to connect as a regular user and use sudo to elevate privileges.

Since publishing rotate-backups I've found that the default rotation algorithm is not to everyone's satisfaction and because the suggested alternatives were just as valid as the choices that I initially made, options were added to expose the alternative behaviors:

The latest version of rotate-backups is available on PyPI and GitHub. The documentation is hosted on Read the Docs. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.

This software is licensed under the MIT license.

© 2016 Peter Odding.