1. Install and introduction:

    This is a backup and restore tool for Ceph RBD image. You can use it to
    backup and restore your RBD images or transfer RBD images to other cluster.

    To install the RBD backup restore tool, change directory to the
    source directory and exec './install.sh'.

    The 'rbd-brctl' is entry/manage command for the RBD backup restore and
    is installed in /bin directory.

    All configuration related file are installed in /etc/rbd_backup_restore.
    You need to config it to reflect your needs.

    All source code or library are be installed in /opt/rbd_backup_restore
    directory.

    Prerequisites:
    This tool requires rbd command and librbd and rados library installed
    in system and python version 2.7.x.


2. Config:

    - config.ini
      This is default/global configuration file of the RBD backup restore tool.
      Following is a default setting of config.ini file.

        [default]
        ceph_cluster_name = ceph
        ceph_conf_file = /etc/ceph/ceph.conf
        ceph_keyring_file = /etc/ceph/ceph.client.admin.keyring

        log_file_path = /var/log/rbd_backup_restore
        log_level = DEBUG
        log_max_bytes = 20971520
        log_backup_count = 10
        log_delay = False
        log_module_name = False

        backup_list_from_openstack_yaml_file = False
        backup_list_file_path = /etc/rbd_backup_restore/backup_list.yaml
        backup_destination_path = /mnt/disk
        backup_concurrent_worker_count = 2
        backup_circle_retain_count = 2
        backup_snapshot_retain_count = 2
        backup_read_options = True
        backup_max_incremental = 6

        weekly_full_backup = 7
        weekly_incr_backup = 1, 2, 3, 4, 5, 6

        openstack_yaml_file_path = /etc/rbd_backup_restore/openstack.yaml
        openstack_yaml_section = default
        openstack_distribution = helion
        openstack_ceph_pool = volumes


      Description of the configuration.

        ceph_cluster_name
          name of Ceph cluster to be backuped.

        ceph_conf_file
          absolute path of Ceph configuration file.

        ceph_keyring_file
          absolute path of Ceph keyring file.

        log_file_path
          directory path for logging.

        log_level
          logging level. 'INFO', 'WARNING', 'ERROR' or 'DEBUG'

        log_max_bytes
          max bytes of logging file. integer value.

        log_backup_count
          max number of logging file for rotation. integer value.

        log_delay
          delay of opening logging file. 'True' or 'False'.

        log_module_name
          log module name in logging file. 'True' or 'False'.

        backup_list_from_openstack_yaml_file
          read rbd list from openstack yaml file. 'True' or 'False'.

        backup_list_file_path
          absolute file path of rbd backup list in yaml format.

        backup_destination_path
          absolute directory path for storing RBD image backups.

        backup_concurrent_worker_count
          number of concurrent RBD images to be backuped. integer value.

        backup_circle_retain_count
          number of RBD backup circle to be retained. integer value.

        backup_snapshot_retain_count
          number of RBD snapshot to be retained from the cluster. integer value.

        backup_read_options
          read backup options form backup list file. 'True' or 'False'.

        backup_max_incremental
          max number of incremental backup in a backup circle. integer value.

        weekly_full_backup
          set day of week to perform full backup. single or multiple integer
          value separate with a comma and space.

        weekly_incr_backup
          set day of week to perform incremental backup. single or multiple
          integer value separate with a comma and space.


    - backup_list.yaml
      An YAML format of file that defines RBD list to be backuped.
      Definition structure:

        <cluster name>:
          - <pool name>:
            - name: <RBD name>
              max_incremental_backup_count:       (optional)
              max_backup_circle_count:            (optional)
              backup_snapshot_retain_count:       (optional)
              backup_type:                        (optional)
            ....

      For example, if you have a Ceph cluster named 'my_ceph' and a pool name
      'my_pool' and there is a RBD called 'my_rbd' in the pool. You define the
      YAML as following to backup the RBD.

        my_ceph:
          - my_pool:
            - name: my_rbd

      There are four optional attributes can be configured to
      control the backup, they will overwrite the global configuration in
      config.ini file.

        max_incremental_backup_count
          integer value, max count of incremental backup keep in a backup circle.

        max_backup_circle_counts
          integer value, max count of backup circle to retain.

        backup_snapshot_retain_count:
          integer value, max count of RBD snapshot keep in Ceph cluster.

        backup_type:
          string of 'full' or 'incr', type of RBD backup. full backup or
          incremental backup.


3. Command Usage:

    Use 'rbd-brctl' with following instruction commands.

        show [ rbd | backup | cluster | config ] ...
            - show information of RBD backup. select one of info type below.

            rbd [ <pool name> ] [ <rbd name> ]
                list backuped RBD images.
                with pool name and rbd name, list backup time of the RBD

            backup [ <backup name> ]
                list backup name.
                with backup name, list RBDs from the backup.

            cluster
                show cluster information of backup directory.

            config [ rbd list ]
                show global configurations.
                with rbd list, show rbd list from backup list yaml file.

        backup
            - perform RBD backup job. no any arguments required.

        restore <pool name> <rbd name> <backup time>
            - perform RBD restore job. need three arguments to perform
              restoration.

            pool name
                pool name of backuped RBD. (source backup pool name)

            rbd name
                RBD name of backuped RBD. (source backup RBD name)

            backup time
                time point of RBD to be restored to.

        delete <pool name> <rbd name>
            - delete a backuped RBD.  need two arguments.

            pool name
                pool name of backuped RBD.

            rbd name
                RBD name of backuped RBD.


    Arguments:

        --config-file <file path>
            specify a file path of backup restore configuration to be used.

        --config-section <section name>
            specify name configuration section to be used.

        --cluster-name <name of Ceph cluster>
            used in restoration, show and delete instruction. specify Ceph
            cluster name.

        --backup-directory <directory path>
            used in restoration, show and delete instruction. specify a backup
            directory path to operate.

        --show-details
            used in show instruction. display detail information.

        --backup-name <backup name>
            used in backup instruction. specify backup name. if ignored, set
            a datetime formated name.

        --dest-pool-name <new pool name>
            used for restoration instruction. specify pool name RBD image
            will be restored to. if ignored, set to source pool name.

        --dest-rbd-name <new rbd name>
            used in restoration instruction. specify name RBD image will
            be restored to. if ignored, set to source RBD name.

        --force-restore
            used in restoration instruction. force restore the RBD.

        --yes
            used in delete instruction. assume yes to delete backuped RBD.


4. Command Output Examples:

    # Show Global/Default Configuration.
    #--------------------------------------------------------------

        [root@yumon1 ~]# rbd-brctl show config

        *Show backup config.

        ceph_cluster_name = ceph
        ceph_conf_file = /etc/ceph/ceph.conf
        ceph_keyring_file = /etc/ceph/ceph.client.admin.keyring
        log_file_path = /var/log/rbd_backup_restore
        log_level = DEBUG
        log_max_bytes = 20971520
        log_backup_count = 10
        log_delay = False
        log_module_name = False
        backup_list_from_openstack_yaml_file = False
        backup_list_file_path = /etc/rbd_backup_restore/backup_list.yaml
        backup_destination_path = /mnt/disk
        backup_concurrent_worker_count = 4
        backup_circle_retain_count = 4
        backup_snapshot_retain_count = 1
        backup_read_options = True
        backup_max_incremental = 6
        weekly_full_backup = 7
        weekly_incr_backup = 1, 2, 3, 4, 5, 6
        openstack_yaml_file_path = /etc/rbd_backup_restore/openstack.yaml
        openstack_yaml_section = default
        openstack_distribution = helion
        openstack_ceph_pool = volumes


    # Show RBD List From Backup List File.
    #--------------------------------------------------------------

        [root@yumon1 ~]# rbd-brctl show config rbd list

        *Show RBD backup list.
        *Yaml file: /etc/rbd_backup_restore/backup_list.yaml
        *Cluster name: ceph

        [Pool name]
          [RBD name]
        ------------------------
        rbd
          new
          rbd0
          rbd1
        volumes
          old
          rbd0
          rbd1


        [root@yumon1 ~]# rbd-brctl show config rbd list --show-details

        *Show RBD backup list.
        *Yaml file: /etc/rbd_backup_restore/backup_list.yaml
        *Cluster name: ceph

        [Pool name]
          [RBD name] [backup_type] [max_incr] [max_circ] [max_snap]
        --------------------------------------------------------------------------
        rbd
          new False 2 2 1
          rbd0 False False False False
          rbd1 False False False False
        volumes
          old full False False False
          rbd0 False False False False
          rbd1 False False False False


    # Show Backup Information of Backuped Directory
    #--------------------------------------------------------------

        [root@yumon1 ~]# rbd-brctl show backup

        *Show all backup name.

        [Backup Name List]
        --------------------
        2018_05_25_16_47_25
        2018_05_25_17_01_41
        2018_05_25_17_03_15
        2018_05_25_17_30_23

        [root@yumon1 ~]# rbd-brctl show backup --show-details

        *Show all backup name.

        [Backup Name]       [RBD Count]
        -------------------------------
        2018_05_25_16_47_25 6
        2018_05_25_17_01_41 6
        2018_05_25_17_03_15 6
        2018_05_25_17_30_23 6


        [root@yumon1 ~]# rbd-brctl show backup 2018_05_25_17_30_23

        *Show RBD list in backup name '2018_05_25_17_30_23'.

        [Backup RBD List]
        --------------------
        rbd/new
        rbd/rbd0
        rbd/rbd1
        volumes/rbd0
        volumes/old
        volumes/rbd1


        [root@yumon1 ~]# rbd-brctl show backup 2018_05_25_17_30_23 --show-details

        *Show RBD list in backup name '2018_05_25_17_30_23'.

        [Backup Time]       [Circle name]       [Pool name/RBD name]  ... [Status]
        --------------------------------------------------------------------------
        2018_05_25_17_30_26 2018_05_25_17_30_26 rbd/new ... OK
        2018_05_25_16_47_27 2018_05_25_17_30_26 rbd/rbd0 ... OK
        2018_05_25_16_47_28 2018_05_25_17_30_27 rbd/rbd1 ... OK
        2018_05_25_16_47_30 2018_05_25_17_30_29 volumes/rbd0 ... OK
        2018_05_25_17_30_27 2018_05_25_17_30_27 volumes/old ... OK
        2018_05_25_16_47_30 2018_05_25_17_30_29 volumes/rbd1 ... OK


    # Show RBD Information of Backuped Directory
    #--------------------------------------------------------------

        [root@yumon1 ~]# rbd-brctl show rbd

        *Show all backuped RBD name.

        [Pool name]
          [RBD name]
        --------------------
        rbd
          rbd0
          rbd1
          new
        volumes
          old
          rbd0


        [root@yumon1 ~]# rbd-brctl show rbd --show-details

        *Show all backuped RBD name.

        [Pool name]
          [RBD name] [block name prefix] [Num objects] [size (bytes)]
        --------------------------------------------------------------------------
        rbd
          rbd0 rbd_data.103e74b0dc51 2048 8589934592
          rbd1 rbd_data.103f74b0dc51 750 3145728000
          new rbd_data.2021b2643c9869 500 2097152000
        volumes
          old rbd_data.2021c2643c9869 500 2097152000
          rbd0 rbd_data.104574b0dc51 804 3369074688
          rbd1 rbd_data.109c643c9869 450 1887436800


        [root@yumon1 ~]# rbd-brctl show rbd volumes rbd0

        *Show backup time of RBD 'volumes/rbd0'
        .
        [Backup time]
        --------------------
        2018_05_25_16_47_30
        2018_05_25_17_01_47
        2018_05_25_17_03_21
        2018_05_25_17_30_29


        [root@yumon1 ~]# rbd-brctl show rbd volumes rbd0 --show-details

        *Show backup time of RBD 'volumes/rbd0'
        .
        [Backup time]       [Backup name]       [Backup circle]     [Backup size]
        --------------------------------------------------------------------------
        2018_05_25_16_47_30 2018_05_25_16_47_25 2018_05_25_16_47_30 3369078784
        2018_05_25_17_01_47 2018_05_25_17_01_41 2018_05_25_16_47_30 4096
        2018_05_25_17_03_21 2018_05_25_17_03_15 2018_05_25_16_47_30 4096
        2018_05_25_17_30_29 2018_05_25_17_30_23 2018_05_25_16_47_30 4096


    # Start RBD Backup
    #--------------------------------------------------------------

        [root@yumon1 ~]# rbd-brctl backup

        2018-05-31 16:05:25 - Start RBD Backup.

        - process start running, pid 1175723
        - check backup directory.
          190329 Mbytes available.
          11052 Mbytes used.
        - check backup rbd list.
          get backup list from /etc/rbd_backup_restore/test.yaml.
          3 RBD(s) to be backuped.
        - verify RBD backup list.
          rbd/rbda - 2097152000 bytes.
          rbd/rbdb - 3145728000 bytes.
          rbd/new1 - 3145728000 bytes.
          volumes/rbda - 2097152000 bytes.
          volumes/rbdb - 4194304000 bytes.
          volumes/old1 - 2097152000 bytes.
          6 RBD(s) can be backuped.
          total RBDs has 16000 Mbytes.
        - check rbd backup type.
          rbd rbda - incr backup.
          rbd rbdb - incr backup.
          rbd new1 - full backup.
          volumes rbda - full backup.
          volumes rbdb - full backup.
          volumes old1 - full backup.
        - sort rbd backup list order.
          rbd rbda 2097152000
          rbd rbdb 3145728000
          rbd new1 3145728000
          volumes rbda 2097152000
          volumes rbdb 4194304000
          volumes old1 2097152000
        - start task workers.
          2 worker(s) started.
        - start RBD snapshot procedure.
          take snapshot of rbd/rbda
          take snapshot of rbd/rbdb
          take snapshot of rbd/new1
          take snapshot of volumes/rbda
          take snapshot of volumes/rbdb
          take snapshot of volumes/old1
          snapshot of rbd/rbda completed.
          snapshot of rbd/rbdb completed.
          snapshot of rbd/new1 completed.
          snapshot of volumes/rbda completed.
          snapshot of volumes/rbdb completed.
          snapshot of volumes/old1 completed.
        - start RBD export procedure.
          export rbd/rbda
          export rbd/rbdb
          export rbd/new1
          export volumes/rbda
          export volumes/rbdb
          export volumes/old1
          export of rbd/rbda completed.
          export of rbd/rbdb completed.
          export of rbd/new1 completed.
          export of volumes/rbda completed.
          export of volumes/rbdb completed.
          export of volumes/old1 completed.
        - check exceed RBD snapshot.
          delete snapshot 2018_05_31_14_25_04 of rbd/rbda
          delete snapshot 2018_05_31_14_25_04 of rbd/rbdb
          delete snapshot 2018_05_31_15_19_04 of rbd/new1
          delete snapshot 2018_05_31_14_25_05 of volumes/rbda
          delete snapshot 2018_05_31_14_25_07 of volumes/rbdb
          delete snapshot 2018_05_31_14_25_07 of volumes/old1
        - check execeed RBD backup circle.

        2018-05-31 16:09:46 - Finish RBD Backup.


        [root@yumon1 test]# rbd-brctl backup rbd/rbda --backup-name mybackup

        2018-05-31 16:19:19 - Start RBD Backup.

        - process start running, pid 1198754
        - check backup directory.
          185069 Mbytes available.
          16312 Mbytes used.
        - check backup rbd list.
          get backup list from command line input.
          1 - rbd rbda
          1 RBD(s) to be backuped.
        - verify RBD backup list.
          rbd/rbda - 2097152000 bytes.
          1 RBD(s) can be backuped.
          total RBDs has 2000 Mbytes.
        - check rbd backup type.
          rbd rbda - incr backup.
        - sort rbd backup list order.
          rbd rbda 2097152000
        - start task workers.
          4 worker(s) started.
        - start RBD snapshot procedure.
          take snapshot of rbd/rbda
          snapshot of rbd/rbda completed.
        - start RBD export procedure.
          export rbd/rbda
          export of rbd/rbda completed.
        - check exceed RBD snapshot.
          delete snapshot 2018_05_31_16_05_28 of rbd/rbda
        - check execeed RBD backup circle.

        2018-05-31 16:19:23 - Finish RBD Backup.


    # Start RBD Restore
    #--------------------------------------------------------------

        [root@yumon1 test]# rbd-brctl restore rbd rbda 2018_05_31_16_05_28 --dest-pool-name test --dest-rbd-name rbda_restored

        2018-05-31 16:14:23 - Start RBD Restore.

        - from backup directory: /mnt/disk
        - from cluster name: ceph
        - restore to datetime: 2018_05_31_16_05_28
        - source pool name: rbd
        - source RBD name: rbda
        - destionation pool name: test
        - destionation RBD name: rbda_restored
        - belonging backup circle name: 2018_05_31_11_54_57
        - start RBD restoring to 2018_05_31_16_05_28
          restoring to 2018_05_31_11_54_57. (full)
          restore to 2018_05_31_11_54_57 successfully.
          restoring to 2018_05_31_14_25_04. (incr)
          restore to 2018_05_31_14_25_04 successfully.
          restoring to 2018_05_31_16_05_28. (incr)
          restore to 2018_05_31_16_05_28 successfully.
        - purge RBD snapshots
          purge RBD snapshots successfully.

        2018-05-31 16:18:06 - Finish RBD Restore.


    # Delete a Backuped RBD From Backup Directory
    #--------------------------------------------------------------

        [root@yumon1 test]# rbd-brctl delete rbd volumes rbdb
        - delete backuped RBD:
          pool name: volumes
          rbd name: rbdb
        Are you sure to delete this RBD backup (yes or no)? : yes
        - the backuped RBD is deleted successfully.