Merge https://github.com/postgrespro/pg_arman into ptrack

README.md

@@ -1,5 +1,8 @@
-pg_arman
-========
+pg_arman fork from Postgres Professional
+========================================
+
+This repository contains fork of pg_arman by Postgres Professional with
+block level incremental backup support.
 
 pg_arman is a backup and recovery manager for PostgreSQL servers able to do
 differential and full backup as well as restore a cluster to a
@@ -14,7 +17,8 @@ Download
 --------
 
 The latest version of this software can be found on the project website at
-https://github.com/michaelpq/pg_arman.
+https://github.com/postgrespro/pg_arman.  Original fork of pg_arman can be
+found at https://github.com/michaelpq/pg_arman.
 
 Installation
 ------------
@@ -48,6 +52,112 @@ launched in a way similar to common PostgreSQL extensions and modules:
 
     make installcheck
 
+Block level incremental backup
+------------------------------
+
+Idea of block level incremental backup is that you may backup only blocks
+changed since last full backup.  It gives two major benefits: taking backups
+faster and making backups smaller.
+
+The major question here is how to get the list of changed blocks.  Since
+each block contains LSN number, changed blocks could be retrieved by full scan
+of all the blocks.  But this approach consumes as much server IO as full
+backup.
+
+This is why we implemented alternative approaches to retrieve
+list of changed blocks.
+
+1. Scan WAL archive and extract changed blocks from it.  However, shortcoming
+of these approach is requirement to have WAL archive.
+
+2. Track bitmap of changes blocks inside PostgreSQL (ptrack).  It introduces
+some overhead to PostgreSQL performance.  On our experiments it appears to be
+less than 3%.
+
+These two approaches were implemented in this fork of pg_arman.  The second
+approach requires [patch for PostgreSQL](https://gist.github.com/stalkerg/7ce2394c4f3b36f995895190a633b4fa).
+
+Testing block level incremental backup
+--------------------------------------
+
+You need build and install [PGPRO9_5_ptrack branch of PostgreSQL](https://github.com/postgrespro/postgrespro/tree/PGPRO9_5_ptrack) or [apply this patch to PostgreSQL 9.5](https://gist.github.com/stalkerg/7ce2394c4f3b36f995895190a633b4fa).
+
+### Retrieving changed blocks from WAL archive
+
+You need to enable WAL archive by adding following lines to postgresql.conf:
+
+```
+wal_level = archive
+archive_command = 'test ! -f /home/postgres/backup/arman/wal/%f && cp %p /home/postgres/backup/arman/wal/%f'
+wal_log_hints = on
+```
+
+Example backup (assuming PostgreSQL is running):
+```bash
+# Init pg_aramn backup folder
+pg_arman init -B /home/postgres/backup/pgarman
+cat << __EOF__ >> /home/postgres/backup/pgarman/pg_arman.ini
+ARCLOG_PATH = '/home/postgres/backup/arman/wal'
+__EOF__
+# Make full backup with 2 thread and verbose mode.
+pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b full -v -j 2
+# Validate backup
+pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
+# Show backups information
+pg_arman show -B /home/postgres/backup/pgarman
+
+# Now you can insert or update some data in your database
+
+# Then start the incremental backup.
+pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b page -v -j 2
+pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
+# You should see that increment is really small
+pg_arman show -B /home/postgres/backup/pgarman
+```
+
+For restore after remove your pgdata you can use:
+```
+pg_arman restore -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -j 4 --verbose
+```
+
+### Retrieving changed blocks from ptrack
+
+The advantage of this approach is that you don't have to save WAL archive.  You will need to enable ptrack in postgresql.conf (restart required).
+
+```
+ptrack_enable = on
+```
+
+Also, some WALs still need to be fetched in order to get consistent backup.  pg_arman can fetch them trough the streaming replication protocol.  Thus, you also need to [enable streaming replication connection](https://wiki.postgresql.org/wiki/Streaming_Replication).
+
+Example backup (assuming PostgreSQL is running):
+```bash
+# Init pg_aramn backup folder
+pg_arman init -B /home/postgres/backup/pgarman
+cat << __EOF__ >> /home/postgres/backup/pgarman/pg_arman.ini
+ARCLOG_PATH = '/home/postgres/backup/arman/wal'
+__EOF__
+# Make full backup with 2 thread and verbose mode.
+pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b full -v -j 2 --stream
+# Validate backup
+pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
+# Show backups information
+pg_arman show -B /home/postgres/backup/pgarman
+
+# Now you can insert or update some data in your database
+
+# Then start the incremental backup.
+pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b ptrack -v -j 2 --stream
+pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
+# You should see that increment is really small
+pg_arman show -B /home/postgres/backup/pgarman
+```
+
+For restore after remove your pgdata you can use:
+```
+pg_arman restore -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -j 4 --verbose --stream
+```
+
 License
 -------
 

doc/pg_arman.md

@@ -223,7 +223,7 @@ absolute paths; relative paths are not allowed.
 
 **-b** BACKUPMODE / **--backup-mode**=BACKUPMODE:  
     Specify backup target files. Available options are: "full",
-    "page".
+    "page", "ptrack".
 
 **-C** / **--smooth-checkpoint**:  
     Checkpoint is performed on every backups. If the option is specified,