1
0
mirror of https://github.com/postgrespro/pg_probackup.git synced 2024-12-05 11:00:22 +02:00
pg_probackup/README.md

208 lines
13 KiB
Markdown
Raw Normal View History

2020-04-01 14:48:51 +02:00
[![Build Status](https://travis-ci.com/postgrespro/pg_probackup.svg?branch=master)](https://travis-ci.com/postgrespro/pg_probackup)
2020-05-12 16:47:22 +02:00
[![GitHub release](https://img.shields.io/github/v/release/postgrespro/pg_probackup?include_prereleases)](https://github.com/postgrespro/pg_probackup/releases/latest)
2020-04-01 14:48:51 +02:00
2017-07-10 17:08:32 +02:00
# pg_probackup
`pg_probackup` is a utility to manage backup and recovery of PostgreSQL database clusters. It is designed to perform periodic backups of the PostgreSQL instance that enable you to restore the server in case of a failure.
2017-10-27 13:30:07 +02:00
The utility is compatible with:
* PostgreSQL 9.5, 9.6, 10, 11, 12;
2017-07-10 17:13:49 +02:00
2017-07-10 17:08:32 +02:00
As compared to other backup solutions, `pg_probackup` offers the following benefits that can help you implement different backup strategies and deal with large amounts of data:
2019-11-06 17:58:18 +02:00
* Incremental backup: page-level incremental backup allows you to save disk space, speed up backup and restore. With three different incremental modes, you can plan the backup strategy in accordance with your data flow.
2020-06-25 15:00:17 +02:00
* Incremental restore: page-level incremental restore allows you dramatically speed up restore by reusing valid unchanged pages in destination directory.
2020-05-03 22:03:55 +02:00
* Merge: using this feature allows you to implement "incrementally updated backups" strategy, eliminating the need to do periodical full backups.
2019-10-12 14:00:38 +02:00
* Validation: automatic data consistency checks and on-demand backup validation without actual data recovery
2019-11-06 17:58:18 +02:00
* Verification: on-demand verification of PostgreSQL instance with the `checkdb` command.
* Retention: managing WAL archive and backups in accordance with retention policy. You can configure retention policy based on recovery time or the number of backups to keep, as well as specify `time to live` (TTL) for a particular backup. Expired backups can be merged or deleted.
2019-10-12 14:00:38 +02:00
* Parallelization: running backup, restore, merge, delete, verificaton and validation processes on multiple parallel threads
* Compression: storing backup data in a compressed state to save disk space
2019-11-06 17:58:18 +02:00
* Deduplication: saving disk space by not copying unchanged non-data files, such as `_vm` or `_fsm`
* Remote operations: backing up PostgreSQL instance located on a remote system or restoring a backup remotely
* Backup from standby: avoid extra load on master by taking backups from a standby server
* External directories: backing up files and directories located outside of the PostgreSQL `data directory` (PGDATA), such as scripts, configuration files, logs, or SQL dump files.
* Backup Catalog: get list of backups and corresponding meta information in plain text or JSON formats
* Archive catalog: getting the list of all WAL timelines and the corresponding meta information in plain text or JSON formats
2019-10-12 14:00:38 +02:00
* Partial Restore: restore only the specified databases or exclude the specified databases from restore.
2017-07-10 17:08:32 +02:00
To manage backup data, `pg_probackup` creates a backup catalog. This directory stores all backup files with additional meta information, as well as WAL archives required for [point-in-time recovery](https://postgrespro.com/docs/postgresql/current/continuous-archiving.html). You can store backups for different instances in separate subdirectories of a single backup catalog.
Using `pg_probackup`, you can take full or incremental backups:
* `Full` backups contain all the data files required to restore the database cluster from scratch.
* `Incremental` backups only store the data that has changed since the previous backup. It allows to decrease the backup size and speed up backup operations. `pg_probackup` supports the following modes of incremental backups:
* `PAGE` backup. In this mode, `pg_probackup` scans all WAL files in the archive from the moment the previous full or incremental backup was taken. Newly created backups contain only the pages that were mentioned in WAL records. This requires all the WAL files since the previous backup to be present in the WAL archive. If the size of these files is comparable to the total size of the database cluster files, speedup is smaller, but the backup still takes less space.
* `DELTA` backup. In this mode, `pg_probackup` read all data files in PGDATA directory and only those pages, that where changed since previous backup, are copied. Continuous archiving is not necessary for it to operate. Also this mode could impose read-only I/O pressure equal to `Full` backup.
2017-07-10 17:08:32 +02:00
* `PTRACK` backup. In this mode, PostgreSQL tracks page changes on the fly. Continuous archiving is not necessary for it to operate. Each time a relation page is updated, this page is marked in a special `PTRACK` bitmap for this relation. As one page requires just one bit in the `PTRACK` fork, such bitmaps are quite small. Tracking implies some minor overhead on the database server operation, but speeds up incremental backups significantly.
Regardless of the chosen backup type, all backups taken with `pg_probackup` support the following strategies of WAL delivery:
* `Autonomous backups` streams via replication protocol all the WAL files required to restore the cluster to a consistent state at the time the backup was taken. Even if continuous archiving is not set up, the required WAL segments are included into the backup.
* `Archive backups` rely on continuous archiving.
2017-07-10 17:08:32 +02:00
2019-07-27 04:07:41 +02:00
## ptrack support
2018-11-30 08:19:32 +02:00
`PTRACK` backup support provided via following options:
* vanilla PostgreSQL compiled with ptrack patch. Currently there are patches for [PostgreSQL 9.6](https://gist.githubusercontent.com/gsmol/5b615c971dfd461c76ef41a118ff4d97/raw/e471251983f14e980041f43bea7709b8246f4178/ptrack_9.6.6_v1.5.patch) and [PostgreSQL 10](https://gist.githubusercontent.com/gsmol/be8ee2a132b88463821021fd910d960e/raw/de24f9499f4f314a4a3e5fae5ed4edb945964df8/ptrack_10.1_v1.5.patch)
* vanilla PostgreSQL 12 with [ptrack extension](https://github.com/postgrespro/ptrack)
* Postgres Pro Standard 9.6, 10, 11, 12
* Postgres Pro Enterprise 9.6, 10, 11, 12
2018-11-30 08:19:32 +02:00
2017-07-10 17:08:32 +02:00
## Limitations
`pg_probackup` currently has the following limitations:
* The server from which the backup was taken and the restored server must be compatible by the [block_size](https://postgrespro.com/docs/postgresql/current/runtime-config-preset#GUC-BLOCK-SIZE) and [wal_block_size](https://postgrespro.com/docs/postgresql/current/runtime-config-preset#GUC-WAL-BLOCK-SIZE) parameters and have the same major release number.
2020-04-30 18:09:02 +02:00
* Remote backup via ssh on Windows currently is not supported.
2020-05-13 01:15:26 +02:00
* When running remote operations via ssh, remote and local pg_probackup versions must be the same.
2017-07-10 17:08:32 +02:00
2019-10-27 22:07:44 +02:00
## Documentation
2020-05-03 22:05:50 +02:00
Documentation can be found at [github](https://postgrespro.github.io/pg_probackup) and [Postgres Professional documentation](https://postgrespro.com/docs/postgrespro/current/app-pgprobackup)
2017-07-10 17:08:32 +02:00
## Installation and Setup
2019-05-17 17:09:59 +02:00
### Windows Installation
2020-06-25 15:00:17 +02:00
Installers are available in release **assets**. [Latests](https://github.com/postgrespro/pg_probackup/releases/2.4.1).
2019-05-17 17:09:59 +02:00
### Linux Installation
2019-10-27 22:07:44 +02:00
#### pg_probackup for vanilla PostgreSQL
```shell
#DEB Ubuntu|Debian Packages
2020-05-03 22:03:55 +02:00
sudo echo "deb [arch=amd64] https://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" > /etc/apt/sources.list.d/pg_probackup.list
sudo wget -O - https://repo.postgrespro.ru/pg_probackup/keys/GPG-KEY-PG_PROBACKUP | sudo apt-key add - && sudo apt-get update
sudo apt-get install pg-probackup-{12,11,10,9.6,9.5}
sudo apt-get install pg-probackup-{12,11,10,9.6,9.5}-dbg
#DEB-SRC Packages
2020-05-03 22:03:55 +02:00
sudo echo "deb-src [arch=amd64] https://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" >>\
/etc/apt/sources.list.d/pg_probackup.list && sudo apt-get update
sudo apt-get source pg-probackup-{12,11,10,9.6,9.5}
#RPM Centos Packages
2020-05-03 22:03:55 +02:00
rpm -ivh https://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-centos.noarch.rpm
yum install pg_probackup-{12,11,10,9.6,9.5}
yum install pg_probackup-{12,11,10,9.6,9.5}-debuginfo
#RPM RHEL Packages
2020-05-03 22:03:55 +02:00
rpm -ivh https://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-rhel.noarch.rpm
yum install pg_probackup-{12,11,10,9.6,9.5}
yum install pg_probackup-{12,11,10,9.6,9.5}-debuginfo
#RPM Oracle Linux Packages
2020-05-03 22:03:55 +02:00
rpm -ivh https://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-oraclelinux.noarch.rpm
yum install pg_probackup-{12,11,10,9.6,9.5}
yum install pg_probackup-{12,11,10,9.6,9.5}-debuginfo
#SRPM Centos|RHEL|OracleLinux Packages
yumdownloader --source pg_probackup-{12,11,10,9.6,9.5}
#RPM SUSE|SLES Packages
zypper install --allow-unsigned-rpm -y https://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-suse.noarch.rpm
zypper --gpg-auto-import-keys install -y pg_probackup-{12,11,10,9.6,9.5}
#SRPM SUSE|SLES Packages
zypper si pg_probackup-{12,11,10,9.6,9.5}
2020-06-08 13:53:14 +02:00
#RPM ALT Linux 7
2020-05-03 22:03:55 +02:00
sudo echo "rpm https://repo.postgrespro.ru/pg_probackup/rpm/latest/altlinux-p7 x86_64 vanilla" > /etc/apt/sources.list.d/pg_probackup.list
2019-11-19 11:39:37 +02:00
sudo apt-get update
sudo apt-get install pg_probackup-{12,11,10,9.6,9.5}
sudo apt-get install pg_probackup-{12,11,10,9.6,9.5}-debuginfo
2020-06-08 13:53:14 +02:00
#RPM ALT Linux 8
2020-05-03 22:03:55 +02:00
sudo echo "rpm https://repo.postgrespro.ru/pg_probackup/rpm/latest/altlinux-p8 x86_64 vanilla" > /etc/apt/sources.list.d/pg_probackup.list
2019-11-19 11:39:37 +02:00
sudo apt-get update
sudo apt-get install pg_probackup-{12,11,10,9.6,9.5}
sudo apt-get install pg_probackup-{12,11,10,9.6,9.5}-debuginfo
2020-06-08 13:53:14 +02:00
#RPM ALT Linux 9
2020-05-03 22:03:55 +02:00
sudo echo "rpm https://repo.postgrespro.ru/pg_probackup/rpm/latest/altlinux-p9 x86_64 vanilla" > /etc/apt/sources.list.d/pg_probackup.list
2019-11-19 11:39:37 +02:00
sudo apt-get update
sudo apt-get install pg_probackup-{12,11,10,9.6,9.5}
sudo apt-get install pg_probackup-{12,11,10,9.6,9.5}-debuginfo
```
2017-07-10 17:08:32 +02:00
#### pg_probackup for PostgresPro Standard and Enterprise
2019-10-27 22:07:44 +02:00
```shell
#DEB Ubuntu|Debian Packages
2020-05-03 22:03:55 +02:00
sudo echo "deb [arch=amd64] https://repo.postgrespro.ru/pg_probackup-forks/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" > /etc/apt/sources.list.d/pg_probackup-forks.list
sudo wget -O - https://repo.postgrespro.ru/pg_probackup-forks/keys/GPG-KEY-PG_PROBACKUP | sudo apt-key add - && sudo apt-get update
2020-06-08 13:53:14 +02:00
sudo apt-get install pg-probackup-{std,ent}-{12,11,10,9.6}
sudo apt-get install pg-probackup-{std,ent}-{12,11,10,9.6}-dbg
2019-10-27 22:07:44 +02:00
#RPM Centos Packages
2020-05-03 22:03:55 +02:00
rpm -ivh https://repo.postgrespro.ru/pg_probackup-forks/keys/pg_probackup-repo-forks-centos.noarch.rpm
2020-06-08 13:53:14 +02:00
yum install pg_probackup-{std,ent}-{12,11,10,9.6}
yum install pg_probackup-{std,ent}-{12,11,10,9.6}-debuginfo
2019-10-27 22:07:44 +02:00
#RPM RHEL Packages
2020-05-03 22:03:55 +02:00
rpm -ivh https://repo.postgrespro.ru/pg_probackup-forks/keys/pg_probackup-repo-forks-rhel.noarch.rpm
2020-06-08 13:53:14 +02:00
yum install pg_probackup-{std,ent}-{12,11,10,9.6}
yum install pg_probackup-{std,ent}-{12,11,10,9.6}-debuginfo
2019-10-27 22:07:44 +02:00
#RPM Oracle Linux Packages
2020-05-03 22:03:55 +02:00
rpm -ivh https://repo.postgrespro.ru/pg_probackup-forks/keys/pg_probackup-repo-forks-oraclelinux.noarch.rpm
2020-06-08 13:53:14 +02:00
yum install pg_probackup-{std,ent}-{12,11,10,9.6}
yum install pg_probackup-{std,ent}-{12,11,10,9.6}-debuginfo
2020-06-08 13:53:14 +02:00
#RPM ALT Linux 7
2020-05-03 22:03:55 +02:00
sudo echo "rpm https://repo.postgrespro.ru/pg_probackup-forks/rpm/latest/altlinux-p7 x86_64 forks" > /etc/apt/sources.list.d/pg_probackup_forks.list
2019-11-19 11:39:37 +02:00
sudo apt-get update
2020-06-08 13:53:14 +02:00
sudo apt-get install pg_probackup-{std,ent}-{12,11,10,9.6}
sudo apt-get install pg_probackup-{std,ent}-{12,11,10,9.6}-debuginfo
2020-06-08 13:53:14 +02:00
#RPM ALT Linux 8
2020-05-03 22:03:55 +02:00
sudo echo "rpm https://repo.postgrespro.ru/pg_probackup-forks/rpm/latest/altlinux-p8 x86_64 forks" > /etc/apt/sources.list.d/pg_probackup_forks.list
2019-11-19 11:39:37 +02:00
sudo apt-get update
2020-06-08 13:53:14 +02:00
sudo apt-get install pg_probackup-{std,ent}-{12,11,10,9.6}
sudo apt-get install pg_probackup-{std,ent}-{12,11,10,9.6}-debuginfo
2020-06-08 13:53:14 +02:00
#RPM ALT Linux 9
2020-05-03 22:03:55 +02:00
sudo echo "rpm https://repo.postgrespro.ru/pg_probackup-forks/rpm/latest/altlinux-p9 x86_64 forks" > /etc/apt/sources.list.d/pg_probackup_forks.list && sudo apt-get update
2020-06-08 13:53:14 +02:00
sudo apt-get install pg_probackup-{std,ent}-{12,11,10,9.6}
sudo apt-get install pg_probackup-{std,ent}-{12,11,10,9.6}-debuginfo
2019-10-27 22:07:44 +02:00
```
Once you have `pg_probackup` installed, complete [the setup](https://github.com/postgrespro/pg_probackup/blob/master/Documentation.md#installation-and-setup).
2018-12-27 12:32:46 +02:00
## Building from source
### Linux
2019-12-23 17:23:29 +02:00
To compile `pg_probackup`, you must have a PostgreSQL installation and raw source tree. Execute this in the module's directory:
2017-07-10 17:08:32 +02:00
```shell
make USE_PGXS=1 PG_CONFIG=<path_to_pg_config> top_srcdir=<path_to_PostgreSQL_source_tree>
```
2019-12-23 17:23:29 +02:00
The alternative way, without using the PGXS infrastructure, is to place `pg_probackup` source directory into `contrib` directory and build it there. Example:
```shell
cd <path_to_PostgreSQL_source_tree> && git clone https://github.com/postgrespro/pg_probackup contrib/pg_probackup && cd contrib/pg_probackup && make
```
2018-12-27 12:32:46 +02:00
### Windows
2017-07-10 17:08:32 +02:00
2018-12-27 12:32:46 +02:00
Currently pg_probackup can be build using only MSVC 2013.
Build PostgreSQL using [pgwininstall](https://github.com/postgrespro/pgwininstall) or [PostgreSQL instruction](https://www.postgresql.org/docs/10/install-windows-full.html) with MSVC 2013.
If zlib support is needed, src/tools/msvc/config.pl must contain path to directory with compiled zlib. [Example](https://gist.githubusercontent.com/gsmol/80989f976ce9584824ae3b1bfb00bd87/raw/240032950d4ac4801a79625dd00c8f5d4ed1180c/gistfile1.txt)
```shell
CALL "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall" amd64
SET PATH=%PATH%;C:\Perl64\bin
SET PATH=%PATH%;C:\msys64\usr\bin
gen_probackup_project.pl C:\path_to_postgresql_source_tree
```
## License
This module available under the [license](LICENSE) similar to [PostgreSQL](https://www.postgresql.org/about/license/).
## Feedback
Do not hesitate to post your issues, questions and new ideas at the [issues](https://github.com/postgrespro/pg_probackup/issues) page.
2017-07-10 17:08:32 +02:00
## Authors
Postgres Professional, Moscow, Russia.
## Credits
2018-10-09 14:01:08 +02:00
`pg_probackup` utility is based on `pg_arman`, that was originally written by NTT and then developed and maintained by Michael Paquier.