1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
pgbackrest/INSTALL.md

535 lines
19 KiB
Markdown
Raw Normal View History

2014-09-18 20:13:04 +03:00
# PgBackRest Installation
2014-06-08 00:29:11 +03:00
2014-09-15 03:06:45 +03:00
## sample ubuntu 12.04 install
1. Starting from a clean install, update the OS:
```
apt-get update
apt-get upgrade (reboot if required)
```
2. Install ssh, git and cpanminus
```
apt-get install ssh
apt-get install git
apt-get install cpanminus
```
3. Install Postgres (instructions from http://www.postgresql.org/download/linux/ubuntu/)
Create the file /etc/apt/sources.list.d/pgdg.list, and add a line for the repository:
```
deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main
```
Then run the following:
```
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
apt-get install postgresql-9.3
apt-get install postgresql-server-dev-9.3
```
4. Install required Perl modules:
```
cpanm JSON
cpanm Net::OpenSSH
cpanm DBI
cpanm DBD::Pg
cpanm IPC::System::Simple
cpanm Digest::SHA
cpanm IO::Compress::Gzip
cpanm IO::Uncompress::Gunzip
```
2014-09-18 20:13:04 +03:00
5. Install PgBackRest
2014-09-15 03:06:45 +03:00
Backrest can be installed by downloading the most recent release:
2014-10-14 22:44:50 +03:00
https://github.com/pgmasters/backrest/releases
2014-09-15 03:06:45 +03:00
6. To run unit tests:
* Create backrest_dev user
* Setup trusted ssh between test user account and backrest_dev
* Backrest user and test user must be in the same group
## configuration examples
2014-06-08 00:29:11 +03:00
2014-09-18 20:13:04 +03:00
PgBackRest takes some command-line parameters, but depends on a configuration file for most of the settings. The default location for the configuration file is /etc/pg_backrest.conf.
2014-09-14 20:12:37 +03:00
2014-09-15 03:06:45 +03:00
#### confguring postgres for archiving with backrest
2014-09-14 20:12:37 +03:00
2014-09-15 03:06:45 +03:00
Modify the following settings in postgresql.conf:
```
wal_level = archive
archive_mode = on
archive_command = '/path/to/backrest/bin/pg_backrest.pl --stanza=db archive-push %p'
```
2014-09-18 20:13:04 +03:00
Replace the path with the actual location where PgBackRest was installed. The stanza parameter should be changed to the actual stanza name you used for your database in pg_backrest.conf.
2014-09-15 03:06:45 +03:00
#### simple single host install
2014-09-14 20:12:37 +03:00
This configuration is appropriate for a small installation where backups are being made locally or to a remote file system that is mounted locally.
`/etc/pg_backrest.conf`:
```
[global:command]
2014-09-15 03:06:45 +03:00
psql=/usr/bin/psql
2014-09-14 20:12:37 +03:00
2014-09-15 03:06:45 +03:00
[global:backup]
path=/var/lib/postgresql/backup
[global:retention]
full-retention=2
differential-retention=2
archive-retention-type=diff
archive-retention=2
[db]
path=/var/lib/postgresql/9.3/main
```
#### simple multiple host install
This configuration is appropriate for a small installation where backups are being made remotely. Make sure that postgres@db-host has trusted ssh to backrest@backup-host and vice versa. This configuration assumes that you have pg_backrest_remote.pl and pg_backrest.pl in the same path on both servers.
2014-09-15 03:06:45 +03:00
`/etc/pg_backrest.conf on the db host`:
```
[global:command]
psql=/usr/bin/psql
2014-09-14 22:31:56 +03:00
[global:backup]
2014-09-15 03:06:45 +03:00
host=backup-host@mydomain.com
user=postgres
path=/var/lib/postgresql/backup
[db]
path=/var/lib/postgresql/9.3/main
2014-09-16 01:18:40 +03:00
```
2014-09-15 03:06:45 +03:00
`/etc/pg_backrest.conf on the backup host`:
```
[global:command]
psql=/usr/bin/psql
[global:backup]
path=/var/lib/postgresql/backup
2014-09-14 22:31:56 +03:00
[global:retention]
2014-09-14 20:12:37 +03:00
full-retention=2
archive-retention-type=full
[db]
2014-09-15 03:06:45 +03:00
host=db-host@mydomain.com
user=postgres
path=/var/lib/postgresql/9.3/main
2014-09-16 01:18:40 +03:00
```
2014-09-14 20:12:37 +03:00
2015-01-31 23:27:32 +02:00
## operations
2014-09-15 03:06:45 +03:00
2015-01-31 23:27:32 +02:00
PgBackRest is intended to be run from a scheduler like cron as there is no built-in scheduler.
2015-01-31 23:27:32 +02:00
### general options
These options are either global or used by all operations.
#### stanza
Defines the stanza for the operation. A stanza is the configuration for a database that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one Postgres cluster and therefore one stanza, whereas backup servers will have a stanza for every database that needs to be backed up.
Examples of how to configure a stanza can be found in the `configuration examples` section.
#### config
By default PgBackRest expects the its configuration file to be located at `/etc/pg_backrest.conf`. Use this option to specify another location.
#### version
Returns the PgBackRest version.
#### help
Prints help with a summary of all options.
### backup
Perform a database backup.
#### options
##### type
The following backup types are supported:
- `full` - all database files will be copied and there will be no dependencies on previous backups.
- `incr` - incremental from the last successful backup.
- `diff` - like an incremental backup but always based on the last full backup.
##### no-start-stop
2015-02-01 00:10:19 +02:00
This option prevents PgBackRest from running `pg_start_backup()` and `pg_stop_backup()` on the database. In order for this to work Postgres should be shut down and PgBackRest will generate an error if it is not.
2015-01-31 23:27:32 +02:00
The purpose of this option is to allow cold backups. The `pg_xlog` directory is copied as-is and `backup::archive-required` is automatically set to `n` for the backup.
##### force
When used with `--no-start-stop` a backup will be run even if PgBackRest thinks that Postgres is running. **This option should be used with extreme care as it will likely result in a bad backup.**
2015-02-01 00:10:19 +02:00
There are some scenarios where a backup might still be desirable under these conditions. For example, if a server crashes and database volume can only be mounted read-only, it would be a good idea to take a backup even if `postmaster.pid` is present. In this case it would be better to revert to the prior backup and replay WAL, but possibly there is a very important transaction in a WAL log that did not get archived.
2015-01-31 23:27:32 +02:00
#### usage examples
```
/path/to/pg_backrest.pl --stanza=db --type=full backup
```
2015-01-31 23:27:32 +02:00
Run a `full` backup on the `db` stanza. `--type` can also be set to `incr` or `diff` for incremental or differential backups. However, if no `full` backup exists then a `full` backup will be forced even if `incr` or `diff` is requested.
2015-01-31 23:27:32 +02:00
### archive-push
```
/path/to/pg_backrest.pl --stanza=db archive-push %p
```
Accepts an archive file from Postgres and pushes it to the backup. `%p` is how Postgres specifies the location of the file to be archived. This command has no other purpose.
2015-01-31 23:27:32 +02:00
### archive-get
```
/path/to/pg_backrest.pl --stanza=db archive-get %f %p
```
Retrieves an archive log from the backup. This is used in `restore.conf` to restore a backup to that last archive log, do PITR, or as an alternative to streaming for keep a replica up to date. `%f` is how Postgres specifies the archive log it needs, and `%p` is the location where it should be copied.
2015-01-31 23:27:32 +02:00
### expire
PgBackRest does backup rotation, but it is not concerned with when the backups were created. So if two full backups are configured in rentention, PgBackRest will keep two full backup no matter whether they occur 2 hours apart or two weeks apart.
```
/path/to/pg_backrest.pl --stanza=db expire
```
2015-01-31 23:27:32 +02:00
Expire (rotate) any backups that exceed the defined retention. Expiration is run automatically after every successful backup, so there's no need to run this command on its own unless you have reduced rentention, usually to free up some space.
### restore
2015-02-01 00:10:19 +02:00
Restore a database from the PgBackRest repository.
2015-02-01 00:36:25 +02:00
* restore options
2015-02-01 00:10:19 +02:00
2015-02-01 00:36:25 +02:00
* set
2015-02-01 00:10:19 +02:00
The backup set to be restored. `latest` will restore the latest backup, otherwise provide the name of the backup to restore. For example: `20150131-153358F` or `20150131-153358F_20150131-153401I`.
2015-02-01 00:36:25 +02:00
* delta
2015-02-01 00:10:19 +02:00
By default the database base and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums if available and timestamp/size when checksums are not available.
2015-02-01 00:36:25 +02:00
* force
2015-02-01 00:10:19 +02:00
By itself this option forces the database base and tablespace paths to be completely overwritten. In combination with `--delta` a timestamp/size delta will be performed even if checksums are available.
2015-02-01 00:36:25 +02:00
* recovery options
2015-02-01 00:10:19 +02:00
2015-02-01 00:36:25 +02:00
* type
2015-02-01 00:10:19 +02:00
The following recovery types are supported:
2015-01-31 23:27:32 +02:00
2015-02-01 00:10:19 +02:00
- `default` - recover to the end of the archive stream.
- `name` - recover the restore point specified in `--target`.
- `xid` - recover to the transaction id specified in `--target`.
- `time` - recover to the time specified in `--target`.
- `preserve` - preserve the existing `recovery.conf` file.
- `none` - no recovery past database becoming consistent.
2015-01-31 23:27:32 +02:00
2015-02-01 00:10:19 +02:00
Note that the `none` option may produce duplicate WAL if the database is started with archive logging enabled. It is recommended that a new stanza be created for production databases restored in this way.
2015-01-31 23:27:32 +02:00
2015-02-01 00:36:25 +02:00
* target
2015-02-01 00:10:19 +02:00
Defines the recovery target when `--type` is `name`, `xid`, or `time`. For example, `--type=time` and `--target=2015-01-30 14:15:11 EST`.
2015-02-01 00:36:25 +02:00
* target-exclusive
2015-02-01 00:10:19 +02:00
Defines whether recovery to the target whould be exclusive (the default is inclusive) and is only valid when `--type` is `time` or `xid`. For example, using `--target-exclusive` would exclude the contents of transaction `1007` when `--type=xid` and `-target=1007`. See `recovery_target_inclusive` option in Postgres docs for more information.
2015-02-01 00:36:25 +02:00
* target-resume
2015-02-01 00:10:19 +02:00
Specifies whether recovery should resume when the recovery target is reached. See `pause_at_recovery_target` in Postgres docs for more information.
2015-02-01 00:36:25 +02:00
* target-timeline
2015-02-01 00:10:19 +02:00
Recovers along the specified timeline. See `recovery_target_timeline` in Postgres docs for more information.
2015-02-01 00:36:25 +02:00
* usage examples
2015-02-01 00:10:19 +02:00
```
/path/to/pg_backrest.pl --stanza=db --set=latest --type=name --target=release
```
Restores the latest database backup and then recovers to the `release` restore point.
**MORE TO BE ADDED HERE**
2015-01-31 23:27:32 +02:00
[reference this when writing about tablespace remapping]
http://www.databasesoup.com/2013/11/moving-tablespaces.html
## structure
PgBackRest stores files in a way that is easy for users to work with directly. Each backup directory has one file and two subdirectories:
2015-02-01 00:32:16 +02:00
* `backup.manifest` file
Stores information about all the directories, links, and files in the backup. The file is plaintext and should be very clear, but documentation of the format is planned in a future release.
2015-02-01 00:32:16 +02:00
* `base` directory
Contains the Postgres data directory as defined by the data_directory setting in `postgresql.conf`.
2015-02-01 00:32:16 +02:00
* `tablespace` directory
If tablespaces are present in the database, contains each tablespace in a separate subdirectory. Tablespace names are used for the subdirectories unless --no-start-stop is specified in which case oids will be used instead. The links in `base/pg_tblspc` are rewritten to the tablespace directory in either case.
2014-09-15 03:06:45 +03:00
## configuration options
2014-09-14 20:12:37 +03:00
2014-09-16 01:18:40 +03:00
Each section defines important aspects of the backup. All configuration sections below should be prefixed with `global:` as demonstrated in the configuration samples.
2014-09-14 20:12:37 +03:00
#### command section
2014-09-18 20:13:04 +03:00
The command section defines external commands that are used by PgBackRest.
2014-09-14 20:12:37 +03:00
##### psql key
Defines the full path to psql. psql is used to call pg_start_backup() and pg_stop_backup().
2014-09-16 01:18:40 +03:00
```
required: y
example: psql=/usr/bin/psql
```
2014-09-14 20:12:37 +03:00
##### remote key
Defines the file path to pg_backrest_remote.pl.
Required only if the path to pg_backrest_remote.pl is different on the local and remote systems. If not defined, the remote path will be assumed to be the same as the local path.
2014-09-16 01:26:03 +03:00
```
required: n
example: remote=/home/postgres/backrest/bin/pg_backrest_remote.pl
```
2014-09-14 20:12:37 +03:00
#### command-option section
The command-option section allows abitrary options to be passed to any command in the command section.
##### psql key
Allows command line parameters to be passed to psql.
2014-09-16 01:26:03 +03:00
```
required: no
example: psql=--port=5433
```
2015-02-01 00:10:19 +02:00
2014-09-14 20:12:37 +03:00
#### log section
The log section defines logging-related settings. The following log levels are supported:
- `off `- No logging at all (not recommended)
- `error `- Log only errors
- `warn `- Log warnings and errors
- `info `- Log info, warnings, and errors
- `debug `- Log debug, info, warnings, and errors
- `trace `- Log trace (very verbose debugging), debug, info, warnings, and errors
##### level-file
Sets file log level.
2014-09-16 01:26:03 +03:00
```
default: info
example: level-file=warn
```
2014-09-14 20:12:37 +03:00
##### level-console
Sets console log level.
2014-09-16 01:26:03 +03:00
```
default: error
example: level-file=info
```
2014-09-14 20:12:37 +03:00
#### backup section
The backup section defines settings related to backup and archiving.
##### host
Sets the backup host.
2014-09-16 01:26:03 +03:00
```
required: n (but must be set if user is defined)
example: host=backup.mydomain.com
```
2014-09-14 20:12:37 +03:00
##### user
Sets user account on the backup host.
2014-09-16 01:26:03 +03:00
```
required: n (but must be set if host is defined)
example: user=backrest
```
2014-09-14 20:12:37 +03:00
##### path
Path where backups are stored on the local or remote host.
2014-09-16 01:26:03 +03:00
```
required: y
example: path=/backup/backrest
```
2014-09-14 20:12:37 +03:00
##### compress
Enable gzip compression. Files stored in the backup are compatible with command-line gzip tools.
2014-09-16 01:26:03 +03:00
```
default: y
example: compress=n
```
2014-09-14 20:12:37 +03:00
##### checksum
Enable SHA-1 checksums. Backup checksums are stored in backup.manifest while archive checksums are stored in the filename.
2014-09-16 01:26:03 +03:00
```
default: y
example: checksum=n
```
2014-10-07 03:53:34 +03:00
##### start-fast
2014-09-14 20:12:37 +03:00
Forces an immediate checkpoint (by passing true to the fast parameter of pg_start_backup()) so the backup begins immediately.
2014-09-16 01:26:03 +03:00
```
default: n
2014-10-07 03:53:34 +03:00
example: start-fast=y
2014-09-16 01:26:03 +03:00
```
2014-09-14 20:12:37 +03:00
##### hardlink
Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each
backup is a full backup. Be care though, because modifying files that are hard-linked can affect all the backups in the set.
2014-09-16 01:26:03 +03:00
```
default: y
example: hardlink=n
```
2014-09-14 20:12:37 +03:00
##### thread-max
Defines the number of threads to use for backup. Each thread will perform compression and transfer to make the backup run faster, but don't set `thread-max` so high that it impacts database performance.
2014-09-16 01:26:03 +03:00
```
default: 1
example: thread-max=4
```
2014-09-14 20:12:37 +03:00
##### thread-timeout
2014-10-07 03:53:34 +03:00
Maximum amount of time (in seconds) that a backup thread should run. This limits the amount of time that a thread might be stuck due to unforeseen issues during the backup.
2014-09-16 01:26:03 +03:00
```
default: <none>
2014-10-07 03:53:34 +03:00
example: thread-timeout=86400
2014-09-16 01:26:03 +03:00
```
2014-09-14 20:12:37 +03:00
##### archive-required
Are archive logs required to to complete the backup? It's a good idea to leave this as the default unless you are using another
method for archiving.
2014-09-16 01:26:03 +03:00
```
default: y
example: archive-required=n
```
2014-09-14 20:12:37 +03:00
#### archive section
The archive section defines parameters when doing async archiving. This means that the archive files will be stored locally, then a background process will pick them and move them to the backup.
##### path
Path where archive logs are stored before being asynchronously transferred to the backup. Make sure this is not the same path as the backup is using if the backup is local.
2014-09-16 01:26:03 +03:00
```
required: y
example: path=/backup/archive
```
2014-09-14 20:12:37 +03:00
##### compress-async
When set then archive logs are not compressed immediately, but are instead compressed when copied to the backup host. This means that more space will be used on local storage, but the initial archive process will complete more quickly allowing greater throughput from Postgres.
2014-09-16 01:26:03 +03:00
```
default: n
example: compress-async=y
```
2014-09-14 20:12:37 +03:00
##### archive-max-mb
Limits the amount of archive log that will be written locally. After the limit is reached, the following will happen:
2014-09-18 20:13:04 +03:00
1. PgBackRest will notify Postgres that the archive was succesfully backed up, then DROP IT.
2014-09-14 20:12:37 +03:00
2. An error will be logged to the console and also to the Postgres log.
3. A stop file will be written in the lock directory and no more archive files will be backed up until it is removed.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
The purpose of this feature is to prevent the log volume from filling up at which point Postgres will stop all operation. Better to lose the backup than have the database go down completely.
To start normal archiving again you'll need to remove the stop file which will be located at `${archive-path}/lock/${stanza}-archive.stop` where `${archive-path}` is the path set in the archive section, and ${stanza} is the backup stanza.
2014-09-16 01:26:03 +03:00
```
required: n
example: archive-max-mb=1024
```
2014-09-14 20:12:37 +03:00
#### retention section
The rentention section defines how long backups will be retained. Expiration only occurs when the number of complete backups exceeds the allowed retention. In other words, if full-retention is set to 2, then there must be 3 complete backups before the oldest will be expired. Make sure you always have enough space for rentention + 1 backups.
##### full-retention
Number of full backups to keep. When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When not defined then all full backups will be kept.
2014-09-16 01:26:03 +03:00
```
required: n
example: full-retention=2
```
2014-09-14 20:12:37 +03:00
##### differential-retention
Number of differential backups to keep. When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.
2014-09-16 01:26:03 +03:00
```
required: n
example: differential-retention=3
```
2014-09-14 20:12:37 +03:00
##### archive-retention-type
2014-09-18 20:13:04 +03:00
Type of backup to use for archive retention (full or differential). If set to full, then PgBackRest will keep archive logs for the number of full backups defined by `archive-retention`. If set to differential, then PgBackRest will keep archive logs for the number of differential backups defined by `archive-retention`.
2014-09-14 20:12:37 +03:00
If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs that are older than the oldest backup, but there may be reasons for doing so.
2014-09-16 01:26:03 +03:00
```
required: n
example: archive-retention-type=full
```
2014-09-14 20:12:37 +03:00
##### archive-retention
Number of backups worth of archive log to keep. If not defined, then `full-retention` will be used when `archive-retention-type=full` and `differential-retention` will be used when `archive-retention-type=differential`.
2014-09-16 01:26:03 +03:00
```
required: n
example: archive-retention=2
```
2015-01-26 00:27:46 +02:00
### restore section
?????
### restore:option section
`Archive Recovery` and `Standby Server` `restore.conf` options can be specified here. See http://www.postgresql.org/docs/X.X/static/recovery-config.html for details on `restore.conf` options (replace `X.X` with your database version).
Note: `restore_command` will automatically be generated unless overridden in this section. Be careful about specifying your own `restore_command` as PgBackRest is designed to handle this for you.
'Target Recovery` options are specified on the command-line since they end to change from restore to restore (or not be needed at all in the case of a standby server).
Since PgBackRest does not start PostgreSQL after writing the recovery.conf file, it is always possible to edit/check the file before manually restarting.
2014-09-14 20:12:37 +03:00
### stanza sections
A stanza defines a backup for a specific database. The stanza section must define the base database path and host/user if the database is remote. Also, any global configuration sections can be overridden to define stanza-specific settings.
##### host
Sets the database host.
2014-09-16 01:26:03 +03:00
```
required: n (but must be set if user is defined)
example: host=db.mydomain.com
```
2014-09-14 20:12:37 +03:00
##### user
Sets user account on the db host.
2014-09-16 01:26:03 +03:00
```
required: n (but must be set if host is defined)
example: user=postgres
```
2014-09-14 20:12:37 +03:00
##### path
Path to the db data directory (data_directory setting in postgresql.conf).
2014-09-16 01:26:03 +03:00
```
required: y
example: path=/var/postgresql/data
```