db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-03-17 20:58:34 +02:00
Code Issues Releases Activity

First pass at building automated docs for markdown/html. This works pretty well, but the config sections of doc.xml still require too much maintenance. With the new Config code, it should be possible to generate those sections automatically.

Browse Source
This commit is contained in:
David Steele 2015-03-08 14:05:41 -04:00
parent ae6bdecfaf
commit 7675a11ded
10 changed files with 2139 additions and 1034 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

562
INSTALL.md
View File

@ -1,562 +0,0 @@
# PgBackRest Installation
## sample ubuntu 12.04 install
1. Starting from a clean install, update the OS:
```
apt-get update
apt-get upgrade (reboot if required)
```
2. Install ssh, git and cpanminus
```
apt-get install ssh
apt-get install git
apt-get install cpanminus
```
3. Install Postgres (instructions from http://www.postgresql.org/download/linux/ubuntu/)
Create the file /etc/apt/sources.list.d/pgdg.list, and add a line for the repository:
```
deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main
```
Then run the following:
```
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
apt-get install postgresql-9.3
apt-get install postgresql-server-dev-9.3
```
4. Install required Perl modules:
```
cpanm JSON
cpanm Net::OpenSSH
cpanm DBI
cpanm DBD::Pg
cpanm IPC::System::Simple
cpanm Digest::SHA
cpanm Compress::ZLib
```
5. Install PgBackRest
Backrest can be installed by downloading the most recent release:
https://github.com/pgmasters/backrest/releases
6. To run unit tests:
* Create backrest_dev user
* Setup trusted ssh between test user account and backrest_dev
* Backrest user and test user must be in the same group
## configuration examples
PgBackRest takes some command-line parameters, but depends on a configuration file for most of the settings. The default location for the configuration file is /etc/pg_backrest.conf.
#### confguring postgres for archiving with backrest
Modify the following settings in postgresql.conf:
```
wal_level = archive
archive_mode = on
archive_command = '/path/to/backrest/bin/pg_backrest.pl --stanza=db archive-push %p'
```
Replace the path with the actual location where PgBackRest was installed. The stanza parameter should be changed to the actual stanza name you used for your database in pg_backrest.conf.
#### simple single host install
This configuration is appropriate for a small installation where backups are being made locally or to a remote file system that is mounted locally.
`/etc/pg_backrest.conf`:
```
[global:command]
psql=/usr/bin/psql
[global:backup]
path=/var/lib/postgresql/backup
[global:retention]
full-retention=2
differential-retention=2
archive-retention-type=diff
archive-retention=2
[db]
path=/var/lib/postgresql/9.3/main
```
#### simple multiple host install
This configuration is appropriate for a small installation where backups are being made remotely. Make sure that postgres@db-host has trusted ssh to backrest@backup-host and vice versa. This configuration assumes that you have pg_backrest_remote.pl and pg_backrest.pl in the same path on both servers.
`/etc/pg_backrest.conf on the db host`:
```
[global:command]
psql=/usr/bin/psql
[global:backup]
host=backup-host@mydomain.com
user=postgres
path=/var/lib/postgresql/backup
[db]
path=/var/lib/postgresql/9.3/main
```
`/etc/pg_backrest.conf on the backup host`:
```
[global:command]
psql=/usr/bin/psql
[global:backup]
path=/var/lib/postgresql/backup
[global:retention]
full-retention=2
archive-retention-type=full
[db]
host=db-host@mydomain.com
user=postgres
path=/var/lib/postgresql/9.3/main
```
## operations
PgBackRest is intended to be run from a scheduler like cron as there is no built-in scheduler.
### general options
These options are either global or used by all operations.
#### stanza
Defines the stanza for the operation. A stanza is the configuration for a database that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one Postgres cluster and therefore one stanza, whereas backup servers will have a stanza for every database that needs to be backed up.
Examples of how to configure a stanza can be found in the `configuration examples` section.
#### config
By default PgBackRest expects the its configuration file to be located at `/etc/pg_backrest.conf`. Use this option to specify another location.
#### version
Returns the PgBackRest version.
#### help
Prints help with a summary of all options.
### backup
Perform a database backup.
#### options
##### type
The following backup types are supported:
- `full` - all database files will be copied and there will be no dependencies on previous backups.
- `incr` - incremental from the last successful backup.
- `diff` - like an incremental backup but always based on the last full backup.
##### no-start-stop
This option prevents PgBackRest from running `pg_start_backup()` and `pg_stop_backup()` on the database. In order for this to work Postgres should be shut down and PgBackRest will generate an error if it is not.
The purpose of this option is to allow cold backups. The `pg_xlog` directory is copied as-is and `backup::archive-required` is automatically set to `n` for the backup.
##### force
When used with `--no-start-stop` a backup will be run even if PgBackRest thinks that Postgres is running. **This option should be used with extreme care as it will likely result in a bad backup.**
There are some scenarios where a backup might still be desirable under these conditions. For example, if a server crashes and database volume can only be mounted read-only, it would be a good idea to take a backup even if `postmaster.pid` is present. In this case it would be better to revert to the prior backup and replay WAL, but possibly there is a very important transaction in a WAL log that did not get archived.
#### usage examples
```
/path/to/pg_backrest.pl --stanza=db --type=full backup
```
Run a `full` backup on the `db` stanza. `--type` can also be set to `incr` or `diff` for incremental or differential backups. However, if no `full` backup exists then a `full` backup will be forced even if `incr` or `diff` is requested.
### archive-push
```
/path/to/pg_backrest.pl --stanza=db archive-push %p
```
Accepts an archive file from Postgres and pushes it to the backup. `%p` is how Postgres specifies the location of the file to be archived. This command has no other purpose.
### archive-get
```
/path/to/pg_backrest.pl --stanza=db archive-get %f %p
```
Retrieves an archive log from the backup. This is used in `restore.conf` to restore a backup to that last archive log, do PITR, or as an alternative to streaming for keep a replica up to date. `%f` is how Postgres specifies the archive log it needs, and `%p` is the location where it should be copied.
### expire
PgBackRest does backup rotation, but it is not concerned with when the backups were created. So if two full backups are configured in rentention, PgBackRest will keep two full backup no matter whether they occur 2 hours apart or two weeks apart.
```
/path/to/pg_backrest.pl --stanza=db expire
```
Expire (rotate) any backups that exceed the defined retention. Expiration is run automatically after every successful backup, so there's no need to run this command on its own unless you have reduced rentention, usually to free up some space.
### restore
Restore a database from the PgBackRest repository.
#### restore options
##### set
The backup set to be restored. `latest` will restore the latest backup, otherwise provide the name of the backup to restore. For example: `20150131-153358F` or `20150131-153358F_20150131-153401I`.
##### delta
By default the database base and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
##### force
By itself this option forces the database base and tablespace paths to be completely overwritten. In combination with `--delta` a timestamp/size delta will be performed instead of using checksums.
#### recovery options
##### type
The following recovery types are supported:
- `default` - recover to the end of the archive stream.
- `name` - recover the restore point specified in `--target`.
- `xid` - recover to the transaction id specified in `--target`.
- `time` - recover to the time specified in `--target`.
- `preserve` - preserve the existing `recovery.conf` file.
- `none` - no recovery past database becoming consistent.
Note that the `none` option may produce duplicate WAL if the database is started with archive logging enabled. It is recommended that a new stanza be created for production databases restored in this way.
##### target
Defines the recovery target when `--type` is `name`, `xid`, or `time`. For example, `--type=time` and `--target=2015-01-30 14:15:11 EST`.
##### target-exclusive
Defines whether recovery to the target whould be exclusive (the default is inclusive) and is only valid when `--type` is `time` or `xid`. For example, using `--target-exclusive` would exclude the contents of transaction `1007` when `--type=xid` and `-target=1007`. See `recovery_target_inclusive` option in Postgres docs for more information.
##### target-resume
Specifies whether recovery should resume when the recovery target is reached. See `pause_at_recovery_target` in Postgres docs for more information.
##### target-timeline
Recovers along the specified timeline. See `recovery_target_timeline` in Postgres docs for more information.
#### usage examples
```
/path/to/pg_backrest.pl --stanza=db --set=latest --type=name --target=release
```
Restores the latest database backup and then recovers to the `release` restore point.
**MORE TO BE ADDED HERE**
PITR should start after the stop time in the .backup file.
[reference this when writing about tablespace remapping]
http://www.databasesoup.com/2013/11/moving-tablespaces.html
## structure
PgBackRest stores files in a way that is easy for users to work with directly. Each backup directory has one file and two subdirectories:
* `backup.manifest` file
Stores information about all the directories, links, and files in the backup. The file is plaintext and should be very clear, but documentation of the format is planned in a future release.
* `base` directory
Contains the Postgres data directory as defined by the data_directory setting in `postgresql.conf`.
* `tablespace` directory
If tablespaces are present in the database, contains each tablespace in a separate subdirectory. Tablespace names are used for the subdirectories unless --no-start-stop is specified in which case oids will be used instead. The links in `base/pg_tblspc` are rewritten to the tablespace directory in either case.
## configuration options
Each section defines important aspects of the backup. All configuration sections below should be prefixed with `global:` as demonstrated in the configuration samples.
#### command section
The command section defines external commands that are used by PgBackRest.
##### psql key
Defines the full path to psql. psql is used to call pg_start_backup() and pg_stop_backup().
```
required: y
example: psql=/usr/bin/psql
```
##### remote key
Defines the file path to pg_backrest_remote.pl.
Required only if the path to pg_backrest_remote.pl is different on the local and remote systems. If not defined, the remote path will be assumed to be the same as the local path.
```
required: n
example: remote=/home/postgres/backrest/bin/pg_backrest_remote.pl
```
#### command-option section
The command-option section allows abitrary options to be passed to any command in the command section.
##### psql key
Allows command line parameters to be passed to psql.
```
required: no
example: psql=--port=5433
```
#### log section
The log section defines logging-related settings. The following log levels are supported:
- `off `- No logging at all (not recommended)
- `error `- Log only errors
- `warn `- Log warnings and errors
- `info `- Log info, warnings, and errors
- `debug `- Log debug, info, warnings, and errors
- `trace `- Log trace (very verbose debugging), debug, info, warnings, and errors
##### level-file
Sets file log level.
```
default: info
example: level-file=warn
```
##### level-console
Sets console log level.
```
default: error
example: level-file=info
```
#### general section
The `general` section defines settings that are shared between multiple operations.
##### buffer-size
Set the buffer size used for copy, compress, and uncompress functions. A maximum of 3 buffers will be in use at a time per thread. An additional maximum of 256K per thread may be used for zlib buffers.
```
default: 1048576
allowed: 4096 - 8388608
example: buffer-size=8192
```
##### compress-level
Sets the zlib level to be used for file compression when `compress=y`.
This setting can be overridden in the `backup` and `archive` sections.
```
default: 6
allowed: 0 - 9
example: compress-level=9
```
##### compress-level-network
Sets the zlib level to be used for protocol compression when `compress=n` and the database is not on the same host as the backup. Protocol compression is used to reduce network traffic but can be disabled by setting `compress-level-network=0`. When `compress=y` the `compress-level-network` setting is ignored and `compress-level` is used instead so that the file is only compressed once. SSH compression is always disabled.
This setting can be overridden in the `backup` and `archive`, and `restore` sections.
```
default: 3
allowed: 0 - 9
example: compress-level-network=1
```
#### backup section
The backup section defines settings related to backup and archiving.
##### host
Sets the backup host.
```
required: n (but must be set if user is defined)
example: host=backup.mydomain.com
```
##### user
Sets user account on the backup host.
```
required: n (but must be set if host is defined)
example: user=backrest
```
##### path
Path where backups are stored on the local or remote host.
```
required: y
example: path=/backup/backrest
```
##### compress
Enable gzip compression. Files stored in the backup are compatible with command-line gzip tools.
```
default: y
example: compress=n
```
##### start-fast
Forces an immediate checkpoint (by passing true to the fast parameter of pg_start_backup()) so the backup begins immediately.
```
default: n
example: start-fast=y
```
##### hardlink
Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each
backup is a full backup. Be care though, because modifying files that are hard-linked can affect all the backups in the set.
```
default: y
example: hardlink=n
```
##### thread-max
Defines the number of threads to use for backup. Each thread will perform compression and transfer to make the backup run faster, but don't set `thread-max` so high that it impacts database performance.
```
default: 1
example: thread-max=4
```
##### thread-timeout
Maximum amount of time (in seconds) that a backup thread should run. This limits the amount of time that a thread might be stuck due to unforeseen issues during the backup.
```
default: <none>
example: thread-timeout=86400
```
##### archive-required
Are archive logs required to to complete the backup? It's a good idea to leave this as the default unless you are using another
method for archiving.
```
default: y
example: archive-required=n
```
#### archive section
The archive section defines parameters when doing async archiving. This means that the archive files will be stored locally, then a background process will pick them and move them to the backup.
##### path
Path where archive logs are stored before being asynchronously transferred to the backup. Make sure this is not the same path as the backup is using if the backup is local.
```
required: y
example: path=/backup/archive
```
##### compress-async
When set then archive logs are not compressed immediately, but are instead compressed when copied to the backup host. This means that more space will be used on local storage, but the initial archive process will complete more quickly allowing greater throughput from Postgres.
```
default: n
example: compress-async=y
```
##### archive-max-mb
Limits the amount of archive log that will be written locally. After the limit is reached, the following will happen:
1. PgBackRest will notify Postgres that the archive was succesfully backed up, then DROP IT.
2. An error will be logged to the console and also to the Postgres log.
3. A stop file will be written in the lock directory and no more archive files will be backed up until it is removed.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
The purpose of this feature is to prevent the log volume from filling up at which point Postgres will stop all operation. Better to lose the backup than have the database go down completely.
To start normal archiving again you'll need to remove the stop file which will be located at `${archive-path}/lock/${stanza}-archive.stop` where `${archive-path}` is the path set in the archive section, and ${stanza} is the backup stanza.
```
required: n
example: archive-max-mb=1024
```
#### retention section
The rentention section defines how long backups will be retained. Expiration only occurs when the number of complete backups exceeds the allowed retention. In other words, if full-retention is set to 2, then there must be 3 complete backups before the oldest will be expired. Make sure you always have enough space for rentention + 1 backups.
##### full-retention
Number of full backups to keep. When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When not defined then all full backups will be kept.
```
required: n
example: full-retention=2
```
##### differential-retention
Number of differential backups to keep. When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.
```
required: n
example: differential-retention=3
```
##### archive-retention-type
Type of backup to use for archive retention (full or differential). If set to full, then PgBackRest will keep archive logs for the number of full backups defined by `archive-retention`. If set to differential, then PgBackRest will keep archive logs for the number of differential backups defined by `archive-retention`.
If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs that are older than the oldest backup, but there may be reasons for doing so.
```
required: n
example: archive-retention-type=full
```
##### archive-retention
Number of backups worth of archive log to keep. If not defined, then `full-retention` will be used when `archive-retention-type=full` and `differential-retention` will be used when `archive-retention-type=differential`.
```
required: n
example: archive-retention=2
```
### restore section
?????
### restore:option section
`Archive Recovery` and `Standby Server` `restore.conf` options can be specified here. See http://www.postgresql.org/docs/X.X/static/recovery-config.html for details on `restore.conf` options (replace `X.X` with your database version).
Note: `restore_command` will automatically be generated unless overridden in this section. Be careful about specifying your own `restore_command` as PgBackRest is designed to handle this for you.
'Target Recovery` options are specified on the command-line since they end to change from restore to restore (or not be needed at all in the case of a standby server).
Since PgBackRest does not start PostgreSQL after writing the recovery.conf file, it is always possible to edit/check the file before manually restarting.
### stanza sections
A stanza defines a backup for a specific database. The stanza section must define the base database path and host/user if the database is remote. Also, any global configuration sections can be overridden to define stanza-specific settings.
##### host
Sets the database host.
```
required: n (but must be set if user is defined)
example: host=db.mydomain.com
```
##### user
Sets user account on the db host.
```
required: n (but must be set if host is defined)
example: user=postgres
```
##### path
Path to the db data directory (data_directory setting in postgresql.conf).
```
required: y
example: path=/var/postgresql/data
```

2
LICENSE
View File

@ -1,6 +1,6 @@
The MIT License (MIT)
Copyright (c) 2013-2014 David Steele
Copyright (c) 2013-2015 David Steele
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in

563
README.md
View File

@ -2,111 +2,550 @@
PgBackRest aims to be a simple backup and restore system that can seamlessly scale up to the largest databases and workloads.
Instead of relying on traditional backup tools like tar and rsync, PgBackRest implements all backup features internally and features a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security. Each thread requires only one SSH connection for remote backups.
Primary PgBackRest features:
* Local or remote backup
* Multi-threaded backup/restore for performance
* Checksums
* Safe backups (checks that logs required for consistency are present before backup completes)
* Full, differential, and incremental backups
* Backup rotation (and minimum retention rules with optional separate retention for archive)
* In-stream compression/decompression
* Archiving and retrieval of logs for replicas/restores built in
* Async archiving for very busy systems (including space limits)
* Backup directories are consistent Postgres clusters (when hardlinks are on and compression is off)
* Tablespace support
* Restore delta option
* Restore using timestamp/size or checksum
* Restore remapping base/tablespaces
- Local or remote backup
- Multi-threaded backup/restore for performance
- Checksums
- Safe backups (checks that logs required for consistency are present before backup completes)
- Full, differential, and incremental backups
- Backup rotation (and minimum retention rules with optional separate retention for archive)
- In-stream compression/decompression
- Archiving and retrieval of logs for replicas/restores built in
- Async archiving for very busy systems (including space limits)
- Backup directories are consistent Postgres clusters (when hardlinks are on and compression is off)
- Tablespace support
- Restore delta option
- Restore using timestamp/size or checksum
- Restore remapping base/tablespaces
## release notes
Instead of relying on traditional backup tools like tar and rsync, PgBackRest implements all backup features internally and features a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security. Each thread requires only one SSH connection for remote backups.
## Install
!!! Perl-based blah, blah, blah
### Ubuntu 12.04
* Starting from a clean install, update the OS:
```
apt-get update
apt-get upgrade (reboot if required)
```
* Install ssh, git and cpanminus:
```
apt-get install ssh
apt-get install git
apt-get install cpanminus
```
* Install Postgres (instructions from http://www.postgresql.org/download/linux/ubuntu/)
Create the file /etc/apt/sources.list.d/pgdg.list, and add a line for the repository:
```
deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main
```
* Then run the following:
```
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
apt-get install postgresql-9.3
apt-get install postgresql-server-dev-9.3
```
* Install required Perl modules:
```
cpanm JSON
cpanm Net::OpenSSH
cpanm DBI
cpanm DBD::Pg
cpanm IPC::System::Simple
cpanm Digest::SHA
cpanm Compress::ZLib
```
* Install PgBackRest
Backrest can be installed by downloading the most recent release:
https://github.com/pgmasters/backrest/releases
* To run unit tests:
```
Create backrest_dev user
Setup trusted ssh between test user account and backrest_dev
Backrest user and test user must be in the same group
```
## Configuration
PgBackRest takes some command-line parameters, but depends on a configuration file for most of the settings. The default location for the configuration file is `/etc/pg_backrest.conf`.
### Examples
#### Confguring Postgres for Archiving
Modify the following settings in `postgresql.conf`:
```
wal_level = archive
archive_mode = on
archive_command = '/path/to/backrest/bin/pg_backrest.pl --stanza=db archive-push %p'
```
Replace the path with the actual location where PgBackRest was installed. The stanza parameter should be changed to the actual stanza name for your database in `pg_backrest.conf`.
#### Simple Single Host Install
This configuration is appropriate for a small installation where backups are being made locally or to a remote file system that is mounted locally.
`/etc/pg_backrest.conf`:
```
[global:command]
psql=/usr/bin/psql
[global:backup]
path=/var/lib/postgresql/backup
[global:retention]
full-retention=2
differential-retention=2
archive-retention-type=diff
archive-retention=2
[db]
path=/var/lib/postgresql/9.3/main
```
#### Simple Multiple Host Install
This configuration is appropriate for a small installation where backups are being made remotely. Make sure that postgres@db-host has trusted ssh to backrest@backup-host and vice versa. This configuration assumes that you have pg_backrest_remote.pl and pg_backrest.pl in the same path on both servers.
`/etc/pg_backrest.conf` on the db host:
```
[global:command]
psql=/usr/bin/psql
[global:backup]
host=backup-host@mydomain.com
user=postgres
path=/var/lib/postgresql/backup
[db]
path=/var/lib/postgresql/9.3/main
```
`/etc/pg_backrest.conf` on the backup host:
```
[global:command]
psql=/usr/bin/psql
[global:backup]
path=/var/lib/postgresql/backup
[global:retention]
full-retention=2
archive-retention-type=full
[db]
host=db-host@mydomain.com
user=postgres
path=/var/lib/postgresql/9.3/main
```
### Options
#### `command` section
The `command` section defines the location of external commands that are used by PgBackRest.
##### `psql` key
Defines the full path to `psql`. `psql` is used to call `pg_start_backup()` and `pg_stop_backup()`.
If addtional parameters need to be passed to `psql` (such as `--port` or `--cluster`) then add `%option%` to the command line and use `command-option::psql` to set options.
```
required: y
example: psql=/usr/bin/psql -X %option%
```
##### `remote` key
Defines the location of `pg_backrest_remote.pl`.
Required only if the path to `pg_backrest_remote.pl` is different on the local and remote systems. If not defined, the remote path will be assumed to be the same as the local path.
```
required: n
default: same as local
example: remote=/usr/lib/backrest/bin/pg_backrest_remote.pl
```
#### `command-option` section
The `command-option` section allows abitrary options to be passed to any command in the `command` section.
##### `psql` key
Allows command line parameters to be passed to `psql`.
```
required: n
example: psql=--port=5433
```
#### `log` section
The `log` section defines logging-related settings. The following log levels are supported:
- `off` - No logging at all (not recommended)
- `error` - Log only errors
- `warn` - Log warnings and errors
- `info` - Log info, warnings, and errors
- `debug` - Log debug, info, warnings, and errors
- `trace` - Log trace (very verbose debugging), debug, info, warnings, and errors
##### `level-file` key
Sets file log level.
```
required: n
default: info
example: level-file=debug
```
##### `level-console` key
Sets console log level.
```
required: n
default: warning
example: level-console=error
```
#### `general` section
The `general` section defines settings that are shared between multiple operations.
##### `buffer-size` key
Set the buffer size used for copy, compress, and uncompress functions. A maximum of 3 buffers will be in use at a time per thread. An additional maximum of 256K per thread may be used for zlib buffers.
```
required: n
default: 1048576
allow: 4096 - 8388608
example: buffer-size=16384
```
##### `compress-level` key
Sets the zlib level to be used for file compression when `compress=y`.
```
required: n
default: 6
allow: 0-9
override: backup, archive
example: compress-level=9
```
##### `compress-level-network` key
Sets the zlib level to be used for protocol compression when `compress=n` and the database is not on the same host as the backup. Protocol compression is used to reduce network traffic but can be disabled by setting `compress-level-network=0`. When `compress=y` the `compress-level-network` setting is ignored and `compress-level` is used instead so that the file is only compressed once. SSH compression is always disabled.
```
required: n
default: 3
allow: 0-9
override: backup, archive, restore
example: compress-level-network=1
```
#### `backup` section
The `backup` section defines settings related to backup.
##### `host` key
Sets the backup host when backup up remotely via SSH. Make sure that trusted SSH authentication is configured between the db host and the backup host.
When backing up to a locally mounted network filesystem this setting is not required.
```
required: n
example: host=backup.domain.com
```
##### `user` key
Sets user account on the backup host.
```
required: n
example: user=backrest
```
##### `path` key
Path where backups are stored on the local or remote host.
```
required: y
example: path=/var/lib/backrest
```
##### `compress` key
Enable gzip compression. Backup files are compatible with command-line gzip tools.
```
required: n
default: y
example: compress=n
```
##### `start-fast` key
Forces a checkpoint (by passing `true` to the `fast` parameter of `pg_start_backup()`) so the backup begins immediately.
```
required: n
default: n
example: start-fast=y
```
##### `hardlink` key
Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each backup is a full backup. Be care though, because modifying files that are hard-linked can affect all the backups in the set.
```
required: n
default: n
example: hardlink=y
```
##### `thread-max` key
Defines the number of threads to use for backup. Each thread will perform compression and transfer to make the backup run faster, but don't set `thread-max` so high that it impacts database performance.
```
required: n
default: 1
example: thread-max=4
```
##### `thread-timeout` key
Maximum amount of time (in seconds) that a backup thread should run. This limits the amount of time that a thread might be stuck due to unforeseen issues during the backup. Has no affect when `thread-max=1`.
```
required: n
example: thread-timeout=3600
```
##### `archive-required` key
Are archive logs required to to complete the backup? It's a good idea to leave this as the default unless you are using another method for archiving.
```
required: n
default: y
example: archive-required=n
```
#### `archive` section
The `archive` section defines parameters when doing async archiving. This means that the archive files will be stored locally, then a background process will pick them and move them to the backup.
##### `path` key
Path where archive logs are stored before being asynchronously transferred to the backup. Make sure this is not the same path as the backup is using if the backup is local.
```
required: n
example: path=/var/lib/backrest
```
##### `compress-async` key
When set archive logs are not compressed immediately, but are instead compressed when copied to the backup host. This means that more space will be used on local storage, but the initial archive process will complete more quickly allowing greater throughput from Postgres.
```
required: n
default: n
example: compress-async=y
```
##### `archive-max-mb` key
Limits the amount of archive log that will be written locally when `compress-async=y`. After the limit is reached, the following will happen:
- PgBackRest will notify Postgres that the archive was succesfully backed up, then DROP IT.
- An error will be logged to the console and also to the Postgres log.
- A stop file will be written in the lock directory and no more archive files will be backed up until it is removed.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
The purpose of this feature is to prevent the log volume from filling up at which point Postgres will stop completely. Better to lose the backup than have the database go down.
To start normal archiving again you'll need to remove the stop file which will be located at `${archive-path}/lock/${stanza}-archive.stop` where `${archive-path}` is the path set in the `archive` section, and `${stanza}` is the backup stanza.
```
required: n
example: archive-max-mb=1024
```
#### `retention` section
The `rentention` section defines how long backups will be retained. Expiration only occurs when the number of complete backups exceeds the allowed retention. In other words, if full-retention is set to 2, then there must be 3 complete backups before the oldest will be expired. Make sure you always have enough space for rentention + 1 backups.
##### `full-retention` key
Number of full backups to keep. When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When not defined then all full backups will be kept.
```
required: n
example: full-retention=2
```
##### `differential-retention` key
Number of differential backups to keep. When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.
```
required: n
example: differential-retention=3
```
##### `differential-retention` key
Number of differential backups to keep. When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.
```
required: n
example: differential-retention=3
```
##### `archive-retention-type` key
Type of backup to use for archive retention (full or differential). If set to full, then PgBackRest will keep archive logs for the number of full backups defined by `archive-retention`. If set to differential, then PgBackRest will keep archive logs for the number of differential backups defined by `archive-retention`.
If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs that are older than the oldest backup, but there may be reasons for doing so.
```
required: n
example: archive-retention-type=full
```
##### `archive-retention` key
Number of backups worth of archive log to keep. If not defined, then `full-retention` will be used when `archive-retention-type=full` and `differential-retention` will be used when `archive-retention-type=differential`.
```
required: n
example: archive-retention=2
```
#### `restore` section
[Not much to put here, but think of something]
#### `restore-option` section
Archive Recovery and Standby Server restore.conf options can be specified here. See http://www.postgresql.org/docs/X.X/static/recovery-config.html for details on restore.conf options (replace X.X with your database version).
Note: `restore_command` will automatically be generated unless overridden in this section. Be careful about specifying your own `restore_command` as PgBackRest is designed to handle this for you.
Target Recovery options are specified on the command-line since they end to change from restore to restore (or not be needed at all in the case of a standby server).
Since PgBackRest does not start PostgreSQL after writing the `recovery.conf` file, it is always possible to edit/check the file before manually restarting.
#### `stanza` section
A stanza defines a backup for a specific database. The stanza section must define the base database path and host/user if the database is remote. Also, any global configuration sections can be overridden to define stanza-specific settings.
##### `host` key
Define the database host. Used for backups where the database host is different from the backup host.
```
required: n
example: host=db.domain.com
```
##### `user` key
Defines user account on the db host when `stanza::host` is defined.
```
required: n
example: user=postgres
```
##### `path` key
Path to the db data directory (data_directory setting in postgresql.conf).
```
required: y
example: path=/data/db
```
## Release Notes
### v0.50: [under development]
* Added restore functionality.
- Added restore functionality.
* De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.
- De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.
* Added option (--no-start-stop) to allow backups when Postgres is shut down. If postmaster.pid is present then --force is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.
- Added option `--no-start-stop` to allow backups when Postgres is shut down. If `postmaster.pid` is present then `--force` is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.
* Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplied a number of things. Issue #28 has been created for checksum deltas.
- Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplied a number of things. Issue #28 has been created for checksum deltas.
* Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.
- Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.
* Removed dependency on Moose. It wasn't being used extensively and makes for longer startup times.
- Removed dependency on Moose. It wasn't being used extensively and makes for longer startup times.
* Checksum for backup.manifest to detect corrupted/modified manifest.
- Checksum for backup.manifest to detect corrupted/modified manifest.
* Link (called latest) always points to the last backup. This has been added for convenience and to make restore simpler.
- Link `latest` always points to the last backup. This has been added for convenience and to make restore simpler.
* More comprehensive backup unit tests.
- More comprehensive backup unit tests.
### v0.30: core restructuring and unit tests
### v0.30: Core Restructuring and Unit Tests
* Complete rewrite of BackRest::File module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.
- Complete rewrite of BackRest::File module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.
* Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.
- Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.
* Removed dependency on Storable and replaced with a custom ini file implementation.
- Removed dependency on Storable and replaced with a custom ini file implementation.
* Added much needed documentation (see INSTALL.md).
- Added much needed documentation
* Numerous other changes that can only be identified with a diff.
- Numerous other changes that can only be identified with a diff.
### v0.19: improved error reporting/handling
### v0.19: Improved Error Reporting/Handling
* Working on improving error handling in the file object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).
- Working on improving error handling in the file object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).
* Found and squashed a nasty bug where file_copy was defaulted to ignore errors. There was also an issue in file_exists that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.
- Found and squashed a nasty bug where `file_copy()` was defaulted to ignore errors. There was also an issue in file_exists that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.
### v0.18: return soft error from archive-get when file is missing
### v0.18: Return Soft Error When Archive Missing
* The archive-get function returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets Postgres know that that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream.
- The `archive-get` operation returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets Postgres know that that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream.
### v0.17: warn when archive directories cannot be deleted
### v0.17: Warn When Archive Directories Cannot Be Deleted
* If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).
- If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).
### v0.16: RequestTTY=yes for SSH sessions
### v0.16: RequestTTY=yes for SSH Sessions
* Added RequestTTY=yes to ssh sesssions. Hoping this will prevent random lockups.
- Added `RequestTTY=yes` to ssh sesssions. Hoping this will prevent random lockups.
### v0.15: added archive-get
### v0.15: RequestTTY=yes for SSH Sessions
* Added archive-get functionality to aid in restores.
- Added archive-get functionality to aid in restores.
* Added option to force a checkpoint when starting the backup (start_fast=y).
- Added option to force a checkpoint when starting the backup `start-fast=y`.
### v0.11: minor fixes
### v0.11: Minor Fixes
Tweaking a few settings after running backups for about a month.
- Removed `master_stderr_discard` option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code.
* Removed master_stderr_discard option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code.
- Changed lock file conflicts on backup and expire commands to ERROR. They were set to DEBUG due to a copy-and-paste from the archive locks.
* Changed lock file conflicts on backup and expire commands to ERROR. They were set to DEBUG due to a copy-and-paste from the archive locks.
### v0.10: Backup and Archiving are Functional
### v0.10: backup and archiving are functional
- No restore functionality, but the backup directories are consistent Postgres data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.
This version has been put into production at Resonate, so it does work, but there are a number of major caveats.
- Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.
* No restore functionality, but the backup directories are consistent Postgres data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.
- Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% threadsafe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.
* Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.
- Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.
* Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% threadsafe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.
- The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well. Would definitely like to save these as human-readable text.
* Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.
- Absolutely no documentation (outside the code). Well, excepting these release notes.
* The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well. Would definitely like to save these as human-readable text.
* Absolutely no documentation (outside the code). Well, excepting these release notes.
* Lots of other little things and not so little things. Much refactoring to follow.
## recognition
## Recognition
Primary recognition goes to Stephen Frost for all his valuable advice and criticism during the development of PgBackRest. It's a far better piece of software than it would have been without him.
Resonate (http://www.resonateinsights.com) also contributed to the development of PgBackRest and allowed me to install early (but well tested) versions as their primary Postgres backup solution.
Resonate (http://www.resonate.com/) also contributed to the development of PgBackRest and allowed me to install early (but well tested) versions as their primary Postgres backup solution.

216
README.old.md Normal file
View File

@ -0,0 +1,216 @@
# PgBackRest Installation
## sample ubuntu 12.04 install
1. Starting from a clean install, update the OS:
```
apt-get update
apt-get upgrade (reboot if required)
```
2. Install ssh, git and cpanminus
```
apt-get install ssh
apt-get install git
apt-get install cpanminus
```
3. Install Postgres (instructions from http://www.postgresql.org/download/linux/ubuntu/)
Create the file /etc/apt/sources.list.d/pgdg.list, and add a line for the repository:
```
deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main
```
Then run the following:
```
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
apt-get install postgresql-9.3
apt-get install postgresql-server-dev-9.3
```
4. Install required Perl modules:
```
cpanm JSON
cpanm Net::OpenSSH
cpanm DBI
cpanm DBD::Pg
cpanm IPC::System::Simple
cpanm Digest::SHA
cpanm Compress::ZLib
```
5. Install PgBackRest
Backrest can be installed by downloading the most recent release:
https://github.com/pgmasters/backrest/releases
6. To run unit tests:
* Create backrest_dev user
* Setup trusted ssh between test user account and backrest_dev
* Backrest user and test user must be in the same group
## operations
PgBackRest is intended to be run from a scheduler like cron as there is no built-in scheduler.
### general options
These options are either global or used by all operations.
#### stanza
Defines the stanza for the operation. A stanza is the configuration for a database that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one Postgres cluster and therefore one stanza, whereas backup servers will have a stanza for every database that needs to be backed up.
Examples of how to configure a stanza can be found in the `configuration examples` section.
#### config
By default PgBackRest expects the its configuration file to be located at `/etc/pg_backrest.conf`. Use this option to specify another location.
#### version
Returns the PgBackRest version.
#### help
Prints help with a summary of all options.
### backup
Perform a database backup.
#### options
##### type
The following backup types are supported:
- `full` - all database files will be copied and there will be no dependencies on previous backups.
- `incr` - incremental from the last successful backup.
- `diff` - like an incremental backup but always based on the last full backup.
##### no-start-stop
This option prevents PgBackRest from running `pg_start_backup()` and `pg_stop_backup()` on the database. In order for this to work Postgres should be shut down and PgBackRest will generate an error if it is not.
The purpose of this option is to allow cold backups. The `pg_xlog` directory is copied as-is and `backup::archive-required` is automatically set to `n` for the backup.
##### force
When used with `--no-start-stop` a backup will be run even if PgBackRest thinks that Postgres is running. **This option should be used with extreme care as it will likely result in a bad backup.**
There are some scenarios where a backup might still be desirable under these conditions. For example, if a server crashes and database volume can only be mounted read-only, it would be a good idea to take a backup even if `postmaster.pid` is present. In this case it would be better to revert to the prior backup and replay WAL, but possibly there is a very important transaction in a WAL log that did not get archived.
#### usage examples
```
/path/to/pg_backrest.pl --stanza=db --type=full backup
```
Run a `full` backup on the `db` stanza. `--type` can also be set to `incr` or `diff` for incremental or differential backups. However, if no `full` backup exists then a `full` backup will be forced even if `incr` or `diff` is requested.
### archive-push
```
/path/to/pg_backrest.pl --stanza=db archive-push %p
```
Accepts an archive file from Postgres and pushes it to the backup. `%p` is how Postgres specifies the location of the file to be archived. This command has no other purpose.
### archive-get
```
/path/to/pg_backrest.pl --stanza=db archive-get %f %p
```
Retrieves an archive log from the backup. This is used in `restore.conf` to restore a backup to that last archive log, do PITR, or as an alternative to streaming for keep a replica up to date. `%f` is how Postgres specifies the archive log it needs, and `%p` is the location where it should be copied.
### expire
PgBackRest does backup rotation, but it is not concerned with when the backups were created. So if two full backups are configured in rentention, PgBackRest will keep two full backup no matter whether they occur 2 hours apart or two weeks apart.
```
/path/to/pg_backrest.pl --stanza=db expire
```
Expire (rotate) any backups that exceed the defined retention. Expiration is run automatically after every successful backup, so there's no need to run this command on its own unless you have reduced rentention, usually to free up some space.
### restore
Restore a database from the PgBackRest repository.
#### restore options
##### set
The backup set to be restored. `latest` will restore the latest backup, otherwise provide the name of the backup to restore. For example: `20150131-153358F` or `20150131-153358F_20150131-153401I`.
##### delta
By default the database base and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
##### force
By itself this option forces the database base and tablespace paths to be completely overwritten. In combination with `--delta` a timestamp/size delta will be performed instead of using checksums.
#### recovery options
##### type
The following recovery types are supported:
- `default` - recover to the end of the archive stream.
- `name` - recover the restore point specified in `--target`.
- `xid` - recover to the transaction id specified in `--target`.
- `time` - recover to the time specified in `--target`.
- `preserve` - preserve the existing `recovery.conf` file.
- `none` - no recovery past database becoming consistent.
Note that the `none` option may produce duplicate WAL if the database is started with archive logging enabled. It is recommended that a new stanza be created for production databases restored in this way.
##### target
Defines the recovery target when `--type` is `name`, `xid`, or `time`. For example, `--type=time` and `--target=2015-01-30 14:15:11 EST`.
##### target-exclusive
Defines whether recovery to the target whould be exclusive (the default is inclusive) and is only valid when `--type` is `time` or `xid`. For example, using `--target-exclusive` would exclude the contents of transaction `1007` when `--type=xid` and `-target=1007`. See `recovery_target_inclusive` option in Postgres docs for more information.
##### target-resume
Specifies whether recovery should resume when the recovery target is reached. See `pause_at_recovery_target` in Postgres docs for more information.
##### target-timeline
Recovers along the specified timeline. See `recovery_target_timeline` in Postgres docs for more information.
#### usage examples
```
/path/to/pg_backrest.pl --stanza=db --set=latest --type=name --target=release
```
Restores the latest database backup and then recovers to the `release` restore point.
**MORE TO BE ADDED HERE**
PITR should start after the stop time in the .backup file.
[reference this when writing about tablespace remapping]
http://www.databasesoup.com/2013/11/moving-tablespaces.html
## structure
PgBackRest stores files in a way that is easy for users to work with directly. Each backup directory has one file and two subdirectories:
* `backup.manifest` file
Stores information about all the directories, links, and files in the backup. The file is plaintext and should be very clear, but documentation of the format is planned in a future release.
* `base` directory
Contains the Postgres data directory as defined by the data_directory setting in `postgresql.conf`.
* `tablespace` directory
If tablespaces are present in the database, contains each tablespace in a separate subdirectory. Tablespace names are used for the subdirectories unless --no-start-stop is specified in which case oids will be used instead. The links in `base/pg_tblspc` are rewritten to the tablespace directory in either case.

72
doc/doc.dtd Normal file
View File

@ -0,0 +1,72 @@
<!ELEMENT doc (intro, install, config, release, recognition)>
<!ATTLIST doc title CDATA #REQUIRED>
<!ATTLIST doc subtitle CDATA #REQUIRED>
<!ELEMENT intro (text)>
<!ELEMENT install (text, install-system-list)>
<!ATTLIST install title CDATA #REQUIRED>
<!ELEMENT install-system-list (text?, install-system+)>
<!ELEMENT install-system (text)>
<!ATTLIST install-system title CDATA #REQUIRED>
<!ELEMENT config (text, config-example-list, config-section-list)>
<!ATTLIST config title CDATA #REQUIRED>
<!ELEMENT config-example-list (text?, config-example+)>
<!ATTLIST config-example-list title CDATA #REQUIRED>
<!ELEMENT config-example (text)>
<!ATTLIST config-example title CDATA #REQUIRED>
<!ELEMENT config-section-list (text?, config-section+)>
<!ATTLIST config-section-list title CDATA #REQUIRED>
<!ELEMENT config-section (text, config-key-list?)>
<!ATTLIST config-section id CDATA #REQUIRED>
<!ELEMENT config-key-list (config-key+)>
<!ELEMENT config-key (text, (required|default?), allow?, override?, example)>
<!ATTLIST config-key id CDATA #REQUIRED>
<!ELEMENT required EMPTY>
<!ELEMENT default (#PCDATA)>
<!ELEMENT allow (#PCDATA)>
<!ELEMENT override (#PCDATA)>
<!ELEMENT example (#PCDATA)>
<!ELEMENT release (text?, release-version-list)>
<!ATTLIST release title CDATA #REQUIRED>
<!ELEMENT release-version-list (release-version+)>
<!ELEMENT release-version (text?, release-feature-bullet-list)>
<!ATTLIST release-version version CDATA #REQUIRED>
<!ATTLIST release-version title CDATA #REQUIRED>
<!ELEMENT release-feature-bullet-list (release-feature+)>
<!ELEMENT release-feature (text)>
<!ELEMENT recognition (text)>
<!ATTLIST recognition title CDATA #REQUIRED>
<!ELEMENT text (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|backrest)*>
<!ELEMENT i (#PCDATA)>
<!ELEMENT b (#PCDATA)>
<!ELEMENT bi (#PCDATA)>
<!ELEMENT ul (li+)>
<!ELEMENT ol (li+)>
<!ELEMENT li (#PCDATA|id)*>
<!ELEMENT id (#PCDATA)>
<!ELEMENT code (#PCDATA)>
<!ELEMENT code-block (#PCDATA)>
<!ELEMENT file (#PCDATA)>
<!ELEMENT path (#PCDATA)>
<!ELEMENT cmd (#PCDATA)>
<!ELEMENT param (#PCDATA)>
<!ELEMENT setting (#PCDATA)>
<!ELEMENT backrest EMPTY>

734
doc/doc.pl Executable file
View File

@ -0,0 +1,734 @@
#!/usr/bin/perl
####################################################################################################################################
# pg_backrest.pl - Simple Postgres Backup and Restore
####################################################################################################################################
####################################################################################################################################
# Perl includes
####################################################################################################################################
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
use File::Basename qw(dirname);
use Pod::Usage qw(pod2usage);
use Getopt::Long qw(GetOptions);
use XML::Checker::Parser;
use lib dirname($0) . '/../lib';
use BackRest::Utility;
####################################################################################################################################
# Usage
####################################################################################################################################
=head1 NAME
doc.pl - Generate PgBackRest documentation
=head1 SYNOPSIS
doc.pl [options] [operation]
General Options:
--help display usage and exit
=cut
####################################################################################################################################
# DOC_RENDER_TAG - render a tag to another markup language
####################################################################################################################################
my $oRenderTag =
{
'markdown' =>
{
'b' => ['**', '**'],
'i' => ['_', '_'],
'bi' => ['_**', '**_'],
'ul' => ["\n", ''],
'ol' => ["\n", ''],
'li' => ['- ', "\n"],
'id' => ['`', '`'],
'file' => ['`', '`'],
'path' => ['`', '`'],
'cmd' => ['`', '`'],
'param' => ['`', '`'],
'setting' => ['`', '`'],
'code' => ['`', '`'],
'code-block' => ['```', '```'],
'backrest' => ['PgBackRest', ''],
},
'html' =>
{
'b' => ['<b>', '</b>']
}
};
sub doc_render_tag
{
my $oTag = shift;
my $strType = shift;
my $strBuffer = "";
my $strTag = $$oTag{name};
my $strStart = $$oRenderTag{$strType}{$strTag}[0];
my $strStop = $$oRenderTag{$strType}{$strTag}[1];
if (!defined($strStart) || !defined($strStop))
{
confess "invalid type ${strType} or tag ${strTag}";
}
$strBuffer .= $strStart;
if ($strTag eq 'li')
{
$strBuffer .= doc_render_text($oTag, $strType);
}
elsif (defined($$oTag{value}))
{
$strBuffer .= $$oTag{value};
}
elsif (defined($$oTag{children}[0]))
{
foreach my $oSubTag (@{doc_list($oTag)})
{
$strBuffer .= doc_render_tag($oSubTag, $strType);
}
}
$strBuffer .= $strStop;
}
####################################################################################################################################
# DOC_RENDER_TEXT - Render a text node
####################################################################################################################################
sub doc_render_text
{
my $oText = shift;
my $strType = shift;
my $strBuffer = "";
if (defined($$oText{children}))
{
for (my $iIndex = 0; $iIndex < @{$$oText{children}}; $iIndex++)
{
if (ref(\$$oText{children}[$iIndex]) eq "SCALAR")
{
$strBuffer .= $$oText{children}[$iIndex];
}
else
{
$strBuffer .= doc_render_tag($$oText{children}[$iIndex], $strType);
}
}
}
return $strBuffer;
}
####################################################################################################################################
# DOC_GET - Get a node
####################################################################################################################################
sub doc_get
{
my $oDoc = shift;
my $strName = shift;
my $bRequired = shift;
my $oNode;
for (my $iIndex = 0; $iIndex < @{$$oDoc{children}}; $iIndex++)
{
if ($$oDoc{children}[$iIndex]{name} eq $strName)
{
if (!defined($oNode))
{
$oNode = $$oDoc{children}[$iIndex];
}
else
{
confess "found more than one child ${strName} in node $$oDoc{name}";
}
}
}
if (!defined($oNode) && (!defined($bRequired) || $bRequired))
{
confess "unable to find child ${strName} in node $$oDoc{name}";
}
return $oNode;
}
####################################################################################################################################
# DOC_GET - Test if a node exists
####################################################################################################################################
sub doc_exists
{
my $oDoc = shift;
my $strName = shift;
my $bExists = false;
for (my $iIndex = 0; $iIndex < @{$$oDoc{children}}; $iIndex++)
{
if ($$oDoc{children}[$iIndex]{name} eq $strName)
{
return true;
}
}
return false;
}
####################################################################################################################################
# DOC_LIST - Get a list of nodes
####################################################################################################################################
sub doc_list
{
my $oDoc = shift;
my $strName = shift;
my $bRequired = shift;
my @oyNode;
for (my $iIndex = 0; $iIndex < @{$$oDoc{children}}; $iIndex++)
{
if (!defined($strName) || $$oDoc{children}[$iIndex]{name} eq $strName)
{
push(@oyNode, $$oDoc{children}[$iIndex]);
}
}
if (@oyNode == 0 && (!defined($bRequired) || $bRequired))
{
confess "unable to find child ${strName} in node $$oDoc{name}";
}
return \@oyNode;
}
####################################################################################################################################
# DOC_VALUE - Get value from a node
####################################################################################################################################
sub doc_value
{
my $oNode = shift;
my $strDefault = shift;
if (defined($oNode) && defined($$oNode{value}))
{
return $$oNode{value};
}
return $strDefault;
}
####################################################################################################################################
# DOC_PARSE - Parse the XML tree into something more usable
####################################################################################################################################
sub doc_parse
{
my $strName = shift;
my $oyNode = shift;
my %oOut;
my $iIndex = 0;
my $bText = $strName eq 'text' || $strName eq 'li';
# Store the node name
$oOut{name} = $strName;
if (keys($$oyNode[$iIndex]))
{
$oOut{param} = $$oyNode[$iIndex];
}
$iIndex++;
# Look for strings and children
while (defined($$oyNode[$iIndex]))
{
# Process string data
if (ref(\$$oyNode[$iIndex]) eq 'SCALAR' && $$oyNode[$iIndex] eq '0')
{
$iIndex++;
my $strBuffer = $$oyNode[$iIndex++];
# Strip tabs, CRs, and LFs
$strBuffer =~ s/\t|\r//g;
# If anything is left
if (length($strBuffer) > 0)
{
# If text node then create array entries for strings
if ($bText)
{
if (!defined($oOut{children}))
{
$oOut{children} = [];
}
push($oOut{children}, $strBuffer);
}
# Don't allow strings mixed with children
elsif (length(trim($strBuffer)) > 0)
{
if (defined($oOut{children}))
{
confess "text mixed with children in node ${strName} (spaces count)";
}
if (defined($oOut{value}))
{
confess "value is already defined in node ${strName} - this shouldn't happen";
}
# Don't allow text mixed with
$oOut{value} = $strBuffer;
}
}
}
# Process a child
else
{
if (defined($oOut{value}) && $bText)
{
confess "text mixed with children in node ${strName} before child " . $$oyNode[$iIndex++] . " (spaces count)";
}
if (!defined($oOut{children}))
{
$oOut{children} = [];
}
push($oOut{children}, doc_parse($$oyNode[$iIndex++], $$oyNode[$iIndex++]));
}
}
return \%oOut;
}
####################################################################################################################################
# DOC_SAVE - save a doc
####################################################################################################################################
sub doc_write
{
my $strFileName = shift;
my $strBuffer = shift;
# Open the file
my $hFile;
open($hFile, '>', $strFileName)
or confess &log(ERROR, "unable to open ${strFileName}");
# Write the buffer
my $iBufferOut = syswrite($hFile, $strBuffer);
# Report any errors
if (!defined($iBufferOut) || $iBufferOut != length($strBuffer))
{
confess "unable to write '${strBuffer}'" . (defined($!) ? ': ' . $! : '');
}
# Close the file
close($hFile);
}
####################################################################################################################################
# Load command line parameters and config
####################################################################################################################################
my $bHelp = false; # Display usage
my $bVersion = false; # Display version
my $bQuiet = false; # Sets log level to ERROR
my $strLogLevel = 'info'; # Log level for tests
GetOptions ('help' => \$bHelp,
'version' => \$bVersion,
'quiet' => \$bQuiet,
'log-level=s' => \$strLogLevel)
or pod2usage(2);
# Display version and exit if requested
if ($bHelp || $bVersion)
{
print 'pg_backrest ' . version_get() . " documentation\n";
if ($bHelp)
{
print "\n";
pod2usage();
}
exit 0;
}
# Set console log level
if ($bQuiet)
{
$strLogLevel = 'off';
}
log_level_set(undef, uc($strLogLevel));
####################################################################################################################################
# Load the doc file
####################################################################################################################################
# Initialize parser object and parse the file
my $oParser = XML::Checker::Parser->new(ErrorContext => 2, Style => 'Tree');
my $strFile = dirname($0) . '/doc.xml';
my $oTree;
eval
{
local $XML::Checker::FAIL = sub
{
my $iCode = shift;
die XML::Checker::error_string($iCode, @_);
};
$oTree = $oParser->parsefile(dirname($0) . '/doc.xml');
};
# Report any error that stopped parsing
if ($@)
{
$@ =~ s/at \/.*?$//s; # remove module line number
die "malformed xml in '$strFile}':\n" . trim($@);
}
####################################################################################################################################
# Parse the doc
####################################################################################################################################
# my %oDocOut;
# Doc Build
#-----------------------------------------------------------------------------------------------------------------------------------
# $oDocOut{title} = $$oDocIn{param}{title};
# $oDocOut{subtitle} = $$oDocIn{param}{subtitle};
# $oDocOut{children} = [];
# my $strReadMe = "# ${strTitle} - ${strSubTitle}";
# Intro Build
#-----------------------------------------------------------------------------------------------------------------------------------
# my $oIntroOut = {name => 'intro'};
#
# $$oIntroOut{text} = doc_get(doc_get($oDocIn, $$oIntroOut{name}), 'text');
# push($oDocOut{children}, $oIntroOut);
# Config Build
#-----------------------------------------------------------------------------------------------------------------------------------
# my $oConfigOut = {name => 'config', children => []};
# my $oConfig = doc_get($oDocIn, $$oConfigOut{name});
#
# $$oConfigOut{title} = $$oConfig{param}{title};
# $$oConfigOut{text} = doc_get($oConfig, 'text');
# push($oDocOut{children}, $oConfigOut);
#
# # Config Example List
# my $oConfigExampleOut = {name => 'config-example-list', children => []};
# my $oConfigExampleList = doc_get($oConfig, $$oConfigExampleOut{name});
#
# $$oConfigExampleOut{title} = $$oConfigExampleList{param}{title};
# $$oConfigExampleOut{text} = doc_get($oConfigExampleList, 'text', false);
# push($$oConfigExampleOut{children}, $oConfigExampleOut);
#
# #$strReadMe .= "\n\n## " . $$oConfigExampleList{param}{title};
#
# $oDocOut{config}{exampleListText} = doc_get($oConfigExampleList, 'text', false);
# my @oExampleList;
#
# # if (doc_exists($oConfigExampleList, 'text', false))
# # {
# # $strReadMe .= "\n\n" . doc_render_text(doc_get($oConfigExampleList, 'text'), 'markdown');
# # }
#
# foreach my $oConfigExample (@{doc_list($oConfigExampleList, 'config-example')})
# {
# my %oExampleOut;
#
# $oExampleOut{title} = $$oConfigExample{param}{title};
# $oExampleOut{text} = doc_get($oConfigExample, 'text', false);
# # $strReadMe .= "\n\n### " . $$oConfigExample{param}{title} .
# # "\n\n" . doc_render_text(doc_get($oConfigExample, 'text'), 'markdown');
#
# push(@oExampleList, \%oExampleOut);
# }
#
# $oDocOut{config}{exampleList} = \@oExampleList;
# # Config Section List
# my $oConfigSectionList = doc_get($oConfig, 'config-section-list');
#
# $strReadMe .= "\n\n## " . $$oConfigSectionList{param}{title};
#
# if (doc_exists($oConfigSectionList, 'text', false))
# {
# $strReadMe .= "\n\n" . doc_render_text(doc_get($oConfigSectionList, 'text'), 'markdown');
# }
#
# foreach my $oConfigSection (@{doc_list($oConfigSectionList, 'config-section')})
# {
# my $strConfigSectionId = $$oConfigSection{param}{id};
#
# $strReadMe .= "\n\n#### `" . $strConfigSectionId . "` section" .
# "\n\n" . doc_render_text(doc_get($oConfigSection, 'text'), 'markdown');
#
# foreach my $oConfigKey (@{doc_list($oConfigSection, 'config-key', false)})
# {
# my $strConfigKeyId = $$oConfigKey{param}{id};
# my $strError = "config section ${strConfigSectionId}, key ${strConfigKeyId} requires";
#
# my $bRequired = doc_exists($oConfigKey, 'required');
# my $strDefault = !$bRequired ? doc_value(doc_get($oConfigKey, 'default', false)) : undef;
# my $strAllow = doc_value(doc_get($oConfigKey, 'allow', false));
# my $strOverride = doc_value(doc_get($oConfigKey, 'override', false));
# my $strExample = doc_value(doc_get($oConfigKey, 'example', false));
#
# defined($strExample) or die "${strError} example";
#
# $strReadMe .= "\n\n##### `" . $strConfigKeyId . "` key" .
# "\n\n" . doc_render_text(doc_get($oConfigKey, 'text'), 'markdown') .
# "\n```\n" .
# "required: " . ($bRequired ? 'y' : 'n') . "\n" .
# (defined($strDefault) ? "default: ${strDefault}\n" : '') .
# (defined($strAllow) ? "allow: ${strAllow}\n" : '') .
# (defined($strOverride) ? "override: ${strOverride}\n" : '') .
# "example: ${strConfigKeyId}=${strExample}\n" .
# "```";
# }
# }
# Release Build
#-----------------------------------------------------------------------------------------------------------------------------------
# my $oReleaseOut = {name => 'release'};
# my $oRelease = doc_get($oDocIn, 'release');
#
# $$oReleaseOut{title} = $$oRelease{param}{title};
# $$oReleaseOut{text} = doc_get($oRelease, 'text', false);
# $$oReleaseOut{children} = [];
#
# my $oReleaseList = doc_get($oRelease, 'release-list');
#
# foreach my $oReleaseVersion (@{doc_list($oReleaseList, 'release-version')})
# {
# my %oVersionOut;
# my @oFeatureList;
#
# $oVersionOut{version} = $$oReleaseVersion{param}{version};
# $oVersionOut{title} = $$oReleaseVersion{param}{title};
# $oVersionOut{text} = doc_get($oReleaseVersion, 'text', false);
# $oVersionOut{list} = true;
# $oVersionOut{children} = [];
#
# foreach my $oReleaseFeature (@{doc_list($oReleaseVersion, 'release-feature')})
# {
# my %oFeatureOut;
# $oFeatureOut{text} = doc_get($oReleaseFeature, 'text');
# push ($oVersionOut{children}, \%oFeatureOut);
# }
#
# push($$oReleaseOut{children}, \%oVersionOut);
# }
#
# push($oDocOut{children}, $oReleaseOut);
# Recognition Build
#-----------------------------------------------------------------------------------------------------------------------------------
# my $oRecognitionOut = {name => 'recognition'};
# my $oRecognition = doc_get($oDocIn, $$oRecognitionOut{name});
#
# $$oRecognitionOut{title} = $$oRecognition{param}{title};
# $$oRecognitionOut{text} = doc_get($oRecognition, 'text');
# push($oDocOut{children}, $oRecognitionOut);
# Build
#-----------------------------------------------------------------------------------------------------------------------------------
my $oDocIn = doc_parse(${$oTree}[0], ${$oTree}[1]);
sub doc_build
{
my $oDoc = shift;
# Initialize the node object
my $oOut = {name => $$oDoc{name}, children => []};
my $strError = "in node $$oDoc{name}";
# Get all params
if (defined($$oDoc{param}))
{
for my $strParam (keys $$oDoc{param})
{
$$oOut{param}{$strParam} = $$oDoc{param}{$strParam};
}
}
if (defined($$oDoc{children}))
{
for (my $iIndex = 0; $iIndex < @{$$oDoc{children}}; $iIndex++)
{
my $oSub = $$oDoc{children}[$iIndex];
my $strName = $$oSub{name};
if ($strName eq 'text')
{
$$oOut{field}{text} = $oSub;
}
elsif (defined($$oSub{value}))
{
$$oOut{field}{$strName} = $$oSub{value};
}
elsif (!defined($$oSub{children}))
{
$$oOut{field}{$strName} = true;
}
else
{
push($$oOut{children}, doc_build($oSub));
}
}
}
return $oOut;
}
my $oDocOut = doc_build($oDocIn);
# Render
#-----------------------------------------------------------------------------------------------------------------------------------
sub doc_render
{
my $oDoc = shift;
my $strType = shift;
my $iDepth = shift;
my $bChildList = shift;
my $strBuffer = "";
my $bList = $$oDoc{name} =~ /.*-bullet-list$/;
$bChildList = defined($bChildList) ? $bChildList : false;
my $iChildDepth = $iDepth;
if ($strType eq 'markdown')
{
if (defined($$oDoc{param}{id}))
{
my @stryToken = split('-', $$oDoc{name});
my $strTitle = @stryToken == 0 ? '[unknown]' : $stryToken[@stryToken - 1];
$strBuffer = ('#' x $iDepth) . " `$$oDoc{param}{id}` " . $strTitle;
}
if (defined($$oDoc{param}{title}))
{
$strBuffer = ('#' x $iDepth) . ' ';
if (defined($$oDoc{param}{version}))
{
$strBuffer .= "v$$oDoc{param}{version}: ";
}
$strBuffer .= $$oDoc{param}{title};
}
if (defined($$oDoc{param}{subtitle}))
{
if (!defined($$oDoc{param}{subtitle}))
{
confess "subtitle not valid without title";
}
$strBuffer .= " - " . $$oDoc{param}{subtitle};
}
if ($strBuffer ne "")
{
$iChildDepth++;
}
if (defined($$oDoc{field}{text}))
{
if ($strBuffer ne "")
{
$strBuffer .= "\n\n";
}
if ($bChildList)
{
$strBuffer .= '- ';
}
$strBuffer .= doc_render_text($$oDoc{field}{text}, $strType);
}
if ($$oDoc{name} eq 'config-key')
{
my $strError = "config section ?, key $$oDoc{param}{id} requires";
my $bRequired = defined($$oDoc{field}{required});
my $strDefault = !$bRequired ? $$oDoc{field}{default} : undef;
my $strAllow = $$oDoc{field}{allow};
my $strOverride = $$oDoc{field}{override};
my $strExample = $$oDoc{field}{example};
defined($strExample) or die "${strError} example";
$strBuffer .= "\n```\n" .
"required: " . ($bRequired ? 'y' : 'n') . "\n" .
(defined($strDefault) ? "default: ${strDefault}\n" : '') .
(defined($strAllow) ? "allow: ${strAllow}\n" : '') .
(defined($strOverride) ? "override: ${strOverride}\n" : '') .
"example: $$oDoc{param}{id}=${strExample}\n" .
"```";
}
if ($strBuffer ne "" && $iDepth != 1 && !$bList)
{
$strBuffer = "\n\n" . $strBuffer;
}
}
else
{
confess "unknown type ${strType}";
}
my $bFirst = true;
foreach my $oChild (@{$$oDoc{children}})
{
if ($strType eq 'markdown')
{
}
else
{
confess "unknown type ${strType}";
}
$strBuffer .= doc_render($oChild, $strType, $iChildDepth, $bList);
}
if ($iDepth == 1)
{
if ($strType eq 'markdown')
{
$strBuffer .= "\n";
}
else
{
confess "unknown type ${strType}";
}
}
return $strBuffer;
}
# Write markdown
doc_write(dirname($0) . '/../README.md', doc_render($oDocOut, 'markdown', 1));

613
doc/doc.xml Normal file
View File

@ -0,0 +1,613 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE doc SYSTEM "doc.dtd">
<doc title="PgBackRest" subtitle="Simple Postgres Backup &amp; Restore">
<intro>
<text><backrest/> aims to be a simple backup and restore system that can seamlessly scale up to the largest databases and workloads.
Primary <backrest/> features:
<ul>
<li>Local or remote backup</li>
<li>Multi-threaded backup/restore for performance</li>
<li>Checksums</li>
<li>Safe backups (checks that logs required for consistency are present before backup completes)</li>
<li>Full, differential, and incremental backups</li>
<li>Backup rotation (and minimum retention rules with optional separate retention for archive)</li>
<li>In-stream compression/decompression</li>
<li>Archiving and retrieval of logs for replicas/restores built in</li>
<li>Async archiving for very busy systems (including space limits)</li>
<li>Backup directories are consistent Postgres clusters (when hardlinks are on and compression is off)</li>
<li>Tablespace support</li>
<li>Restore delta option</li>
<li>Restore using timestamp/size or checksum</li>
<li>Restore remapping base/tablespaces</li>
</ul>
Instead of relying on traditional backup tools like tar and rsync, <backrest/> implements all backup features internally and features a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security. Each thread requires only one SSH connection for remote backups.</text>
</intro>
<install title="Install">
<text>!!! Perl-based blah, blah, blah</text>
<install-system-list>
<install-system title="Ubuntu 12.04">
<text>* Starting from a clean install, update the OS:
<code-block>
apt-get update
apt-get upgrade (reboot if required)
</code-block>
* Install ssh, git and cpanminus:
<code-block>
apt-get install ssh
apt-get install git
apt-get install cpanminus
</code-block>
* Install Postgres (instructions from http://www.postgresql.org/download/linux/ubuntu/)
Create the file /etc/apt/sources.list.d/pgdg.list, and add a line for the repository:
<code-block>
deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main
</code-block>
* Then run the following:
<code-block>
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
apt-get install postgresql-9.3
apt-get install postgresql-server-dev-9.3
</code-block>
* Install required Perl modules:
<code-block>
cpanm JSON
cpanm Net::OpenSSH
cpanm DBI
cpanm DBD::Pg
cpanm IPC::System::Simple
cpanm Digest::SHA
cpanm Compress::ZLib
</code-block>
* Install PgBackRest
Backrest can be installed by downloading the most recent release:
https://github.com/pgmasters/backrest/releases
* To run unit tests:
<code-block>
Create backrest_dev user
Setup trusted ssh between test user account and backrest_dev
Backrest user and test user must be in the same group
</code-block></text>
</install-system>
</install-system-list>
</install>
<config title="Configuration">
<text><backrest/> takes some command-line parameters, but depends on a configuration file for most of the settings. The default location for the configuration file is <file>/etc/pg_backrest.conf</file>.</text>
<config-example-list title="Examples">
<config-example title="Confguring Postgres for Archiving">
<text>Modify the following settings in <file>postgresql.conf</file>:
<code-block>
wal_level = archive
archive_mode = on
archive_command = '/path/to/backrest/bin/pg_backrest.pl --stanza=db archive-push %p'
</code-block>
Replace the path with the actual location where <backrest/> was installed. The stanza parameter should be changed to the actual stanza name for your database in <file>pg_backrest.conf</file>.
</text>
</config-example>
<config-example title="Simple Single Host Install">
<text>This configuration is appropriate for a small installation where backups are being made locally or to a remote file system that is mounted locally.
<file>/etc/pg_backrest.conf</file>:
<code-block>
[global:command]
psql=/usr/bin/psql
[global:backup]
path=/var/lib/postgresql/backup
[global:retention]
full-retention=2
differential-retention=2
archive-retention-type=diff
archive-retention=2
[db]
path=/var/lib/postgresql/9.3/main
</code-block>
</text>
</config-example>
<config-example title="Simple Multiple Host Install">
<text>This configuration is appropriate for a small installation where backups are being made remotely. Make sure that postgres@db-host has trusted ssh to backrest@backup-host and vice versa. This configuration assumes that you have pg_backrest_remote.pl and pg_backrest.pl in the same path on both servers.
<file>/etc/pg_backrest.conf</file> on the db host:
<code-block>
[global:command]
psql=/usr/bin/psql
[global:backup]
host=backup-host@mydomain.com
user=postgres
path=/var/lib/postgresql/backup
[db]
path=/var/lib/postgresql/9.3/main
</code-block>
<file>/etc/pg_backrest.conf</file> on the backup host:
<code-block>
[global:command]
psql=/usr/bin/psql
[global:backup]
path=/var/lib/postgresql/backup
[global:retention]
full-retention=2
archive-retention-type=full
[db]
host=db-host@mydomain.com
user=postgres
path=/var/lib/postgresql/9.3/main
</code-block>
</text>
</config-example>
</config-example-list>
<config-section-list title="Options">
<config-section id="command">
<text>The <setting>command</setting> section defines the location of external commands that are used by <backrest/>.</text>
<config-key-list>
<config-key id="psql">
<text>Defines the full path to <cmd>psql</cmd>. <cmd>psql</cmd> is used to call <code>pg_start_backup()</code> and <code>pg_stop_backup()</code>.
If addtional parameters need to be passed to <cmd>psql</cmd> (such as <param>--port</param> or <param>--cluster</param>) then add <param>%option%</param> to the command line and use <setting>command-option::psql</setting> to set options.</text>
<required/>
<example>/usr/bin/psql -X %option%</example>
</config-key>
<config-key id="remote">
<text>Defines the location of <cmd>pg_backrest_remote.pl</cmd>.
Required only if the path to <cmd>pg_backrest_remote.pl</cmd> is different on the local and remote systems. If not defined, the remote path will be assumed to be the same as the local path.</text>
<default>same as local</default>
<example>/usr/lib/backrest/bin/pg_backrest_remote.pl</example>
</config-key>
</config-key-list>
</config-section>
<!-- CONFIG - COMMAND-OPTION SECTION -->
<config-section id="command-option">
<text>The <setting>command-option</setting> section allows abitrary options to be passed to any command in the <setting>command</setting> section.</text>
<!-- CONFIG - COMMAND-OPTION SECTION - PSQL KEY -->
<config-key-list>
<config-key id="psql">
<text>Allows command line parameters to be passed to <cmd>psql</cmd>.</text>
<example>--port=5433</example>
</config-key>
</config-key-list>
</config-section>
<!-- CONFIG - LOG -->
<config-section id="log">
<text>The <setting>log</setting> section defines logging-related settings. The following log levels are supported:
<ul>
<li><id>off</id> - No logging at all (not recommended)</li>
<li><id>error</id> - Log only errors</li>
<li><id>warn</id> - Log warnings and errors</li>
<li><id>info</id> - Log info, warnings, and errors</li>
<li><id>debug</id> - Log debug, info, warnings, and errors</li>
<li><id>trace</id> - Log trace (very verbose debugging), debug, info, warnings, and errors</li>
</ul></text>
<!-- CONFIG - LOG SECTION - LEVEL-FILE KEY -->
<config-key-list>
<config-key id="level-file">
<text>Sets file log level.</text>
<default>info</default>
<example>debug</example>
</config-key>
<!-- CONFIG - LOG SECTION - LEVEL-CONSOLE KEY -->
<config-key id="level-console">
<text>Sets console log level.</text>
<default>warning</default>
<example>error</example>
</config-key>
</config-key-list>
</config-section>
<!-- CONFIG - GENERAL -->
<config-section id="general">
<text>The <setting>general</setting> section defines settings that are shared between multiple operations.</text>
<!-- CONFIG - GENERAL SECTION - BUFFER-SIZE KEY -->
<config-key-list>
<config-key id="buffer-size">
<text>Set the buffer size used for copy, compress, and uncompress functions. A maximum of 3 buffers will be in use at a time per thread. An additional maximum of 256K per thread may be used for zlib buffers.</text>
<default>1048576</default>
<allow>4096 - 8388608</allow>
<example>16384</example>
</config-key>
<!-- CONFIG - GENERAL SECTION - COMPRESS-LEVEL KEY -->
<config-key id="compress-level">
<text>Sets the zlib level to be used for file compression when <setting>compress=y</setting>.</text>
<default>6</default>
<allow>0-9</allow>
<override>backup, archive</override>
<example>9</example>
</config-key>
<!-- CONFIG - GENERAL SECTION - COMPRESS-LEVEL-NETWORK KEY -->
<config-key id="compress-level-network">
<text>Sets the zlib level to be used for protocol compression when <setting>compress=n</setting> and the database is not on the same host as the backup. Protocol compression is used to reduce network traffic but can be disabled by setting <setting>compress-level-network=0</setting>. When <setting>compress=y</setting> the <setting>compress-level-network</setting> setting is ignored and <setting>compress-level</setting> is used instead so that the file is only compressed once. SSH compression is always disabled.</text>
<default>3</default>
<allow>0-9</allow>
<override>backup, archive, restore</override>
<example>1</example>
</config-key>
</config-key-list>
</config-section>
<!-- CONFIG - BACKUP -->
<config-section id="backup">
<text>The <setting>backup</setting> section defines settings related to backup.</text>
<!-- CONFIG - BACKUP SECTION - HOST KEY -->
<config-key-list>
<config-key id="host">
<text>Sets the backup host when backup up remotely via SSH. Make sure that trusted SSH authentication is configured between the db host and the backup host.
When backing up to a locally mounted network filesystem this setting is not required.</text>
<example>backup.domain.com</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - USER KEY -->
<config-key id="user">
<text>Sets user account on the backup host.</text>
<example>backrest</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - PATH KEY -->
<config-key id="path">
<text>Path where backups are stored on the local or remote host.</text>
<required/>
<example>/var/lib/backrest</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - COMPRESS -->
<config-key id="compress">
<text>Enable gzip compression. Backup files are compatible with command-line gzip tools.</text>
<default>y</default>
<example>n</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - START-FAST -->
<config-key id="start-fast">
<text>Forces a checkpoint (by passing <id>true</id> to the <id>fast</id> parameter of <code>pg_start_backup()</code>) so the backup begins immediately.</text>
<default>n</default>
<example>y</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - HARDLINK -->
<config-key id="hardlink">
<text>Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each backup is a full backup. Be care though, because modifying files that are hard-linked can affect all the backups in the set.</text>
<default>n</default>
<example>y</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - THEAD-MAX -->
<config-key id="thread-max">
<text>Defines the number of threads to use for backup. Each thread will perform compression and transfer to make the backup run faster, but don't set <setting>thread-max</setting> so high that it impacts database performance.</text>
<default>1</default>
<example>4</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - THEAD-TIMEOUT -->
<config-key id="thread-timeout">
<text>Maximum amount of time (in seconds) that a backup thread should run. This limits the amount of time that a thread might be stuck due to unforeseen issues during the backup. Has no affect when <setting>thread-max=1</setting>.</text>
<example>3600</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - ARCHIVE-REQUIRED -->
<config-key id="archive-required">
<text>Are archive logs required to to complete the backup? It's a good idea to leave this as the default unless you are using another method for archiving.</text>
<default>y</default>
<example>n</example>
</config-key>
</config-key-list>
</config-section>
<!-- CONFIG - ARCHIVE -->
<config-section id="archive">
<text>The <setting>archive</setting> section defines parameters when doing async archiving. This means that the archive files will be stored locally, then a background process will pick them and move them to the backup.</text>
<!-- CONFIG - ARCHIVE SECTION - PATH KEY -->
<config-key-list>
<config-key id="path">
<text>Path where archive logs are stored before being asynchronously transferred to the backup. Make sure this is not the same path as the backup is using if the backup is local.</text>
<example>/var/lib/backrest</example>
</config-key>
<!-- CONFIG - ARCHIVE SECTION - COMPRESS-ASYNC KEY -->
<config-key id="compress-async">
<text>When set archive logs are not compressed immediately, but are instead compressed when copied to the backup host. This means that more space will be used on local storage, but the initial archive process will complete more quickly allowing greater throughput from Postgres.</text>
<default>n</default>
<example>y</example>
</config-key>
<!-- CONFIG - ARCHIVE SECTION - ARCHIVE-MAX-MB KEY -->
<config-key id="archive-max-mb">
<text>Limits the amount of archive log that will be written locally when <setting>compress-async=y</setting>. After the limit is reached, the following will happen:
<ol>
<li>PgBackRest will notify Postgres that the archive was succesfully backed up, then DROP IT.</li>
<li>An error will be logged to the console and also to the Postgres log.</li>
<li>A stop file will be written in the lock directory and no more archive files will be backed up until it is removed.</li>
</ol>
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
The purpose of this feature is to prevent the log volume from filling up at which point Postgres will stop completely. Better to lose the backup than have the database go down.
To start normal archiving again you'll need to remove the stop file which will be located at <file>${archive-path}/lock/${stanza}-archive.stop</file> where <code>${archive-path}</code> is the path set in the <setting>archive</setting> section, and <code>${stanza}</code> is the backup stanza.</text>
<example>1024</example>
</config-key>
</config-key-list>
</config-section>
<!-- CONFIG - RETENTION -->
<config-section id="retention">
<text>The <setting>rentention</setting> section defines how long backups will be retained. Expiration only occurs when the number of complete backups exceeds the allowed retention. In other words, if full-retention is set to 2, then there must be 3 complete backups before the oldest will be expired. Make sure you always have enough space for rentention + 1 backups.</text>
<!-- CONFIG - RETENTION SECTION - FULL-RETENTION KEY -->
<config-key-list>
<config-key id="full-retention">
<text>Number of full backups to keep. When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When not defined then all full backups will be kept.</text>
<example>2</example>
</config-key>
<!-- CONFIG - RETENTION SECTION - DIFFERENTIAL-RETENTION KEY -->
<config-key id="differential-retention">
<text>Number of differential backups to keep. When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.</text>
<example>3</example>
</config-key>
<!-- CONFIG - RETENTION SECTION - DIFFERENTIAL-RETENTION KEY -->
<config-key id="differential-retention">
<text>Number of differential backups to keep. When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.</text>
<example>3</example>
</config-key>
<!-- CONFIG - RETENTION SECTION - ARCHIVE-RETENTION-TYPE KEY -->
<config-key id="archive-retention-type">
<text>Type of backup to use for archive retention (full or differential). If set to full, then PgBackRest will keep archive logs for the number of full backups defined by <setting>archive-retention</setting>. If set to differential, then PgBackRest will keep archive logs for the number of differential backups defined by <setting>archive-retention</setting>.
If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs that are older than the oldest backup, but there may be reasons for doing so.</text>
<example>full</example>
</config-key>
<!-- CONFIG - RETENTION SECTION - ARCHIVE-RETENTION KEY -->
<config-key id="archive-retention">
<text>Number of backups worth of archive log to keep. If not defined, then <setting>full-retention</setting> will be used when <setting>archive-retention-type=full</setting> and <setting>differential-retention</setting> will be used when <setting>archive-retention-type=differential</setting>.</text>
<example>2</example>
</config-key>
</config-key-list>
</config-section>
<!-- CONFIG - RESTORE -->
<config-section id="restore">
<text>[Not much to put here, but think of something]</text>
</config-section>
<!-- CONFIG - RESTORE-OPTION -->
<config-section id="restore-option">
<text>Archive Recovery and Standby Server restore.conf options can be specified here. See http://www.postgresql.org/docs/X.X/static/recovery-config.html for details on restore.conf options (replace X.X with your database version).
Note: <setting>restore_command</setting> will automatically be generated unless overridden in this section. Be careful about specifying your own <setting>restore_command</setting> as PgBackRest is designed to handle this for you.
Target Recovery options are specified on the command-line since they end to change from restore to restore (or not be needed at all in the case of a standby server).
Since <backrest/> does not start PostgreSQL after writing the <file>recovery.conf</file> file, it is always possible to edit/check the file before manually restarting.</text>
</config-section>
<!-- CONFIG - STANZA -->
<config-section id="stanza">
<text>A stanza defines a backup for a specific database. The stanza section must define the base database path and host/user if the database is remote. Also, any global configuration sections can be overridden to define stanza-specific settings.</text>
<!-- CONFIG - RETENTION SECTION - HOST KEY -->
<config-key-list>
<config-key id="host">
<text>Define the database host. Used for backups where the database host is different from the backup host.</text>
<example>db.domain.com</example>
</config-key>
<!-- CONFIG - RETENTION SECTION - USER KEY -->
<config-key id="user">
<text>Defines user account on the db host when <setting>stanza::host</setting> is defined.</text>
<example>postgres</example>
</config-key>
<!-- CONFIG - RETENTION SECTION - PATH KEY -->
<config-key id="path">
<text>Path to the db data directory (data_directory setting in postgresql.conf).</text>
<required/>
<example>/data/db</example>
</config-key>
</config-key-list>
</config-section>
</config-section-list>
</config>
<release title="Release Notes">
<release-version-list>
<release-version version="0.50" title="[under development]">
<release-feature-bullet-list>
<release-feature>
<text>Added restore functionality.</text>
</release-feature>
<release-feature>
<text>De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.</text>
</release-feature>
<release-feature>
<text>Added option <param>--no-start-stop</param> to allow backups when Postgres is shut down. If <file>postmaster.pid</file> is present then <param>--force</param> is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.</text>
</release-feature>
<release-feature>
<text>Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplied a number of things. Issue #28 has been created for checksum deltas.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.</text>
</release-feature>
<release-feature>
<text>Removed dependency on Moose. It wasn't being used extensively and makes for longer startup times.</text>
</release-feature>
<release-feature>
<text>Checksum for backup.manifest to detect corrupted/modified manifest.</text>
</release-feature>
<release-feature>
<text>Link <path>latest</path> always points to the last backup. This has been added for convenience and to make restore simpler.</text>
</release-feature>
<release-feature>
<text>More comprehensive backup unit tests.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.30" title="Core Restructuring and Unit Tests">
<release-feature-bullet-list>
<release-feature>
<text>Complete rewrite of BackRest::File module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.</text>
</release-feature>
<release-feature>
<text>Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.</text>
</release-feature>
<release-feature>
<text>Removed dependency on Storable and replaced with a custom ini file implementation.</text>
</release-feature>
<release-feature>
<text>Added much needed documentation</text>
</release-feature>
<release-feature>
<text>Numerous other changes that can only be identified with a diff.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.19" title="Improved Error Reporting/Handling">
<release-feature-bullet-list>
<release-feature>
<text>Working on improving error handling in the file object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).</text>
</release-feature>
<release-feature>
<text>Found and squashed a nasty bug where <code>file_copy()</code> was defaulted to ignore errors. There was also an issue in file_exists that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.18" title="Return Soft Error When Archive Missing">
<release-feature-bullet-list>
<release-feature>
<text>The <param>archive-get</param> operation returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets Postgres know that that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.17" title="Warn When Archive Directories Cannot Be Deleted">
<release-feature-bullet-list>
<release-feature>
<text>If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.16" title="RequestTTY=yes for SSH Sessions">
<release-feature-bullet-list>
<release-feature>
<text>Added <setting>RequestTTY=yes</setting> to ssh sesssions. Hoping this will prevent random lockups.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.15" title="RequestTTY=yes for SSH Sessions">
<release-feature-bullet-list>
<release-feature>
<text>Added archive-get functionality to aid in restores.</text>
</release-feature>
<release-feature>
<text>Added option to force a checkpoint when starting the backup <setting>start-fast=y</setting>.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.11" title="Minor Fixes">
<release-feature-bullet-list>
<release-feature>
<text>Removed <setting>master_stderr_discard</setting> option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code.</text>
</release-feature>
<release-feature>
<text>Changed lock file conflicts on backup and expire commands to ERROR. They were set to DEBUG due to a copy-and-paste from the archive locks.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
<release-version version="0.10" title="Backup and Archiving are Functional">
<release-feature-bullet-list>
<release-feature>
<text>No restore functionality, but the backup directories are consistent Postgres data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.</text>
</release-feature>
<release-feature>
<text>Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.</text>
</release-feature>
<release-feature>
<text>Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% threadsafe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.</text>
</release-feature>
<release-feature>
<text>Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.</text>
</release-feature>
<release-feature>
<text>The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well. Would definitely like to save these as human-readable text.</text>
</release-feature>
<release-feature>
<text>Absolutely no documentation (outside the code). Well, excepting these release notes.</text>
</release-feature>
</release-feature-bullet-list>
</release-version>
</release-version-list>
</release>
<recognition title="Recognition">
<text>Primary recognition goes to Stephen Frost for all his valuable advice and criticism during the development of <backrest/>. It's a far better piece of software than it would have been without him.
Resonate (http://www.resonate.com/) also contributed to the development of PgBackRest and allowed me to install early (but well tested) versions as their primary Postgres backup solution.</text>
</recognition>
</doc>

2
doc/default.css → doc/html/default.css
View File

@ -36,7 +36,7 @@ a:link
color: black;
}
a:visited
a:visited
{
text-decoration: none;
color: black;

407
doc/index.html
View File

@ -1,407 +0,0 @@
<!DOCTYPE HTML>
<head>
<title>PgBackRest - Simple Postgres Backup &amp; Restore</title>
<link rel="stylesheet" href="default.css" type="text/css"></link>
<meta name="viewport" content="width=device-width, initial-scale=1"></meta>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"></meta>
</head>
<body>
<!-- HEADER -->
<div class="header">
<div class="header-title">PgBackRest</div>
<div class="header-subtitle">Simple Postgres Backup &amp; Restore</div>
</div>
<!-- MENU -->
<div class="menu-set">
<div class="menu-first">[<a class="menu-link" href="#install">Install</a>]</div>
<div class="menu">[<a class="menu-link" href="#configure">Configure</a>]</div>
</div>
<!-- INTRO -->
<doc-intro>
Some intro text here.
</doc-intro>
<!-- INSTALL -->
<doc-install>
<doc-install-header><a id="install">Install</a></doc-install-header>
Something.<br/><br/>
</doc-install>
<!-- CONFIGURE -->
<doc-configure>
<doc-configure-header><a id="configure">Configure</a></doc-configure-header>
PgBackRest takes some command-line parameters, but depends on a configuration file for most of the settings. The default location for the configuration file is <doc-file>/etc/pg_backrest.conf</doc-file>.
<!-- CONFIGURE - COMMAND SECTION -->
<doc-configure-section>
<doc-configure-section-header><doc-id>command</doc-id> section</doc-configure-section-header>
The <doc-id>command</doc-id> section defines the location external commands that are used by PgBackRest.
<!-- CONFIGURE - COMMAND SECTION - PSQL KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>psql</doc-id> key</doc-configure-key-header>
Defines the full path to <doc-file>psql</doc-file>. <doc-file>psql</doc-file> is used to call <doc-function>pg_start_backup()</doc-function> and <doc-function>pg_stop_backup()</doc-function>.
<doc-code-block>
required: y<br/>
example: psql=/usr/bin/psql
</doc-code-block>
</doc-configure-key>
<!-- CONFIGURE - COMMAND SECTION - REMOTE KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>remote</doc-id> key</doc-configure-key-header>
Defines the file path to pg_backrest_remote.pl.
Required only if the path to <doc-file>pg_backrest_remote.pl</doc-file> is different on the local and remote systems. If not defined, the remote path will be assumed to be the same as the local path.
<doc-code-block>
required: n<br/>
example: remote=/home/postgres/backrest/bin/pg_backrest_remote.pl
</doc-code-block>
</doc-configure-key>
</doc-configure-section>
<!-- CONFIGURE - COMMAND-OPTION SECTION -->
<doc-configure-section>
<doc-configure-section-header><doc-id>command-option</doc-id> section</doc-configure-section-header>
The <doc-id>command-option</doc-id> section allows abitrary options to be passed to any command in the <doc-id>command</doc-id> section.
<!-- CONFIGURE - COMMAND-OPTION SECTION - PSQL KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>psql</doc-id> key</doc-configure-key-header>
Allows command line parameters to be passed to <doc-file>psql</doc-file>.
<doc-code-block>
required: no<br/>
example: psql=--port=5433
</doc-code-block>
</doc-configure-key>
</doc-configure-section>
<!-- CONFIGURE - LOG SECTION -->
<doc-configure-section>
<doc-configure-section-header><doc-id>log</doc-id> section</doc-configure-section-header>
The <doc-id>log</doc-id> section defines logging-related settings. The following log levels are supported:<br/>
<ul>
<li><doc-id>off </doc-id> - No logging at all (not recommended)</li>
<li><doc-id>error</doc-id> - Log only errors</li>
<li><doc-id>warn </doc-id> - Log warnings and errors</li>
<li><doc-id>info </doc-id> - Log info, warnings, and errors</li>
<li><doc-id>debug</doc-id> - Log debug, info, warnings, and errors</li>
<li><doc-id>trace</doc-id> - Log trace (very verbose debugging), debug, info, warnings, and errors</li>
</ul>
<!-- CONFIGURE - LOG SECTION - LEVEL-FILE KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>level-file</doc-id> key</doc-configure-key-header>
Sets file log level.
<doc-code-block>
default: info<br/>
example: level-file=debug
</doc-code-block>
</doc-configure-key>
<!-- CONFIGURE - LOG SECTION - LEVEL-CONSOLE KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>level-console</doc-id> key</doc-configure-key-header>
Sets console log level.
<doc-code-block>
default: warn<br/>
example: level-console=info
</doc-code-block>
</doc-configure-key>
</doc-configure-section>
<!-- CONFIGURE - GENERAL SECTION -->
<doc-configure-section>
<doc-configure-section-header><doc-id>general</doc-id> section</doc-configure-section-header>
The <doc-id>general</doc-id> section defines settings that are shared between multiple operations.
<!-- CONFIGURE - GENERAL SECTION - BUFFER_SIZE KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>buffer-size</doc-id> key</doc-configure-key-header>
Set the buffer size used for copy, compress, and uncompress functions. A maximum of 3 buffers will be in use at a time per thread. An additional maximum of 256K per thread may be used for zlib buffers.
<doc-detail-block>
<doc-detail>default:<doc-detail-value>1048576</doc-detail-value></doc-detail>
<doc-detail>allowed:<doc-detail-value>4096 - 8388608</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>buffer-size=8192</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - LOG SECTION - COMPRESS-LEVEL KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>compress-level</doc-id> key</doc-configure-key-header>
Sets the zlib level to be used for file compression when `compress=y`.
<doc-detail-block>
<doc-detail>default:<doc-detail-value>6</doc-detail-value></doc-detail>
<doc-detail>allowed:<doc-detail-value>0-9</doc-detail-value></doc-detail>
<doc-detail>override:<doc-detail-value>backup, archive</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>compress-level=9</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - LOG SECTION - COMPRESS-LEVEL-NETWORK KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>compress-level-network</doc-id> key</doc-configure-key-header>
Sets the zlib level to be used for protocol compression when <doc-setting>compress=n</doc-setting> and the database is not on the same host as the backup. Protocol compression is used to reduce network traffic but can be disabled by setting <doc-setting>compress-level-network=0</doc-setting>. When <doc-setting>compress=y</doc-setting> the <doc-setting>compress-level-network</doc-setting> setting is ignored and <doc-setting>compress-level</doc-setting> is used instead so that the file is only compressed once. SSH compression is always disabled.
<doc-detail-block>
<doc-detail>default:<doc-detail-value>6</doc-detail-value></doc-detail>
<doc-detail>allowed:<doc-detail-value>0-9</doc-detail-value></doc-detail>
<doc-detail>override:<doc-detail-value>backup, archive</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>compress-level=9</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
</doc-configure-section>
<!-- CONFIGURE - BACKUP SECTION -->
<doc-configure-section>
<doc-configure-section-header><doc-id>backup</doc-id> section</doc-configure-section-header>
The <doc-id>backup</doc-id> section defines settings related to backup.
<!-- CONFIGURE - BACKUP SECTION - HOST KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>host</doc-id> key</doc-configure-key-header>
<p>Sets the backup host when backup up remotely via SSH. Make sure that trusted SSH authentication is configured between the db host and the backup host.</p>
<p>When backing up to a locally mounted network filesystem this setting is not required.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>host=backup.mydomain.com</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - USER KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>user</doc-id> key</doc-configure-key-header>
<p>Sets user account on the backup host.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n (but must be set if host is defined)</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>user=backrest</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - PATH KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>path</doc-id> key</doc-configure-key-header>
<p>Path where backups are stored on the local or remote host.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>y</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>path=/var/lib/backrest</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - COMPRESS KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>compress</doc-id> key</doc-configure-key-header>
<p>Enable gzip compression. Backup files are compatible with command-line gzip tools.</p>
<doc-detail-block>
<doc-detail>default:<doc-detail-value>y</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>compress=n</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - START-FAST KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>start-fast</doc-id> key</doc-configure-key-header>
<p>Forces a checkpoint (by passing <doc-id>true</doc-id> to the <doc-id>fast</doc-id> parameter of <doc-function>pg_start_backup()</doc-function>) so the backup begins immediately.</p>
<doc-detail-block>
<doc-detail>default:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>start-fast=y</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - HARDLINK KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>hardlink</doc-id> key</doc-configure-key-header>
<p>Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each backup is a full backup. Be care though, because modifying files that are hard-linked can affect all the backups in the set.</p>
<doc-detail-block>
<doc-detail>default:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>hardlink=y</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - THREAD-MAX KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>thread-max</doc-id> key</doc-configure-key-header>
<p>Defines the number of threads to use for backup. Each thread will perform compression and transfer to make the backup run faster, but don't set <doc-setting>thread-max</doc-setting> so high that it impacts database performance.</p>
<doc-detail-block>
<doc-detail>default:<doc-detail-value>1</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>thread-max=4</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - THREAD-TIMEOUT KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>thread-timeout</doc-id> key</doc-configure-key-header>
<p>Maximum amount of time (in seconds) that a backup thread should run. This limits the amount of time that a thread might be stuck due to unforeseen issues during the backup. Not valid when <doc-setting>thread-max=1</setting>.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>thread-timeout=3600</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - BACKUP SECTION - ARCHIVE-REQUIRED KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>archive-required</doc-id> key</doc-configure-key-header>
<p>Are archive logs required to to complete the backup? It's a good idea to leave this as the default unless you are using another method for archiving.</p>
<doc-detail-block>
<doc-detail>default:<doc-detail-value>y</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>archive-required=n</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
</doc-configure-section>
<!-- CONFIGURE - ARCHIVE SECTION -->
<doc-configure-section>
<doc-configure-section-header><doc-id>archive</doc-id> section</doc-configure-section-header>
<p>The <doc-id>archive</doc-id> section defines parameters when doing async archiving. This means that the archive files will be stored locally, then a background process will pick them and move them to the backup.</p>
<!-- CONFIGURE - ARCHIVE SECTION - PATH KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>path</doc-id> key</doc-configure-key-header>
<p>Path where archive logs are stored before being asynchronously transferred to the backup. Make sure this is not the same path as the backup is using if the backup is local.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>y</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>/var/lib/backest</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - ARCHIVE SECTION - COMPRESS-ASYNC KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>compress-async</doc-id> key</doc-configure-key-header>
<p>When set archive logs are not compressed immediately, but are instead compressed when copied to the backup host. This means that more space will be used on local storage, but the initial archive process will complete more quickly allowing greater throughput from Postgres.</p>
<doc-detail-block>
<doc-detail>default:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>compress-async=y</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - ARCHIVE SECTION - ARCHIVE-MAX-MB KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>archive-max-mb</doc-id> key</doc-configure-key-header>
<p>Limits the amount of archive log that will be written locally when <doc-setting>compress-async=y</doc-setting>. After the limit is reached, the following will happen:</p>
<ol>
<li>PgBackRest will notify Postgres that the archive was succesfully backed up, then DROP IT.</li>
<li>An error will be logged to the console and also to the Postgres log.</li>
<li>A stop file will be written in the lock directory and no more archive files will be backed up until it is removed.</li>
</ol>
</p>If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.</p>
<p>The purpose of this feature is to prevent the log volume from filling up at which point Postgres will stop completely. Better to lose the backup than have the database go down.</p>
<p>To start normal archiving again you'll need to remove the stop file which will be located at <doc-id>${archive-path}/lock/${stanza}-archive.stop</doc-id> where <doc-id>${archive-path}</doc-id> is the path set in the <doc-id>archive</doc-id> section, and <doc-id>${stanza}</doc-id> is the backup stanza.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>archive-max-mb=1024</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
</doc-configure-section>
<!-- CONFIGURE - RETENTION SECTION -->
<doc-configure-section>
<doc-configure-section-header><doc-id>archive</doc-id> section</doc-configure-section-header>
<p>The <doc-id>rentention</doc-id> section defines how long backups will be retained. Expiration only occurs when the number of complete backups exceeds the allowed retention. In other words, if full-retention is set to 2, then there must be 3 complete backups before the oldest will be expired. Make sure you always have enough space for rentention + 1 backups.</p>
<!-- CONFIGURE - RETENTION SECTION - FULL-RETENTION KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>full-retention</doc-id> key</doc-configure-key-header>
<p>Number of full backups to keep. When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When not defined then all full backups will be kept.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>full-retention=2</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - RETENTION SECTION - DIFFERENTIAL-RETENTION KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>differential-retention</doc-id> key</doc-configure-key-header>
<p>Number of differential backups to keep. When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>differential-retention=3</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - RETENTION SECTION - ARCHIVE-RETENTION-TYPE KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>archive-retention-type</doc-id> key</doc-configure-key-header>
<p>Type of backup to use for archive retention (full or differential). If set to full, then PgBackRest will keep archive logs for the number of full backups defined by <doc-setting>archive-retention</doc-setting>. If set to differential, then PgBackRest will keep archive logs for the number of differential backups defined by <doc-setting>archive-retention</doc-setting>.</p>
<p>If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs that are older than the oldest backup, but there may be reasons for doing so.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>archive-retention-type=full</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
<!-- CONFIGURE - RETENTION SECTION - ARCHIVE-RETENTION KEY -->
<doc-configure-key>
<doc-configure-key-header><doc-id>archive-retention</doc-id> key</doc-configure-key-header>
<p>Number of backups worth of archive log to keep. If not defined, then <doc-setting>full-retention</doc-setting> will be used when <doc-setting>archive-retention-type=full</doc-setting> and <doc-setting>differential-retention</doc-setting> will be used when <doc-setting>archive-retention-type=differential</doc-setting>.</p>
<doc-detail-block>
<doc-detail>required:<doc-detail-value>n</doc-detail-value></doc-detail>
<doc-detail>example:<doc-detail-value>archive-retention=2</doc-detail-value></doc-detail>
</doc-detail-block>
</doc-configure-key>
</doc-configure-section>
</doc-configure>
</body>
</html>

2
test/lib/BackRestTest/BackupTest.pm
View File

@ -1964,7 +1964,7 @@ sub BackRestTestBackup_Test
# Static backup parameters
my $bSynthetic = false;
my $fTestDelay = 1;
my $fTestDelay = .1;
# Variable backup parameters
my $bDelta = true;