1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1349 Commits

Author SHA1 Message Date
David Steele
01b8e2258f
Improve archive-push command fault tolerance.
3b8f0ef missed some cases that could cause archive-push to fail:

* Checking archive info.
* Checking to see if a WAL segment already exists.

These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
Cynthia Shang
2789d3b620
Improve info command fault tolerance.
This improvement reduces the number of errors thrown; these errors will now be reported as a status for the stanza or repo as appropriate. Invalid option configurations are still thrown but all other errors are caught, formatted and reported. This was necessary for multiple repositories so that the command can complete gathering information from each repository and report the results rather than immediately aborting when an error occurs.

Two new error codes were introduced:
6 = requested backup not found
99 = other, which is used to indicate an error has occurred that requires more details to be provided

A new stanza name of "[invalid]" was created for instances where a stanza was not specified and no stanza can be found.

If there is only one repository configured the error will move up to the stanza level with the standard error formatting of 'error (message)' where the message will be "other" and the details of the error will be listed on the next line(s):

stanza: stanza1
    status: error (other)
            [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
            CryptoError: cipher header invalid
            HINT: is or was the repo encrypted?
            FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
            HINT: backup.info cannot be opened and is required to perform a backup.
            HINT: has a stanza-create been performed?
            HINT: use option --stanza if encryption settings are different for the stanza than the global
    cipher: aes-256-cbc

If a backup set is requested but is not found on any repo, a stanza-level status error of 'requested backup not found' is reported when there are no other errors:

pgbackrest info --stanza=demo --set=bogus
stanza: demo
    status: error (requested backup not found)
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

If there are multiple repositories configured and a single repo is in error but the other repos are ok or have a different error:

pgbackrest info --stanza=demo --set=20210322-171211F
stanza: demo
    status: mixed
        repo1: error
               [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
               CryptoError: cipher header invalid
               HINT: is or was the repo encrypted?
               FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
               HINT: backup.info cannot be opened and is required to perform a backup.
               HINT: has a stanza-create been performed?
               HINT: use option --stanza if encryption settings are different for the stanza than the global
        repo2: ok
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

    db (current)
        wal archive min/max (12): 000000010000000000000001/000000010000000000000003

        full backup: 20210322-171211F
            timestamp start/stop: 2021-03-22 17:12:11 / 2021-03-22 17:12:28
            wal start/stop: 000000010000000000000002 / 000000010000000000000002
            database size: 23.4MB, database backup size: 23.4MB
            repo2: backup set size: 2.8MB, backup size: 2.8MB
            database list: postgres (13359)

Json output will include the repository information and any error information. If no stanzas are found, then [invalid] will be set as the name:

[
   {
       "archive":[],
       "backup":[],
       "cipher":"none",
       "db":[],
       "name":"[invalid]",
       "repo":[
           {
               "cipher":"none",
               "key":1,
               "status":{
                   "code":99,
                   "message":"[PathOpenError] unable to list file info for path '/var/lib/pgbackrest/repo2/backup': [13] Permission denied"
               }
           }
       ],
       "status":{
           "code":99,
           "lock":{"backup":{"held":false}},
           "message":"other"
           }
   }
]
2021-03-25 12:29:36 -04:00
David Steele
7d7ac0e0eb
Exclude content-length from S3 signed headers.
The content-length header was being signed since it was the only header that didn't need to be and it seemed simpler just to sign it as well. Also, the S3 documentation encourages signing as many headers as possible to avoid tampering.

However, some proxies munge this header causing authentication failure, so skip signing content-length.
2021-03-25 07:07:16 -04:00
David Steele
2016fac0d9
Improve protocol handlers.
Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.
2021-03-16 13:09:34 -04:00
David Steele
b1d945ebb7
Fix repo-ls / on S3 repositories.
S3 returns 200 for HEAD / which indicates it is a file but does not return the expected headers which causes an error.

Rather than fix this for S3, just automatically return / as not existing for any storage that does not support paths.

Also add some defensive checks to prevent this from generating a segfault if it happens again.
2021-03-16 12:42:55 -04:00
Stefan Fercot
6942ff569d
Include recreated system databases during selective restore.
Some standard system databases (e.g. postgres) may be recreated by the user and have an OID that makes them look like user databases.

Identify the standard three system databases (template0, template1, postgres) and restore them non-zeroed no matter what OID they have.
2021-03-15 12:54:14 -04:00
David Steele
92d12ccb9b
Update selective restore documentation with caveats.
Recovery may error unless --type=immediate is specified. This is because after consistency is reached PostgreSQL will flag zeroed pages as errors even for a full-page write.

For PostgreSQL ≥ 13 the ignore_invalid_pages setting may be used to ignore invalid pages. In this case it is important to check the logs after recovery to ensure that no invalid pages were reported in the selected databases.
2021-03-11 10:19:50 -05:00
David Steele
c862e9654a
Log archive copy during backup.
Copying can be a fairly expensive operation so it makes sense to log it so the user gets some status during long copy operations.
2021-03-11 08:22:44 -05:00
David Steele
9506ffae39
Add compress-type clarification to archive-copy documentation.
It is best if the archive-push and backup commands have the same compress-type (e.g. lz4) when using archive-copy. Otherwise, the WAL segments will need to be recompressed with the compress-type used by the backup, which can be fairly expensive depending on how much WAL was generated during the backup.
2021-03-11 07:53:10 -05:00
David Steele
778adbf19f Fix memory leak in backup during archive copy.
There was already leakage here but when the compression transcoding was added it became a deluge.

There is some argument to be made that the filters should clean themselves up better but a temp mem context makes sense here anyway so do that.
2021-03-10 09:15:35 -05:00
Cynthia Shang
31c7824a4d
Allow stanza-* commands to be run remotely.
The stanza-create, stanza-upgrade and stanza-delete were required to be run on the repository host. When there was only one repository allowed this was not a problem.

However, with the introduction of multiple repository support, this becomes more of a burden to the user, therefore the stanza-create, stanza-upgrade and stanza-delete commands have been improved to allow for them to be run remotely.
2021-03-10 08:10:46 -05:00
David Steele
c4a3dc4e46 Combine multi-repo release notes. 2021-03-10 07:44:18 -05:00
David Steele
1dbb3bf50b
Multiple repository support.
Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.

Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.

Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.

The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
For single repository configurations there should be no change in behavior.
2021-03-08 13:31:13 -05:00
David Steele
088662d986
GCS support for repository storage.
GCS and GCS-compatible object stores can now be used for repository storage.
2021-03-05 12:13:51 -05:00
David Steele
95063f6812 Make --repo optional for remaining commands except stanza-delete.
Some commands (repo-*, verify) still required the --repo option but it makes sense to give them the same treatment as backup and simply use the first repo when one is not specified.

This leaves stanza-delete as the only remaining command that requires --repo. This is by design to enhance safe usage.
2021-03-03 09:21:06 -05:00
David Steele
d1aa765a9d
Consolidate less commonly used repository storage options.
The following options are renamed as specified:

repo1-azure-ca-file -> repo1-storage-ca-file
repo1-azure-ca-path -> repo1-storage-ca-path
repo1-azure-host -> repo1-storage-host
repo1-azure-port -> repo1-storage-port
repo1-azure-verify-tls -> repo1-storage-verify-tls
repo1-s3-ca-file -> repo1-storage-ca-file
repo1-s3-ca-path -> repo1-storage-ca-path
repo1-s3-host -> repo1-storage-host
repo1-s3-port -> repo1-storage-port
repo1-s3-verify-tls -> repo1-storage-verify-tls

The old option names (e.g. repo1-s3-port) will continue to work for repo1, but repo2, etc. will require the new names.
2021-03-02 13:51:40 -05:00
David Steele
e64999db77
Add HttpUrl object.
Parse a URL into component parts.
2021-03-01 13:44:47 -05:00
David Steele
3b8f0ef7ae Add write fault-tolerance to archive-push command.
The archive-push command will continue to push even after it gets a write error on one or more repos. The idea is to archive to as many repos as possible even we still need to throw an error to PostgreSQL to prevent it from removing the WAL file.
2021-02-26 16:52:59 -05:00
David Steele
a1280c41e5 Refactor archive-push command warnings to work like archive-get.
Warnings are logged individually in the async log rather than all together.
2021-02-26 15:58:11 -05:00
Cynthia Shang
13dc8e68d7 Make --repo optional for backup command.
If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 14:49:50 -05:00
Michael Schout
9243962b95
Allow custom config-path default with ./configure --with-configdir.
Add --with-confdir=DIR option to configure, which can be used to override the default configuration directory of /etc/pgbackrest.

Probably in the future it would be better to just leverage ${sysconfdir} which is based on prefix, but since previously the config directory was hard coded to /etc/pgbackrest, we retain that default value by not relying on sysconfdir for now.
2021-02-25 12:03:44 -05:00
Cynthia Shang
0ddc0380ff Remove restore default repo from integration tests.
The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 11:32:13 -05:00
Cynthia Shang
065b2ff230 Refactor info command repoMin/Max. 2021-02-23 16:27:05 -05:00
Cynthia Shang
118d9e64fe Enhance restore command multi-repo support.
The restore command automatically defaults to selecting the latest backup from a single repository. With multiple repositories configured, the restore command will now default to selecting the latest backup from the first repository where backups exist. The order in which the repositories are checked is dictated by the pgbackrest.conf order.

To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
2021-02-23 16:17:27 -05:00
David Steele
bec3e20b2c Add archive-get command multi-repo support.
Repositories will be searched in order for the requested archive file.

Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Cynthia Shang
e28f6f11e9 Expire continues if an error occurs processing a repository.
Errors are logged to the log file rather than thrown. If, after processing all repos, one or more errors occurred, then a single error error will be thrown to indicate there were errors and the log file should be inspected.

Also update log messages to be more consistent with new patterns.
2021-02-23 12:20:02 -05:00
David Steele
3837e61a75
Fix option warnings breaking async archive-get/archive-push.
Option warnings will cause the async process to fail because a warning is logged but stdout is closed so the process aborts.

This bug has existed for quite some time, but it was made worse by abb8ebe because now the async role can have different valid options than the default role. Previously at least a warning would be emitted before the async process died.

Fix this by only allowing warnings for the default role. Warnings were already suppressed for local and remote roles so the logic already exists.
2021-02-18 13:29:09 -05:00
David Steele
d29855bd0b
Fix stack overflow in cipher passphrase generation.
The destination buffer on the stack was not large enough to contain the zero-terminating character.

Increase the buffer size and add an assertion to prevent regressions.

Found on arm64 running musl libc. Other architectures and glibc do not seem to be affected though it is clearly a bug.
2021-02-12 10:08:47 -05:00
Cynthia Shang
3408f1ee2e Enhance expire command multi-repo support.
The expire command has been enhanced to expire backups and archives from all configured repositories by default.

In addition, it will accept the --repo option to expire backups and archives only from the specified repository. Using the --repo options the --set option can also be refined further to the specified repo. If --set is provided but the --repo option has not, then all repositories will be searched and retention settings will be applied on each whether the backup set has been found or not.
2021-02-10 12:03:52 -05:00
David Steele
00f06065e7 Begin v2.33 development. 2021-02-08 13:18:22 -05:00
David Steele
aadc9e2fe6 v2.32: Repository Commands
Bug Fixes:

* Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.)

Features:

* Add repo-ls command. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.)
* Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele, Michael Banck.)

Improvements:

* Improve archive-get performance. (Reviewed by Cynthia Shang.)
2021-02-08 09:08:16 -05:00
Cynthia Shang
d350d1cc21 Improve expire command documentation. 2021-02-05 11:48:07 -05:00
David Steele
b65c370346 Add repo-get command. 2021-02-05 10:39:03 -05:00
David Steele
218cd078a6 Add repo-ls command. 2021-02-05 10:07:43 -05:00
Stefan Fercot
4b46115345
Add archive-mode-check option.
This option disallows the PostgreSQL archive_mode=always setting and disabling it allows the setting.
2021-02-02 13:43:14 -05:00
Cynthia Shang
d5b919e657
Update expire command log messages with repo prefix.
In preparation for multi-repo support, a repo tag is added in this commit to the expire command log and error messages. This change also affects the expect logs and the user-guide. The format of the tag is "repoX:" where X is the repo key used in the configuration.

Until multi-repo support has been completed, this tag will always be "repo1:".
2021-01-27 16:33:01 -05:00
Cynthia Shang
2e60b93709
Add backup verification to internal verify command.
This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered.

Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Cynthia Shang
00fac1c0d1 Improve info command text output and --set handling.
The info command provides total sizes for files in the backup on the database as well as the repository. The text output and associated user documentation has been updated to provide more clarity regarding the sizes being displayed.

In addition, the info command is updated to allow a user to optionally specify the repository when requesting a specific backup set. In this case, the text output will reflect the status of the stanza, the cipher types and archive min/max over all the repositories instead of a single repository when the repo option is specified.
2021-01-25 09:19:05 -05:00
Cynthia Shang
f32eb9b94e
Partial multi-repository implementation.
Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands.

Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations.

Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
David Steele
a8fb285756
Improve archive-get performance.
Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get.

Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process.

Cache results of storageList() when looking for multiple files to reduce storage I/O.

Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
David Steele
aeee83044d
Fix resume after partial delete of backup by prior resume.
If files other than backup.manifest.copy were left in a backup path by a prior resume then the next resume would skip the backup rather than removing it. Since the backup path still existed, it would be found during backup label generation and cause an error if it appeared to be later than the new backup label. This occurred if the skipped backup was full.

The error was only likely on object stores such as S3 because of the order of file deletion. Posix file systems delete from the bottom up because directories containing files cannot be deleted. Object stores do not have directories so files are deleted in whatever order they are provided by the list command. However, the issue can be reproduced on a Posix file system by manually deleting backup.manifest.copy from a resumable backup path.

Fix the issue by removing the resumable backup if it has no manifest files. Also add a new warning message for this condition.

Note that this issue could be resolved by running expire or a new full backup.
2021-01-12 12:38:32 -05:00
David Steele
96fd678662
Add job-retry and job-retry-interval options.
These options specify the number of local worker job retries and the retry interval after one immediate retry.

There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests.

Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally.

These options are currently internal.
2021-01-11 15:15:25 -05:00
David Steele
abb8ebe58b
Limit option validity by command role.
Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here.

The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.
2020-12-29 15:49:37 -05:00
David Steele
8361a97482
Add pack type.
The pack type is an architecture-independent format for serializing data compactly, inspired by ProtocolBuffers and Avro.

Also add ioReadSmall(), which is optimized for small binary reads, similar to ioReadLineParam().
2020-12-09 12:05:14 -05:00
David Steele
87996558d2
Replace double type with time in config module.
The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand.

Also remove Variant doubles since they are no longer used.

Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
David Steele
ab0500789e Begin v2.32 development. 2020-12-07 11:13:45 -05:00
David Steele
e116b535e6 v2.31: Minor Bug Fixes and Improvements
Bug Fixes:

* Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)
* Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)

Features:

* Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
* Add repo-azure-endpoint option. (Reviewed by Cynthia Shang, Brian Peterson. Suggested by Brian Peterson.)
* Add pg-database option. (Reviewed by Cynthia Shang.)

Improvements:

* Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)
* Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang, Oscar.)
* Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)

Documentation Improvements:

* Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.
2020-12-07 09:55:00 -05:00
David Steele
31becf05b7 Add RHEL/CentOS 8 documentation.
Update RHEL/CentOS 7 to cover the versions that were previously covered by RHEL/CentOS 6.

Since RHEL/CentOS 7/8 work the same update the documentation logic and labels to reflect this compatibility.
2020-12-04 10:59:57 -05:00
David Steele
b0ea337965 Add pg-database option.
In some rare cases there is no postgres database so this option may be used to specify an alternate database.
2020-12-02 22:42:50 -05:00
David Steele
d4211d3aaf Add retries to PostgreSQL sleep when starting a backup.
Inaccuracies in sleep time or clock skew might make a single sleep insufficient to reach the next second.

Add a few retries to make the process more reliable but still avoid an infinite loop if something is seriously wrong.
2020-12-02 22:41:14 -05:00