1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

3770 Commits

Author SHA1 Message Date
David Steele
5dba0d6e9b Set option-archive-copy flag in backup.manifest to false when offline.
In offline mode the pg_wal directory is copied, but that is not the same as archive-copy, which copies the exact set of WAL required from the archive.

This flag is purely for informational purposes so there is no live bug here, but the prior behavior was certainly misleading.
2022-04-05 18:42:19 -04:00
David Steele
54b4187527 Show Docker output when building containers if --log-level=detail.
This helps with debugging and monitoring container builds.
2022-04-05 13:14:42 -04:00
Reid Thompson
d8d4132118
Auto-select backup for restore command --type=lsn.
For PITR with --type=lsn, attempt to auto-select the appropriate backup set based on the --target LSN provided. Pick the most recent backup where backup-lsn-stop is less than or equal to the provided LSN.
2022-04-05 11:59:12 -04:00
David Steele
08d9e269c6 Refactor target type checking for clarity in restore module.
This avoids using targetTime != 0 as an indicator that a time target was selected.
2022-04-05 09:14:56 -04:00
Stefan Fercot
f5fdab2989 Fix typo in db-timeout help. 2022-03-31 17:09:50 +02:00
David Steele
c222ce1a5f Fix tls-server-auth example and add clarifications. 2022-03-25 08:52:41 -06:00
David Steele
d4954fcbf4 Fix comment typo. 2022-03-25 07:52:16 -06:00
David Steele
f60ec5055a Cleanup output to stderr in unit tests.
The unit tests were ignoring stderr but nothing being output there was important. Now a test will fail if there is anything on stderr.

This makes it easier to work with -fsanitize, which outputs to stderr.
2022-03-24 18:43:43 -06:00
David Steele
50ee4b19fe Align checksum page error list in manifest file packs.
This was left unaligned on purpose to save space but the sanitizer did not like it. Since this field is seldom used go ahead and align it to make the sanitizer happy.

Also add some macros to make working with alignment easier.

Found with -fsanitize=undefined.
2022-03-24 17:55:38 -06:00
Abubakar Mohammed
3dd7960451 Fix incorrect reference to stanza-update in the user guide.
This should be stanza-upgrade. Also fix in the git history cache since the comment was copied from the user guide.
2022-03-24 15:59:41 -06:00
David Steele
14016a86e7 Check that sha1 checksum is not empty in manifestFileUpdate().
The manifest test module was setting a blank value here and causing a stack overflow because memcpy() is used instead of strcpy().

This was really just a test issue but add an assert just in case the same were to happen in production code.

Also update a bogus checksum in the integration tests to the correct length to avoid running afoul of the assert.

Found with -fsanitize=address.
2022-03-24 13:13:35 -06:00
David Steele
4e5ac11517 Fix incorrect struct type in list initialization.
This looks like a copy-paste error.

The code is only run during development so this is not a live issue.

Found with -fsanitize=address.
2022-03-24 12:56:26 -06:00
David Steele
75b26319ae Use strNewZ() in cases where STRDEF() assignment goes out of scope.
If a variable assigned with STRDEF() is referenced out of scope of the STRDEF() assignment then the value is undefined.

Luckily most of the instances are in tests but there is one in the core code. It is not clear if this is a live bug or not but it certainly needs to be fixed.

Found with -fsanitize=address.
2022-03-24 12:26:09 -06:00
David Steele
edf6c70baa Prevent signed integer overflow in cfgParseSize().
If the value and multiplier were large enough then the return value could overflow unpredictably.

Check the value to make sure it will not overflow with the current multiplier.

It would be better to present an "out of range" error to the user rather than "is not valid" but it doesn't seem worth the effort since the error is extremely unlikely.

Found with -fsanitize=undefined.
2022-03-24 11:00:51 -06:00
David Steele
ccbe2a1f70 Do not pass NULL to memcpy() in Buffer/String objects.
glibc and others seem tolerant of this but the behavior is undefined.

Found with -fsanitize=undefined.
2022-03-24 09:32:18 -06:00
David Steele
98792b1b0c Do not pass NULL to bsearch()/qsort() in List object.
glibc and others seem tolerant of this but the behavior is undefined.

Found with -fsanitize=undefined.
2022-03-24 09:22:05 -06:00
David Steele
333ef84606 Use bufUsed() instead of struct member in Buffer object.
bufUsed() is an inline function so it is just as efficient and should also be easier to read and maintain.
2022-03-23 21:47:56 -06:00
David Steele
424008d293 Allow files that become zero-length after the backup manifest is built.
It is possible that a file will be be truncated to zero-length after the backup manifest has been built. We could build logic into backupFile() to handle this case but it is hard to test well because of the race condition so tests would need to written directly against backupFile() and backupJobResult(). It hardly seems worth all that effort for a condition that occurs rarely, if ever.

Instead just remove the manifest check and add tests to restore to make sure it handles bundled zero-length files correctly. Logging will show that the file was bundled so if it happens a lot (which seems very unlikely) then we can think about an alternate implementation.
2022-03-23 10:41:36 -06:00
David Steele
fe9fd2ff2d Disable repo-hardlink option when repo-bundle option is enabled.
Hardlinking will not work with bundles because files are not stored individually.
2022-03-22 09:02:33 -06:00
David Steele
7afaac0a3d
Allow repo-hardlink option to be changed after full backup.
This rule was added because there were not sufficient tests to demonstrate that the repo-hardlink option could be changed in a backup set.

Remove the restriction and add/update tests to show that it works.

This is necessary now because bundling requires that hardlinking be disabled. Rather than add code complexity, it seems better just to address this limitation.
2022-03-22 08:35:34 -06:00
Reid Thompson
5ae84d5e47
Improve path validation for repo-* commands.
Check for invalid path in repo-* commands. Perform path validation and throw an error when appropriate. Path may not contain '//'. Strip trailing '/' from path. Absolute path must fall under repo path.
2022-03-22 07:50:26 -06:00
nunopi
21cef09dfd
Add AWS IMDSv2 support.
IMDSv2 provides additional security to prevent instance metadata from being read by an attacker.

All AWS instances should provide IMDSv2 but still fail back to IMDSv1 if the IMDSv2 token request fails. This is in case there are any services outside AWS that are emulating IMDSv1 but have not implemented IMDSv2.
2022-03-16 11:02:29 -06:00
David Steele
2c96327e65 Remove extraneous double spaces in code and comments. 2022-03-15 17:55:48 -06:00
David Steele
3f66f42ef9
Rename bundle-* options to repo-bundle-*.
It seems best for these to be repo options so they can be configured per repo, rather than globally.

All clarify usage for repo-bundle-size and repo-bundle-limit.
2022-03-14 17:49:52 -06:00
Reid Thompson
7c9208ba85
Improve error message for invalid repo-azure-key.
Check that repo-azure-key is valid base64 when repo-azure-key-type = shared.
2022-03-11 10:10:02 -06:00
David Steele
0054677147 Add bundle logging to backup command.
This was added to the restore command so add it to the backup command as well.
2022-03-09 15:34:15 -06:00
David Steele
dca6da86bf
Optimize restore command for file bundling.
Since files are stored sequentially in a bundle, it is often possible to restore multiple files with a single read. Previously, each restored file required a separate read. Reducing the number of reads is particularly beneficial for object stores, but performance should benefit on any file system.

Currently if there is a gap then a new read is required. In the future we might set a limit for how large a gap we'll skip without starting a new read.
2022-03-09 15:03:28 -06:00
Reid Thompson
f7ab002aa7
Improve stop command to honor stanza option.
Improve the stop command, when force and stanza options are specified, to terminate only processes holding lock files for the given stanza. Prior to these changes, termination of all processes holding lock files regardless of stanza occurred.
2022-03-08 12:18:23 -06:00
David Steele
514137040e Add limit parameter to ioCopyP().
Allows the number of bytes copied to be limited.
2022-03-08 08:23:31 -06:00
David Steele
166039c0da Fix example for repo-gcs-key-type option in configuration reference.
This looks like a copy-paste error from another option.
2022-03-08 08:09:58 -06:00
David Steele
91b718bd4f Fix comment typo. 2022-03-06 14:58:43 -06:00
Reid Thompson
330e19900e
Increase precision of percent complete logging for backup and restore.
For very large backups only getting an update per percent may not be often enough.

Add hundredths to the percent complete logging to provide more timely information.
2022-03-06 13:01:24 -06:00
David Steele
be731f8254 Fix incorrect comment.
This appears to have been blindly copied from the archive-get command.
2022-03-06 12:03:57 -06:00
David Steele
8f23b46b4b Replace percentage and size with a constant in restore test logs.
Checking percentage and size in every test can cause quite a bit of churn when changes are made.

Follow the example of the backup tests and replace percentage and size after the few tests to reduce churn.
2022-03-06 11:57:20 -06:00
David Steele
4d2fef1c37 Remove redundant restoreFile() test and improve coverage.
These tests were written before the restore command was fully migrated to C so many of them have become redundant.

In the cases were they still provide coverage, add tests to synthetic restores to replace them. In general, these higher level tests provide better coverage than poking at the restoreFile() function directly.
2022-03-06 11:48:22 -06:00
David Steele
009d33aa8e Update config.guess and config.sub to latest versions. 2022-03-06 11:17:01 -06:00
David Steele
c242b966e0 Begin v2.39 development. 2022-03-06 11:14:01 -06:00
David Steele
5249b89a2e v2.38: Minor Bug Fixes and Improvements
IMPORTANT NOTE: Repository size reported by the info command is now entirely based on what pgBackRest has written to storage. Previously, in certain cases, pgBackRest could detect if additional compression was being applied by the storage but this is no longer supported.

Bug Fixes:

* Retry errors in S3 batch file delete. (Reviewed by Reid Thompson. Reported by Alex Richman.)
* Allow case-insensitive matching of HTTP connection header values. (Reviewed by Reid Thompson. Reported by Rémi Vidier.)

Features:

* Add support for AWS S3 server-side encryption using KMS. (Contributed by Christoph Berg. Reviewed by David Steele, Tharindu Amila.)
* Add archive-missing-retry option. (Reviewed by Stefan Fercot.)
* Add backup type filter to info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)

Improvements:

* Retry on page validation failure during backup. (Reviewed by Stephen Frost, David Christensen.)
* Handle TLS servers that do not close connections gracefully. (Reviewed by Rémi Vidier, David Christensen, Stephen Frost.)
* Add backup LSNs to info command output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Automatically strip trailing slashes for repo-ls paths. (Contributed by David Christensen. Reviewed by David Steele.)
* Do not retry fatal errors. (Reviewed by Reid Thompson.)
* Remove support for PostgreSQL 8.3/8.4. (Reviewed by Reid Thompson, Stefan Fercot.)
* Remove logic that tried to determine additional file system compression. (Reviewed by Reid Thompson, Stefan Fercot.)

Documentation Bug Fixes:

* Move repo options in TLS documentation to the global section. (Reported by Anton Kurochkin.)
* Remove unused backup-standby option from stanza commands. (Reported by Stefan Fercot.)
* Fix typos in help and release notes. (Fixed by Daniel Gustafsson. Reviewed by David Steele.)

Documentation Improvements:

* Add aliveness check to systemd service configuration. (Suggested by Yogesh Sharma.)
* Add FAQ explaining WAL archive suffix. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Note that replications slots are not restored. (Contributed by Reid Thompson. Reviewed by David Steele, Stefan Fercot. Suggested by Christophe Courtois.)
2022-03-06 10:30:59 -06:00
David Steele
59a5373cf8
Handle TLS servers that do not close connections gracefully.
Some TLS server implementations will simply close the socket rather than correctly closing the TLS connection. This causes problems when connection: close is specified with no content-length or chunked encoding and we are forced to read to EOF. It is hard to know if this is a real EOF or a network error.

In cases where we can parse the content and (hopefully) ensure it is correct, allow the closed socket to serve as EOF. This is not ideal, but the change in 8e1807c means that currently working servers with this issue will stop working after 2.35 is installed, which seems too risky.
2022-03-02 11:38:52 -06:00
David Steele
f1bdf3e04b Add aliveness check to systemd service configuration.
If the pgbackrest service service fails to start then the user will get an error. However, retries will continue in the background.
2022-03-02 10:59:06 -06:00
David Steele
a66ec8d549 Revert PGDG yum repo workaround for aarch64.
da0f3a855 used a workaround to get the documentation building on aarch64 but recent changes to the PGDG yum repo have broken this workaround. Installing the regular way still doesn't work, either.

Reverting for now to get the CI pipeline working again.
2022-03-02 10:22:15 -06:00
David Steele
fb5051fde7 Use vagrant user in the Docker container.
This is a bit of legacy from the current Vagrant environment used to do the release, but since it is not as easy to change the user in Vagrant, just make the Docker environment conform.

This allows documentation to be built in a Vagrant environment (or any environment with the same user name) and to be deployed in a Docker environment.
2022-02-26 13:50:30 -06:00
David Steele
f716e98ad0 Suppress errors when there is stderr output for Docker builds.
Docker outputs build info to stderr even when the build is successful. This seems to be especially true on Mac M1.

ContainerTest.pm already does this suppression so add it the other places where containers are built.
2022-02-26 11:31:52 -06:00
David Steele
53de3e3aeb Move repo options in TLS documentation to the global section.
These options were mistakenly added to the stanza section, which works in certain cases, but is not best practice.
2022-02-26 11:17:05 -06:00
Stefan Fercot
98d525dba4
Add FAQ explaining WAL archive suffix. 2022-02-25 14:03:09 -06:00
David Steele
b33cabe08c
Allow case-insensitive matching of HTTP connection header values.
The specification allows values for the connection header to be case-insensitive. See https://www.rfc-editor.org/rfc/rfc7230#section-6.1.
2022-02-25 10:51:40 -06:00
David Christensen
6320712323
Automatically strip trailing slashes for repo-ls paths.
Trailing slashes in at least some of the repository storage types were preventing repo-ls from displaying any content (presumably due to storage-specific behavior).

Since the path with the slash should be equivalent to the path without the slash, just remove it if provided by the user.
2022-02-23 13:53:02 -06:00
David Steele
53f1b25204
Improve validation of zero pages.
Checking that pd_upper == 0 is not enough since this field may be corrupted. Still use pd_upper as a quick check, but when it is zero proceed to check the rest of the page to ensure it is also all zeroes.
2022-02-23 13:17:14 -06:00
David Steele
9eec98c613
Retry on page checksum validation failure during backup.
Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL.

Also remove tests made redundant by the test merge in b4897077.
2022-02-23 12:05:53 -06:00
David Steele
5c332a0b10 Fix release note attribution in cacfdd94 and f798458e. 2022-02-23 11:23:32 -06:00