1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4370 Commits

Author SHA1 Message Date
David Steele
995a8e9669
Reload GCS credentials before renewing authentication token.
This allows the service key to be updated while a command is running. The new key should be written atomically and ideally the old key should remain valid for some period of time to avoid a race condition if the old token happens to expire at the same time that the new key is being generated.
2023-08-07 13:30:50 +01:00
David Steele
cbafcfabf2 Remove UNCONSTIFY() from gz compression module.
The ZLIB_CONST macro may be used since 1.2.5.2 (17 Dec 2011) to specify that the input buffers are const. This is sufficiently old to cover all non-EOL distributions, i.e. everything we test on.

Compiling on older distributions may generate warnings but will continue to work.
2023-08-07 12:38:33 +01:00
David Steele
6c75c99d0b Update xxHash to v0.8.2.
This release fixes compilation errors on some platforms and contains numerous performance improvements.
2023-07-31 11:09:17 +02:00
David Steele
f55c8f1b81 Rename HRN_PG_CONTROL_OVERRIDE_PUT().
Rename to HRN_PG_CONTROL_OVERRIDE_VERSION_PUT() since other types of overrides are possible (and coming).
2023-07-26 10:45:12 +02:00
David Steele
65fb3b3b2d Update config.guess and config.sub to latest versions. 2023-07-24 09:17:46 +02:00
David Steele
c585ddaf48 Begin v2.48 development. 2023-07-24 09:15:38 +02:00
David Steele
b5b033cc1c v2.47: Performance Improvements and Bug Fixes
Bug Fixes:

* Preserve block incremental info in manifest during delta backup. (Reviewed by Stephen Frost. Reported by Francisco Miguel Biete Banon.)
* Fix block incremental file names in verify command. (Reviewed by Reid Thompson. Reported by Francisco Miguel Biete Banon.)
* Fix spurious automatic delta backup on backup from standby. (Reviewed by Stephen Frost. Reported by krmozejko, Don Seiler.)
* Skip recovery.signal for PostgreSQL >= 12 when recovery type=none. (Reviewed by Stefan Fercot. Reported by T.Anastacio.)
* Fix unique label generation for diff/incr backup. (Fixed by Andrey Sokolov. Reviewed by David Steele.)
* Fix time-based archive expiration when no backups are expired. (Reviewed by Stefan Fercot.)

Improvements:

* Improve performance of SFTP storage driver. (Contributed by Stephen Frost, Reid Thompson. Reviewed by David Steele.)
* Add timezone offset to info command date/time output. (Reviewed by Stefan Fercot, Philip Hurst. Suggested by Philip Hurst.)
* Centralize error handling for unsupported features. (Reviewed by Stefan Fercot.)

Documentation Improvements:

* Clarify preference to install from packages in the user guide. (Reviewed by Stefan Fercot. Suggested by dr-kd.)
2023-07-24 09:12:30 +02:00
David Steele
5ed6f8df14
Fix spurious automatic delta backup on backup from standby.
When performing backup from standby the file sizes on the standby may not be equal to file sizes on the primary. This is because replication continues during the backup and by the time the file is copied from the standby it may have changed. Since we cap the size of all files copied from the standby this practically applies to truncation and in particular truncation of free space maps (at least, every case we have seen so far is an fsm). Free space maps are especially vulnerable since they are only partially replicated, which amplifies the difference between the primary and standby.

On an incremental backup it may look like the size has changed on the primary (because of the final size recorded by the standby in the prior backup) but the timestamp may not have changed on the primary and this will trigger a checksum delta for safety. While this has no impact on backup integrity, checksum delta incrementals can run much longer than regular incrementals and backup schedules may be impacted.

The solution is to preserve the original size in the manifest and use it to do the time/size check. In the case of backup from standby the original size will always be the size on the primary, which makes comparisons against subsequent file sizes on the primary consistent. Original size is only stored in the manifest when it differs from final size, so there should not be any noticeable manifest bloat.
2023-07-18 07:35:12 +02:00
David Steele
4c27d74bbd
Preserve block incremental info in manifest during delta backup.
It was possible for block incremental info to be lost if a file had been modified in such a way that block incremental would be disabled if the file were new, e.g. if the file shrank below the block incremental limit or the file timestamp regressed far enough into the past. In those cases the block incremental info would not be copied in manifestBuildIncr().

Instead always copy the block incremental info in case the file ends up being referenced to a prior backup.

The validation tests were not robust enough to catch this issue so they were improved in 1d42aed.

In the particular case that exposed this bug, a file had a timestamp that was almost four weeks in the past at full backup time. A few days later a fail over occurred and the next incremental ran on the new primary (old standby) in delta mode. The same file had a timestamp just a few hours older than in the full backup, but now four weeks older than the current backup. Block incremental was disabled for the file on initial manifest build because of its age, which meant the block incremental info was not copied into the new manifest. The delta then determined the file had not changed and referenced it to the full backup. On restore, the file appeared to be a normal file stored in a bundle but could not be decompressed because it was in fact a block incremental.
2023-07-17 22:37:02 +02:00
David Steele
cdb7e26350
Fix block incremental file names in verify command.
The verify command was not appending the .pgbi extension instead of the compression extension when verifying block incremental files stored outside a bundle.

Originally the idea was that verify would not need any changes (since it just examines repo-size and checksum) but at some point the new extension was added and broke that assumption.

Use backupFileRepoPathP() to generate the correct filename (Just like backup, restore, etc).
2023-07-14 17:53:58 +03:00
David Steele
1d42aed152 Improve validation of referenced files in command/backup unit test.
Referenced files were not being checked for validity unless they were hard linked to the current backup (which a lot of the tests did). Newer tests with bundling do not have hard links and so missed these checks.

Improve the validation code to check referenced files in the manifest even when they are not hard linked into the current backup.

Add a delta test for bundling/block incremental that includes a file old enough to get a block size of zero. These are good tests by themselves but they also reduce the churn in an upcoming bug fix.
2023-07-14 11:18:25 +03:00
David Steele
c9703b3530 Log when file size changes during backup.
It is possible for the size of a file to change during the backup. In most cases we won't notice since files sizes are usually capped but it is possible for some files to grow or shrink between when they are recorded in the manifest and when they are actually copied. The new size is recorded in the manifest but the old size may be useful for debugging.

The new code has coverage but no test changes because it is covered by the parallel backup testing, which does not have deterministic log output. It doesn't seem worth creating a new test to check the log format as it is not very critical (though the output was manually verified).
2023-07-13 13:47:27 +03:00
David Steele
b6b13bd634 Mention block-level backups in feature list. 2023-07-12 13:23:16 +03:00
David Steele
c8afbed530 Increase protocolServerError() logging to debug level.
Errors should be rare enough that it makes sense to log them at debug level. Right now if there is an error if won't be logged at debug level, which makes it harder to tell why the main process may have terminated the local/remote process.
2023-07-12 13:09:34 +03:00
David Steele
06536b5814 Rename macros in command/verify unit test.
These macros were redefined, which worked since they were identical to their prior definitions, but this will not always be true.
2023-07-12 10:20:09 +03:00
David Steele
aa229a1dee Modify time_t casts based on complaints from Coverity.
Coverity complained about time_t being cast directly to unsigned int, so instead cast the result of the operation.

We are confident in both cases that the time_t values will not be out of unsigned int range but Coverity has no way to know that.

One of these is new (introduced by 9efd5cd0) but the other one (from a9867cb0) remained unnoticed for a while, though it has not caused any production impact.
2023-07-09 21:56:05 +03:00
Stephen Frost
28b6b2d465
Improve performance of SFTP storage driver.
The initial implementation used simple waits when having to loop due to getting a LIBSSH2_ERROR_EAGAIN, but we don't want to just wait some amount of time, we want to wait until we're able to read or write on the fd that we would have blocked on.

This change removes all of the wait code from the SFTP driver and changes the loops to call the newly introduced storageSftpWaitFd(), which in turn checks with libssh2 to determine the appropriate direction to wait on (read, write, or both) and then calls fdReady() to perform the wait using the provided timeout.

This also removes the need to pass ioSession or timeout down into the SFTP read/write code.
2023-07-07 10:36:15 +02:00
David Steele
125676ae0e Cleanup of init/handshake in storageSftpNew().
Rename handshakeStatus to rc to be consistent with the rest of the module.

Add comments and do some reformatting.
2023-07-07 09:56:26 +02:00
David Steele
9efd5cd0bb
Add timezone offset to info command date/time output.
This makes it easier to use timestamps from the info command directly in PostgreSQL recovery settings, especially the --target option.
2023-07-06 18:46:31 +02:00
David Steele
762498f4cd
Fix time-based archive expiration when no backups are expired.
In the case that no backups were expired but time-based retention was met no archive expiration would occur and the following would be logged:

INFO: time-based archive retention not met - archive logs will not be expired

In most cases this was harmless, but when retention was first met or if retention was increased, it would require one additional backup to expire earlier WAL. After that expiration worked as normal.

Even once expiration was working normally the message would continue to be output, which was pretty misleading since retention had been met, even though there was nothing to do.

Bring this code in line with count-based retention, i.e. always log what should be expired at detail level (even if nothing will be expired) and then log info about what was expired (even if nothing is expired). For example:

DETAIL: repo1: 11-1 archive retention on backup 20181119-152138F, start = 000000010000000000000002
INFO: repo1: 11-1 no archive to remove
2023-07-06 11:27:00 +02:00
David Steele
e280ed9098 Make result code handling in storage/sftp more consistent.
Initializers are useless since rc is always set later. Make rc checks consistent with the rest of the module.
2023-07-06 08:58:16 +02:00
David Steele
1fd8845c7f Remove unresolved todo from 87087fac.
Seems easiest just to make the additional config required since it tests that custom ports are being used correctly. The test for synthetic was a noop since SFTP is not used in synthetic tests.
2023-06-29 11:08:58 +02:00
David Steele
0051d7ca87 Update comments for removal of chunking and block numbers.
dd4e526 should have updated this comment but failed to do so.
2023-06-29 09:42:12 +02:00
Andrey Sokolov
0ac09344dc
Fix unique label generation for diff/incr backup.
If there were at least two full backups and the last one was expired, it was impossible to make either a differential or incremental backup without first making a new full backup. The backupLabelCreate() function identified this situation as clock skew because the new backup label was compared with label of the expired full backup.

If the new backup is differential or incremental, then its label is now compared with the labels of differential or incremental backups related to the same full backup.

Also convert a hard-coded date length to a macro.
2023-06-28 18:19:20 +02:00
David Steele
5cbef3ade2 Fix missed memory auditing in FUNCTION_LOG_RETURN_VOID().
9ca492c missed adding auditing to this macro and as a result a few memory leaks have slipped through. Add auditing to the macro to close this hole.

Of the leaks found the only possibly serious one is in blockIncrProcess(), which would leak a PackRead of about eight bytes with every superblock. Since superblocks max out at a few thousand per file this was probably not too bad.

Also change the ordering of auditing in FUNCTION_TEST_RETURN_VOID(). Even though the order does not matter, having it different from the other macros is confusing and looks like an error.
2023-06-25 17:36:57 +02:00
David Steele
ecae001653 Add test for configuration hash type override behavior.
This behavior is different than regular options where a repeated value will result in an error. It appears to be a legacy of the original Perl implementation, which used a hash as the underlying data type in the built-in command-line parser, and the C command-line parser was written to match.
2023-06-24 10:18:29 +02:00
David Steele
434938e32b Update Fedora test image to Fedora 38.
This adds testing for the latest tool versions, e.g. gcc.
2023-06-22 18:23:06 +02:00
David Steele
5531e2662d Add --tz param missed in C unit test migration.
This was missed in the C unit test migration and since then a new test was added that was not setting its timezone correctly.

This feature exists to make sure the tests will run on systems with different timezones and has no impact on the core code.
2023-06-18 12:55:29 +03:00
David Steele
a36f480c85
Clarify preference to install from packages in the user guide. 2023-06-14 14:39:40 +02:00
David Steele
9b1bc7d11b Use new LibXML error handler.
The old error handler has been deprecated.
2023-06-14 12:28:00 +02:00
David Steele
4adf6eed09 Clarify comments for the handling of pg_control during restore. 2023-06-12 23:10:14 +02:00
David Steele
396e237d53
Update NEWS.md for new version and features. 2023-06-12 15:41:39 +02:00
David Steele
6875358d11 Fix typo in "Upgrading PostgreSQL" section of user guide. 2023-06-12 11:26:03 +02:00
David Steele
818ef4b71d Improve comments in protocol/command module. 2023-06-08 16:36:25 +03:00
David Steele
0bf1b4a3b8 Move delete stanza section in user guide.
Deleting a stanza after all the storage driver stanzas were created was causing problems because the SFTP driver is slow and the GCS driver has no server (so it threw errors). This delayed the shutdown of PostgreSQL, which for some reason caused systemctl to hang when the documentation was being built on a RHEL host.

Move the section up and add a comment about why the location is required. Also add a comment to the GCS section about its location.

This does not address the issue of systemctl hanging on RHEL container hosts but it will hopefully make it less common.
2023-05-26 19:50:08 +03:00
David Steele
5d671c63d8 New CI container build for PostgreSQL 16 beta1. 2023-05-25 20:08:12 +03:00
David Steele
9cceed6ac4
Skip recovery.signal for PostgreSQL >= 12 when recovery type=none.
Bring PostgreSQL >= 12 behavior in line with other versions when recovery type=none.

We are fairly sure this did not work correctly when PostgreSQL 12 was released, but apparently the issue has been fixed since then. Either way, after testing we have determined that the behavior is now as expected.
2023-05-24 16:34:21 +03:00
David Steele
36ff81dc6f
Centralize error handling for unsupported features.
Some features are conditionally compiled into pgBackRest (e.g. lz4). Previously checking to see if the feature existed was the responsibility of the feature's module.

Centralize this logic in the config/parse module to make the errors more detailed and consistent.

This also fixes the assert that is thrown when SFTP storage is specified but SFTP support is not compiled into pgBackRest.
2023-05-24 14:17:07 +03:00
David Steele
de46276bf6 Refactor allow list processing in config/parse module.
Combine StringId and int checking into a single loop. This seems more compact and makes it easier to add code that affects both types (and possibly more types in the future).
2023-05-24 09:56:32 +03:00
David Steele
48b26bf569 Fix sftp-all=y in the user guide so it creates a valid configuration.
This was not tested in 87087fac and the generated config was only valid for pushing from the primary. Also do some general cleanup.

Update the SFTP server user to be "pgbackrest" instead of "postgres".

Even though sftp-all=y now creates a valid configuration, the user guide build still fails because SFTP is too slow and operations time out (particularly starting PostgreSQL). This will need to be addressed in a future commit.
2023-05-23 12:50:12 +03:00
David Steele
c633b187db Remove user-facing documentation references to --vm=none.
This parameter is now optional and defaults to none so there is no reason to explicitly show it in user-facing documentation.

Also make the vm parameter in ci.pl optional to be consistent with how test.pl behaves.
2023-05-23 10:58:51 +03:00
David Steele
4ec51cdb2f Explicitly create SSH directory in SFTP setup.
RHEL 9 (at least) will warn on stderr that the directory has been created which causes the documentation build to fail.
2023-05-23 10:40:56 +03:00
David Steele
5bbe987589 Build u20 image to speed contributing document generation.
This image was left out of the last round of builds, which forced the contributing document to build it from scratch.
2023-05-23 10:14:00 +03:00
David Steele
c2c60350d3 Add missed --no-log-timestamp in unit tests and improved formatting.
The --no-log-timestamp option was missed when unit test building was migrated to C, which caused test timings to show up in the contributing guide. This caused no harm but did create churn in this file during releases.

Also improve the formatting when test timing is disabled.
2023-05-23 08:25:17 +03:00
David Steele
9dbf76d8e8 Begin v2.47 development. 2023-05-22 11:19:00 +03:00
David Steele
b461f7c6f8 v2.46: Block Incremental Backup and SFTP Storage
Features:

* Block incremental backup. (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)
* SFTP support for repository storage. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)
* PostgreSQL 16 support. (Reviewed by Stefan Fercot.)

Improvements:

* Allow page header checks to be skipped. (Reviewed by David Christensen. Suggested by David Christensen.)
* Avoid chown() on recovery files during restore. (Reviewed by Stefan Fercot, Marcelo Henrique Neppel. Suggested by Marcelo Henrique Neppel.)
* Add error retry detail for HTTP retries.

Documentation Improvements:

* Add warning about using recovery type=none. (Reviewed by Stefan Fercot.)
* Add note about running stanza-create on already-created repositories.
2023-05-22 11:13:13 +03:00
David Steele
a28b72c624
Add warning about using recovery type=none. 2023-05-18 09:32:44 +03:00
David Steele
c61115e8ad Add bundling and block incremental to stress testing.
This makes the stress testing more robust and provides additional testing for important features.
2023-05-15 17:46:41 +03:00
Reid Thompson
87087fac23
SFTP support for repository storage. 2023-05-13 19:16:16 +03:00
David Steele
0c1f823e7a Add note about running stanza-create on already-created repositories. 2023-05-12 15:46:04 +03:00