1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4454 Commits

Author SHA1 Message Date
Stephen Frost
28b6b2d465
Improve performance of SFTP storage driver.
The initial implementation used simple waits when having to loop due to getting a LIBSSH2_ERROR_EAGAIN, but we don't want to just wait some amount of time, we want to wait until we're able to read or write on the fd that we would have blocked on.

This change removes all of the wait code from the SFTP driver and changes the loops to call the newly introduced storageSftpWaitFd(), which in turn checks with libssh2 to determine the appropriate direction to wait on (read, write, or both) and then calls fdReady() to perform the wait using the provided timeout.

This also removes the need to pass ioSession or timeout down into the SFTP read/write code.
2023-07-07 10:36:15 +02:00
David Steele
125676ae0e Cleanup of init/handshake in storageSftpNew().
Rename handshakeStatus to rc to be consistent with the rest of the module.

Add comments and do some reformatting.
2023-07-07 09:56:26 +02:00
David Steele
9efd5cd0bb
Add timezone offset to info command date/time output.
This makes it easier to use timestamps from the info command directly in PostgreSQL recovery settings, especially the --target option.
2023-07-06 18:46:31 +02:00
David Steele
762498f4cd
Fix time-based archive expiration when no backups are expired.
In the case that no backups were expired but time-based retention was met no archive expiration would occur and the following would be logged:

INFO: time-based archive retention not met - archive logs will not be expired

In most cases this was harmless, but when retention was first met or if retention was increased, it would require one additional backup to expire earlier WAL. After that expiration worked as normal.

Even once expiration was working normally the message would continue to be output, which was pretty misleading since retention had been met, even though there was nothing to do.

Bring this code in line with count-based retention, i.e. always log what should be expired at detail level (even if nothing will be expired) and then log info about what was expired (even if nothing is expired). For example:

DETAIL: repo1: 11-1 archive retention on backup 20181119-152138F, start = 000000010000000000000002
INFO: repo1: 11-1 no archive to remove
2023-07-06 11:27:00 +02:00
David Steele
e280ed9098 Make result code handling in storage/sftp more consistent.
Initializers are useless since rc is always set later. Make rc checks consistent with the rest of the module.
2023-07-06 08:58:16 +02:00
David Steele
1fd8845c7f Remove unresolved todo from 87087fac.
Seems easiest just to make the additional config required since it tests that custom ports are being used correctly. The test for synthetic was a noop since SFTP is not used in synthetic tests.
2023-06-29 11:08:58 +02:00
David Steele
0051d7ca87 Update comments for removal of chunking and block numbers.
dd4e526 should have updated this comment but failed to do so.
2023-06-29 09:42:12 +02:00
Andrey Sokolov
0ac09344dc
Fix unique label generation for diff/incr backup.
If there were at least two full backups and the last one was expired, it was impossible to make either a differential or incremental backup without first making a new full backup. The backupLabelCreate() function identified this situation as clock skew because the new backup label was compared with label of the expired full backup.

If the new backup is differential or incremental, then its label is now compared with the labels of differential or incremental backups related to the same full backup.

Also convert a hard-coded date length to a macro.
2023-06-28 18:19:20 +02:00
David Steele
5cbef3ade2 Fix missed memory auditing in FUNCTION_LOG_RETURN_VOID().
9ca492c missed adding auditing to this macro and as a result a few memory leaks have slipped through. Add auditing to the macro to close this hole.

Of the leaks found the only possibly serious one is in blockIncrProcess(), which would leak a PackRead of about eight bytes with every superblock. Since superblocks max out at a few thousand per file this was probably not too bad.

Also change the ordering of auditing in FUNCTION_TEST_RETURN_VOID(). Even though the order does not matter, having it different from the other macros is confusing and looks like an error.
2023-06-25 17:36:57 +02:00
David Steele
ecae001653 Add test for configuration hash type override behavior.
This behavior is different than regular options where a repeated value will result in an error. It appears to be a legacy of the original Perl implementation, which used a hash as the underlying data type in the built-in command-line parser, and the C command-line parser was written to match.
2023-06-24 10:18:29 +02:00
David Steele
434938e32b Update Fedora test image to Fedora 38.
This adds testing for the latest tool versions, e.g. gcc.
2023-06-22 18:23:06 +02:00
David Steele
5531e2662d Add --tz param missed in C unit test migration.
This was missed in the C unit test migration and since then a new test was added that was not setting its timezone correctly.

This feature exists to make sure the tests will run on systems with different timezones and has no impact on the core code.
2023-06-18 12:55:29 +03:00
David Steele
a36f480c85
Clarify preference to install from packages in the user guide. 2023-06-14 14:39:40 +02:00
David Steele
9b1bc7d11b Use new LibXML error handler.
The old error handler has been deprecated.
2023-06-14 12:28:00 +02:00
David Steele
4adf6eed09 Clarify comments for the handling of pg_control during restore. 2023-06-12 23:10:14 +02:00
David Steele
396e237d53
Update NEWS.md for new version and features. 2023-06-12 15:41:39 +02:00
David Steele
6875358d11 Fix typo in "Upgrading PostgreSQL" section of user guide. 2023-06-12 11:26:03 +02:00
David Steele
818ef4b71d Improve comments in protocol/command module. 2023-06-08 16:36:25 +03:00
David Steele
0bf1b4a3b8 Move delete stanza section in user guide.
Deleting a stanza after all the storage driver stanzas were created was causing problems because the SFTP driver is slow and the GCS driver has no server (so it threw errors). This delayed the shutdown of PostgreSQL, which for some reason caused systemctl to hang when the documentation was being built on a RHEL host.

Move the section up and add a comment about why the location is required. Also add a comment to the GCS section about its location.

This does not address the issue of systemctl hanging on RHEL container hosts but it will hopefully make it less common.
2023-05-26 19:50:08 +03:00
David Steele
5d671c63d8 New CI container build for PostgreSQL 16 beta1. 2023-05-25 20:08:12 +03:00
David Steele
9cceed6ac4
Skip recovery.signal for PostgreSQL >= 12 when recovery type=none.
Bring PostgreSQL >= 12 behavior in line with other versions when recovery type=none.

We are fairly sure this did not work correctly when PostgreSQL 12 was released, but apparently the issue has been fixed since then. Either way, after testing we have determined that the behavior is now as expected.
2023-05-24 16:34:21 +03:00
David Steele
36ff81dc6f
Centralize error handling for unsupported features.
Some features are conditionally compiled into pgBackRest (e.g. lz4). Previously checking to see if the feature existed was the responsibility of the feature's module.

Centralize this logic in the config/parse module to make the errors more detailed and consistent.

This also fixes the assert that is thrown when SFTP storage is specified but SFTP support is not compiled into pgBackRest.
2023-05-24 14:17:07 +03:00
David Steele
de46276bf6 Refactor allow list processing in config/parse module.
Combine StringId and int checking into a single loop. This seems more compact and makes it easier to add code that affects both types (and possibly more types in the future).
2023-05-24 09:56:32 +03:00
David Steele
48b26bf569 Fix sftp-all=y in the user guide so it creates a valid configuration.
This was not tested in 87087fac and the generated config was only valid for pushing from the primary. Also do some general cleanup.

Update the SFTP server user to be "pgbackrest" instead of "postgres".

Even though sftp-all=y now creates a valid configuration, the user guide build still fails because SFTP is too slow and operations time out (particularly starting PostgreSQL). This will need to be addressed in a future commit.
2023-05-23 12:50:12 +03:00
David Steele
c633b187db Remove user-facing documentation references to --vm=none.
This parameter is now optional and defaults to none so there is no reason to explicitly show it in user-facing documentation.

Also make the vm parameter in ci.pl optional to be consistent with how test.pl behaves.
2023-05-23 10:58:51 +03:00
David Steele
4ec51cdb2f Explicitly create SSH directory in SFTP setup.
RHEL 9 (at least) will warn on stderr that the directory has been created which causes the documentation build to fail.
2023-05-23 10:40:56 +03:00
David Steele
5bbe987589 Build u20 image to speed contributing document generation.
This image was left out of the last round of builds, which forced the contributing document to build it from scratch.
2023-05-23 10:14:00 +03:00
David Steele
c2c60350d3 Add missed --no-log-timestamp in unit tests and improved formatting.
The --no-log-timestamp option was missed when unit test building was migrated to C, which caused test timings to show up in the contributing guide. This caused no harm but did create churn in this file during releases.

Also improve the formatting when test timing is disabled.
2023-05-23 08:25:17 +03:00
David Steele
9dbf76d8e8 Begin v2.47 development. 2023-05-22 11:19:00 +03:00
David Steele
b461f7c6f8 v2.46: Block Incremental Backup and SFTP Storage
Features:

* Block incremental backup. (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)
* SFTP support for repository storage. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)
* PostgreSQL 16 support. (Reviewed by Stefan Fercot.)

Improvements:

* Allow page header checks to be skipped. (Reviewed by David Christensen. Suggested by David Christensen.)
* Avoid chown() on recovery files during restore. (Reviewed by Stefan Fercot, Marcelo Henrique Neppel. Suggested by Marcelo Henrique Neppel.)
* Add error retry detail for HTTP retries.

Documentation Improvements:

* Add warning about using recovery type=none. (Reviewed by Stefan Fercot.)
* Add note about running stanza-create on already-created repositories.
2023-05-22 11:13:13 +03:00
David Steele
a28b72c624
Add warning about using recovery type=none. 2023-05-18 09:32:44 +03:00
David Steele
c61115e8ad Add bundling and block incremental to stress testing.
This makes the stress testing more robust and provides additional testing for important features.
2023-05-15 17:46:41 +03:00
Reid Thompson
87087fac23
SFTP support for repository storage. 2023-05-13 19:16:16 +03:00
David Steele
0c1f823e7a Add note about running stanza-create on already-created repositories. 2023-05-12 15:46:04 +03:00
David Steele
c7ac173304 Increase check timeouts in real/all integration tests.
The prior timeouts were a bit aggressive and were causing timeouts in the Azure tests. There have also been occasional timeouts in other storage drivers.

The performance of CI environments is pretty variable so increased timeouts should make the tests more stable.
2023-05-10 09:22:26 +03:00
David Steele
2fc6413837 Fix typos in test titles. 2023-05-06 12:16:58 +03:00
David Steele
6f250bf686 Fix groupId type in StorageInfo.
gid_t is the correct type but there was no bug because uid_t and gid_t have the same underlying type.
2023-05-06 12:15:21 +03:00
David Steele
ad9196cb49 Update Debian user guide to build on Ubuntu 20.04.
Ubuntu 18.04 will be EOL at the end of the month so update to a newer version.
2023-05-05 10:08:49 +03:00
David Christensen
023d3c99d8 Fix incorrect commit hash in 2f322e2a. 2023-05-04 08:52:31 +03:00
David Steele
2f322e2a49 Add efac3474 (remove double spacing) to .git-blame-ignore-revs. 2023-05-03 09:49:24 +03:00
David Steele
d49907239e Remove double spaces missed in 1bd5530a. 2023-05-03 09:37:59 +03:00
David Steele
efac3474cf Add 1bd5530a (remove double spacing) to .git-blame-ignore-revs. 2023-05-02 12:59:20 +03:00
David Steele
1bd5530a59 Remove double spaces from comments and documentation.
Double spaces have fallen out of favor in recent years because they no longer contribute to readability.

We have been using single spaces and editing related paragraphs for some time, but now it seems best to update the remaining instances to avoid churn in unrelated commits and to make it clearer what spacing contributors should use.
2023-05-02 12:57:12 +03:00
David Steele
5ea7b91bf9 Remove extraneous space from error messages. 2023-05-02 12:07:28 +03:00
David Steele
9637d94490 Use boolean variant constants where possible instead of redeclaring. 2023-05-02 09:37:42 +03:00
David Steele
822f2a5842
Finalize block incremental feature.
Remove beta status and update documentation to remove beta references and warnings.

The repo-block-* sub-options have been marked internal. Most users will be best off with the default behavior and we may still decide to change these options for remove them in the future.
2023-05-01 19:26:48 +03:00
David Steele
c510046dd7 Remove extraneous spaces in user guide. 2023-05-01 14:27:56 +03:00
David Steele
dd4e52679e
Removing chunking and block numbers from incremental block list.
These were intended to allow the block list to be scanned without reading the map but were never utilized. They were left in "just in case" and because they did not seem to be doing any harm.

In fact, it is better not to have the block numbers because this allows us set the block size at a future time as long as it is a factor of the super block size. One way this could be useful is to store older files without super blocks or a map in the full backup and then build a map for them if the file gets modified in a diff/incr backup. This would require reading the file from the full backup to build the map but it would be more space efficient and we could make more intelligent decisions about block size. It would also be possible to change the block size even if one had already been selected in a prior backup.

Omitting the block numbers makes the chunking unnecessary since there is now no way to make sense of the block list without the map. Also, we might want to build maps for unchunked block lists, i.e. files that were copied normally.
2023-04-27 23:29:12 +03:00
David Steele
3fc3690dd7
PostgreSQL 16 Support.
Add catalog version and WAL magic for PostgreSQL 16.

The GUC to force parallel mode has be renamed so update that in the tests.
2023-04-27 10:30:50 +03:00
David Steele
39059dc4e7 Remove extra linefeed. 2023-04-26 15:00:36 +03:00