1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

4238 Commits

Author SHA1 Message Date
David Steele
6875358d11 Fix typo in "Upgrading PostgreSQL" section of user guide. 2023-06-12 11:26:03 +02:00
David Steele
818ef4b71d Improve comments in protocol/command module. 2023-06-08 16:36:25 +03:00
David Steele
0bf1b4a3b8 Move delete stanza section in user guide.
Deleting a stanza after all the storage driver stanzas were created was causing problems because the SFTP driver is slow and the GCS driver has no server (so it threw errors). This delayed the shutdown of PostgreSQL, which for some reason caused systemctl to hang when the documentation was being built on a RHEL host.

Move the section up and add a comment about why the location is required. Also add a comment to the GCS section about its location.

This does not address the issue of systemctl hanging on RHEL container hosts but it will hopefully make it less common.
2023-05-26 19:50:08 +03:00
David Steele
5d671c63d8 New CI container build for PostgreSQL 16 beta1. 2023-05-25 20:08:12 +03:00
David Steele
9cceed6ac4
Skip recovery.signal for PostgreSQL >= 12 when recovery type=none.
Bring PostgreSQL >= 12 behavior in line with other versions when recovery type=none.

We are fairly sure this did not work correctly when PostgreSQL 12 was released, but apparently the issue has been fixed since then. Either way, after testing we have determined that the behavior is now as expected.
2023-05-24 16:34:21 +03:00
David Steele
36ff81dc6f
Centralize error handling for unsupported features.
Some features are conditionally compiled into pgBackRest (e.g. lz4). Previously checking to see if the feature existed was the responsibility of the feature's module.

Centralize this logic in the config/parse module to make the errors more detailed and consistent.

This also fixes the assert that is thrown when SFTP storage is specified but SFTP support is not compiled into pgBackRest.
2023-05-24 14:17:07 +03:00
David Steele
de46276bf6 Refactor allow list processing in config/parse module.
Combine StringId and int checking into a single loop. This seems more compact and makes it easier to add code that affects both types (and possibly more types in the future).
2023-05-24 09:56:32 +03:00
David Steele
48b26bf569 Fix sftp-all=y in the user guide so it creates a valid configuration.
This was not tested in 87087fac and the generated config was only valid for pushing from the primary. Also do some general cleanup.

Update the SFTP server user to be "pgbackrest" instead of "postgres".

Even though sftp-all=y now creates a valid configuration, the user guide build still fails because SFTP is too slow and operations time out (particularly starting PostgreSQL). This will need to be addressed in a future commit.
2023-05-23 12:50:12 +03:00
David Steele
c633b187db Remove user-facing documentation references to --vm=none.
This parameter is now optional and defaults to none so there is no reason to explicitly show it in user-facing documentation.

Also make the vm parameter in ci.pl optional to be consistent with how test.pl behaves.
2023-05-23 10:58:51 +03:00
David Steele
4ec51cdb2f Explicitly create SSH directory in SFTP setup.
RHEL 9 (at least) will warn on stderr that the directory has been created which causes the documentation build to fail.
2023-05-23 10:40:56 +03:00
David Steele
5bbe987589 Build u20 image to speed contributing document generation.
This image was left out of the last round of builds, which forced the contributing document to build it from scratch.
2023-05-23 10:14:00 +03:00
David Steele
c2c60350d3 Add missed --no-log-timestamp in unit tests and improved formatting.
The --no-log-timestamp option was missed when unit test building was migrated to C, which caused test timings to show up in the contributing guide. This caused no harm but did create churn in this file during releases.

Also improve the formatting when test timing is disabled.
2023-05-23 08:25:17 +03:00
David Steele
9dbf76d8e8 Begin v2.47 development. 2023-05-22 11:19:00 +03:00
David Steele
b461f7c6f8 v2.46: Block Incremental Backup and SFTP Storage
Features:

* Block incremental backup. (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)
* SFTP support for repository storage. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)
* PostgreSQL 16 support. (Reviewed by Stefan Fercot.)

Improvements:

* Allow page header checks to be skipped. (Reviewed by David Christensen. Suggested by David Christensen.)
* Avoid chown() on recovery files during restore. (Reviewed by Stefan Fercot, Marcelo Henrique Neppel. Suggested by Marcelo Henrique Neppel.)
* Add error retry detail for HTTP retries.

Documentation Improvements:

* Add warning about using recovery type=none. (Reviewed by Stefan Fercot.)
* Add note about running stanza-create on already-created repositories.
2023-05-22 11:13:13 +03:00
David Steele
a28b72c624
Add warning about using recovery type=none. 2023-05-18 09:32:44 +03:00
David Steele
c61115e8ad Add bundling and block incremental to stress testing.
This makes the stress testing more robust and provides additional testing for important features.
2023-05-15 17:46:41 +03:00
Reid Thompson
87087fac23
SFTP support for repository storage. 2023-05-13 19:16:16 +03:00
David Steele
0c1f823e7a Add note about running stanza-create on already-created repositories. 2023-05-12 15:46:04 +03:00
David Steele
c7ac173304 Increase check timeouts in real/all integration tests.
The prior timeouts were a bit aggressive and were causing timeouts in the Azure tests. There have also been occasional timeouts in other storage drivers.

The performance of CI environments is pretty variable so increased timeouts should make the tests more stable.
2023-05-10 09:22:26 +03:00
David Steele
2fc6413837 Fix typos in test titles. 2023-05-06 12:16:58 +03:00
David Steele
6f250bf686 Fix groupId type in StorageInfo.
gid_t is the correct type but there was no bug because uid_t and gid_t have the same underlying type.
2023-05-06 12:15:21 +03:00
David Steele
ad9196cb49 Update Debian user guide to build on Ubuntu 20.04.
Ubuntu 18.04 will be EOL at the end of the month so update to a newer version.
2023-05-05 10:08:49 +03:00
David Christensen
023d3c99d8 Fix incorrect commit hash in 2f322e2a. 2023-05-04 08:52:31 +03:00
David Steele
2f322e2a49 Add efac3474 (remove double spacing) to .git-blame-ignore-revs. 2023-05-03 09:49:24 +03:00
David Steele
d49907239e Remove double spaces missed in 1bd5530a. 2023-05-03 09:37:59 +03:00
David Steele
efac3474cf Add 1bd5530a (remove double spacing) to .git-blame-ignore-revs. 2023-05-02 12:59:20 +03:00
David Steele
1bd5530a59 Remove double spaces from comments and documentation.
Double spaces have fallen out of favor in recent years because they no longer contribute to readability.

We have been using single spaces and editing related paragraphs for some time, but now it seems best to update the remaining instances to avoid churn in unrelated commits and to make it clearer what spacing contributors should use.
2023-05-02 12:57:12 +03:00
David Steele
5ea7b91bf9 Remove extraneous space from error messages. 2023-05-02 12:07:28 +03:00
David Steele
9637d94490 Use boolean variant constants where possible instead of redeclaring. 2023-05-02 09:37:42 +03:00
David Steele
822f2a5842
Finalize block incremental feature.
Remove beta status and update documentation to remove beta references and warnings.

The repo-block-* sub-options have been marked internal. Most users will be best off with the default behavior and we may still decide to change these options for remove them in the future.
2023-05-01 19:26:48 +03:00
David Steele
c510046dd7 Remove extraneous spaces in user guide. 2023-05-01 14:27:56 +03:00
David Steele
dd4e52679e
Removing chunking and block numbers from incremental block list.
These were intended to allow the block list to be scanned without reading the map but were never utilized. They were left in "just in case" and because they did not seem to be doing any harm.

In fact, it is better not to have the block numbers because this allows us set the block size at a future time as long as it is a factor of the super block size. One way this could be useful is to store older files without super blocks or a map in the full backup and then build a map for them if the file gets modified in a diff/incr backup. This would require reading the file from the full backup to build the map but it would be more space efficient and we could make more intelligent decisions about block size. It would also be possible to change the block size even if one had already been selected in a prior backup.

Omitting the block numbers makes the chunking unnecessary since there is now no way to make sense of the block list without the map. Also, we might want to build maps for unchunked block lists, i.e. files that were copied normally.
2023-04-27 23:29:12 +03:00
David Steele
3fc3690dd7
PostgreSQL 16 Support.
Add catalog version and WAL magic for PostgreSQL 16.

The GUC to force parallel mode has be renamed so update that in the tests.
2023-04-27 10:30:50 +03:00
David Steele
39059dc4e7 Remove extra linefeed. 2023-04-26 15:00:36 +03:00
David Steele
3ff88ffbb4
Avoid chown() on recovery files during restore.
The chown() was already skipped on the files restored from the repository but the same logic was not applied to the generated recovery files, probably because chown'ing a few recovery files does not have performance implications. Use the same logic for recovery files to determined if they need to be chown'd.

Ultimately this behavior is pretty hard to test, so add a fail safe into the Posix driver that will skip chown if the permissions are already as required.
2023-04-25 11:52:28 +03:00
David Steele
750ab8e55c Add MacOS back to CirrusCI.
9e29c01 removed MacOS testing due to breaking changes in the update to arm on the platform.

Update the scripts to correctly work with the version of Homebrew deployed with the arm images.
2023-04-21 16:21:10 +03:00
David Steele
23d7d67d49 Fix missing void parameter. 2023-04-21 15:08:59 +03:00
David Steele
f5e6bc2698
Allow page header checks to be skipped.
These checks cause false negatives for page checksum verification when the page is encrypted because pd_upper might end up as 0 in the encrypted data. This issue is rare but reproducible given a large enough cluster.

Make these checks optional, but leave them enabled by default.
2023-04-20 13:24:12 +03:00
David Steele
8240eb5da5 Autogenerate PostgreSQL versions.
This will make adding/removing versions of PostgreSQL more reliable.
2023-04-16 17:41:27 +03:00
David Steele
a05bf6bb15 Rename PG_VERSION_*_STR constants to PG_VERSION_*_Z.
This is more consistent with other zero-terminated string constants and also has the benefit of being shorter.
2023-04-16 17:32:24 +03:00
David Steele
75254c9285 Parameterize configLoad() as cfgLoadP().
There is one existing optional parameter and there are more to come.
2023-04-11 16:28:29 +04:00
David Steele
d6cb3de17a Update command/check module to recent coding standards.
Add const as appropriate and remove an unneeded declaration.
2023-04-11 14:50:59 +04:00
David Steele
a9c5cd9749 Update command/backup module to recent coding standards.
Add const as appropriate and avoid setting results to NULL if the result will definitely be set later on.
2023-04-11 12:46:00 +04:00
David Steele
f33e1cfb16 Add error retry detail for HTTP retries.
This should make it clearer when retries have happened and for how long.
2023-04-06 11:38:18 +04:00
David Steele
df419b34b5 Improve retry error messages.
Centralize the code to allow it to be used in more places and update the protocol/server module to use the new code.

Since the time measurements make testing difficult, also add time and errorRetry harnesses to allow specific data to be used for testing. In the case of errorRetry, the production behavior is turned off by default during testing and only enabled for the errorRetry test module.
2023-04-06 11:03:46 +04:00
David Steele
801e396dac Move error modules to common/error directory.
There are already enough files to warrant a separate directory and more are coming.

Also remove extraneous includes.
2023-04-06 10:38:49 +04:00
David Steele
ff98636e41 Update 32-bit CI host to Debian 10 from Ubuntu 18.04.
Ubuntu 18.04 will be EOL before the next release, so update to the oldest available Debian version.

Also fix one incorrect return value type, a test cast, and adjust some test timeouts.
2023-04-06 08:22:14 +04:00
David Steele
a9f39857cf Remove unused strOS parameter in Common::HostTest. 2023-04-05 11:24:53 +04:00
David Steele
8f7f73e4af Sleep using nanosleep() instead of select().
This is a safer way to sleep due to select's not-portable interaction with signals.

Based on https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a948e49e2ef11815be0b211723bfc5b53b7f75a8 from the PostgreSQL project.
2023-03-30 17:46:25 +05:00
David Steele
2cfbee903a Use Buffer object for page buffer in PageChecksum filter.
This avoids the object needing to support allocations.
2023-03-30 15:43:37 +05:00