1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-07 00:35:37 +02:00
Commit Graph

157 Commits

Author SHA1 Message Date
2706168f60 Add progress-only detail level for info command output.
The info command fetches a lot of information from the repository about backups and archives, so this operation can be slow. Because progress data is stored in local lock files, accessing the repository is unnecessary when only progress information is required.

This patch introduces the `--detail-level=[progress|full]` option, with `full` as the default. The `progress` level limits the info command output to progress details without querying the repository. The only remaining operations are scanning the folder structure to list available stanzas and reading lock files.

Note: When `progress` is selected, the info command performs no checks beyond verifying stanza availability.
2025-06-26 17:40:05 -04:00
638815b4cd Fix issue with adhoc expiration when no backups in a repository.
If there are no backups in one or more repositories then the following error occurs during adhoc expiration:

ASSERT: [025]: cannot get index 0 from list with 0 value(s)

Fix this by skipping the adhoc logic when there are no backups in a repository.
2025-06-24 12:35:18 -04:00
5e14692588 Retry failed reads on object stores.
There is currently a retry if the initial get request fails (depending on the error code) but if the read fails later on while fetching blocks of data it is fatal. In most cases there is a higher level retry (e.g. restore) but restarting the restore job might be expensive depending on how many files are being restored.

Add a retry that will catch read errors and retry from where the last data was successfully read.

A bit of history -- this patch was first started three years ago but the memory context model at that time would not allow the interface (StorageRead) to own the driver (e.g. StorageReadS3). Subsequent improvements in memory contexts have allowed this ownership model and in fact it is now the default so no ownership changes are required in this patch except in StorageReadRemote which was not updated in f6e3073.
2025-06-16 16:25:50 -04:00
2e76e9fc69 Describe discrete option values in a list where appropriate.
In the documentation and help output the possible values for most options are described as a list but there were several commands for which formatting was done in a different style.

Formatting has been applied to commands and their options to display option values as a list:

* `info` command, `--type` option
* `verify` command, `--output` option
* `version` command, `--output` option
* `manifest` command, `--output` option
2025-06-09 10:34:17 -04:00
ad7ba46bc6 Improve the predictability of floating point numbers formatted as strings.
Some C libraries (e.g.musl) render floating point numbers differently when using printf(). This does not cause any problems for the core code but the unit tests require more predictability to function smoothly without a lot of exceptions.

To accomplish this, wrap the various floating point operations in functions that mostly continue doing math using double but format the output string using integers. This leads to more predictable output at the cost of some complexity.

Rounding could also be accomplished using nearbyint() but this would require linking the math library, which does not seem worth it for a fairly simple operation.
2025-06-05 13:27:45 -04:00
cb673225e9 Fix "less than" in help output for archive-mode option. 2025-05-28 15:33:38 -04:00
8737cad566 Fix defaults in command-line help.
This is another attempt to fix broken defaults in the command-line help. PR #2395 tried to do this by removing current and default values (at least from the list) but this removed much of the value of command-line help being context sensitive. Improvements were made to current values in b3ca2e34 but defaults were left as-is, i.e. broken in some cases.

This new approach handles defaults much the same way as current values. If there are multiple defaults for an option then it is displayed in the option list as <multi> and enumerated in the specific option help. For example, if repo1-type=s3 and repo2-type=azure then the defaults for repo1-storage-upload-chunk-size and repo2-storage-upload-chunk-size will differ.

However, getting the above right means we can no longer display defaults for options that are not valid in the current context. For instance, if repo1-type=posix then none of the repo1-s3-* options are valid and therefore their defaults are not displayed in help. In this case they don't have defaults and if you tried to set one of these options you would get an error.
2025-05-12 13:33:42 -04:00
e38e3e0090 Merge v2.55.1 release. 2025-05-05 13:40:14 -04:00
96dfdce085 Revert "calculate content-md5 on S3 only when required".
20bfd14 removed content-md5 where allowed by the specification but failed to notice that either content-md5 or x-amz-content-* is required for PUT when object lock is enabled.

On top of that it appears Scality S3 (at least?) won't accept alternate content checksums when object lock is enabled. Technically this is a violation of the specification but nonetheless the change breaks working installations.

For now it seems safer to revert this change and pursue a better solution for a future feature release.
2025-04-29 15:12:11 -04:00
9958b17834 Fix lower bounds checking for option keys.
Specifying an indexed option with a key of 0, e.g. pg0-path, would lead to a segfault.

Add lower bounds checking to fix the issue.
2025-04-28 14:33:56 -04:00
01c8c0f864 Begin v2.56.0 development. 2025-04-21 18:51:50 -04:00
6c90196e74 v2.55.0: Verification Improvements and PostgreSQL 18 Support
Bug Fixes:

* Fix block incremental restore issue on non-default repository. (Reviewed by David Christensen, Aleksander Łukasz. Reported by Aleksander Łukasz.)
* Do not set recovery_target_timeline=current for PostgreSQL < 12. (Reviewed by Stefan Fercot.)
* Fix expire archive range logging. (Reviewed by Stefan Fercot. Reported by Aleš Zelený.)
* Fix error reporting for queries with no results. (Reviewed by Stefan Fercot. Reported by Susantha Bathige.)

Features:

* Verify recovery target timeline. (Reviewed by Stefan Fercot.)
* Allow verification of a specified backup. (Contributed by Maxim Michkov. Reviewed by David Steele.)
* Add support for S3/GCS requester pays. (Contributed by Timothée Peignier. Reviewed by David Steele.)
* PostgreSQL 18 experimental support. (Reviewed by Stefan Fercot.)
* Allow connections to PostgreSQL on abstract domain sockets. (Reviewed by Chris Bandy. Suggested by Chris Bandy.)
* Add numeric output to version command. (Contributed by Stefan Fercot. Reviewed by David Steele.)

Improvements:

* Allow backup command to operate on remote repositories. (Reviewed by Stefan Fercot.)
* Use lz4 for protocol compression. (Reviewed by Stefan Fercot.)
* Calculate content-md5 on S3 only when required. (Reviewed by David Christensen.)
* Warn when a value for a multi-key option is overwritten. (Reviewed by David Christensen, Stefan Fercot.)
* Add detail logging for expired archive path. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Remove support for PostgreSQL 9.4. (Reviewed by Stefan Fercot.)
* Remove autoconf/make build. (Reviewed by David Christensen.)

Documentation Improvements:

* Fix documentation for specifying multiple stanzas with tls-server-auth. (Reviewed by David Christensen, Stefan Fercot. Suggested by Terry MacAndrew.)
* Clarify incremental backup expiration. (Reviewed by Stefan Fercot.)
* Clarify requirement for local/remote pgBackRest versions to match. (Contributed by Greg Clough. Reviewed by David Steele.)
* Add FAQ about exporting self-contained cluster. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Caveat --tablespace-map-all regarding tablespace creation. (Reviewed by Stefan Fercot, Christophe Courtois. Suggested by Christophe Courtois.)
* Clarify behavior of --repo-retention-full-type. (Reviewed by Antoine Beaupré. Suggested by Antoine Beaupré.)
* Change --process-max recommendation for object stores to --repo-bundle. (Reviewed by Stefan Fercot.)
* Update unix_socket_directory to unix_socket_directories. (Contributed by hyunkyu han. Reviewed by David Steele.)
* Recommend not placing spool-path within pg_xlog/pg_wal. (Reviewed by Martín Marqués, Don Seiler. Suggested by Martín Marqués.)
2025-04-21 18:25:00 -04:00
4bb7cb1ac6 Fix incorrect not-equal comparison in TEST_RESULT_Z_NE().
This comparison was not being used for strings in the tests but best to fix it so it does not cause confusion in the future.
2025-04-21 09:02:56 -04:00
9386920b79 Add FAQ about exporting self-contained cluster. 2025-04-16 12:26:15 -05:00
8bdba74798 Clarify incremental backup expiration.
The documentation was a bit misleading regarding how incremental backups are expired. Update the misleading part ("Differential and Incremental backups are count-based...") and move the explanation of how incremental expiration works out of differential expiration into the introductory paragraph.

Also add a note about how full backups are considered as differential for the purpose of expiration.
2025-04-14 13:34:35 -05:00
565d2e0c47 PostgreSQL 18 experimental support.
Support is experimental since PostgreSQL 18 is still in development and has not released a beta, but it may be useful for testing.
2025-04-14 12:29:31 -05:00
dd3a7d99a7 Recommend not placing spool-path within pg_xlog/pg_wal.
This is generally not a good idea and it can cause problems for PostgreSQL tools such as pg_rewind.
2025-04-10 12:08:35 -05:00
cab4d6fd5c Warn when a value for a multi-key option is overwritten.
Some options can contain multiple key/value pairs. However, if if the key is specified again the value will be silently overwritten. In most cases one value per key is appropriate, but it makes sense to warn the user about the overwrite.
2025-04-10 10:26:50 -05:00
68f22aea66 Allow connections to PostgreSQL on abstract domain sockets.
Currently the pg-socket-path option is required to be a valid absolute path but this precludes the use of abstract domain sockets.

Set the option type to string so abstract domain sockets are allowed. This removes some validation but libpq will report if the path is invalid and we don't use it for any other purpose.
2025-04-09 14:50:16 -05:00
20bfd14b73 Calculate content-md5 on S3 only when required.
The content-md5 header was generated for all requests with content but it is only required for batch delete requests. It is not clear why this header is required when x-amz-content-sha256 is also provided or why it
is required only for this request but the documentation is clear on the matter. However, the content for these requests is relatively small compared to uploading files so omitting content-md5 where possible will
save some CPU cycles.

Current AWS S3 and recent Minio don't complain if this header is missing but since it is still required by older versions of Minio and it is specified in the documentation for batch delete it is makes sense to
keep it.
2025-04-09 12:27:27 -05:00
53a45d25c3 Fix documentation for specifying multiple stanzas with tls-server-auth.
The prior documentation said that multiple stanzas should be specified by repeating the tls-server-auth option. This is incorrect -- in fact a comma-separated list of stanza should be used.

Fix the documentation and add a test to confirm this behavior.

In passing add some const qualifiers that were missing in the relevant code.
2025-04-08 10:45:06 -05:00
e18ca19895 Remove extraneous const from cvtIntToZ() prototype. 2025-04-04 10:26:03 -05:00
aeea81a388 Fix test logging in storageHelperDryRunInit(). 2025-04-04 10:07:32 -05:00
cc50e1aecd Revert "full/incremental backup method".
This method was introduced in cad595f but on further reflection it does not seem worth the added complexity just to make restore consistency faster without improving the speed of the overall backup. The most common recovery case is PITR and this method produces diminishing returns as the recovery time gets further from the backup end time.

A better solution (not implemented here) is to copy unmodified files from prior backups. This is much faster than recopying and compressing files from the cluster (especially on object stores with a copy command) and can even be done after the backup window to further reduce WAL replay required for consistency. It also reduces load on the host where the backup is made.
2025-04-02 15:25:23 -05:00
dcfe30e322 Fix instances of "of of". 2025-03-31 14:18:29 -05:00
b28ae98cec Update unix_socket_directory to unix_socket_directories.
unix_socket_directory has not been valid since PostgreSQL 9.2 and since 9.5 is now the minimum supported version there is no reason to document both.
2025-03-31 12:24:03 -05:00
e4def8845e Allow verification of a specified backup.
Add support for the verify command --set option. This (internal) option was already accepted without errors but was not implemented.

The default behavior for verify is to check all the backups present. With the --set option only the specified backup will be verified. If the specified backup label is not valid an error is added to the result and verification is skipped. In addition, only WAL required to make the specified backup consistent will be verified.
2025-03-26 08:44:29 -06:00
84eb9a742d Clarify behavior of --repo-retention-full-type.
Make some clarifications and add explicit documentation for repo-retention-full-type=count.
2025-03-24 12:21:31 -06:00
5d2c67c3c6 Fix block incremental restore issue on non-default repository.
If the selected backup to restore was not in the default (lowest number) repository and block incremental was used, then restore would erroneously try to load the file super block list from the default repository. Specifying --repo would fix this since it changed the default repository.

Fix by updating the super block read to the specified repository.
2025-03-21 10:37:20 -06:00
723f900eaa Fix missing return in FUNCTION_LOG_RETURN_VOID().
This macro is only ever called last in functions so this is not an active issue, but it makes sense to fix since it would pose a risk for future development.
2025-03-21 09:01:02 -06:00
d10ad3780a Fix test logging. 2025-03-19 11:17:59 -06:00
f00c0d4e85 Fix comments in unit tests. 2025-03-19 09:34:32 -06:00
750a051b7a Add numeric output to version command.
Add a new output option to the version command that defaults to text (current behavior) but can be set to num to output the version in a numeric format, e.g. 2055000.

This makes it easier to compare and identify versions of pgBackRest.
2025-02-28 16:53:03 -06:00
814bf487c3 Add support for GCS requester pays.
GCS user projects allow the bucket requester to be billed rather than the bucket owner. This is useful for hosted services that create backups and want to provide direct (probably read-only) access to the backups.
2025-02-25 16:37:22 -06:00
4db0263fd6 Add support for S3 requester pays.
S3 requester pays allows the bucket requester to be billed rather than the bucket owner. This is useful for hosted services that create backups and want to provide direct (probably read-only) access to the backups.
2025-02-25 16:12:28 -06:00
2155a12a3d Add detail logging for expired archive path.
When archive expiration has a large number of files to remove it may look like the process has hung, at least at higher log levels.

Add a log message at detail level to show that progress is being made.
2025-02-17 11:51:51 -06:00
82299eb4cd Clarify requirement for local/remote pgBackRest versions to match. 2025-02-15 12:00:00 -06:00
47f050f8fd Fix expire archive range logging.
The prior logging only output the last path to be removed since start was overwritten as each path was deleted. This had no affect on expire functionality but was confusing since many more files might be expired than the logs indicated.

Fix logging so the correct start path is logged.
2025-02-15 11:17:04 -06:00
aa7210a183 Change --process-max recommendation for object stores to --repo-bundle.
While process-max is as useful for object stores as any other storage type, for file creation time in particular file bundling is far more effective since fewer files are created.

Update the recommendation to reflect this.
2025-02-04 11:15:51 -05:00
922e9f0775 Verify recovery target timeline.
If the user picks an invalid timeline (or the default is invalid) they will not discover it until after the restore is complete and recovery starts. In that case they'll receive a message like this:

FATAL:  requested timeline 2 is not a child of this server's history
DETAIL:  Latest checkpoint is at 0/7000028 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/600AA20.

This message generally causes confusion unless one is familiar with it. In this case 1) a standby was promoted creating a new timeline 2) a new backup was made from the primary 3) the new backup was restored but could not follow the new timeline because the backup was made after the new timeline forked off. Since PostgreSQL 12 following the latest timeline has been the default so this error has become common in split brain situations.

Improve pgBackRest to read the history files and provide better error messages. Now this error is thrown before the restore starts:

ERROR: [058]: target timeline 2 forked from backup timeline 1 at 0/600aa20 which is before backup lsn of 0/7000028
       HINT: was the target timeline created by accidentally promoting a standby?
       HINT: was the target timeline created by testing a restore without --archive-mode=off?
       HINT: was the backup made after the target timeline was created?

This saves time since it happens before the restore and gives more information about what has gone wrong.

If the backup timeline is not an ancestor of the target timeline the error message is:

ERROR: [058]: backup timeline 6, lsn 0/4ffffff is not in the history of target timeline B
       HINT: was the target timeline created by promoting from a timeline < latest?

This situation should be rare but can happen during complex recovery scenarios where the user is explicitly setting the target time.
2025-02-04 10:06:17 -05:00
d5cefb7290 Fix error reporting for queries with no results.
If a query that expected no results returned an error then it would incorrectly report that no results were expected because the error was interpreted as a result.

Switch the order of the checks so that an error is reported instead and add a test to prevent regression.
2025-01-29 13:48:26 -05:00
e625ed8be2 Caveat --tablespace-map-all regarding tablespace creation.
If a tablespace is created after the backup starts then it cannot be mapped using --tablespace-map-all since there is no record of it in the manifest.

This would be extremely complex to fix but it can be documented.
2025-01-28 09:14:30 -05:00
6776940c3b Use three part version in development builds.
This makes the versioning more consistent and is required by a subsequent commit that will separate the version components.
2025-01-23 14:55:44 -05:00
6fbb28fa2d Do not set recovery_target_timeline=current for PostgreSQL < 12.
PostgreSQL < 12 defaults recovery_target_timeline to current but if current is explicitly set it behaves as if latest was set. Since current is not handled in the PostgreSQL code it looks as if there should be an error during the integer conversion but that doesn't happen due to incorrect strtoul() usage (not checking endptr).

Handle this by omitting recovery_target_timeline from recovery.conf when it is explicitly set by the user to current.
2025-01-23 07:58:41 -05:00
931435c017 Allow backup command to operate on remote repositories.
The backup command has always been limited to working only when the repository is local. This was due to some limitations in storage (addressed in 01b81f9) and the protocol helper (addressed in 4a94b6be).

Now that there a no limitations preventing this feature it makes sense to enable it. This allows for more flexibility in where backups are run.
2025-01-21 11:45:50 -05:00
23bd392bdc Improve hex encode performance with bytewise lookup.
Previously, hex encode looked up each nibble of the input separately. Instead use a larger lookup table containing the two-byte encoding of every possible input byte, resulting in a 1/3 reduction in encoding time.

Inspired by and mostly cribbed from PostgreSQL commit e24d7708.
2025-01-20 14:09:54 -05:00
713f6657d3 Merge v2.54.2 release. 2025-01-20 10:57:27 -05:00
7a33d6168b Replace constant version with macro in backup test module. 2025-01-14 13:10:32 -05:00
13f23f2168 Fix issue after disabling bundling with block incremental enabled.
When bundling and block incremental are both enabled the bundleRaw flag is set to indicate that headers are omitted (whenever possible) for encryption and compression. This is intended to save space, especially when there are very large numbers of small files.

If bundling is disabled this flag needs to be preserved so that existing bundles from prior backups are read correctly. However, the prior code was only saving the flag when bundling was enabled, which caused prior backups to be unreadable if bundling was disabled.

Fix so that the flag is preserved and backups are not broken.
2024-12-26 12:01:59 -05:00
48ecbe422d Clarify behavior of multiple configuration files. 2024-12-19 13:52:59 -05:00