1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Commit Graph

1501 Commits

Author SHA1 Message Date
David Steele
aeecb500f5 Improve implementation of cfgOptionIdxName().
Cache option names after they are generated rather than regenerating them each time.
2022-01-10 14:47:29 -05:00
David Steele
aced5d47ed Replace cfgOptionGroupIdxToKey() with cfgOptionGroupName().
Do the replacement anywhere cfgOptionGroupIdxToKey() is being used to construct a group name in a message. cfgOptionGroupName() is better for this case since it also includes the name of the group so that it does not need to be repeated in each message.
2022-01-10 09:10:06 -05:00
David Steele
9657f1b325 Dynamically allocate index to key index map.
Now the config/config module has no notion of max option indexes. The config/parse still needs the max but this will be removed at a later date.
2022-01-09 19:53:05 -05:00
David Steele
e4b48eb430 Fix inconsistent group display names in messages.
In other instances there are no dashes, e.g. repo1 or pg1. Make these messages match.
2022-01-09 19:43:44 -05:00
Stefan Fercot
d866dd5c29
Add backup LSNs to info command output.
The backup LSNs are useful for performing LSN-based PITR. LSNs will not be displayed in the general text output (without --set) because they are probably not useful enough to deserve their own line.
2022-01-07 14:09:58 -05:00
David Steele
bb4b30ddd3
Remove support for PostgreSQL 8.3/8.4.
There is no evidence that users need 8.3/8.4 anymore but it does cost us in terms of development and testing, especially now that we have a number of new backup/restore features planned.

It seems to make sense to remove this support now. If there are users who need to use/migrate from these versions they can use an older version of pgBackRest.
2022-01-06 15:34:04 -05:00
Reid Thompson
a82f0179cd
Note that replications slots are not restored.
Update documentation and help to note that replication slots are not restored and reference the PostgreSQL documentation to explain why.
2022-01-04 16:11:27 -05:00
David Steele
75e4580919 Begin v2.38 development. 2022-01-03 08:52:25 -05:00
David Steele
f18f2d9991 v2.37: TLS Server
Bug Fixes:

* Fix restore delta link mapping when path/file already exists. (Reviewed by Reid Thompson. Reported by Younes Alhroub.)
* Fix socket leak on connection retries. (Reviewed by Reid Thompson. Reported by James Coleman.)

Features:

* Add TLS server. (Reviewed by Stephen Frost, Reid Thompson, Andrew L'Ecuyer.)
* Add --cmd option. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele. Suggested by Virgile CREVON.)

Improvements:

* Check archive immediately after backup start. (Reviewed by Reid Thompson, David Christensen.)
* Add timeline and checkpoint checks to backup. (Reviewed by Stefan Fercot, Reid Thompson.)
* Check that clusters are alive and correctly configured during a backup. (Reviewed by Stefan Fercot.)
* Error when restore is unable to find a backup to match the time target. (Reviewed by Reid Thompson, Douglas J Hunley. Suggested by Douglas J Hunley.)
* Parse protocol/port in S3/Azure endpoints. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add warning when checkpoint_timeout exceeds db-timeout. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Add verb to HTTP error output. (Contributed by Christoph Berg. Reviewed by David Steele.)
* Allow y/n arguments for boolean command-line options. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Make backup size logging exactly match info command output. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Mahomed Hussein.)

Documentation Improvements:

* Display size option default and allowed values with appropriate units. (Reviewed by Reid Thompson.)
* Fix typos and improve documentation for the tablespace-map-all option. (Reviewed by Reid Thompson. Suggested by Reid Thompson.)
* Remove obsolete statement about future multi-repository support. (Suggested by David Christensen.)
2022-01-03 08:43:55 -05:00
David Steele
fccb7f7dd4 Add release note regarding IANA approval of the default TLS port. 2021-12-28 17:39:22 -05:00
Reid Thompson
6a12458440
Parse protocol/port in S3/Azure endpoints.
Utilize httpUrlNewParseP() to parse endpoint and port from the URL in the S3 and Azure helpers to avoid issues where protocol was not expected to be part of the URL.
2021-12-16 10:30:59 -05:00
David Steele
f06101de77 Add TLS server documentation.
Add documentation and make the feature visible.
2021-12-16 09:47:04 -05:00
David Steele
615bdff403
Fix socket leak on connection retries.
This leak was caused by the file descriptor variable getting clobbered after a long jump. Mark it as volatile to fix.

Testing this is a bit complex because the issue only happens in optimized builds, if at all. Put the test into the performance suite, which is always optimized, until a better idea presents itself.
2021-12-14 14:53:41 -05:00
David Steele
a73fe4eb96
Fix restore delta link mapping when path/file already exists.
If a path/file was remapped to a link using either --link-map or --link-all there would be no affect if the path/file already existed. If a link existed it would be properly updated and converting a link to a path/file also worked.

The issue happened during delta cleanup, which failed to check if the existing path/file had been remapped to a link.

Add checks for newly mapped path/file links and remove the old path/file we required.
2021-12-10 15:53:40 -05:00
Christoph Berg
c38e2d3170 Add verb to HTTP error output.
This makes it easier to debug HTTP errors.
2021-12-08 15:00:19 -05:00
David Steele
be4ac3923c
Error when restore is unable to find a backup to match the time target.
This was previously a warning but the warning is easy to miss so a lot of time may be lost restoring and recovering a backup that will not hit the target.

Since this is technically a breaking change, add an "important note" about the change to the release.
2021-12-08 13:57:26 -05:00
Stefan Fercot
6723305937
Add warning when checkpoint_timeout exceeds db-timeout.
In the backup command, add a warning if start-fast is disabled and the PostgreSQL checkpoint_timeout is greater than db-timeout.

In such cases, we might timeout before the checkpoint occurs and the backup really starts.
2021-12-08 12:29:20 -05:00
David Steele
bd2ba802db
Check that clusters are alive and correctly configured during a backup.
Fail the backup if a cluster stops or the standby is promoted. Previously, shutting down the primary would cause an error but it was not detected until the end of the backup. Now the error will happen sooner and a promotion on the standby will also cause an error.
2021-12-08 10:16:41 -05:00
David Steele
7b3ea883c7
Add SIGTERM and SIGHUP handling to TLS server.
SIGHUP allows the configuration to be reloaded. Note that the configuration will not be updated in child processes that have already started.

SIGTERM terminates the server process gracefully and sends SIGTERM to all child processes. This also gives the tests an easy way to stop the server.
2021-12-07 18:18:43 -05:00
David Steele
49145d72ba
Add timeline and checkpoint checks to backup.
Add the following checks:

* Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
* Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
* Standby is on the same timeline as the primary. If not, this standby is not following the primary.
* Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.

This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.

The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.
2021-12-07 09:21:07 -05:00
Reid Thompson
dcb4f09d83 Revert changes to backupFilePut() made in 1e77fc3d.
These changes were made obsolete by a3d7a23a.
2021-11-23 09:37:12 -05:00
Reid Thompson
a3d7a23a9d
Use infoBackupDataByLabel() to log backup size.
Eliminate summing and passing of copied files sizes for logging backup size.

Instead, utilize infoBackupDataByLabel() to pull the backup size for the log message.
2021-11-22 12:52:37 -05:00
Reid Thompson
1a0560d363
Allow y/n arguments for boolean command-line options.
This allows boolean boolean command-line options to work like their config file equivalents.

At least for now this behavior will remain undocumented since all examples in the documentation will continue to use the standard syntax. The idea is that it will "just work" when options are copied out of config files rather than generating an error.
2021-11-19 12:22:09 -05:00
David Steele
2d963ce947 Rename server-start command to server. 2021-11-18 17:23:11 -05:00
David Steele
1f14f45dfb
Check archive immediately after backup start.
Previously the archive was only checked at the end of the backup to ensure all WAL required to make the backup consistent was present. The problem was that if archiving was not functioning then the backup had to complete before the user found out, which could be a while if the database was large enough.

Add an archive check immediately after backup start so failures are reported earlier.

The trick is to determine which WAL to check. If the repo is new there may not be any WAL in it and pg_start_backup() will not switch the WAL segment if it is empty. These are both likely scenarios when setting up and/or testing pgBackRest.

If the WAL segment is switched by pg_start_backup(), then check the archive for the segment that was detected prior to backup start. This should be common on normal running clusters with regular activity. Note that this might not be the segment immediately prior to the backup start segment if WAL volume is high.

If pg_start_backup() did not switch the WAL then we can force a switch on PostgreSQL >= 9.3 by creating a restore point. In that case the WAL to check will be the backup start WAL. This is most likely to happen on idle systems, during testing, or immediately after a repo switch.

An advantage of this approach other than earlier notification is that the backup directory will not be created so no resume will be attempted on the next backup.

Note that some additional churn was created in backup.c because the load of archive.info needs to be done earlier.
2021-11-18 16:18:10 -05:00
David Steele
dea752477a Remove obsolete statement about future multi-repository support. 2021-11-17 16:39:04 -05:00
Reid Thompson
1e77fc3d75
Include backup_label and tablespace_map file sizes in log output.
In cases where they are returned by postgres, include backup_label and tablespace_map file sizes in the backup size value output in the log.
2021-11-16 10:21:32 -05:00
David Steele
df89eff429 Fix typos and improve documentation for the tablespace-map-all option. 2021-11-15 16:53:41 -05:00
David Steele
afe77e76e0 Update contributor for 6e635764. 2021-11-10 07:31:02 -05:00
Reid Thompson
6e635764a6
Match backup log size with size reported by info command.
Properly log the size of files copied during the backup, matching the backup size returned from the info command.

In the reference issue, the incremental backup after switchover logs the size of all files evaluated rather than only the size of the files copied in the backup.
2021-11-09 13:24:56 -05:00
David Steele
038abaa71d
Display size option default and allowed values with appropriate units.
Size option default and allowed values were displayed in bytes, which was confusing for the user.

This also lays the groundwork for adding units to time options.

Move option parsing functions into a common module so they can be used from the build module.
2021-11-03 15:23:08 -04:00
Reid Thompson
2a576477b3
Add --cmd option.
Allows users to provide an executable to be used when pgbackrest generates command strings that expect to invoke pgbackrest. These generated commands are written to files by pgbackrest, e.g. recovery.conf.
2021-11-03 11:36:34 -04:00
David Steele
c5b5b58806
Simplify error handler.
The error handler used a loop to process try, catch, and finally blocks. This worked fine but static analysis tools like Coverity did not understand that the finally block would always run and so there were false positives about double-free, unfreed resource, etc.

This implementation removes the loop, which simplifies everything, and makes it clear that the finally block will always run. This cuts down on Coverity false positives.

This implementation also catches lack of coverage on empty catch blocks so a few test fixes were committed separately in d74fe7a.

A small refactor in backup.c is required because gcc 10.3.1 on Fedora 33 complains that the reason variable may be used uninitialized. It's not clear why this is the case, but reducing the scope of the TRY block fixes the issue.
2021-11-03 10:36:31 -04:00
David Steele
7f6c513be9
Add StringId as an option type.
Rather the converting String to StringIds at runtime, store defaults in StringId format in parse.auto.c and convert user input to StringId during parsing.
2021-11-03 07:27:26 -04:00
David Steele
b13844086d
Use cfgOptionStrId() instead of cfgOptionStr() where appropriate.
The compress-type, repo-type and log-level-* options have allow lists, which means it is more efficient to treat them as StringIds.

For compress-type and log-level-* also update the functions that convert them to enums.
2021-11-01 17:35:19 -04:00
David Steele
bc352fa6a8
Simplify strIdFrom*() functions.
The strIdFrom*() forced the caller to pick an encoding, which led to a number of TRY...CATCH blocks in the code. In practice the caller does not care which encoding is used as long as the string is valid for some encoding.

Update the strIdFrom*() function to try all possible encodings and only throw an error when the string is not valid for any of them.
2021-11-01 10:08:56 -04:00
David Steele
904b897f5e Begin v2.37 development. 2021-11-01 09:03:42 -04:00
David Steele
42fd6ce4e0 v2.36: Minor Bug Fixes and Improvements
Bug Fixes:

* Allow "global" as a stanza prefix. (Reviewed by Stefan Fercot. Reported by Younes Alhroub.)
* Fix segfault on invalid GCS key file. (Reviewed by Stephen Frost. Reported by Henrik Feldt.)

Improvements:

* Allow link-map option to create new links. (Reviewed by Don Seiler, Stefan Fercot, Chris Bandy. Suggested by Don Seiler.)
* Increase max index allowed for pg/repo options to 256. (Reviewed by Cynthia Shang.)
* Add WebIdentity authentication for AWS S3. (Reviewed by James Callahan, Reid Thompson, Benjamin Blattberg, Andrew L'Ecuyer.)
* Report backup file validation errors in backup.info. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Add recovery start time to online backup restore log. (Reviewed by Tom Swartz, Stefan Fercot. Suggested by Tom Swartz.)
* Report original error and retries on local job failure. (Reviewed by Stefan Fercot.)
* Rename page checksum error to error list in info text output. (Reviewed by Stefan Fercot.)
* Add hints to standby replay timeout message. (Reviewed by Cynthia Shang, Stefan Fercot. Suggested by Leigh Downs.)
2021-11-01 08:59:14 -04:00
David Steele
1336657326 Restore some linefeed rendering behavior from before def7d513.
The new rendering behavior is correct in normal cases, but for the pre-rendered HTML blocks in the command and configuration references it causes a lot of churn. This would be OK if the new HTML was diff-able, but it is not.

Go back to the old behavior of using br tags for this case to reduce churn until a more permanent solution is found.
2021-10-29 10:35:56 -04:00
David Steele
4f10441574 Add missing paragraph tags in coding standards. 2021-10-26 08:25:21 -04:00
David Steele
13d4559708 Check return value of getsockopt().
Checking the return value is not terribly important here, but if setsockopt() fails it is likely that bind() will fail as well. May as well get it over with and this makes Coverity happy.
2021-10-25 15:31:39 -04:00
Reid Thompson
1152f7a7d6 Fix mismatched parameters in tlsClientNew() call.
3879bc69 added this call and the parameters were not quite right but in way that the compiler decided they were OK. It was mostly working but TLS verification was disabled if caPath was NULL, which is not OK.
2021-10-25 12:56:33 -04:00
David Steele
3879bc69b8
Add WebIdentity authentication for AWS S3.
This allows credentials to be automatically acquired in an EKS environment.
2021-10-22 18:31:55 -04:00
David Steele
51785739f4
Store config values as a union instead of a variant.
The variants were needed to easily serialize configurations for the Perl code.

Unions are more efficient and will allow us to add new types that are not supported by variants, e.g. StringId.
2021-10-22 18:02:20 -04:00
David Steele
2cea005f74 Fix segfault on invalid GCS key file. 2021-10-22 17:19:16 -04:00
David Steele
e443e3c6c0 Add br tags for HTML documentation rendering missed in def7d513. 2021-10-19 09:06:06 -04:00
David Steele
ccc255d3e0 Add TLS Server.
The TLS server is an alternative to using SSH for protocol connections to remote hosts.

This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
David Steele
498902e885
Allow "global" as a stanza prefix.
A stanza name like global_stanza was not allowed because the code was not selective enough about how a global section should be formatted.

Update the config parser to correctly recognize global sections.
2021-10-07 12:18:24 -04:00
David Steele
68c5f3eaf1
Allow link-map option to create new links.
Currently link-map only allows links that exist in the backup manifest to be remapped to a new destination.

Allow link-map to create a new link as long as a valid path/file from the backup is referenced.
2021-10-05 17:59:05 -04:00
David Steele
6af827cbb1
Report original error and retries on local job failure.
The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync.

Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00