1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3845 Commits

Author SHA1 Message Date
David Steele
0895cfcdf7 Add HRN_PG_CONTROL_PUT() and HRN_PG_CONTROL_TIME().
These macros simplify management of pg_control test files.

Centralize time updates for pg_control in the command/backup module. This caused some time updates in the logs.

Finally, move the postgres module after the storage module so it can use storage macros.
2021-11-30 13:23:11 -05:00
David Steele
01ac6b6cac Autogenerate test system identifiers.
hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL.

Add constants to represent version system ids in tests. These will eventually be auto-generated.

This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
David Steele
3f7409019d Ensure ASSERT() macro is always available in test modules.
Tests that run without DEBUG for performance did not have ASSERT() and were using CHECK() instead.

Instead ensure that the ASSERT() macro is always available in tests.
2021-11-24 16:09:45 -05:00
Reid Thompson
dcb4f09d83 Revert changes to backupFilePut() made in 1e77fc3d.
These changes were made obsolete by a3d7a23a.
2021-11-23 09:37:12 -05:00
David Steele
7e35245dc3 Use ASSERT() or TEST_RESULT*() instead of CHECK() in test modules. 2021-11-23 08:07:31 -05:00
Reid Thompson
a3d7a23a9d
Use infoBackupDataByLabel() to log backup size.
Eliminate summing and passing of copied files sizes for logging backup size.

Instead, utilize infoBackupDataByLabel() to pull the backup size for the log message.
2021-11-22 12:52:37 -05:00
Reid Thompson
1a0560d363
Allow y/n arguments for boolean command-line options.
This allows boolean boolean command-line options to work like their config file equivalents.

At least for now this behavior will remain undocumented since all examples in the documentation will continue to use the standard syntax. The idea is that it will "just work" when options are copied out of config files rather than generating an error.
2021-11-19 12:22:09 -05:00
David Steele
2d963ce947 Rename server-start command to server. 2021-11-18 17:23:11 -05:00
David Steele
1f14f45dfb
Check archive immediately after backup start.
Previously the archive was only checked at the end of the backup to ensure all WAL required to make the backup consistent was present. The problem was that if archiving was not functioning then the backup had to complete before the user found out, which could be a while if the database was large enough.

Add an archive check immediately after backup start so failures are reported earlier.

The trick is to determine which WAL to check. If the repo is new there may not be any WAL in it and pg_start_backup() will not switch the WAL segment if it is empty. These are both likely scenarios when setting up and/or testing pgBackRest.

If the WAL segment is switched by pg_start_backup(), then check the archive for the segment that was detected prior to backup start. This should be common on normal running clusters with regular activity. Note that this might not be the segment immediately prior to the backup start segment if WAL volume is high.

If pg_start_backup() did not switch the WAL then we can force a switch on PostgreSQL >= 9.3 by creating a restore point. In that case the WAL to check will be the backup start WAL. This is most likely to happen on idle systems, during testing, or immediately after a repo switch.

An advantage of this approach other than earlier notification is that the backup directory will not be created so no resume will be attempted on the next backup.

Note that some additional churn was created in backup.c because the load of archive.info needs to be done earlier.
2021-11-18 16:18:10 -05:00
David Steele
dea752477a Remove obsolete statement about future multi-repository support. 2021-11-17 16:39:04 -05:00
David Steele
0949b4d35f Add linefeed and remove space. 2021-11-16 18:26:21 -05:00
David Steele
809f0bbc63 Add infoBackupLabelExists().
This is easier to read than using infoBackupDataByLabel() != NULL.

It also allows an assertion to be added to infoBackupDataByLabel() to ensure that a NULL return value is not used unsafely.
2021-11-16 11:34:53 -05:00
Reid Thompson
1e77fc3d75
Include backup_label and tablespace_map file sizes in log output.
In cases where they are returned by postgres, include backup_label and tablespace_map file sizes in the backup size value output in the log.
2021-11-16 10:21:32 -05:00
David Steele
6b5322cdad Add findutils package to RHEL 8 documentation container.
This package was dropped from the most recent Rocky Linux 8 image.
2021-11-16 09:27:15 -05:00
David Steele
df89eff429 Fix typos and improve documentation for the tablespace-map-all option. 2021-11-15 16:53:41 -05:00
Reid Thompson
fcae9d3503 Fix parameter test logging in parseOptionIdxValue(). 2021-11-15 16:42:46 -05:00
David Steele
b3a5f7a8e2 Add tablespace_map file to command/backup test module.
The code worked fine but better to have explicit tests for this file.
2021-11-15 14:32:22 -05:00
David Steele
e62ba8e85e Add path to pgbench used for stress test in user guide.
This allows the stress test to run on RHEL.
2021-11-12 17:15:45 -05:00
David Steele
43cfa9cef7 Revive archive performance test.
This test was lost due to a syntax issue in a58635ac.

Update the test to use system() to better mimic what postgres does and add logging so pgBackRest timing can be determined.
2021-11-10 12:14:41 -05:00
David Steele
dd96c29f96 Refactor postgres/client module with inline getters/setters.
Extend the pattern introduced in 79a2d02 to the postgres/client module.
2021-11-10 07:53:46 -05:00
David Steele
afe77e76e0 Update contributor for 6e635764. 2021-11-10 07:31:02 -05:00
Reid Thompson
6e635764a6
Match backup log size with size reported by info command.
Properly log the size of files copied during the backup, matching the backup size returned from the info command.

In the reference issue, the incremental backup after switchover logs the size of all files evaluated rather than only the size of the files copied in the backup.
2021-11-09 13:24:56 -05:00
David Steele
d05d6b8714 Do not delete manifests individually during stanza delete.
This appears to have been an attempt to not delete files that we don't recognize, but it only works in narrow cases and could leave the user is a position of not being able to complete the stanza delete without manual intervention. It seems better just to proceed with the delete, especially since the info files have already been removed.

In addition, deleting the manifests individually could be slow on object stores if there were a very large number of backups.
2021-11-08 09:39:58 -05:00
David Steele
bb03b3f419 Refactor switch statements in strIdBitFromZN().
Coverity does not like fall-throughs either to or from the default case so refactor to avoid that.
2021-11-04 09:44:31 -04:00
David Steele
676b9d95dd Optional parameters for tlsClientNew().
There are a number of optional parameters with the same type so this makes them easier to track and reduces churn when new ones are added.
2021-11-04 08:19:18 -04:00
David Steele
038abaa71d
Display size option default and allowed values with appropriate units.
Size option default and allowed values were displayed in bytes, which was confusing for the user.

This also lays the groundwork for adding units to time options.

Move option parsing functions into a common module so they can be used from the build module.
2021-11-03 15:23:08 -04:00
David Steele
1b93a77236 Use void * instead of List * to avoid Coverity false positives.
Coverity complains that this should be "List" but that is clearly not correct.
2021-11-03 12:14:17 -04:00
Reid Thompson
2a576477b3
Add --cmd option.
Allows users to provide an executable to be used when pgbackrest generates command strings that expect to invoke pgbackrest. These generated commands are written to files by pgbackrest, e.g. recovery.conf.
2021-11-03 11:36:34 -04:00
David Steele
c5b5b58806
Simplify error handler.
The error handler used a loop to process try, catch, and finally blocks. This worked fine but static analysis tools like Coverity did not understand that the finally block would always run and so there were false positives about double-free, unfreed resource, etc.

This implementation removes the loop, which simplifies everything, and makes it clear that the finally block will always run. This cuts down on Coverity false positives.

This implementation also catches lack of coverage on empty catch blocks so a few test fixes were committed separately in d74fe7a.

A small refactor in backup.c is required because gcc 10.3.1 on Fedora 33 complains that the reason variable may be used uninitialized. It's not clear why this is the case, but reducing the scope of the TRY block fixes the issue.
2021-11-03 10:36:31 -04:00
David Steele
cff961ede7 Centralize logic to build value lists during config rendering.
This reduces duplication and makes it easier to add new types.
2021-11-03 07:38:06 -04:00
David Steele
7f6c513be9
Add StringId as an option type.
Rather the converting String to StringIds at runtime, store defaults in StringId format in parse.auto.c and convert user input to StringId during parsing.
2021-11-03 07:27:26 -04:00
David Steele
b13844086d
Use cfgOptionStrId() instead of cfgOptionStr() where appropriate.
The compress-type, repo-type and log-level-* options have allow lists, which means it is more efficient to treat them as StringIds.

For compress-type and log-level-* also update the functions that convert them to enums.
2021-11-01 17:35:19 -04:00
David Steele
b237d0cd59 Remove placeholder bz2 helper data.
This placeholder data should have been removed when bz2 support was added in a021c9fe05.
2021-11-01 10:43:08 -04:00
David Steele
f4e281399a Remove unused protocol log level.
This log level was used in the Perl code but was never ported to C.
2021-11-01 10:27:57 -04:00
David Steele
bc352fa6a8
Simplify strIdFrom*() functions.
The strIdFrom*() forced the caller to pick an encoding, which led to a number of TRY...CATCH blocks in the code. In practice the caller does not care which encoding is used as long as the string is valid for some encoding.

Update the strIdFrom*() function to try all possible encodings and only throw an error when the string is not valid for any of them.
2021-11-01 10:08:56 -04:00
David Steele
a92d793819 Update automake version.
There were no changes to install.sh in this version.
2021-11-01 09:11:43 -04:00
David Steele
904b897f5e Begin v2.37 development. 2021-11-01 09:03:42 -04:00
David Steele
42fd6ce4e0 v2.36: Minor Bug Fixes and Improvements
Bug Fixes:

* Allow "global" as a stanza prefix. (Reviewed by Stefan Fercot. Reported by Younes Alhroub.)
* Fix segfault on invalid GCS key file. (Reviewed by Stephen Frost. Reported by Henrik Feldt.)

Improvements:

* Allow link-map option to create new links. (Reviewed by Don Seiler, Stefan Fercot, Chris Bandy. Suggested by Don Seiler.)
* Increase max index allowed for pg/repo options to 256. (Reviewed by Cynthia Shang.)
* Add WebIdentity authentication for AWS S3. (Reviewed by James Callahan, Reid Thompson, Benjamin Blattberg, Andrew L'Ecuyer.)
* Report backup file validation errors in backup.info. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Add recovery start time to online backup restore log. (Reviewed by Tom Swartz, Stefan Fercot. Suggested by Tom Swartz.)
* Report original error and retries on local job failure. (Reviewed by Stefan Fercot.)
* Rename page checksum error to error list in info text output. (Reviewed by Stefan Fercot.)
* Add hints to standby replay timeout message. (Reviewed by Cynthia Shang, Stefan Fercot. Suggested by Leigh Downs.)
2021-11-01 08:59:14 -04:00
David Steele
6abb06248c Make analytics optional for HTML documentation.
Analytics should only be added to the current HTML documentation on the website, so exclude them by default.
2021-10-29 11:45:50 -04:00
David Steele
1336657326 Restore some linefeed rendering behavior from before def7d513.
The new rendering behavior is correct in normal cases, but for the pre-rendered HTML blocks in the command and configuration references it causes a lot of churn. This would be OK if the new HTML was diff-able, but it is not.

Go back to the old behavior of using br tags for this case to reduce churn until a more permanent solution is found.
2021-10-29 10:35:56 -04:00
David Steele
c32e000ab9 Use Rocky Linux for documentation builds instead of CentOS.
Since CentOS 8 will be EOL at the end of the year it makes sense to do this now. The centos:8 image is still used in documentation.xml because changes there require manual testing, which will need to be done at a later date. The changes are not user-facing, however, and can be done at any time.

Also update CentOS references to RHEL since that is what we are emulating for testing purposes.
2021-10-28 15:15:49 -04:00
David Steele
30c589ace7 Fix typo in contributing guide.
Not sure how this got broken but it was probably an errant search and replace.
2021-10-28 13:28:49 -04:00
David Steele
2f1a287737 Add missing assert. 2021-10-28 11:49:00 -04:00
David Steele
adc09ffc3b Minor fix for lower-casing of option summaries.
This works with existing cases and fixes "I/O".
2021-10-28 08:10:43 -04:00
David Steele
fa564ee196 Improve documentation for cmd-ssh, repo-host-cmd, pg-host-cmd options.
Use "command" instead of "exe" and make the descriptions more consistent.
2021-10-27 11:08:32 -04:00
David Steele
e1f6c066b3 Improve documentation for buffer-size option. 2021-10-27 10:52:39 -04:00
David Steele
1f7c7b7dda Fix test descriptions in common/typeVariantTest. 2021-10-26 16:56:44 -04:00
David Steele
d74fe7a222 Add coverage for empty CATCH() blocks.
Currently empty CATCH() blocks are always marked as covered because of the loop structure of error handling.

A prototype implementation of error handling without looping has shown that these CATCH() blocks are not covered without new tests. Whether or not that prototype gets committed it is worth adding the tests.
2021-10-26 13:53:44 -04:00
David Steele
e2eea974c1 Add assertion for Coverity.
Coverity thinks this value might be NULL but that should not be possible because of the TRY...CATCH block.
2021-10-26 12:09:41 -04:00
David Steele
4f10441574 Add missing paragraph tags in coding standards. 2021-10-26 08:25:21 -04:00