1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1414 Commits

Author SHA1 Message Date
David Steele
8e1807cdbe
Detect errors in S3 multi-part upload finalize.
Multi-part upload may fail despite returning an HTTP success code. Check for the ETag field in the result and if not present consider the upload to have failed. This will trigger a retry at the local job level.
2021-07-08 13:06:52 -04:00
David Steele
43b874628c
Fix detection of circular symlinks.
Links were followed before they were checked for validity so a circular link would send the manifest build into endless recursion leading to a crash. Fix by moving the recursion after the link check.

Note that this issue has existed since the C migration and was not introduced by the refactor in eba013b.
2021-07-08 11:30:23 -04:00
Stefan Fercot
056a3070e0
Automatically create data directory on restore.
Data directory creation was added during the C migration, but creation of the base data directory (PGDATA) was prevented by a check migrated from Perl.

Remove the check and update tests to create the data directory at least once.
2021-07-08 10:33:19 -04:00
Cynthia Shang
62e8d97af0
Update command/archive tests to use standard patterns.
Includes archiveCommon, archiveGet and archivePush.

Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
Eric Radman
23bdc3deb6
Fix documentation and comment typos.
Identified using `ag -l | igor`.
2021-07-01 11:50:03 -04:00
Cynthia Shang
6a90d46909
Update info/manifest test to use standard patterns. 2021-06-30 08:51:11 -04:00
Cynthia Shang
c6208113bf Add timeModified parameter to HRN_STORAGE_PUT() macro. 2021-06-28 12:14:26 -04:00
Cynthia Shang
065a8c8454
Replace defines with string constants in unit tests.
For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes.

Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
Cynthia Shang
fb2d111dcd
Update info/infoBackup test to use standard patterns. 2021-06-24 16:11:14 -04:00
David Steele
6a1c0337dd
Binary protocol.
Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol.

There are a number advantages:

* The pack type is more compact than JSON and are more efficient to render/parse.
* Packs are more strictly typed than JSON.
* Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state.
* There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files.

Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C.

There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
Cynthia Shang
bffb43ea3f
Update command/check test to use standard patterns. 2021-06-24 12:23:09 -04:00
Cynthia Shang
17c9ed0ef0
Update command/info test to use standard patterns.
Some tests had to be reordered or updated, as follows:

* Reordered tests at line 317 and 331 to avoid unnecessary file removal.

* Change "stanza found" test at line 1735 to reflect real-life scenario. Originally this test had the cipher-pass environment key set up which caused the RepoGrp to be 2 but with no valid repo path. This resulted in the repo loops executing for the repo2 but since the path was not defined, the tests just reported "none" for cipher which is incorrect since the repo IS encrypted.

* Moved order of HRN_CFG_LOAD in some tests when able to avoid using storageTest.
2021-06-24 08:55:44 -04:00
Cynthia Shang
ff95eddc36
Update command/verify test to use standard patterns. 2021-06-23 18:09:46 -04:00
Cynthia Shang
8dd882bade
Update info/infoArchive test to use standard patterns. 2021-06-23 16:20:47 -04:00
David Steele
e697f5705a
Clear error when a CATCH() block finishes.
It is better to clear errors after the catch block completes rather than leave them set until the next error. This also make is possible to tell when a error is currently being handled, which a function further down the stack might use to modify its behavior. Currently this is only useful in testing, but clearing the error seems like a good idea in general.

Two places used errors outside the CATCH() block. Mem context cleanup now uses a FINALLY() which is a better implementation anyway. The error handling in main() now calls exitSafe() from withing the CATCH() block.
2021-06-23 13:02:19 -04:00
Cynthia Shang
03021c6a17 Update command/stanza test to use standard patterns.
No core code changes; only changes for stanza-create, stanza-upgrade, and stanza-delete command unit tests.
2021-06-14 13:28:27 -04:00
David Steele
4a075b7252
Add support for more Pack types.
Since the pack type was stored in 4 bits, only 15 values were allowed (0 was reserved).

Allow virtually unlimited types by storing type info in a base-128 encoded integer following the tag when the type bits in the tag are set to 0xF.

Also separate the type IDs used in the pack (PackTypeMap) from those presented to the user (PackType). The prior PackType enum exposed implementation details to the user, e.g. pckTypeUnknown.
2021-06-08 12:55:00 -04:00
Cynthia Shang
3f9fbc3c24
Update command/expire test to use standard patterns.
The way tests are written has evolved over time. Update the command/expire test to use the new test patterns.
2021-06-08 11:00:28 -04:00
David Steele
a7b1d2b9d0 Begin v2.35 development. 2021-06-07 07:32:34 -04:00
David Steele
d10a99d73b v2.34: PostgreSQL 14 Support
Bug Fixes:

* Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.)
* Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.)
* Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.)
* Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)

Features:

* Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)
* Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.)
* Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
* Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)

Improvements:

* Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)

Documentation Improvements:

* Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)
* Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)
* Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
2021-06-07 06:51:08 -04:00
Cynthia Shang
aceb956815
Change archive expiration logging from detail to info level.
To assist with issue debugging, removal of files during archive expiration has been changed from detail level logging to info level.
2021-05-27 08:03:44 -04:00
Stefan Fercot
5b332b22b5
Add repo-retention-history option to expire backup history.
The default is to keep all backup history to match the current behavior. In minimal configuration (0 days), unexpired backups are always kept in history.

When a full backup manifest expires, all dependent differential/incremental manifests expire as well.
2021-05-26 13:07:18 -04:00
David Steele
ba351e9c5c Refactor storage/remote unit test using the protocol remote shim.
Using the local process shim improves coverage and simplifies the tests.
2021-05-26 12:38:23 -04:00
David Steele
441c000b5c Factor remote process exec out of protocolRemoteGet().
This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing.

The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
David Steele
6732806d89 Add local process shim to archive-get/archive-push unit tests.
Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
David Steele
d55b9471a8 Protocol shim improvements.
Add executable to parameter list to avoid first option being lost. The backup, restore, and verify tests worked OK with their first option being defaulted because it ended up being job-retry which worked fine as the default.

Add hrnProtocolLocalShimUninstall() allow the shim to be uninstalled.

Log shim at debug level to make it obvious in the logs when a shim is in use.
2021-05-25 11:00:24 -04:00
David Steele
55f52955a5 Fix shims with more than one function.
Each subsequent function would double every line in the C module.

Move the loop to detect shimmed functions down to prevent this issues.
2021-05-24 17:45:31 -04:00
David Steele
2452c4d5a4
Add PostgreSQL 14 support.
There are no code changes from PostgreSQL 13 so simply add the new version.

Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.

Update the integration tests to handle new PostgreSQL startup messages.
2021-05-24 17:17:03 -04:00
David Steele
eba013b49b
Fix issue when checking links for large numbers of tablespaces.
manifestLinkCheck() was pretty inefficient so large numbers of links caused it to use a lot of memory and eventually crash. This is a more efficient implementation which runs O(nlogn) and uses far less memory.

Checking for duplicate file links has been added, which represents a change in behavior, but hopefully a good one.
2021-05-24 16:29:36 -04:00
David Steele
ccac75e7de
Consolidate RHEL/CentOS user guide into a single document.
The user guide was split primarily to provide documentation for the stop-auto option in PostgreSQL <= 9.5. Now that 9.5 is EOL there does not seem to be a good reason to generate an extra user guide. The stop-auto function is still documented in the reference.

Leave the stop-auto documentation in the user guide in case we want to manually generate documentation for older versions.

Also rename centos to rhel for most identifiers since that is the core platform we are building for, similar to how we label 'debian' builds even though we generally use Ubuntu. With CentOS set to become an upstream for RHEL later this year, we'll likely need to pick a new test distribution, perhaps Rocky Linux if that gets off the ground.
2021-05-24 16:03:15 -04:00
Cynthia Shang
68faf1482a
Update contributing documentation.
Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 13:21:25 -04:00
David Steele
15b8b9207d Add log shim.
This allows DEBUG_UNIT and DEBUG_UNIT_EXTERN to be removed since static log variables can now be exposed by functions in the harness.
2021-05-21 12:51:32 -04:00
David Steele
ef63750e0b Add local process shim.
Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test.

The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
David Steele
45a4e801ed
Replace getopt_long() with custom implementation.
getopt_long() requires an exhaustive list of all possible options that may be found on the command line. Because of the way options are indexed (e.g. repo1-4, pg1-8) optionList[] has 827 entries and we have kept it small by curtailing the maximum indexes very severely. Another issue is that getopt_long() scans the array sequentially so parsing gets slower as the index maximums increase.

Replace getopt_long() with a custom implementation that behaves the same but allows options to be parsed with a function instead of using optionList[]. This commit leaves the list in place in order to focus on the getopt_long() replacement, but cfgParseOption() could be replaced with a more efficient implementation that removes the need for optionList[].

This implementation also fixes an issue where invalid options were misreported in the error message if they only had one dash, e.g. -config. This seems to have been some kind of problem in getopt_long(), but no investigation was done since the new implementation fixes it.

Tests were added at 0825428, 2b8d2da, 34dd663, and 384f247 to check that previously untested getopt_long() behavior doesn't change.
2021-05-20 16:02:31 -04:00
David Steele
831ee81466
Rename default command role to main.
Main makes more sense because we refer to the main process in the code, not the default process. The word default is pretty overloaded anyway.
2021-05-20 14:39:47 -04:00
Cynthia Shang
c92bc43c1e Simplify defaults for --force option.
All commands have the same default so it is more efficient to set the default for the entire option.
2021-05-18 16:33:45 -04:00
David Steele
320c6e1aad
Remove stanza archive spool path on restore.
Remove stanza archive spool path so existing files do not interfere with the new cluster. For instance, old archive-push acknowledgements could cause a new cluster to skip archiving. This should not happen if a new timeline is selected but better to be safe. Missing stanza spool paths are ignored.

Also add new path expression STORAGE_SPOOL_ARCHIVE to easily access this path.
2021-05-18 15:49:22 -04:00
David Steele
9af033194a
Add automatic GCS authentication for GCE instances.
When running on a GCE instance the authentication token can be pulled directly from the instance metadata. This is configured with repo-gcs-key-type=auto.

In a separate commit (26fefa6), move the code that parses the token response into a separate function, storageGcsAuthToken(), since it is now needed by two key types. This drastically improves the readability of the main commit.
2021-05-17 14:55:50 -04:00
David Steele
9235c62c6b Revert ignoring catalog version when identifying a PostgreSQL version.
927d9adb changed the way CATALOG_VERSION_NO is used to identify PostgreSQL versions since PG_CONTROL_VERSION is generally bumped with each release. The goal was to make the beta/rc period less painful because any CATALOG_VERSION_NO bump renders pgBackRest inoperative.

This worked, but in fact we'd rather be stricter about which CATALOG_VERSION_NO we accept when identifying a version of PostgreSQL. It is not just about identifying a major version, but making sure the build contains all the functions and catalogs we expect to make pgBackRest work correctly. It is better to reject early dev/beta/rc builds that may not work.

Since 927d9adb was relatively recent the chance that this stricter checking will cause a problem seems minimal, so revert to checking CATALOG_VERSION_NO for every PostgreSQL version.

Leave in place the code that pulls CATALOG_VERSION_NO from pg_control rather than the internal constant since the plan is still to allow catalog versions to "float" during the PostgreSQL beta/rc phase, which will be the subject of a future commit.
2021-05-17 07:12:46 -04:00
David Steele
f45e76fa2e
Fix issues with leftover spool files from a prior restore.
If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue.

Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process.

Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
David Steele
5464ac83d1
Convert option values in commands to StringId.
Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.

For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.
2021-05-11 17:24:30 -04:00
David Steele
87df6d7a58
Convert BackupType enum to StringId.
Allows removal of backupType()/backupTypeStr() and improves debug logging of the enum.

Move BackupType enum and string constants to info/infoBackup.h so they are available to more modules. Also convert InfoBackup to use BackupType instead of a String.
2021-05-03 12:15:39 -04:00
David Steele
fb7ddce807 Convert InfoPgType enum to StringId.
Improves debug logging by outputting archive/backup instead of 0/1.
2021-04-28 13:21:24 -04:00
David Steele
bd0081fec8 Update IoClient/IoSession to use StringIds.
Using StringId for the client/session type removes String constants and some awkward referencing/dereferencing needed to use a String constant in the interface.

Converting IoSessionRole to StringId removes a conditional in ioSessionToLog() and improves debug logging by outputting client/server instead of 0/1.
2021-04-28 12:37:22 -04:00
David Steele
0ec91f61c6 Convert ArchivePushFileIoType enum to StringId.
Improves debug logging by outputting open/write/close instead of 0/1/2.
2021-04-28 12:23:37 -04:00
David Steele
8394577c6a Convert ArchiveMode enum to StringId.
Improves debug logging by outputting get/push instead of 0/1.
2021-04-28 12:05:01 -04:00
David Steele
7dd01897fd Convert ProtocolStorageType enum to StringId.
Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
David Steele
bd68ed63ba Convert ProtocolParallelJobState enum to StringId.
Allows removal of protocolParallelJobToConstZ(), which was used only for debugging.
2021-04-28 11:43:08 -04:00
David Steele
85fc3da4c3
Update CipherType/CipherMode to StringId.
As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
David Steele
1edcfde93e
Add cfgOptionDisplay()/cfgOptionIdxDisplay().
Centralize the formatting of the configuration value for display to the user or passing on a command line.

For the new functions, if the value was set by the user via the command line, config, etc., then that exact value will be displayed. This makes it easier for the user to recognize the value and saves having to format it into something reasonable, especially for time and size option types.

Note that cfgOptTypeHash and cfgOptTypeList option types are not supported by these functions, but they are generally not displayed to the user as a whole.

This also fixes a bug in config/load.c where time values where not being formatted correctly in an error message.
2021-04-27 12:12:43 -04:00