1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1910 Commits

Author SHA1 Message Date
David Steele
d6797009f8 Add ioFdReadNewOpen() and ioFdWriteNewOpen().
These functions construct and open in one call, which allows them to be used as function parameters.
2021-07-13 11:13:57 -04:00
Cynthia Shang
26dd918856 Add compressType parameter to TEST_STORAGE_GET() macro.
Also remove duplicate extension output in HRN_STORAGE_PUT().
2021-07-12 15:36:52 -04:00
Cynthia Shang
f2ec4e00a4 Add cipher parameters to TEST_STORAGE_GET() macro. 2021-07-09 15:59:17 -04:00
David Steele
849ab343aa Change level of backup/restore copied file logging to detail.
The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.
2021-07-09 13:50:35 -04:00
David Steele
a7a041e241 Fix flapping coverage in storage/remote test.
The protocol does not put an end message on exit so there was a race between the main process exiting and the remote processes exiting. If the main process exited first then the remote processes might not write coverage data causing coverage to fail.

Fix by calling exit explicitly at the end of the test and update the harness to put an end message so the exits are synchronized.
2021-07-09 11:18:45 -04:00
Cynthia Shang
8bb0b28455
Update command/control and command/command tests to use standard patterns.
In the commandTest the HRN_STORAGE_REMOVE replacement uses .errorOnMissing when the code being tested added the file. The reason for this is 3 fold:

1. to ensure that an inadvertent typo in the path/file name does not go undetected,
2. to ensure that nothing else has removed the file prior to the call, and
3. consistency

Also, added "stanza" to comment when a stanza stop file is removed vs an "all" stop file.
2021-07-08 16:34:11 -04:00
David Steele
8e1807cdbe
Detect errors in S3 multi-part upload finalize.
Multi-part upload may fail despite returning an HTTP success code. Check for the ETag field in the result and if not present consider the upload to have failed. This will trigger a retry at the local job level.
2021-07-08 13:06:52 -04:00
David Steele
43b874628c
Fix detection of circular symlinks.
Links were followed before they were checked for validity so a circular link would send the manifest build into endless recursion leading to a crash. Fix by moving the recursion after the link check.

Note that this issue has existed since the C migration and was not introduced by the refactor in eba013b.
2021-07-08 11:30:23 -04:00
David Steele
6839335633 Increase harness log level to detail in command/backup test.
Also clean up some unneeded calls to harnessLogLevelReset().
2021-07-08 11:17:13 -04:00
Stefan Fercot
056a3070e0
Automatically create data directory on restore.
Data directory creation was added during the C migration, but creation of the base data directory (PGDATA) was prevented by a check migrated from Perl.

Remove the check and update tests to create the data directory at least once.
2021-07-08 10:33:19 -04:00
David Steele
34e51ee7b6 Allow NULL path in HRN_STORAGE_MODE() macro. 2021-07-08 09:25:42 -04:00
Cynthia Shang
62e8d97af0
Update command/archive tests to use standard patterns.
Includes archiveCommon, archiveGet and archivePush.

Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
Eric Radman
23bdc3deb6
Fix documentation and comment typos.
Identified using `ag -l | igor`.
2021-07-01 11:50:03 -04:00
Cynthia Shang
6a90d46909
Update info/manifest test to use standard patterns. 2021-06-30 08:51:11 -04:00
David Steele
39c1c10120 Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants.
It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
Cynthia Shang
c6208113bf Add timeModified parameter to HRN_STORAGE_PUT() macro. 2021-06-28 12:14:26 -04:00
Cynthia Shang
065a8c8454
Replace defines with string constants in unit tests.
For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes.

Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
Cynthia Shang
fb2d111dcd
Update info/infoBackup test to use standard patterns. 2021-06-24 16:11:14 -04:00
Cynthia Shang
d0f3a3f2af Add HRN_STORAGE_COPY() macro. 2021-06-24 14:01:03 -04:00
Cynthia Shang
50c129ae9d Add expression parameter to TEST_STORAGE_LIST() macro. 2021-06-24 13:57:55 -04:00
David Steele
6a1c0337dd
Binary protocol.
Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol.

There are a number advantages:

* The pack type is more compact than JSON and are more efficient to render/parse.
* Packs are more strictly typed than JSON.
* Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state.
* There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files.

Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C.

There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
Cynthia Shang
bffb43ea3f
Update command/check test to use standard patterns. 2021-06-24 12:23:09 -04:00
Cynthia Shang
17c9ed0ef0
Update command/info test to use standard patterns.
Some tests had to be reordered or updated, as follows:

* Reordered tests at line 317 and 331 to avoid unnecessary file removal.

* Change "stanza found" test at line 1735 to reflect real-life scenario. Originally this test had the cipher-pass environment key set up which caused the RepoGrp to be 2 but with no valid repo path. This resulted in the repo loops executing for the repo2 but since the path was not defined, the tests just reported "none" for cipher which is incorrect since the repo IS encrypted.

* Moved order of HRN_CFG_LOAD in some tests when able to avoid using storageTest.
2021-06-24 08:55:44 -04:00
Cynthia Shang
ff95eddc36
Update command/verify test to use standard patterns. 2021-06-23 18:09:46 -04:00
Cynthia Shang
8dd882bade
Update info/infoArchive test to use standard patterns. 2021-06-23 16:20:47 -04:00
David Steele
e697f5705a
Clear error when a CATCH() block finishes.
It is better to clear errors after the catch block completes rather than leave them set until the next error. This also make is possible to tell when a error is currently being handled, which a function further down the stack might use to modify its behavior. Currently this is only useful in testing, but clearing the error seems like a good idea in general.

Two places used errors outside the CATCH() block. Mem context cleanup now uses a FINALLY() which is a better implementation anyway. The error handling in main() now calls exitSafe() from withing the CATCH() block.
2021-06-23 13:02:19 -04:00
Cynthia Shang
7ba5aef763 Add mode parameter to HRN_STORAGE_PUT() macro. 2021-06-23 12:07:19 -04:00
Cynthia Shang
03021c6a17 Update command/stanza test to use standard patterns.
No core code changes; only changes for stanza-create, stanza-upgrade, and stanza-delete command unit tests.
2021-06-14 13:28:27 -04:00
David Steele
80d63a5e8b Ignore mismatched close requests in the pq shim during error processing. 2021-06-12 18:00:59 -04:00
David Steele
fba0437179 Improve error handling in db test module.
Errors could cause segfaults since server objects were freed immediately but client destructors were run much later.
2021-06-12 17:48:35 -04:00
David Steele
0a65e59b55 Update pg-path options in db test module to be valid.
This allows files to be saved in the paths, which would have failed when they were invalid.
2021-06-11 19:16:25 -04:00
David Steele
97155bad86 Add mode, Pack, and StringId to the Pack type.
Add StringList, which is not a primitive type but rather an array of String types.

Also update pckWriteToLog() to work after pckWriteEnd(), i.e. this->tagStackTop is NULL.
2021-06-10 12:40:55 -04:00
David Steele
85a55bd401 Add pckReadMove() and pckWriteMove().
Move a PackRead or PackWrite object to a new mem context.

Also note that these functions may not work as expected with pack objects created by pckReadNewBuf() and pckWriteNewBuf() since the pack object does not have ownership of the passed buffer and cannot move it.
2021-06-10 09:25:57 -04:00
David Steele
788f7c1f30 Skip sleep in sleepMSec() when sleep time is zero.
There is no point in sleeping when there is no sleep time.

The advantage is that callers do not need to perform the check themselves.
2021-06-10 09:24:42 -04:00
David Steele
5e1a8e6895 Add error test harness/shim.
The hrnErrorThrowP() macro allows errors with specified fields to be generated, which simplifies testing.

Update the common/exit test to use the new macro.
2021-06-10 09:21:15 -04:00
Cynthia Shang
c5897007c4 Add HRN_STORAGE_MOVE() macro.
Update command/expire test to show how it is used.
2021-06-09 12:41:23 -04:00
David Steele
15dfbf4c1d Use lower-case names from Azure identifiers in integration tests.
Azurite, which is used for testing, did not enforce this before so the capital letters were not a problem. Now Azurite enforces the same rules as Azure so use lower-case identifiers instead.

These names were only used in integration tests so there was no production impact.
2021-06-09 12:28:40 -04:00
David Steele
c6a8528e31 Add optional remove to TEST_STORAGE_EXISTS().
This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before.

Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
David Steele
4a075b7252
Add support for more Pack types.
Since the pack type was stored in 4 bits, only 15 values were allowed (0 was reserved).

Allow virtually unlimited types by storing type info in a base-128 encoded integer following the tag when the type bits in the tag are set to 0xF.

Also separate the type IDs used in the pack (PackTypeMap) from those presented to the user (PackType). The prior PackType enum exposed implementation details to the user, e.g. pckTypeUnknown.
2021-06-08 12:55:00 -04:00
Cynthia Shang
3f9fbc3c24
Update command/expire test to use standard patterns.
The way tests are written has evolved over time. Update the command/expire test to use the new test patterns.
2021-06-08 11:00:28 -04:00
David Steele
4ccb42bb7a Rename param structs to match function names.
The functions were named with short integer representations (e.g. I32) but the param structs were using longer ones, e.g. UInt32. Shorten the integer representations in the param structs to match.

Also rename pckReadUInt64Internal() to pckReadU64Internal() for the same reason.
2021-06-08 08:44:10 -04:00
David Steele
df8276f59f Switch order of remote startup in storage/remote test.
The pg storage must be started before the repo storage to set the max remotes allowed to 2. The protocol helper expects all remotes to have the same type so we are cheating here a bit, but without this ordering the second remote will never be sent an explicit exit and may not save coverage data.
2021-06-07 11:53:25 -04:00
David Steele
a607750be1 Handle NULL VariantList in JSON.
This worked if the Variant was NULL but not if the Variant contained a NULL.
2021-06-07 07:46:51 -04:00
David Steele
d10a99d73b v2.34: PostgreSQL 14 Support
Bug Fixes:

* Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.)
* Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.)
* Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.)
* Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)

Features:

* Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)
* Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.)
* Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
* Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)

Improvements:

* Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)

Documentation Improvements:

* Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)
* Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)
* Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
2021-06-07 06:51:08 -04:00
Cynthia Shang
c9a8ff27f2 Add HRN_STORAGE_PATH_REMOVE() macro.
Update command/archive-push test to show how it is used.
2021-06-02 11:46:54 -04:00
David Steele
a97fc6e708 Add storage/remote test for unknown user/group name. 2021-06-01 17:10:15 -04:00
David Steele
8250990afb Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro.
HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases.

It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
David Steele
c1277677a3 Add HRN_STORAGE_PATH_CREATE() macro.
Update command/archive-get test to show how it is used.

Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
David Steele
c8427682ea Improve storage harness/test macros.
Make the macros more consistent in format and make sure that each macro outputs a line number before doing any work so when errors happen it is clear where they happened.

Add noRecurse option to TEST_STORAGE_LIST().

Add comment option to all storage macros.
2021-05-28 14:39:43 -04:00
David Steele
0a43be0183 Use L to denote line numbers in test output.
The lower-case l was too easy to confuse with a 1.
2021-05-28 13:08:41 -04:00