1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3537 Commits

Author SHA1 Message Date
Cynthia Shang
4c9ddf5ef2
Update storage tests to use standard patterns.
The storage tests were not modified to the HRN_STORAGE_* nor TEST_STORAGE_* macros as these test are testing the storage drivers.

Note that posixTest.c removed an extraneous #endif // TEST_CONTAINER_REQUIRED and #ifdef TEST_CONTAINER_REQUIRED.

This PR includes all files in the storage/* test directory, namely: azureTest.c, cifsTest.c, gcsTest.c, posixTest.c, remoteTest.c, s3Test.c
2021-07-22 18:17:08 -04:00
David Steele
3a37482de9 Update db test module error for newer libpq versions.
PostgreSQL 14 libpq throws a more detailed error, so adjust the regular expression to handle the old and the new errors.
2021-07-22 15:54:30 -04:00
David Steele
eb98b8d2db Fix typo. 2021-07-21 13:19:09 -04:00
Cynthia Shang
e3d05e2b4a
Update contributing documentation and add pull request template. 2021-07-21 13:14:06 -04:00
David Steele
46992acd80 Update Vagrantfile box version. 2021-07-20 15:43:48 -04:00
Cynthia Shang
4992508764
Update command/restore test to use standard patterns. 2021-07-20 13:30:17 -04:00
David Steele
9ee9b1fad6 Remove test.pl --smart, --dev, and --dev-test options.
--smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it.

Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 12:01:10 -04:00
Cynthia Shang
1522cb4ed2 Allow NULL path in HRN_STORAGE_PATH_REMOVE() macro.
Update command/backup test to pass NULL where appropriate.
2021-07-19 15:21:40 -04:00
Cynthia Shang
a6cdf5b22f
Update command/repo test to use standard patterns. 2021-07-19 10:47:49 -04:00
Cynthia Shang
ebe5848494
Update command/local and command/remote tests to use standard patterns.
Note that the logging output display of a parent/child test may look jumbled on some systems since the child and parent are attempting to log information at the same time. This is not an issue with the actual test, rather a harness issue that would be beyond the scope of this project to fix.
2021-07-19 09:51:11 -04:00
David Steele
c5ae047e76 Partial migration of config code generation to C.
Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.

This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.

The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
2021-07-18 19:02:01 -04:00
David Steele
18ffec72da Update order of configuration options.
The upcoming C parser requires that options be in dependency order, i.e. inherited options must come before the options that inherit them.

This change has been separated to demonstrate that it does no affect the auto-generated code.
2021-07-18 18:31:43 -04:00
David Steele
6397d73535 Update test container OS versions.
Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial.
Update Ubuntu 18.04 to 20.04.
Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up.

Add --no-install-recommends to apt-get commands to save a bit of time and space.

Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small.

Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 17:29:20 -04:00
David Steele
81602f1593 Remove test.pl --vm-host option.
This option was once used to optimize bin builds but has been defunct for some time.
2021-07-17 11:09:53 -04:00
Cynthia Shang
d57299ada8 Allow NULL path in TEST_STORAGE_LIST() macro. 2021-07-16 13:07:16 -04:00
Cynthia Shang
117ffe8896
Update command/help tests to use standard patterns.
In the first test (helpRenderSplitSize) added test for empty list and in that and some other tests, the test comment was updated to clarify a bit more what the actual tests is trying to accomplish.

Note that help test parameters can only use the harnessConfig system when testing option values that have been set since options passed to the help command are not "set" options.
2021-07-15 17:51:48 -04:00
Cynthia Shang
4ad0bbda53
Update command/backup tests to use standard patterns.
Includes backup and backupCommon tests.

Some tests in backupTest were split out where they were originally combined into a single boolean check - which made it difficult to determine which part of the conditional failed.

String values were also removed where they were no longer needed.
2021-07-15 17:00:20 -04:00
David Steele
cdc89fc975 Allow NULL path in HRN_STORAGE_PATH_CREATE() macro. 2021-07-15 16:05:32 -04:00
David Steele
25508a48e0 Add contributors missed in 8e187cdb. 2021-07-15 14:34:47 -04:00
Cynthia Shang
c0feca4673 Add noParentCreate parameter to HRN_STORAGE_PATH_CREATE() macro. 2021-07-15 13:31:54 -04:00
David Steele
4a6ca54b47 Remove last vestiges of key replacement missed in b270253a. 2021-07-15 13:19:49 -04:00
Stefan Fercot
563d117967
Allow restore --type=lsn.
The LSN recovery target was added in PostgreSQL 10 but did not get added to pgBackRest.
2021-07-15 13:02:41 -04:00
Stefan Fercot
8fac1ac50d
Loop while waiting for checkpoint LSN to reach replay LSN.
It is possible for the checkpoint LSN to lag slightly behind the replay LSN until pg_control has been updated.

Add a loop to keep checking rather than failing when the checkpoint LSN has not been updated.
2021-07-15 07:45:06 -04:00
Cynthia Shang
a6691c6f61 Clarify restore --type behavior in command reference. 2021-07-14 16:06:42 -04:00
David Steele
d791bb7298 Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros.
This removes a lot of boiler plate where every instance needs to create these interfaces.

Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes.

In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
Cynthia Shang
be7b8a485b Error in TEST_STORAGE_LIST() when path does not exist. 2021-07-13 14:28:58 -04:00
David Steele
1ace1ac938 Improve HRN_FORK*() macros.
Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults.

Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
2021-07-13 14:22:53 -04:00
David Steele
76cfbf833d Rename HARNESS_FORK*() macros to HRN_FORK*().
This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
David Steele
d6797009f8 Add ioFdReadNewOpen() and ioFdWriteNewOpen().
These functions construct and open in one call, which allows them to be used as function parameters.
2021-07-13 11:13:57 -04:00
Stefan Fercot
f70ddc16d7 Fix typo. 2021-07-13 07:09:55 -04:00
Cynthia Shang
26dd918856 Add compressType parameter to TEST_STORAGE_GET() macro.
Also remove duplicate extension output in HRN_STORAGE_PUT().
2021-07-12 15:36:52 -04:00
Cynthia Shang
f2ec4e00a4 Add cipher parameters to TEST_STORAGE_GET() macro. 2021-07-09 15:59:17 -04:00
David Steele
3c8819e10f
Add CodeQL static code analysis.
Also fix some minor issues identified, specifically using gmtime_r()/localtime_r() vs gmtime()/localtime().
2021-07-09 14:16:10 -04:00
David Steele
849ab343aa Change level of backup/restore copied file logging to detail.
The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.
2021-07-09 13:50:35 -04:00
David Steele
a7a041e241 Fix flapping coverage in storage/remote test.
The protocol does not put an end message on exit so there was a race between the main process exiting and the remote processes exiting. If the main process exited first then the remote processes might not write coverage data causing coverage to fail.

Fix by calling exit explicitly at the end of the test and update the harness to put an end message so the exits are synchronized.
2021-07-09 11:18:45 -04:00
David Steele
3369440eef Make pgControlFromBuffer() static.
This function is no longer used externally.
2021-07-09 09:07:41 -04:00
Cynthia Shang
8bb0b28455
Update command/control and command/command tests to use standard patterns.
In the commandTest the HRN_STORAGE_REMOVE replacement uses .errorOnMissing when the code being tested added the file. The reason for this is 3 fold:

1. to ensure that an inadvertent typo in the path/file name does not go undetected,
2. to ensure that nothing else has removed the file prior to the call, and
3. consistency

Also, added "stanza" to comment when a stanza stop file is removed vs an "all" stop file.
2021-07-08 16:34:11 -04:00
David Steele
8e1807cdbe
Detect errors in S3 multi-part upload finalize.
Multi-part upload may fail despite returning an HTTP success code. Check for the ETag field in the result and if not present consider the upload to have failed. This will trigger a retry at the local job level.
2021-07-08 13:06:52 -04:00
David Steele
43b874628c
Fix detection of circular symlinks.
Links were followed before they were checked for validity so a circular link would send the manifest build into endless recursion leading to a crash. Fix by moving the recursion after the link check.

Note that this issue has existed since the C migration and was not introduced by the refactor in eba013b.
2021-07-08 11:30:23 -04:00
David Steele
6839335633 Increase harness log level to detail in command/backup test.
Also clean up some unneeded calls to harnessLogLevelReset().
2021-07-08 11:17:13 -04:00
Stefan Fercot
056a3070e0
Automatically create data directory on restore.
Data directory creation was added during the C migration, but creation of the base data directory (PGDATA) was prevented by a check migrated from Perl.

Remove the check and update tests to create the data directory at least once.
2021-07-08 10:33:19 -04:00
David Steele
34e51ee7b6 Allow NULL path in HRN_STORAGE_MODE() macro. 2021-07-08 09:25:42 -04:00
Cynthia Shang
62e8d97af0
Update command/archive tests to use standard patterns.
Includes archiveCommon, archiveGet and archivePush.

Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
Eric Radman
23bdc3deb6
Fix documentation and comment typos.
Identified using `ag -l | igor`.
2021-07-01 11:50:03 -04:00
Cynthia Shang
6a90d46909
Update info/manifest test to use standard patterns. 2021-06-30 08:51:11 -04:00
David Steele
39c1c10120 Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants.
It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
Cynthia Shang
c6208113bf Add timeModified parameter to HRN_STORAGE_PUT() macro. 2021-06-28 12:14:26 -04:00
Cynthia Shang
065a8c8454
Replace defines with string constants in unit tests.
For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes.

Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
Cynthia Shang
fb2d111dcd
Update info/infoBackup test to use standard patterns. 2021-06-24 16:11:14 -04:00
Cynthia Shang
d0f3a3f2af Add HRN_STORAGE_COPY() macro. 2021-06-24 14:01:03 -04:00