1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

3344 Commits

Author SHA1 Message Date
David Steele
ba351e9c5c Refactor storage/remote unit test using the protocol remote shim.
Using the local process shim improves coverage and simplifies the tests.
2021-05-26 12:38:23 -04:00
David Steele
58369c02df Add remote process shim.
Run the remote process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the remote process rather than needing to test the remote protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test and the test does not need to run in a container.
2021-05-25 18:16:59 -04:00
David Steele
441c000b5c Factor remote process exec out of protocolRemoteGet().
This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing.

The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
David Steele
e1bc1b3e53 Set buffer-size in the configuration test harness.
Most tests can set buffer-size as needed, but local/remote process shims are forked, so the best way to set buffer-size is via the configuration.
2021-05-25 17:48:42 -04:00
David Steele
cd88f82329 Move protocol test module before config module.
The protocol module should be tested before modules that have a dependency on it.
2021-05-25 11:08:51 -04:00
David Steele
6732806d89 Add local process shim to archive-get/archive-push unit tests.
Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
David Steele
d55b9471a8 Protocol shim improvements.
Add executable to parameter list to avoid first option being lost. The backup, restore, and verify tests worked OK with their first option being defaulted because it ended up being job-retry which worked fine as the default.

Add hrnProtocolLocalShimUninstall() allow the shim to be uninstalled.

Log shim at debug level to make it obvious in the logs when a shim is in use.
2021-05-25 11:00:24 -04:00
David Steele
55f52955a5 Fix shims with more than one function.
Each subsequent function would double every line in the C module.

Move the loop to detect shimmed functions down to prevent this issues.
2021-05-24 17:45:31 -04:00
David Steele
2452c4d5a4
Add PostgreSQL 14 support.
There are no code changes from PostgreSQL 13 so simply add the new version.

Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.

Update the integration tests to handle new PostgreSQL startup messages.
2021-05-24 17:17:03 -04:00
David Steele
eba013b49b
Fix issue when checking links for large numbers of tablespaces.
manifestLinkCheck() was pretty inefficient so large numbers of links caused it to use a lot of memory and eventually crash. This is a more efficient implementation which runs O(nlogn) and uses far less memory.

Checking for duplicate file links has been added, which represents a change in behavior, but hopefully a good one.
2021-05-24 16:29:36 -04:00
David Steele
ccac75e7de
Consolidate RHEL/CentOS user guide into a single document.
The user guide was split primarily to provide documentation for the stop-auto option in PostgreSQL <= 9.5. Now that 9.5 is EOL there does not seem to be a good reason to generate an extra user guide. The stop-auto function is still documented in the reference.

Leave the stop-auto documentation in the user guide in case we want to manually generate documentation for older versions.

Also rename centos to rhel for most identifiers since that is the core platform we are building for, similar to how we label 'debian' builds even though we generally use Ubuntu. With CentOS set to become an upstream for RHEL later this year, we'll likely need to pick a new test distribution, perhaps Rocky Linux if that gets off the ground.
2021-05-24 16:03:15 -04:00
David Steele
f6303152f5 Minor fixes for command/archive-get unit test.
Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test.

Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet().

Move some harness headers down to the correct section.
2021-05-24 12:40:27 -04:00
David Steele
e5e0b21196 Replace system() in tests with HRN_SYSTEM*(). 2021-05-22 15:07:18 -04:00
David Steele
bd40156c22 Rename TEST_SYSTEM*() to HRN_SYSTEM*().
These calls are not tests, rather they setup data for tests.
2021-05-22 14:22:51 -04:00
David Steele
73885f8c2e Replace hrnLogResult() with TEST_RESULT_LOG/_FMT().
The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
David Steele
6baad5cdd1 Replace TEST_ERROR_FMT() with TEST_ERROR() where possible.
Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.
2021-05-22 11:59:43 -04:00
David Steele
a4f057bb70 Remove comment formatting from TEST_*() macros.
Comment formatting was not used much but it incurred a heavy cost in each macro to process possible formatting.

Remove formatted comments where they did not contain valuable information and replace with strZ(strNewFmt()) otherwise.
2021-05-22 11:28:56 -04:00
David Steele
b270253a69 Add defines for many test*() getter functions.
A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests.

Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE.

Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements.

In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
David Steele
aed3d468a1 Rename strNew() to strNewZ() and add parameter-less strNew().
Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not.

In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
Cynthia Shang
68faf1482a
Update contributing documentation.
Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 13:21:25 -04:00
David Steele
15b8b9207d Add log shim.
This allows DEBUG_UNIT and DEBUG_UNIT_EXTERN to be removed since static log variables can now be exposed by functions in the harness.
2021-05-21 12:51:32 -04:00
David Steele
ef63750e0b Add local process shim.
Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test.

The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
David Steele
cab7a97ab6 Add shim feature for unit tests.
A shim allows a test harness to access static functions and variables in a C module, and also allows functions to be shimmed (i.e. overridden) for the purposes of testing.

For instance, coverage testing works when a process that is normally exec'd is run as a forked child process instead.
2021-05-20 18:47:31 -04:00
David Steele
e31df55c8d Factor local process exec out of protocolLocalGet().
This allows protocolLocalExec() to be shimmed, which means the local can be run as a child of the test process, simplifying coverage testing.
2021-05-20 18:35:30 -04:00
David Steele
45a4e801ed
Replace getopt_long() with custom implementation.
getopt_long() requires an exhaustive list of all possible options that may be found on the command line. Because of the way options are indexed (e.g. repo1-4, pg1-8) optionList[] has 827 entries and we have kept it small by curtailing the maximum indexes very severely. Another issue is that getopt_long() scans the array sequentially so parsing gets slower as the index maximums increase.

Replace getopt_long() with a custom implementation that behaves the same but allows options to be parsed with a function instead of using optionList[]. This commit leaves the list in place in order to focus on the getopt_long() replacement, but cfgParseOption() could be replaced with a more efficient implementation that removes the need for optionList[].

This implementation also fixes an issue where invalid options were misreported in the error message if they only had one dash, e.g. -config. This seems to have been some kind of problem in getopt_long(), but no investigation was done since the new implementation fixes it.

Tests were added at 0825428, 2b8d2da, 34dd663, and 384f247 to check that previously untested getopt_long() behavior doesn't change.
2021-05-20 16:02:31 -04:00
David Steele
831ee81466
Rename default command role to main.
Main makes more sense because we refer to the main process in the code, not the default process. The word default is pretty overloaded anyway.
2021-05-20 14:39:47 -04:00
David Steele
8453d89a92 Add recursion to TEST_STORAGE_LIST() macro.
This makes the macro useful when subpaths are present.

Identify types other than files (path, link, etc.) with a single appended character for easier debugging.
2021-05-19 14:18:43 -04:00
David Steele
93eb2f8362 Fix invalid type in test function.
This should have been char * from the start, but of course void * worked fine.
2021-05-19 14:03:42 -04:00
David Steele
4d48850aad Update negate to indicate that it should not be used for new options. 2021-05-19 12:58:48 -04:00
David Steele
870bcf30af Add missing header.
This worked because all places where this header was included, common/type/stringId.h was included first.
2021-05-18 16:48:30 -04:00
David Steele
4fa95af447 Fix typo in struct typedef.
This worked because the type is never declared as struct ExecPub.
2021-05-18 16:38:57 -04:00
Cynthia Shang
c92bc43c1e Simplify defaults for --force option.
All commands have the same default so it is more efficient to set the default for the entire option.
2021-05-18 16:33:45 -04:00
David Steele
320c6e1aad
Remove stanza archive spool path on restore.
Remove stanza archive spool path so existing files do not interfere with the new cluster. For instance, old archive-push acknowledgements could cause a new cluster to skip archiving. This should not happen if a new timeline is selected but better to be safe. Missing stanza spool paths are ignored.

Also add new path expression STORAGE_SPOOL_ARCHIVE to easily access this path.
2021-05-18 15:49:22 -04:00
David Steele
dc0e6645cd Move includes after typedef in info/manifest.h.
This allows the Manifest type to be used in headers that need it, e.g. command/backup/common.h.
2021-05-18 09:08:06 -04:00
David Steele
9af033194a
Add automatic GCS authentication for GCE instances.
When running on a GCE instance the authentication token can be pulled directly from the instance metadata. This is configured with repo-gcs-key-type=auto.

In a separate commit (26fefa6), move the code that parses the token response into a separate function, storageGcsAuthToken(), since it is now needed by two key types. This drastically improves the readability of the main commit.
2021-05-17 14:55:50 -04:00
David Steele
0152075e6b Remove default VM for test.pl --coverage-only option.
When running outside of our standard Vagrantfile the default will not be set correctly, so require the user to set it.

In any case, this option is primarily useful for reporting so note that in the command line help.
2021-05-17 11:35:22 -04:00
David Steele
384f247077 Add config/parse tests for config/env partial options.
Partial option matching is valid on the command line but should never be used in config files or environment options.
2021-05-17 09:33:36 -04:00
David Steele
ae7f0af202 Move PostgreSQL version interface test functions to a test harness.
Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core.

Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
David Steele
18cc02238b Remove overzealous cast.
There does not seem to be a good reason to have an explicit cast for such a small number, so remove it.

Leave PG_CONTROL_SIZE alone since it will be removed in an upcoming commit.
2021-05-17 07:15:13 -04:00
David Steele
9235c62c6b Revert ignoring catalog version when identifying a PostgreSQL version.
927d9adb changed the way CATALOG_VERSION_NO is used to identify PostgreSQL versions since PG_CONTROL_VERSION is generally bumped with each release. The goal was to make the beta/rc period less painful because any CATALOG_VERSION_NO bump renders pgBackRest inoperative.

This worked, but in fact we'd rather be stricter about which CATALOG_VERSION_NO we accept when identifying a version of PostgreSQL. It is not just about identifying a major version, but making sure the build contains all the functions and catalogs we expect to make pgBackRest work correctly. It is better to reject early dev/beta/rc builds that may not work.

Since 927d9adb was relatively recent the chance that this stricter checking will cause a problem seems minimal, so revert to checking CATALOG_VERSION_NO for every PostgreSQL version.

Leave in place the code that pulls CATALOG_VERSION_NO from pg_control rather than the internal constant since the plan is still to allow catalog versions to "float" during the PostgreSQL beta/rc phase, which will be the subject of a future commit.
2021-05-17 07:12:46 -04:00
David Steele
f45e76fa2e
Fix issues with leftover spool files from a prior restore.
If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue.

Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process.

Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
David Steele
34dd6636b8 Add config/parse tests for options and option args with spaces.
If an option name has a space at the beginning then it will be considered an invalid command, but a space at the end is an invalid option. Add tests for these conditions.

Spaces in option arguments should be preserved, so add a test to be sure this is true.
2021-05-13 17:01:21 -04:00
David Steele
2b8d2daca1 Add config/parse tests for partial options.
Add coverage for partial options, i.e. an option that is partially specified but only prefix matches with a single valid option.
2021-05-13 16:54:25 -04:00
David Steele
add0f9b77d Fix an ungrammatical error message in config parsing. 2021-05-13 12:37:59 -04:00
David Steele
0825428fef Add config/parse test where the option/value are not in the same arg.
All the tests in this module use --option=value syntax so add one test with --option value syntax for coverage.
2021-05-13 12:35:11 -04:00
David Steele
26fefa6aee Refactor GCS token response parsing into a separate function.
This function is also useful for the upcoming auto authentication. Refactoring in a separate commit to make the feature commit more readable.
2021-05-13 06:59:57 -04:00
David Steele
7b71604def Use existing variable for GCS test server port.
Also fix incorrect separator comment.
2021-05-12 13:21:59 -04:00
David Steele
5464ac83d1
Convert option values in commands to StringId.
Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.

For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.
2021-05-11 17:24:30 -04:00
David Steele
87ba2ca253 Change CentOS 7 documentation test to CentOS 8.
The CentOS 7 documentation test relies on PostgreSQL 9.5 which has been removed from the yum.p.o repository package. Switch the test to CentOS 8 to fix the immediate issue, but a decision on the PostgreSQL 9.5 documentation will need to be made before the next release.
2021-05-11 16:54:42 -04:00
David Steele
2bfebff2ec Add instructions to install lcov for MacOS. 2021-05-04 15:51:46 -04:00