1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3513 Commits

Author SHA1 Message Date
David Steele
a76cb57c80 Remove parse.auto.h.
At one time there was a lot more in this header but it got merged with the enums and constants in config.auto.c, leaving only the ConfigOptionType enum.

Auto-generating the ConfigOptionType enum is not very useful since new option types require code in parse.c.
2021-06-07 08:48:09 -04:00
David Steele
7a449f0e49 Remove unused command string constants. 2021-06-07 08:18:02 -04:00
David Steele
a607750be1 Handle NULL VariantList in JSON.
This worked if the Variant was NULL but not if the Variant contained a NULL.
2021-06-07 07:46:51 -04:00
David Steele
aaca4df983 Update config.guess and config.sub to latest versions. 2021-06-07 07:36:56 -04:00
David Steele
a7b1d2b9d0 Begin v2.35 development. 2021-06-07 07:32:34 -04:00
David Steele
d10a99d73b v2.34: PostgreSQL 14 Support
Bug Fixes:

* Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.)
* Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.)
* Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.)
* Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)

Features:

* Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)
* Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.)
* Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
* Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)

Improvements:

* Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)

Documentation Improvements:

* Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)
* Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)
* Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
2021-06-07 06:51:08 -04:00
Cynthia Shang
c9a8ff27f2 Add HRN_STORAGE_PATH_REMOVE() macro.
Update command/archive-push test to show how it is used.
2021-06-02 11:46:54 -04:00
David Steele
a97fc6e708 Add storage/remote test for unknown user/group name. 2021-06-01 17:10:15 -04:00
David Steele
d35315aaf9 Add missing parameter log debug. 2021-06-01 15:33:07 -04:00
David Steele
8250990afb Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro.
HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases.

It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
David Steele
c1277677a3 Add HRN_STORAGE_PATH_CREATE() macro.
Update command/archive-get test to show how it is used.

Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
David Steele
c8427682ea Improve storage harness/test macros.
Make the macros more consistent in format and make sure that each macro outputs a line number before doing any work so when errors happen it is clear where they happened.

Add noRecurse option to TEST_STORAGE_LIST().

Add comment option to all storage macros.
2021-05-28 14:39:43 -04:00
David Steele
0a43be0183 Use L to denote line numbers in test output.
The lower-case l was too easy to confuse with a 1.
2021-05-28 13:08:41 -04:00
David Steele
cfe8e51910 Remove line parameter from hrnTestResultBegin().
Instead store the line number in hrnTestLogPrefix() so it doesn't need to be passed to hrnTestResultBegin().

Also add missing linefeed in hrnStorageList().
2021-05-28 08:50:28 -04:00
David Steele
b3ac9e8010 Remove unused padding parameter from hrnTestLogPrefix().
Also add fflush() so log prefix is displayed immediately.
2021-05-28 08:03:22 -04:00
David Steele
52217f1244 Replace storageTest with storagePg*() in command/archive-get test.
All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted.

Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
Cynthia Shang
aceb956815
Change archive expiration logging from detail to info level.
To assist with issue debugging, removal of files during archive expiration has been changed from detail level logging to info level.
2021-05-27 08:03:44 -04:00
Stefan Fercot
5b332b22b5
Add repo-retention-history option to expire backup history.
The default is to keep all backup history to match the current behavior. In minimal configuration (0 days), unexpired backups are always kept in history.

When a full backup manifest expires, all dependent differential/incremental manifests expire as well.
2021-05-26 13:07:18 -04:00
David Steele
a69a6c2f55 Show diff when test.pl --gen-check detects unexpected autogeneration.
This will help track down autogeneration issues if they can't be reproduced locally.
2021-05-26 12:41:15 -04:00
David Steele
ba351e9c5c Refactor storage/remote unit test using the protocol remote shim.
Using the local process shim improves coverage and simplifies the tests.
2021-05-26 12:38:23 -04:00
David Steele
58369c02df Add remote process shim.
Run the remote process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the remote process rather than needing to test the remote protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test and the test does not need to run in a container.
2021-05-25 18:16:59 -04:00
David Steele
441c000b5c Factor remote process exec out of protocolRemoteGet().
This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing.

The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
David Steele
e1bc1b3e53 Set buffer-size in the configuration test harness.
Most tests can set buffer-size as needed, but local/remote process shims are forked, so the best way to set buffer-size is via the configuration.
2021-05-25 17:48:42 -04:00
David Steele
cd88f82329 Move protocol test module before config module.
The protocol module should be tested before modules that have a dependency on it.
2021-05-25 11:08:51 -04:00
David Steele
6732806d89 Add local process shim to archive-get/archive-push unit tests.
Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
David Steele
d55b9471a8 Protocol shim improvements.
Add executable to parameter list to avoid first option being lost. The backup, restore, and verify tests worked OK with their first option being defaulted because it ended up being job-retry which worked fine as the default.

Add hrnProtocolLocalShimUninstall() allow the shim to be uninstalled.

Log shim at debug level to make it obvious in the logs when a shim is in use.
2021-05-25 11:00:24 -04:00
David Steele
55f52955a5 Fix shims with more than one function.
Each subsequent function would double every line in the C module.

Move the loop to detect shimmed functions down to prevent this issues.
2021-05-24 17:45:31 -04:00
David Steele
2452c4d5a4
Add PostgreSQL 14 support.
There are no code changes from PostgreSQL 13 so simply add the new version.

Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.

Update the integration tests to handle new PostgreSQL startup messages.
2021-05-24 17:17:03 -04:00
David Steele
eba013b49b
Fix issue when checking links for large numbers of tablespaces.
manifestLinkCheck() was pretty inefficient so large numbers of links caused it to use a lot of memory and eventually crash. This is a more efficient implementation which runs O(nlogn) and uses far less memory.

Checking for duplicate file links has been added, which represents a change in behavior, but hopefully a good one.
2021-05-24 16:29:36 -04:00
David Steele
ccac75e7de
Consolidate RHEL/CentOS user guide into a single document.
The user guide was split primarily to provide documentation for the stop-auto option in PostgreSQL <= 9.5. Now that 9.5 is EOL there does not seem to be a good reason to generate an extra user guide. The stop-auto function is still documented in the reference.

Leave the stop-auto documentation in the user guide in case we want to manually generate documentation for older versions.

Also rename centos to rhel for most identifiers since that is the core platform we are building for, similar to how we label 'debian' builds even though we generally use Ubuntu. With CentOS set to become an upstream for RHEL later this year, we'll likely need to pick a new test distribution, perhaps Rocky Linux if that gets off the ground.
2021-05-24 16:03:15 -04:00
David Steele
f6303152f5 Minor fixes for command/archive-get unit test.
Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test.

Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet().

Move some harness headers down to the correct section.
2021-05-24 12:40:27 -04:00
David Steele
e5e0b21196 Replace system() in tests with HRN_SYSTEM*(). 2021-05-22 15:07:18 -04:00
David Steele
bd40156c22 Rename TEST_SYSTEM*() to HRN_SYSTEM*().
These calls are not tests, rather they setup data for tests.
2021-05-22 14:22:51 -04:00
David Steele
73885f8c2e Replace hrnLogResult() with TEST_RESULT_LOG/_FMT().
The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
David Steele
6baad5cdd1 Replace TEST_ERROR_FMT() with TEST_ERROR() where possible.
Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.
2021-05-22 11:59:43 -04:00
David Steele
a4f057bb70 Remove comment formatting from TEST_*() macros.
Comment formatting was not used much but it incurred a heavy cost in each macro to process possible formatting.

Remove formatted comments where they did not contain valuable information and replace with strZ(strNewFmt()) otherwise.
2021-05-22 11:28:56 -04:00
David Steele
b270253a69 Add defines for many test*() getter functions.
A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests.

Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE.

Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements.

In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
David Steele
aed3d468a1 Rename strNew() to strNewZ() and add parameter-less strNew().
Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not.

In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
Cynthia Shang
68faf1482a
Update contributing documentation.
Add more examples of how to write code, add configuration options, test, etc.
2021-05-21 13:21:25 -04:00
David Steele
15b8b9207d Add log shim.
This allows DEBUG_UNIT and DEBUG_UNIT_EXTERN to be removed since static log variables can now be exposed by functions in the harness.
2021-05-21 12:51:32 -04:00
David Steele
ef63750e0b Add local process shim.
Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test.

The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
David Steele
cab7a97ab6 Add shim feature for unit tests.
A shim allows a test harness to access static functions and variables in a C module, and also allows functions to be shimmed (i.e. overridden) for the purposes of testing.

For instance, coverage testing works when a process that is normally exec'd is run as a forked child process instead.
2021-05-20 18:47:31 -04:00
David Steele
e31df55c8d Factor local process exec out of protocolLocalGet().
This allows protocolLocalExec() to be shimmed, which means the local can be run as a child of the test process, simplifying coverage testing.
2021-05-20 18:35:30 -04:00
David Steele
45a4e801ed
Replace getopt_long() with custom implementation.
getopt_long() requires an exhaustive list of all possible options that may be found on the command line. Because of the way options are indexed (e.g. repo1-4, pg1-8) optionList[] has 827 entries and we have kept it small by curtailing the maximum indexes very severely. Another issue is that getopt_long() scans the array sequentially so parsing gets slower as the index maximums increase.

Replace getopt_long() with a custom implementation that behaves the same but allows options to be parsed with a function instead of using optionList[]. This commit leaves the list in place in order to focus on the getopt_long() replacement, but cfgParseOption() could be replaced with a more efficient implementation that removes the need for optionList[].

This implementation also fixes an issue where invalid options were misreported in the error message if they only had one dash, e.g. -config. This seems to have been some kind of problem in getopt_long(), but no investigation was done since the new implementation fixes it.

Tests were added at 0825428, 2b8d2da, 34dd663, and 384f247 to check that previously untested getopt_long() behavior doesn't change.
2021-05-20 16:02:31 -04:00
David Steele
831ee81466
Rename default command role to main.
Main makes more sense because we refer to the main process in the code, not the default process. The word default is pretty overloaded anyway.
2021-05-20 14:39:47 -04:00
David Steele
8453d89a92 Add recursion to TEST_STORAGE_LIST() macro.
This makes the macro useful when subpaths are present.

Identify types other than files (path, link, etc.) with a single appended character for easier debugging.
2021-05-19 14:18:43 -04:00
David Steele
93eb2f8362 Fix invalid type in test function.
This should have been char * from the start, but of course void * worked fine.
2021-05-19 14:03:42 -04:00
David Steele
4d48850aad Update negate to indicate that it should not be used for new options. 2021-05-19 12:58:48 -04:00
David Steele
870bcf30af Add missing header.
This worked because all places where this header was included, common/type/stringId.h was included first.
2021-05-18 16:48:30 -04:00
David Steele
4fa95af447 Fix typo in struct typedef.
This worked because the type is never declared as struct ExecPub.
2021-05-18 16:38:57 -04:00