1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2120 Commits

Author SHA1 Message Date
David Steele
c38d6926d6 Revert Azurite version for testing to 3.14.0.
3.14.2 is causing breakage in the documentation. There is no obvious cause so for now just revert to the last working version.
2021-09-09 08:48:45 -04:00
David Steele
f4e1babf6b Migrate command-line help generation to C.
Command-line help is now generated at build time so it does not need to be committed. This reduces churn on commits that add configuration and/or update the help.

Since churn is no longer an issue, help.auto.c is bzip2 compressed to save space in the binary.

The Perl config parser (Data.pm) has been moved to doc/lib since the Perl build path is no longer required.

Likewise doc/xml/reference.xml has been moved to src/build/help/help.xml since it is required at build time.
2021-09-08 18:16:06 -04:00
David Steele
1afea449e2 Build newer valgrind and move CA cert install in test containers.
The newer version of valgrind helps with some arm64 issues that have been fixed since the architecture has become more popular. Also add the valgrind builds to the Vagrantfile and Dockerfile.

Move the CA cert install from the base container to the test container. This means the CA cert can be changed without rebuilding all the base containers.
2021-09-02 13:26:21 -04:00
David Steele
475b57c89b Allow additional memory to be allocated with a mem context.
The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer.

Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size.

Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall.

Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
David Steele
02b06aa495
Increase max index allowed for pg/repo options to 256.
The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.

Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.

Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.

Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.

cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 12:09:50 -04:00
David Steele
5d857e196a Do not use certs in container for testing.
The certs are available in test/certificate so it makes more sense to use them there. In addition the container does not need to be rebuilt unless the CA cert changes.
2021-08-25 07:33:31 -04:00
David Steele
d72d4415a7 Use contextParentIdx instead of a loop when moving a context.
contextParentIdx was introduced in 90709dfd to improve the performance of mem context frees. memContextMove() did not get the message, however, and continued to use a loop to find the mem context in the old parent.

Use contextParentIdx to find the mem context in the old parent to avoid a loop.
2021-08-23 08:27:07 -04:00
David Steele
3787cf7803 v2.35: Binary Protocol
IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.

Bug Fixes:

* Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang, Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)
* Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)
* Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)

Improvements:

* Binary protocol. (Reviewed by Cynthia Shang.)
* Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)
* Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)
* Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)
* Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)
* Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)

Documentation Bug Fixes:

* Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)

Documentation Improvements:

* Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)
* Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)

Test Suite Improvements:

* Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
* Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)
* Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2021-08-23 06:52:51 -04:00
David Steele
ad5e063990 Update containers for new PostgreSQL releases.
Also add a package that is required for older Debian versions.
2021-08-19 17:25:57 -04:00
David Steele
590e759968 Rename stringz.h to stringZ.h.
This is more consistent with our file naming conventions.
2021-08-19 11:57:37 -04:00
David Steele
9b45df7057 Rearrange MemContext struct members to save space on 64-bit.
On 64-bit systems this saves a bit of space (10%) due to alignment. 32-bit systems see no benefit.

Also add tests for the individual struct sizes.
2021-08-19 11:12:56 -04:00
David Steele
d69a596358 Fix infinite loop in protocolServerProcess() on error.
The error was written to the client and then another command read. If the write did not fail then the loop would never exit.

Instead exit on any error that is not raised by the command handler as we can pretty safely assume this is an unrecoverable protocol error. The command handler might throw a protocol error itself, but this should be caught in the next read or write in the main loop.
2021-08-18 09:48:37 -04:00
David Steele
9a6afe3fc1 Fix EOF behavior of ioReadSmall() to match ioRead().
If the buffer was not full at EOF then ioReadSmall() would get stuck in an infinite loop. Instead, return on EOF even if the buffer is not full.

This is not an issue in released versions since ioReadSmall() is not being used.

Also fix a comment typo.
2021-08-18 09:32:20 -04:00
Cynthia Shang
eca2fc6958
Update config/parse test to use standard patterns. 2021-08-12 12:38:07 -04:00
Cynthia Shang
e17865a03a
Update protocol/protocol test to use standard patterns. 2021-08-12 11:57:17 -04:00
David Steele
a0bdfa436c
Log backup file total and restore size/file total.
The backup size was a bit off because it did not include any files (e.g. backup_label, WAL files) that were added to the manifest after the main copy. To fix this move the log message to the very end of the backup.

Add size/file total log message to restore since it did not exist before.
2021-08-11 13:39:36 -04:00
David Steele
6789ec420e Add additional checks to performance/storage test.
The storageInfoList() test was broken by 54c4eb0c when the remote was changed to use writeable storage. Since the test driver was being injected into the wrong location, new default storage was created and the test effectively did nothing but still "succeeded".

To prevent this type of regression, add checks to ensure the expected test driver is being used and the callback runs the expected number of times.
2021-08-10 10:37:37 -04:00
David Steele
c8492aac41 Cleanup inherited clients in protocol test harness.
Cleanup all clients inherited from the parent process so they cannot be accidentally used to send messages to servers that do not belong to this process.

We need to do this carefully so that exit commands are not sent and processes are not terminated, so clear the mem context callback on each object before freeing it.
2021-08-09 17:11:39 -04:00
Cynthia Shang
f653b59664
Update db/db test to use standard patterns. 2021-08-09 16:35:48 -04:00
David Steele
33775e53c9 Fix test added for lstComparatorZ() in 9abf6a27.
strcmp() returns < 0 and > 0 but these are not guaranteed to be -1 and 1.
2021-08-06 11:26:12 -04:00
David Steele
935de1e7a0
Only pass selected repo options to the remote.
Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster.

Also fix an outdated comment.
2021-08-05 14:29:26 -04:00
David Steele
74c0c44fc8 Migrate error code generation to C.
Parse src/build/error.yaml and write to src/config/error.auto.h and src/config/error.auto.c.
2021-08-02 18:32:11 -04:00
David Steele
930fee3a0c Move bldStrId() into a C file.
This function was included in a header but not declared inline, so linker errors happened when the header was included into more than one file.

Because of the setjmp() in TRY_BEGIN() it can't be inlined so put it in a C file.

Also add some missing headers.
2021-08-02 17:49:05 -04:00
David Steele
7542eadc9f Increase timeouts in storage/remote test.
There have been intermittent failures on f33 (with coverage) but not on u16 (without coverage).

Reproducing this reliably has been very difficult, so just try increasing the timeouts. This is based on the observation that tests with coverage take longer than tests without, which may lead the f33 tests to fail if CI is running slower than usual.

This will not increase the runtime of the test unless there is an error.
2021-08-02 14:48:31 -04:00
David Steele
4e71c077f1 Clean src path in test.pl before attempting vpath builds.
If configure/make has been run in the src path it can conflict with tests, which may require different build options.

Also add a comment when rebuilding for code generation.
2021-08-02 10:00:42 -04:00
David Steele
c192ec4561 Fix rsync include list removed from test.pl in 9ee9b1fa.
This caused the include list to be ignored and all files to be rsync'd, which worked but took a much longer.
2021-08-02 08:52:49 -04:00
David Steele
9abf6a2709 Add lstComparatorZ().
Works just like lstComparatorStr() but with zero-terminated strings.
2021-07-30 18:08:04 -04:00
David Steele
8bca6946b4 Add line and column to build yaml error messages.
This makes it much easier to debug errors in the yaml files.
2021-07-30 17:51:56 -04:00
David Steele
b47a07b8b9 Remove generated config.auto.c file.
This file duplicated the command list that already exists in parse.auto.c.

Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
2021-07-29 14:40:30 -04:00
David Steele
e32f9e146b
Add check for test path inside repo path.
If the test path is inside the repo path then it can cause strange issues during testing because the entire repo path is duplicated into the test path so that all tests see a consistent view of the repo.

Another solution might be to pick a better test path name and exclude it from the rsync, but this fix at least addresses the immediate issue.
2021-07-29 11:09:51 -04:00
David Steele
f3f0c64a78 Complete migration of config code generation to C.
This was started in c5ae047e but did not include generation of parse.auto.c.

The parser has also been improved with better errors and multiple passes to reduce dependency on ordering and produce and cleaner output.

Option order resolution now includes cycle detection.
2021-07-28 19:59:23 -04:00
David Steele
52d4574460 Automatically generate option value StringIds in make build-config.
Remove strIdGenerate() since bldStrId() performs the same function without cluttering the core code. Since bldStrId() is intended to work in non-debug builds, move the validity checks for input strings out of the DEBUG block.

StringIds are generated as 5/6 bit, whichever is most efficient, for each option value. cfgOptionStrIdInternal() has been updated for this logic.
2021-07-26 12:02:31 -04:00
Cynthia Shang
eeaab6a3d1
Update config tests (except parse) to use standard patterns. 2021-07-23 16:18:50 -04:00
David Steele
947c7a84cb Refactor cmdLocal()/cmdRemote() to accept a ProtocolServer object.
This allows a local/remote to be started independently of server initialization, which will be useful for implementing new transport types, e.g. TLS.

Also remove some dead code in localTest.c.
2021-07-23 10:35:36 -04:00
David Steele
0999de0279 Move noop from protocolClientNew() to local/remote initialization.
protocolServerNew() does not automatically process a noop so this made the handshake in the constructors asymmetric. This made testing a bit confusing since an extra noop was needed when cmdLocal()/cmdRemote() were not called (since they processed the noop sent by protocolClientNew()).

The extra noops also make complex protocol negotiation (coming in a future commit) more complicated and slower because of the additional round trips.
2021-07-23 08:32:30 -04:00
Cynthia Shang
4c9ddf5ef2
Update storage tests to use standard patterns.
The storage tests were not modified to the HRN_STORAGE_* nor TEST_STORAGE_* macros as these test are testing the storage drivers.

Note that posixTest.c removed an extraneous #endif // TEST_CONTAINER_REQUIRED and #ifdef TEST_CONTAINER_REQUIRED.

This PR includes all files in the storage/* test directory, namely: azureTest.c, cifsTest.c, gcsTest.c, posixTest.c, remoteTest.c, s3Test.c
2021-07-22 18:17:08 -04:00
David Steele
3a37482de9 Update db test module error for newer libpq versions.
PostgreSQL 14 libpq throws a more detailed error, so adjust the regular expression to handle the old and the new errors.
2021-07-22 15:54:30 -04:00
David Steele
eb98b8d2db Fix typo. 2021-07-21 13:19:09 -04:00
David Steele
46992acd80 Update Vagrantfile box version. 2021-07-20 15:43:48 -04:00
Cynthia Shang
4992508764
Update command/restore test to use standard patterns. 2021-07-20 13:30:17 -04:00
David Steele
9ee9b1fad6 Remove test.pl --smart, --dev, and --dev-test options.
--smart is now the default mode. Since --dev is now just an alias for --no-optimize, remove it. --dev-test has been a noop for a while, so this seems like a good time to remove it.

Also make the C auto-generator skip writing files that have not changed to avoid updating the timestamp.
2021-07-20 12:01:10 -04:00
Cynthia Shang
1522cb4ed2 Allow NULL path in HRN_STORAGE_PATH_REMOVE() macro.
Update command/backup test to pass NULL where appropriate.
2021-07-19 15:21:40 -04:00
Cynthia Shang
a6cdf5b22f
Update command/repo test to use standard patterns. 2021-07-19 10:47:49 -04:00
Cynthia Shang
ebe5848494
Update command/local and command/remote tests to use standard patterns.
Note that the logging output display of a parent/child test may look jumbled on some systems since the child and parent are attempting to log information at the same time. This is not an issue with the actual test, rather a harness issue that would be beyond the scope of this project to fix.
2021-07-19 09:51:11 -04:00
David Steele
c5ae047e76 Partial migration of config code generation to C.
Parse enough of config.yaml to auto-generate config.auto.h and config.auto.c.

This commit implements most of the infrastructure needed to migrate the rest of the build code to C, but each set of auto-generated files will present its own challenges.

The build is now dependent on libyaml. At this point there is no need for a hard requirement, but that will come soon so it seems better to add the dependency now.
2021-07-18 19:02:01 -04:00
David Steele
6397d73535 Update test container OS versions.
Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial.
Update Ubuntu 18.04 to 20.04.
Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up.

Add --no-install-recommends to apt-get commands to save a bit of time and space.

Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small.

Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 17:29:20 -04:00
David Steele
81602f1593 Remove test.pl --vm-host option.
This option was once used to optimize bin builds but has been defunct for some time.
2021-07-17 11:09:53 -04:00
Cynthia Shang
d57299ada8 Allow NULL path in TEST_STORAGE_LIST() macro. 2021-07-16 13:07:16 -04:00
Cynthia Shang
117ffe8896
Update command/help tests to use standard patterns.
In the first test (helpRenderSplitSize) added test for empty list and in that and some other tests, the test comment was updated to clarify a bit more what the actual tests is trying to accomplish.

Note that help test parameters can only use the harnessConfig system when testing option values that have been set since options passed to the help command are not "set" options.
2021-07-15 17:51:48 -04:00
Cynthia Shang
4ad0bbda53
Update command/backup tests to use standard patterns.
Includes backup and backupCommon tests.

Some tests in backupTest were split out where they were originally combined into a single boolean check - which made it difficult to determine which part of the conditional failed.

String values were also removed where they were no longer needed.
2021-07-15 17:00:20 -04:00
David Steele
cdc89fc975 Allow NULL path in HRN_STORAGE_PATH_CREATE() macro. 2021-07-15 16:05:32 -04:00
Cynthia Shang
c0feca4673 Add noParentCreate parameter to HRN_STORAGE_PATH_CREATE() macro. 2021-07-15 13:31:54 -04:00
David Steele
4a6ca54b47 Remove last vestiges of key replacement missed in b270253a. 2021-07-15 13:19:49 -04:00
Stefan Fercot
563d117967
Allow restore --type=lsn.
The LSN recovery target was added in PostgreSQL 10 but did not get added to pgBackRest.
2021-07-15 13:02:41 -04:00
Stefan Fercot
8fac1ac50d
Loop while waiting for checkpoint LSN to reach replay LSN.
It is possible for the checkpoint LSN to lag slightly behind the replay LSN until pg_control has been updated.

Add a loop to keep checking rather than failing when the checkpoint LSN has not been updated.
2021-07-15 07:45:06 -04:00
Cynthia Shang
a6691c6f61 Clarify restore --type behavior in command reference. 2021-07-14 16:06:42 -04:00
David Steele
d791bb7298 Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros.
This removes a lot of boiler plate where every instance needs to create these interfaces.

Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes.

In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
Cynthia Shang
be7b8a485b Error in TEST_STORAGE_LIST() when path does not exist. 2021-07-13 14:28:58 -04:00
David Steele
1ace1ac938 Improve HRN_FORK*() macros.
Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults.

Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
2021-07-13 14:22:53 -04:00
David Steele
76cfbf833d Rename HARNESS_FORK*() macros to HRN_FORK*().
This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
David Steele
d6797009f8 Add ioFdReadNewOpen() and ioFdWriteNewOpen().
These functions construct and open in one call, which allows them to be used as function parameters.
2021-07-13 11:13:57 -04:00
Cynthia Shang
26dd918856 Add compressType parameter to TEST_STORAGE_GET() macro.
Also remove duplicate extension output in HRN_STORAGE_PUT().
2021-07-12 15:36:52 -04:00
Cynthia Shang
f2ec4e00a4 Add cipher parameters to TEST_STORAGE_GET() macro. 2021-07-09 15:59:17 -04:00
David Steele
849ab343aa Change level of backup/restore copied file logging to detail.
The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail.
2021-07-09 13:50:35 -04:00
David Steele
a7a041e241 Fix flapping coverage in storage/remote test.
The protocol does not put an end message on exit so there was a race between the main process exiting and the remote processes exiting. If the main process exited first then the remote processes might not write coverage data causing coverage to fail.

Fix by calling exit explicitly at the end of the test and update the harness to put an end message so the exits are synchronized.
2021-07-09 11:18:45 -04:00
Cynthia Shang
8bb0b28455
Update command/control and command/command tests to use standard patterns.
In the commandTest the HRN_STORAGE_REMOVE replacement uses .errorOnMissing when the code being tested added the file. The reason for this is 3 fold:

1. to ensure that an inadvertent typo in the path/file name does not go undetected,
2. to ensure that nothing else has removed the file prior to the call, and
3. consistency

Also, added "stanza" to comment when a stanza stop file is removed vs an "all" stop file.
2021-07-08 16:34:11 -04:00
David Steele
8e1807cdbe
Detect errors in S3 multi-part upload finalize.
Multi-part upload may fail despite returning an HTTP success code. Check for the ETag field in the result and if not present consider the upload to have failed. This will trigger a retry at the local job level.
2021-07-08 13:06:52 -04:00
David Steele
43b874628c
Fix detection of circular symlinks.
Links were followed before they were checked for validity so a circular link would send the manifest build into endless recursion leading to a crash. Fix by moving the recursion after the link check.

Note that this issue has existed since the C migration and was not introduced by the refactor in eba013b.
2021-07-08 11:30:23 -04:00
David Steele
6839335633 Increase harness log level to detail in command/backup test.
Also clean up some unneeded calls to harnessLogLevelReset().
2021-07-08 11:17:13 -04:00
Stefan Fercot
056a3070e0
Automatically create data directory on restore.
Data directory creation was added during the C migration, but creation of the base data directory (PGDATA) was prevented by a check migrated from Perl.

Remove the check and update tests to create the data directory at least once.
2021-07-08 10:33:19 -04:00
David Steele
34e51ee7b6 Allow NULL path in HRN_STORAGE_MODE() macro. 2021-07-08 09:25:42 -04:00
Cynthia Shang
62e8d97af0
Update command/archive tests to use standard patterns.
Includes archiveCommon, archiveGet and archivePush.

Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
Eric Radman
23bdc3deb6
Fix documentation and comment typos.
Identified using `ag -l | igor`.
2021-07-01 11:50:03 -04:00
Cynthia Shang
6a90d46909
Update info/manifest test to use standard patterns. 2021-06-30 08:51:11 -04:00
David Steele
39c1c10120 Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants.
It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
Cynthia Shang
c6208113bf Add timeModified parameter to HRN_STORAGE_PUT() macro. 2021-06-28 12:14:26 -04:00
Cynthia Shang
065a8c8454
Replace defines with string constants in unit tests.
For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes.

Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
Cynthia Shang
fb2d111dcd
Update info/infoBackup test to use standard patterns. 2021-06-24 16:11:14 -04:00
Cynthia Shang
d0f3a3f2af Add HRN_STORAGE_COPY() macro. 2021-06-24 14:01:03 -04:00
Cynthia Shang
50c129ae9d Add expression parameter to TEST_STORAGE_LIST() macro. 2021-06-24 13:57:55 -04:00
David Steele
6a1c0337dd
Binary protocol.
Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol.

There are a number advantages:

* The pack type is more compact than JSON and are more efficient to render/parse.
* Packs are more strictly typed than JSON.
* Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state.
* There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files.

Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C.

There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
Cynthia Shang
bffb43ea3f
Update command/check test to use standard patterns. 2021-06-24 12:23:09 -04:00
Cynthia Shang
17c9ed0ef0
Update command/info test to use standard patterns.
Some tests had to be reordered or updated, as follows:

* Reordered tests at line 317 and 331 to avoid unnecessary file removal.

* Change "stanza found" test at line 1735 to reflect real-life scenario. Originally this test had the cipher-pass environment key set up which caused the RepoGrp to be 2 but with no valid repo path. This resulted in the repo loops executing for the repo2 but since the path was not defined, the tests just reported "none" for cipher which is incorrect since the repo IS encrypted.

* Moved order of HRN_CFG_LOAD in some tests when able to avoid using storageTest.
2021-06-24 08:55:44 -04:00
Cynthia Shang
ff95eddc36
Update command/verify test to use standard patterns. 2021-06-23 18:09:46 -04:00
Cynthia Shang
8dd882bade
Update info/infoArchive test to use standard patterns. 2021-06-23 16:20:47 -04:00
David Steele
e697f5705a
Clear error when a CATCH() block finishes.
It is better to clear errors after the catch block completes rather than leave them set until the next error. This also make is possible to tell when a error is currently being handled, which a function further down the stack might use to modify its behavior. Currently this is only useful in testing, but clearing the error seems like a good idea in general.

Two places used errors outside the CATCH() block. Mem context cleanup now uses a FINALLY() which is a better implementation anyway. The error handling in main() now calls exitSafe() from withing the CATCH() block.
2021-06-23 13:02:19 -04:00
Cynthia Shang
7ba5aef763 Add mode parameter to HRN_STORAGE_PUT() macro. 2021-06-23 12:07:19 -04:00
Cynthia Shang
03021c6a17 Update command/stanza test to use standard patterns.
No core code changes; only changes for stanza-create, stanza-upgrade, and stanza-delete command unit tests.
2021-06-14 13:28:27 -04:00
David Steele
80d63a5e8b Ignore mismatched close requests in the pq shim during error processing. 2021-06-12 18:00:59 -04:00
David Steele
fba0437179 Improve error handling in db test module.
Errors could cause segfaults since server objects were freed immediately but client destructors were run much later.
2021-06-12 17:48:35 -04:00
David Steele
0a65e59b55 Update pg-path options in db test module to be valid.
This allows files to be saved in the paths, which would have failed when they were invalid.
2021-06-11 19:16:25 -04:00
David Steele
97155bad86 Add mode, Pack, and StringId to the Pack type.
Add StringList, which is not a primitive type but rather an array of String types.

Also update pckWriteToLog() to work after pckWriteEnd(), i.e. this->tagStackTop is NULL.
2021-06-10 12:40:55 -04:00
David Steele
85a55bd401 Add pckReadMove() and pckWriteMove().
Move a PackRead or PackWrite object to a new mem context.

Also note that these functions may not work as expected with pack objects created by pckReadNewBuf() and pckWriteNewBuf() since the pack object does not have ownership of the passed buffer and cannot move it.
2021-06-10 09:25:57 -04:00
David Steele
788f7c1f30 Skip sleep in sleepMSec() when sleep time is zero.
There is no point in sleeping when there is no sleep time.

The advantage is that callers do not need to perform the check themselves.
2021-06-10 09:24:42 -04:00
David Steele
5e1a8e6895 Add error test harness/shim.
The hrnErrorThrowP() macro allows errors with specified fields to be generated, which simplifies testing.

Update the common/exit test to use the new macro.
2021-06-10 09:21:15 -04:00
Cynthia Shang
c5897007c4 Add HRN_STORAGE_MOVE() macro.
Update command/expire test to show how it is used.
2021-06-09 12:41:23 -04:00
David Steele
15dfbf4c1d Use lower-case names from Azure identifiers in integration tests.
Azurite, which is used for testing, did not enforce this before so the capital letters were not a problem. Now Azurite enforces the same rules as Azure so use lower-case identifiers instead.

These names were only used in integration tests so there was no production impact.
2021-06-09 12:28:40 -04:00
David Steele
c6a8528e31 Add optional remove to TEST_STORAGE_EXISTS().
This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before.

Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
David Steele
4a075b7252
Add support for more Pack types.
Since the pack type was stored in 4 bits, only 15 values were allowed (0 was reserved).

Allow virtually unlimited types by storing type info in a base-128 encoded integer following the tag when the type bits in the tag are set to 0xF.

Also separate the type IDs used in the pack (PackTypeMap) from those presented to the user (PackType). The prior PackType enum exposed implementation details to the user, e.g. pckTypeUnknown.
2021-06-08 12:55:00 -04:00
Cynthia Shang
3f9fbc3c24
Update command/expire test to use standard patterns.
The way tests are written has evolved over time. Update the command/expire test to use the new test patterns.
2021-06-08 11:00:28 -04:00
David Steele
4ccb42bb7a Rename param structs to match function names.
The functions were named with short integer representations (e.g. I32) but the param structs were using longer ones, e.g. UInt32. Shorten the integer representations in the param structs to match.

Also rename pckReadUInt64Internal() to pckReadU64Internal() for the same reason.
2021-06-08 08:44:10 -04:00
David Steele
df8276f59f Switch order of remote startup in storage/remote test.
The pg storage must be started before the repo storage to set the max remotes allowed to 2. The protocol helper expects all remotes to have the same type so we are cheating here a bit, but without this ordering the second remote will never be sent an explicit exit and may not save coverage data.
2021-06-07 11:53:25 -04:00
David Steele
a607750be1 Handle NULL VariantList in JSON.
This worked if the Variant was NULL but not if the Variant contained a NULL.
2021-06-07 07:46:51 -04:00
David Steele
d10a99d73b v2.34: PostgreSQL 14 Support
Bug Fixes:

* Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.)
* Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.)
* Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete.)
* Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)

Features:

* Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)
* Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.)
* Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
* Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)

Improvements:

* Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)

Documentation Improvements:

* Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)
* Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)
* Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
2021-06-07 06:51:08 -04:00
Cynthia Shang
c9a8ff27f2 Add HRN_STORAGE_PATH_REMOVE() macro.
Update command/archive-push test to show how it is used.
2021-06-02 11:46:54 -04:00
David Steele
a97fc6e708 Add storage/remote test for unknown user/group name. 2021-06-01 17:10:15 -04:00
David Steele
8250990afb Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro.
HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases.

It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
David Steele
c1277677a3 Add HRN_STORAGE_PATH_CREATE() macro.
Update command/archive-get test to show how it is used.

Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
David Steele
c8427682ea Improve storage harness/test macros.
Make the macros more consistent in format and make sure that each macro outputs a line number before doing any work so when errors happen it is clear where they happened.

Add noRecurse option to TEST_STORAGE_LIST().

Add comment option to all storage macros.
2021-05-28 14:39:43 -04:00
David Steele
0a43be0183 Use L to denote line numbers in test output.
The lower-case l was too easy to confuse with a 1.
2021-05-28 13:08:41 -04:00
David Steele
cfe8e51910 Remove line parameter from hrnTestResultBegin().
Instead store the line number in hrnTestLogPrefix() so it doesn't need to be passed to hrnTestResultBegin().

Also add missing linefeed in hrnStorageList().
2021-05-28 08:50:28 -04:00
David Steele
b3ac9e8010 Remove unused padding parameter from hrnTestLogPrefix().
Also add fflush() so log prefix is displayed immediately.
2021-05-28 08:03:22 -04:00
David Steele
52217f1244 Replace storageTest with storagePg*() in command/archive-get test.
All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted.

Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
Cynthia Shang
aceb956815
Change archive expiration logging from detail to info level.
To assist with issue debugging, removal of files during archive expiration has been changed from detail level logging to info level.
2021-05-27 08:03:44 -04:00
Stefan Fercot
5b332b22b5
Add repo-retention-history option to expire backup history.
The default is to keep all backup history to match the current behavior. In minimal configuration (0 days), unexpired backups are always kept in history.

When a full backup manifest expires, all dependent differential/incremental manifests expire as well.
2021-05-26 13:07:18 -04:00
David Steele
a69a6c2f55 Show diff when test.pl --gen-check detects unexpected autogeneration.
This will help track down autogeneration issues if they can't be reproduced locally.
2021-05-26 12:41:15 -04:00
David Steele
ba351e9c5c Refactor storage/remote unit test using the protocol remote shim.
Using the local process shim improves coverage and simplifies the tests.
2021-05-26 12:38:23 -04:00
David Steele
58369c02df Add remote process shim.
Run the remote process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the remote process rather than needing to test the remote protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test and the test does not need to run in a container.
2021-05-25 18:16:59 -04:00
David Steele
441c000b5c Factor remote process exec out of protocolRemoteGet().
This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing.

The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
David Steele
e1bc1b3e53 Set buffer-size in the configuration test harness.
Most tests can set buffer-size as needed, but local/remote process shims are forked, so the best way to set buffer-size is via the configuration.
2021-05-25 17:48:42 -04:00
David Steele
cd88f82329 Move protocol test module before config module.
The protocol module should be tested before modules that have a dependency on it.
2021-05-25 11:08:51 -04:00
David Steele
6732806d89 Add local process shim to archive-get/archive-push unit tests.
Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
David Steele
d55b9471a8 Protocol shim improvements.
Add executable to parameter list to avoid first option being lost. The backup, restore, and verify tests worked OK with their first option being defaulted because it ended up being job-retry which worked fine as the default.

Add hrnProtocolLocalShimUninstall() allow the shim to be uninstalled.

Log shim at debug level to make it obvious in the logs when a shim is in use.
2021-05-25 11:00:24 -04:00
David Steele
55f52955a5 Fix shims with more than one function.
Each subsequent function would double every line in the C module.

Move the loop to detect shimmed functions down to prevent this issues.
2021-05-24 17:45:31 -04:00
David Steele
2452c4d5a4
Add PostgreSQL 14 support.
There are no code changes from PostgreSQL 13 so simply add the new version.

Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.

Update the integration tests to handle new PostgreSQL startup messages.
2021-05-24 17:17:03 -04:00
David Steele
eba013b49b
Fix issue when checking links for large numbers of tablespaces.
manifestLinkCheck() was pretty inefficient so large numbers of links caused it to use a lot of memory and eventually crash. This is a more efficient implementation which runs O(nlogn) and uses far less memory.

Checking for duplicate file links has been added, which represents a change in behavior, but hopefully a good one.
2021-05-24 16:29:36 -04:00
David Steele
f6303152f5 Minor fixes for command/archive-get unit test.
Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test.

Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet().

Move some harness headers down to the correct section.
2021-05-24 12:40:27 -04:00
David Steele
e5e0b21196 Replace system() in tests with HRN_SYSTEM*(). 2021-05-22 15:07:18 -04:00
David Steele
bd40156c22 Rename TEST_SYSTEM*() to HRN_SYSTEM*().
These calls are not tests, rather they setup data for tests.
2021-05-22 14:22:51 -04:00
David Steele
73885f8c2e Replace hrnLogResult() with TEST_RESULT_LOG/_FMT().
The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
David Steele
6baad5cdd1 Replace TEST_ERROR_FMT() with TEST_ERROR() where possible.
Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.
2021-05-22 11:59:43 -04:00
David Steele
a4f057bb70 Remove comment formatting from TEST_*() macros.
Comment formatting was not used much but it incurred a heavy cost in each macro to process possible formatting.

Remove formatted comments where they did not contain valuable information and replace with strZ(strNewFmt()) otherwise.
2021-05-22 11:28:56 -04:00
David Steele
b270253a69 Add defines for many test*() getter functions.
A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests.

Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE.

Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements.

In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
David Steele
aed3d468a1 Rename strNew() to strNewZ() and add parameter-less strNew().
Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not.

In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
David Steele
15b8b9207d Add log shim.
This allows DEBUG_UNIT and DEBUG_UNIT_EXTERN to be removed since static log variables can now be exposed by functions in the harness.
2021-05-21 12:51:32 -04:00
David Steele
ef63750e0b Add local process shim.
Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test.

The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
David Steele
cab7a97ab6 Add shim feature for unit tests.
A shim allows a test harness to access static functions and variables in a C module, and also allows functions to be shimmed (i.e. overridden) for the purposes of testing.

For instance, coverage testing works when a process that is normally exec'd is run as a forked child process instead.
2021-05-20 18:47:31 -04:00
David Steele
45a4e801ed
Replace getopt_long() with custom implementation.
getopt_long() requires an exhaustive list of all possible options that may be found on the command line. Because of the way options are indexed (e.g. repo1-4, pg1-8) optionList[] has 827 entries and we have kept it small by curtailing the maximum indexes very severely. Another issue is that getopt_long() scans the array sequentially so parsing gets slower as the index maximums increase.

Replace getopt_long() with a custom implementation that behaves the same but allows options to be parsed with a function instead of using optionList[]. This commit leaves the list in place in order to focus on the getopt_long() replacement, but cfgParseOption() could be replaced with a more efficient implementation that removes the need for optionList[].

This implementation also fixes an issue where invalid options were misreported in the error message if they only had one dash, e.g. -config. This seems to have been some kind of problem in getopt_long(), but no investigation was done since the new implementation fixes it.

Tests were added at 0825428, 2b8d2da, 34dd663, and 384f247 to check that previously untested getopt_long() behavior doesn't change.
2021-05-20 16:02:31 -04:00
David Steele
831ee81466
Rename default command role to main.
Main makes more sense because we refer to the main process in the code, not the default process. The word default is pretty overloaded anyway.
2021-05-20 14:39:47 -04:00
David Steele
8453d89a92 Add recursion to TEST_STORAGE_LIST() macro.
This makes the macro useful when subpaths are present.

Identify types other than files (path, link, etc.) with a single appended character for easier debugging.
2021-05-19 14:18:43 -04:00
David Steele
93eb2f8362 Fix invalid type in test function.
This should have been char * from the start, but of course void * worked fine.
2021-05-19 14:03:42 -04:00
David Steele
320c6e1aad
Remove stanza archive spool path on restore.
Remove stanza archive spool path so existing files do not interfere with the new cluster. For instance, old archive-push acknowledgements could cause a new cluster to skip archiving. This should not happen if a new timeline is selected but better to be safe. Missing stanza spool paths are ignored.

Also add new path expression STORAGE_SPOOL_ARCHIVE to easily access this path.
2021-05-18 15:49:22 -04:00
David Steele
9af033194a
Add automatic GCS authentication for GCE instances.
When running on a GCE instance the authentication token can be pulled directly from the instance metadata. This is configured with repo-gcs-key-type=auto.

In a separate commit (26fefa6), move the code that parses the token response into a separate function, storageGcsAuthToken(), since it is now needed by two key types. This drastically improves the readability of the main commit.
2021-05-17 14:55:50 -04:00
David Steele
0152075e6b Remove default VM for test.pl --coverage-only option.
When running outside of our standard Vagrantfile the default will not be set correctly, so require the user to set it.

In any case, this option is primarily useful for reporting so note that in the command line help.
2021-05-17 11:35:22 -04:00
David Steele
384f247077 Add config/parse tests for config/env partial options.
Partial option matching is valid on the command line but should never be used in config files or environment options.
2021-05-17 09:33:36 -04:00
David Steele
ae7f0af202 Move PostgreSQL version interface test functions to a test harness.
Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core.

Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
David Steele
f45e76fa2e
Fix issues with leftover spool files from a prior restore.
If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue.

Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process.

Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
David Steele
34dd6636b8 Add config/parse tests for options and option args with spaces.
If an option name has a space at the beginning then it will be considered an invalid command, but a space at the end is an invalid option. Add tests for these conditions.

Spaces in option arguments should be preserved, so add a test to be sure this is true.
2021-05-13 17:01:21 -04:00
David Steele
2b8d2daca1 Add config/parse tests for partial options.
Add coverage for partial options, i.e. an option that is partially specified but only prefix matches with a single valid option.
2021-05-13 16:54:25 -04:00
David Steele
add0f9b77d Fix an ungrammatical error message in config parsing. 2021-05-13 12:37:59 -04:00
David Steele
0825428fef Add config/parse test where the option/value are not in the same arg.
All the tests in this module use --option=value syntax so add one test with --option value syntax for coverage.
2021-05-13 12:35:11 -04:00
David Steele
7b71604def Use existing variable for GCS test server port.
Also fix incorrect separator comment.
2021-05-12 13:21:59 -04:00
David Steele
5464ac83d1
Convert option values in commands to StringId.
Convert most of the remaining options that benefit from being StringIds. Since all the command modules can include config.h directly it makes sense to auto-generate these values instead of manually creating an enum for each one.

For the time being StringIds are not being auto-generated because the StringId code does not exist in Perl. However, the *_Z zero-terminated constants for each allowed option value are now auto-generated.
2021-05-11 17:24:30 -04:00
David Steele
87ba2ca253 Change CentOS 7 documentation test to CentOS 8.
The CentOS 7 documentation test relies on PostgreSQL 9.5 which has been removed from the yum.p.o repository package. Switch the test to CentOS 8 to fix the immediate issue, but a decision on the PostgreSQL 9.5 documentation will need to be made before the next release.
2021-05-11 16:54:42 -04:00
David Steele
2bfebff2ec Add instructions to install lcov for MacOS. 2021-05-04 15:51:46 -04:00
David Steele
54074a5697 Exclude Dockerfile from code count. 2021-05-04 07:52:35 -04:00
David Steele
baddec1e9a Basic multi-architecture support for test containers.
The tests worked fine on multiple architectures, but would only run "bare metal", i.e. tests that required containers could not be run.

Enable basic multi-architecture support by allowing containers to be built using whatever architecture the host supports. Also allow cached containers to be defined for multiple architectures in container.yaml.

Add a Dockerfile which can be used as a container for other containers to provide a consistent development environment.

The primary goal is to allow development on Mac M1 but other architectures should find these improvements useful.
2021-05-03 16:31:27 -04:00
David Steele
87df6d7a58
Convert BackupType enum to StringId.
Allows removal of backupType()/backupTypeStr() and improves debug logging of the enum.

Move BackupType enum and string constants to info/infoBackup.h so they are available to more modules. Also convert InfoBackup to use BackupType instead of a String.
2021-05-03 12:15:39 -04:00
David Steele
7dd01897fd Convert ProtocolStorageType enum to StringId.
Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
David Steele
85fc3da4c3
Update CipherType/CipherMode to StringId.
As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
Cynthia Shang
c3b15fc3bd Fix comments where 'output' was misspelled as 'ouput'. 2021-04-28 10:58:45 -04:00
David Steele
066fbcf268 Refactor String, Buffer, and Variant types with inline getters.
Extend the pattern introduced in 79a2d02c to the String, Buffer, and Variant types.
2021-04-27 15:25:10 -04:00
David Steele
1edcfde93e
Add cfgOptionDisplay()/cfgOptionIdxDisplay().
Centralize the formatting of the configuration value for display to the user or passing on a command line.

For the new functions, if the value was set by the user via the command line, config, etc., then that exact value will be displayed. This makes it easier for the user to recognize the value and saves having to format it into something reasonable, especially for time and size option types.

Note that cfgOptTypeHash and cfgOptTypeList option types are not supported by these functions, but they are generally not displayed to the user as a whole.

This also fixes a bug in config/load.c where time values where not being formatted correctly in an error message.
2021-04-27 12:12:43 -04:00
David Steele
6cc521b6b2
Update storage module to use StringIds.
Use StringIds for the storage types (e.g. STORAGE_S3_TYPE) and configuration settings, e.g. cfgOptS3KeyType.

Also add new config functions and harness config functions to support StringIds.
2021-04-23 13:19:47 -04:00
David Steele
aa72c19a83
Do not write files atomically or sync paths during backup copy.
There is no need to write the file atomically (e.g. via a temp file on Posix) because checksums are tested on resume after a failed backup. The path does not need be synced for each file because all paths are synced at the end of the backup.

This functionality was not lost during the migration -- it never existed in the Perl code, though these settings are used in restore. See 59f1353 where backupFile() was migrated to C.
2021-04-23 12:33:25 -04:00
David Steele
aaa15b9709
Add help for all internal options valid for default roles.
Fix the segfault when getting help for an internal option is requested by adding help for all internal options that are valid for a default command role.

Also print warnings about internal options in code rather than putting in each command/option description.
2021-04-23 11:46:03 -04:00
David Steele
2ad497ea4c Remove documentation about enum truncation and add tests.
The enum truncation observed was due to the value getting passed via a protocol function which silently narrowed the enum.

Even so, add some tests to ensure tested platforms support 64-bit enums.
2021-04-23 08:04:03 -04:00
David Steele
bcc925b740 Replace misused kvAdd() with kvPut().
Although kvAdd() works like kvPut() on the first call, kvPut() is more efficient when a key has a single value.

Update the comment to clarify that kvAdd() is seldom required.
2021-04-22 20:04:27 -04:00
David Steele
45f83558ea
Fix help when a valid option is invalid for the specified command.
Getting help for a valid option that was invalid for the command would segfault.

Add a check to ensure the option is valid for the command's default role.
2021-04-22 11:48:04 -04:00
David Steele
20a018a8ae Fix incorrect test title. 2021-04-21 18:27:18 -04:00
David Steele
fd69357302 Add const to inline functions where appropriate.
This lets the compiler know that these variables are not modified which should lead to better optimization.

Smart compilers should be able to figure this out on their own, but marking parameters const is still good for documentation.
2021-04-20 18:43:16 -04:00
David Steele
ed0d48f52c Add StringId type.
It is often useful to represent identifiers as strings when they cannot easily be represented as an enum/integer, e.g. because they are distributed among a number of unrelated modules or need to be passed to remote processes. Strings are also more helpful in debugging since they can be recognized without cross-referencing the source. However, strings are awkward to work with in C since they cannot be directly used in switch statements leading to less efficient if-else structures.

A StringId encodes a short string into an integer so it can be used in switch statements but may also be readily converted back into a string for debugging purposes. StringIds may also be suitable for matching user input providing the strings are short enough.

This patch includes a sample of StringId usage by converting protocol commands to StringIds. There are many other possible use cases. To list a few:

* All "types" in storage, filters. IO , etc. These types are primarily for identification and debugging so they fit well with this model.

* MemContext names would work well as StringIds since these are entirely for debugging.

* Option values could be represented as StringIds which would mean we could remove the functions that convert strings to enums, e.g. CipherType.

* There are a number of places where enums need to be converted back to strings for logging/debugging purposes. An example is protocolParallelJobToConstZ. If ProtocolParallelJobState were defined as:

typedef enum
{
    protocolParallelJobStatePending = STRID5("pend", ...),
    protocolParallelJobStateRunning = STRID5("run", ...),
    protocolParallelJobStateDone = STRID5("done", ...),
} ProtocolParallelJobState;

then protocolParallelJobToConstZ() could be replaced with strIdToZ(). This also applies to many enums that we don't covert to strings for logging, such as CipherMode.

As an example of usage, convert all protocol commands from strings to StringIds.
2021-04-20 15:22:42 -04:00
Stefan Fercot
292f836f12
Add db-exclude option.
Restore excluding the specified databases. Databases excluded will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, those databases will not be accessible but can be removed with the drop database command. The --db-exclude option can be passed multiple times to specify more than one database to exclude.

When used in combination with the --db-include option, --db-exclude will only apply to standard system databases (template0, template1, and postgres).
2021-04-19 15:01:00 -04:00
David Steele
a75b413ac2 Add storageDriver() inlines missed in b715c70b. 2021-04-19 14:22:36 -04:00
David Steele
d859fe8c4d Update to remove files after archive-get test completes.
This cleans up the spool path for the next test. Currently there isn't one but there will be.
2021-04-14 15:48:04 -04:00
David Steele
040ad71f8c Remove lockClear().
This function has not been used since the switch to the fork/exec model.

lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
David Steele
c2d4a0286e Define DEBUG in build.auto.c.
Both NDEBUG and DEBUG were used in the code, which was a bit confusing.

Define DEBUG in build.auto.c so it is available in all C and header files and stop using NDEBUG. This is preferable to using NDEBUG everywhere since there are multiple DEBUG* defines, e.g. DEBUG_COVERAGE.

Note that NDEBUG is still required since it is used by the C libraries.
2021-04-13 18:06:07 -04:00
David Steele
9fec4ce98c Refactor remaining common/io modules with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the remaining common/io modules.
2021-04-13 14:37:02 -04:00
David Steele
8844ced384 Refactor common/io/filter module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/io/filter module.
2021-04-12 16:05:40 -04:00
David Steele
3b9bed9518 Refactor common/type/keyValue module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/type/keyValue module.
2021-04-09 17:29:14 -04:00
David Steele
e18f3c1701 Remove unused XML constructors.
These alternate constructors may have been used in the Perl days but they are no longer used so remove them.
2021-04-09 15:56:42 -04:00
David Steele
ddd37ebf7d Refactor common/exec module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the common/exec module.
2021-04-09 15:31:31 -04:00
David Steele
1d674c352f Refactor db module with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the db module.
2021-04-09 14:32:21 -04:00
David Steele
442b2e41b1 Refactor info modules with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the info modules.
2021-04-09 13:48:40 -04:00
David Steele
4937653a3d Refactor protocol modules with inline getters/setters.
Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
David Steele
6c005afd5b Add Ubuntu 20.04 test VM definition. 2021-04-08 14:38:20 -04:00
David Steele
d30ec9c9ae Replace OBJECT_DEFINE_MOVE() and OBJECT_DEFINE_FREE() with inlines.
Inline functions are more efficient and if they are not used are automatically omitted from the binary.

This also makes the implementation of these functions easier to find and removes the need for a declaration. That is, the complete implementation is located in the header rather than being spread between the header and C file.
2021-04-08 10:04:57 -04:00
David Steele
351e7db4c4 Replace OBJECT_DEFINE_FREE_RESOURCE_BEGIN() with normal functions.
OBJECT_DEFINE_MOVE() and OBJECT_DEFINE_FREE() will be replaced with inlines so this would be the only macro left that is constructing functions.

It is not a great pattern anyway since it makes it hard to find the function implementation.
2021-04-07 16:27:55 -04:00
David Steele
cc85c4f03d Replace OBJECT_DEFINE_GET() with *Pub struct pattern.
This macro was originally intended to simplify the creation of simple getters but it has been superseded by the pattern introduced in 79a2d02c.

Remove instances of OBJECT_DEFINE_GET() to avoid confusion with the new pattern.
2021-04-07 14:27:57 -04:00
David Steele
b715c70b46 Refactor storage modules with inline getters/setters.
Extended the pattern introduced in 79a2d02c to the storage modules: Storage, StorageRead, StorageWrite.
2021-04-07 14:04:38 -04:00
David Steele
79a2d02c9c Refactor List, StringList, and VariantList for performance.
Introduce a standard pattern for exposing public struct members (as documented in CODING.md) and use it to inline lstSize() which should improve the performance of iterating large lists.

Since many functions in these modules are just thin wrappers of other functions, inline where appropriate.

Remove strLstExistsZ() and strLstInsertZ() since they were only used in tests, where the String version of the function is sufficient.

Move strLstNewSplitSizeZ() to command/help/help.c and remove strLstNewSplitSize(). This function has only ever been used by help and does not seem widely applicable.
2021-04-07 12:50:33 -04:00
David Steele
a3d89143d6 v2.33: Multi-Repository and GCS Support
Bug Fixes:

* Fix option warnings breaking async archive-get/archive-push. (Reviewed by Cynthia Shang. Reported by Lev Kokotov.)
* Fix memory leak in backup during archive copy. (Reviewed by Cynthia Shang. Reported by Christian ROUX, Efremov Egor.)
* Fix stack overflow in cipher passphrase generation. (Reviewed by Cynthia Shang. Reported by bsiara.)
* Fix repo-ls / on S3 repositories. (Reviewed by Cynthia Shang. Reported by Lesovsky Alexey.)

Features:

* Multiple repository support. (Contributed by Cynthia Shang, David Steele. Reviewed by Stefan Fercot, Stephen Frost.)
* GCS support for repository storage. (Reviewed by Cynthia Shang.)
* Add archive-header-check option. (Reviewed by Stephen Frost, Cynthia Shang. Suggested by Hans-Jürgen Schönig.)

Improvements:

* Include recreated system databases during selective restore. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
* Exclude content-length from S3 signed headers. (Reviewed by Cynthia Shang. Suggested by Brian P Bockelman.)
* Consolidate less commonly used repository storage options. (Reviewed by Cynthia Shang.)
* Allow custom config-path default with ./configure --with-configdir. (Contributed by Michael Schout. Reviewed by David Steele.)
* Log archive copy during backup. (Reviewed by Cynthia Shang, Stefan Fercot.)

Documentation Improvements:

* Update reference to include links to user guide examples. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Update selective restore documentation with caveats. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Add compress-type clarification to archive-copy documentation. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Add compress-level defaults per compress-type value. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Add note about required NFS settings being the same as PostgreSQL. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2021-04-05 09:18:20 -04:00
David Steele
a6a544c7f9 Fix extraneous hrnReplaceKey() calls in unit tests.
hrnReplaceKey() was added to the TEST_ERROR*() macros in 58760486 but some calls to TEST_ERROR*() already used it. This led to the function being called twice on the same buffer which had no effect but valgrind definitely did not like.

Remove extraneous calls to make valgrind happy. Since this is test code there are no implications for production.
2021-04-02 15:35:41 -04:00
David Steele
1615cb0261 Add --gen-check option to test.pl to check autogenerated code.
Make sure that auto-generated code does not change during CI.

This is useful for catching missed code generation, especially help.auto.c.
2021-03-31 10:36:22 -04:00
Cynthia Shang
d372dd652c
Update reference to include links to user guide examples.
The command-example and command-example-list elements were removed from the documentation rendering some time ago so these tags were dead code. The tags, however, contained some examples and information that were pertinent to the command, so where possible, the information was included in the description of the command and/or the user-guide and links to the relevant user guide sections were added.

Note that some commands could not be updated with user guide references since doing so would cause a cyclical reference in the user guide. These commands have an internal comment to indicate this.

In addition, some clarifications were added (e.g. expire --set option) where information was lacking.
2021-03-31 09:36:56 -04:00
David Steele
b6106f3c1f
Add archive-header-check option.
Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located.

Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
David Steele
01b8e2258f
Improve archive-push command fault tolerance.
3b8f0ef missed some cases that could cause archive-push to fail:

* Checking archive info.
* Checking to see if a WAL segment already exists.

These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
Cynthia Shang
2789d3b620
Improve info command fault tolerance.
This improvement reduces the number of errors thrown; these errors will now be reported as a status for the stanza or repo as appropriate. Invalid option configurations are still thrown but all other errors are caught, formatted and reported. This was necessary for multiple repositories so that the command can complete gathering information from each repository and report the results rather than immediately aborting when an error occurs.

Two new error codes were introduced:
6 = requested backup not found
99 = other, which is used to indicate an error has occurred that requires more details to be provided

A new stanza name of "[invalid]" was created for instances where a stanza was not specified and no stanza can be found.

If there is only one repository configured the error will move up to the stanza level with the standard error formatting of 'error (message)' where the message will be "other" and the details of the error will be listed on the next line(s):

stanza: stanza1
    status: error (other)
            [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
            CryptoError: cipher header invalid
            HINT: is or was the repo encrypted?
            FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
            HINT: backup.info cannot be opened and is required to perform a backup.
            HINT: has a stanza-create been performed?
            HINT: use option --stanza if encryption settings are different for the stanza than the global
    cipher: aes-256-cbc

If a backup set is requested but is not found on any repo, a stanza-level status error of 'requested backup not found' is reported when there are no other errors:

pgbackrest info --stanza=demo --set=bogus
stanza: demo
    status: error (requested backup not found)
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

If there are multiple repositories configured and a single repo is in error but the other repos are ok or have a different error:

pgbackrest info --stanza=demo --set=20210322-171211F
stanza: demo
    status: mixed
        repo1: error
               [CryptoError] unable to load info file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info' or '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy':
               CryptoError: cipher header invalid
               HINT: is or was the repo encrypted?
               FileMissingError: unable to open missing file '/var/lib/pgbackrest/repo/backup/stanza1/backup.info.copy' for read
               HINT: backup.info cannot be opened and is required to perform a backup.
               HINT: has a stanza-create been performed?
               HINT: use option --stanza if encryption settings are different for the stanza than the global
        repo2: ok
    cipher: mixed
        repo1: aes-256-cbc
        repo2: none

    db (current)
        wal archive min/max (12): 000000010000000000000001/000000010000000000000003

        full backup: 20210322-171211F
            timestamp start/stop: 2021-03-22 17:12:11 / 2021-03-22 17:12:28
            wal start/stop: 000000010000000000000002 / 000000010000000000000002
            database size: 23.4MB, database backup size: 23.4MB
            repo2: backup set size: 2.8MB, backup size: 2.8MB
            database list: postgres (13359)

Json output will include the repository information and any error information. If no stanzas are found, then [invalid] will be set as the name:

[
   {
       "archive":[],
       "backup":[],
       "cipher":"none",
       "db":[],
       "name":"[invalid]",
       "repo":[
           {
               "cipher":"none",
               "key":1,
               "status":{
                   "code":99,
                   "message":"[PathOpenError] unable to list file info for path '/var/lib/pgbackrest/repo2/backup': [13] Permission denied"
               }
           }
       ],
       "status":{
           "code":99,
           "lock":{"backup":{"held":false}},
           "message":"other"
           }
   }
]
2021-03-25 12:29:36 -04:00
David Steele
7d7ac0e0eb
Exclude content-length from S3 signed headers.
The content-length header was being signed since it was the only header that didn't need to be and it seemed simpler just to sign it as well. Also, the S3 documentation encourages signing as many headers as possible to avoid tampering.

However, some proxies munge this header causing authentication failure, so skip signing content-length.
2021-03-25 07:07:16 -04:00
David Steele
5876048675 Allow key replacements in TEST_ERROR*() macros.
Allow standard replacements (e.g. {[path]}) in TEST_ERROR*() macros.

Update command/archive-push unit test as an example of usage.
2021-03-23 18:20:26 -04:00