1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-03-03 14:52:21 +02:00

1555 Commits

Author SHA1 Message Date
David Steele
fa564ee196 Improve documentation for cmd-ssh, repo-host-cmd, pg-host-cmd options.
Use "command" instead of "exe" and make the descriptions more consistent.
2021-10-27 11:08:32 -04:00
David Steele
e1f6c066b3 Improve documentation for buffer-size option. 2021-10-27 10:52:39 -04:00
David Steele
e2eea974c1 Add assertion for Coverity.
Coverity thinks this value might be NULL but that should not be possible because of the TRY...CATCH block.
2021-10-26 12:09:41 -04:00
David Steele
7fb99c59c8 Use externed instead of extern'd in comments.
This is mostly to revert some comment changes in b11ab9f7 that will break the ppc64le patch, but at the same time keep the spelling consistent in all comments and documentation.

Also revert some space changes for the same reason.
2021-10-26 07:46:48 -04:00
David Steele
13d4559708 Check return value of getsockopt().
Checking the return value is not terribly important here, but if setsockopt() fails it is likely that bind() will fail as well. May as well get it over with and this makes Coverity happy.
2021-10-25 15:31:39 -04:00
Reid Thompson
1152f7a7d6 Fix mismatched parameters in tlsClientNew() call.
3879bc69 added this call and the parameters were not quite right but in way that the compiler decided they were OK. It was mostly working but TLS verification was disabled if caPath was NULL, which is not OK.
2021-10-25 12:56:33 -04:00
David Steele
a1a2284c88 Fix typos in error messages. 2021-10-25 09:01:22 -04:00
David Steele
3879bc69b8
Add WebIdentity authentication for AWS S3.
This allows credentials to be automatically acquired in an EKS environment.
2021-10-22 18:31:55 -04:00
David Steele
51785739f4
Store config values as a union instead of a variant.
The variants were needed to easily serialize configurations for the Perl code.

Unions are more efficient and will allow us to add new types that are not supported by variants, e.g. StringId.
2021-10-22 18:02:20 -04:00
David Steele
2cea005f74 Fix segfault on invalid GCS key file. 2021-10-22 17:19:16 -04:00
David Steele
b11ab9f799 Fix typos. 2021-10-21 13:31:22 -04:00
David Steele
8ad6b7330e Fix outdated comment.
This check was moved from within the path checks at some point but the comment did not get updated.
2021-10-21 09:20:40 -04:00
David Steele
5dfdd6dd5b Add -Werror -Wfatal-errors -g flags to configure --enable-test.
These flags are used for all tests but it was not possible to add them to configure before the change in 046d6643. This is especially important for adhoc tests to ensure the flags are not forgotten.

Remove the flags from test make commands where they were being applied.

There is no change for production builds.
2021-10-19 12:45:20 -04:00
David Steele
046d664337 Set most compiler flags as late as possible in configure.
Some flags, e.g. -Wfatal-errors, will cause tests in configure to behave incorrectly so we have not been able to add them to --enable-test.

Add the compiler flags as late as possible so configure checks are not affected. This will allow us to add flags that we need for testing without having to explicitly pass them to make.
2021-10-19 12:14:09 -04:00
Reid Thompson
4c2d89eb66 Fix typos. 2021-10-18 16:43:19 -04:00
David Steele
ccc255d3e0 Add TLS Server.
The TLS server is an alternative to using SSH for protocol connections to remote hosts.

This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
David Steele
09fb9393f1 Write command configuration overrides explicitly.
If not written explicitly then it is impossible to distinguish the override from a NULL, which indicates no override.
2021-10-18 14:02:05 -04:00
David Steele
4570c7e275 Allow error buffer to be resized for testing.
Some tests can generate very large error messages for diffs and they often get cut off before the end.

Also fix a test so it does not create too large a buffer on the stack.
2021-10-18 11:32:53 -04:00
David Steele
6b9e19d423 Convert configuration optional rules to pack format.
The previous format was custom for configuration parsing and was not as expressive as the pack format. An immediate benefit is that commands with the same optional rules are merged.

Defaults are now represented correctly (not multiplied), which simplifies the option default functions used by help.
2021-10-16 12:35:47 -04:00
David Steele
360cff94e4 Update 32-bit test container to Debian 9.
Also rebalance PostgreSQL version integration tests.
2021-10-16 12:33:31 -04:00
David Steele
0e84c19a9f Remove allow range from pg-socket-path option.
The allow range was never processed because the string type does not allow ranges, but it is wasteful to have it in the parse rules.

It would be good if auto-generation errored on useless allow ranges, but that will need wait since it does not impact production.
2021-10-15 17:50:54 -04:00
David Steele
144469b977 Add const buffer functions to Pack type.
These allow packs to be created without allocating a buffer in the case that the buffer already exists or the data is in a global constant.

Also fix a rendering issue in hrnPackReadToStr().
2021-10-15 15:50:55 -04:00
David Steele
01b20724da Rename PostgreSQL pid file constants and tests. 2021-10-13 19:36:59 -04:00
David Steele
5701620408 Rename manifest file primary flag in tests. 2021-10-13 19:02:58 -04:00
David Steele
5e84645ac0 Update comments referring to the PostgreSQL primary. 2021-10-13 12:16:47 -04:00
David Steele
bd91ebca75 Remove command overrides for output options.
The overrides are not needed since both commands require the same default and allow list.
2021-10-12 16:16:05 -04:00
David Steele
e8e346bc87 Remove command overrides for restore-only options.
The overrides are not needed since these options are only valid for one command.
2021-10-12 08:53:12 -04:00
David Steele
576b047634 Invert required in set option to simplify generated rules. 2021-10-11 16:25:36 -04:00
David Steele
980b777a4a Fix indentation. 2021-10-09 12:39:54 -04:00
David Steele
cc7f2eea90 Add assert in pckReadNext() to prevent reading into a field. 2021-10-09 12:37:25 -04:00
David Steele
7ab8dcbe6e Read tag size in pckReadTagNext().
Rather than reading the size everywhere it is needed, get it when the tag is read, if it exists.

This simplifies logic around consuming the data when not needed. There are more use cases for this coming up.
2021-10-09 12:15:19 -04:00
David Steele
ed68792e76 Rename strNewN() to strNewZN().
Make the function name consistent with other functions that accept zero-terminated strings, e.g. strNewZ() and strCatZN().
2021-10-07 19:57:28 -04:00
David Steele
b7e17d80ea More efficient memory allocation for Strings and String Variants.
The vast majority of Strings are never modified so for most cases allocate memory for the string with the object. This results in one allocation in most cases instead of two. Use strNew() if strCat*() functions are needed.

Update varNewStr() in the same way since String Variants can never be modified. This results in one allocation in all cases instead of three. Also update varNewStrZ() to use STR() instead of strNewZ() to save two more allocations.
2021-10-07 19:43:28 -04:00
David Steele
74d3131830 More efficient generation of diff/incr backup label. 2021-10-07 14:58:11 -04:00
David Steele
498902e885
Allow "global" as a stanza prefix.
A stanza name like global_stanza was not allowed because the code was not selective enough about how a global section should be formatted.

Update the config parser to correctly recognize global sections.
2021-10-07 12:18:24 -04:00
David Steele
fb3f6928c9 Add configurable storage helpers to create repository storage.
Remove the hardcoded storage helpers from storageRepoGet() except for the the built-in Posix helper and the special remote helper.

The goal is to make storage driver development a bit easier by isolating as much of the code as possible into the driver module. This also makes coverage reporting much simpler for additional drivers since they do not need to provide coverage for storage/helper.

Consolidate the CIFS tests into the Posix tests since CIFS is just a special case of the Posix.

Test all storage features in the Posix test so that other storage driver tests do not need to provide coverage for storage/storage.

Remove some dead code in the storage/s3 test.
2021-10-06 19:27:04 -04:00
David Steele
cfd823355a Refactor S3 storage driver for additional auth methods.
Currently only two auth methods are supported and a lot of refactoring is required to add a third one.

Do the refactoring now to reduce noise in the commit that adds the third auth method.
2021-10-06 12:38:56 -04:00
David Steele
68c5f3eaf1
Allow link-map option to create new links.
Currently link-map only allows links that exist in the backup manifest to be remapped to a new destination.

Allow link-map to create a new link as long as a valid path/file from the backup is referenced.
2021-10-05 17:59:05 -04:00
David Steele
f2aeb30fc7 Add state to ProtocolClient.
This is currently only useful for debugging, but in the future the state may be used for resetting the protocol when something goes wrong.
2021-10-05 14:06:59 -04:00
David Steele
2c65fed80f Add missing asserts and move temp mem context block. 2021-10-05 12:29:16 -04:00
David Steele
6af827cbb1
Report original error and retries on local job failure.
The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync.

Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00
Stefan Fercot
34f7873432
Report backup file validation errors in backup.info.
Currently errors found during the backup are only available in text output when specifying --set.

Add a flag to backup.info that is available in both the text and json output when --set is not specified. This at least provides the basic info that an error was found in the cluster during the backup, though details are still only available as described above.
2021-10-04 13:45:53 -04:00
David Steele
f1ed8f0e51 Sort WAL segment names when reporting duplicates.
Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output.

Found on arm64.
2021-10-02 16:29:31 -04:00
David Steele
71047a9d6d Use strncpy() to limit characters copied to optionName.
Valgrind complained about uninitialized values on arm64 when comparing the reset prefix, probably because "reset" ended up being larger than the option name: Conditional jump or move depends on uninitialised value(s) at cfgParseOption (parse.c:568).

Coverity complained because it could not verify the size of the string to be copied into optionName, probably because it does not understand the purpose of strSize(): You might overrun the 65-character fixed-size string "optionName" by copying the return value of "strZ" without checking the length.

Use strncpy() even though we have already checked the size and make sure the string is terminated. Keep the size check because searching for truncated option names is not a good idea.

This is not a production bug since the code has not been released yet.
2021-10-02 16:17:33 -04:00
David Steele
b792a14cd7 Use temp mem context when calling command handlers.
It is safer and more efficient to free memory after each handler completes.

The db command handlers use the server context so update them to use the top context.
2021-10-01 18:23:03 -04:00
David Steele
136d309dd4 Allow stack trace to be specified for errorInternalThrow().
This allows the stack trace to be set when an error is received by the protocol, rather than appending it to the message. Now these errors will look no different than any other error and the stack trace will be reported in the same way.

One immediate benefit is that test.pl --vm-out --log-level-test=debug will work for tests that check expect log results. Previously, the test would error at the first check because the stack trace included in the message would not match the expected log output.
2021-10-01 15:29:31 -04:00
David Steele
376362475e Move archive-header-check option to the archive reference section. 2021-09-30 16:15:45 -04:00
David Steele
cf1a57518f Refactor restoreManifestMap() to be driven by link-map.
This will allow new links to be added in a future commit. The current implementation is driven by the links that already exist in the manifest, which would make the new use case more complex to implement.

Also, add a more helpful error when a tablespace link is specified.
2021-09-30 14:29:49 -04:00
David Steele
d89a67776c Refactor restoreManifestMap() tests in the command/restore unit.
Add test titles, new tests, and rearrange.

Also manifestTargetFindDefault(), which will soon be used by core code in a refactoring commit.
2021-09-30 13:39:29 -04:00
David Steele
7a53ba7c7f Add note to comment for int64 typedef. 2021-09-30 13:28:14 -04:00