1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

3580 Commits

Author SHA1 Message Date
David Steele
fbd018cd56 Allow S3/Azure Docker images to be specified in user guide.
It is not uncommon for the S3/Azure emulators we use to introduce breaking changes without warning. If that happens the documentation can still be built by specifying a working version of the image. In general, it is better to let the version float so we know when things break.

Azurite has yet another breaking change coming up (see 096829b3, c38d6926, and Azurite issue 1039) so set azure-image at the current version until the breaking change has been released.
2021-10-20 08:22:37 -04:00
David Steele
5dfdd6dd5b Add -Werror -Wfatal-errors -g flags to configure --enable-test.
These flags are used for all tests but it was not possible to add them to configure before the change in 046d6643. This is especially important for adhoc tests to ensure the flags are not forgotten.

Remove the flags from test make commands where they were being applied.

There is no change for production builds.
2021-10-19 12:45:20 -04:00
David Steele
046d664337 Set most compiler flags as late as possible in configure.
Some flags, e.g. -Wfatal-errors, will cause tests in configure to behave incorrectly so we have not been able to add them to --enable-test.

Add the compiler flags as late as possible so configure checks are not affected. This will allow us to add flags that we need for testing without having to explicitly pass them to make.
2021-10-19 12:14:09 -04:00
David Steele
e443e3c6c0 Add br tags for HTML documentation rendering missed in def7d513. 2021-10-19 09:06:06 -04:00
Reid Thompson
4c2d89eb66 Fix typos. 2021-10-18 16:43:19 -04:00
Reid Thompson
6cc8e45df6 Add missing paragraph tag in user guide. 2021-10-18 14:45:36 -04:00
David Steele
ccc255d3e0 Add TLS Server.
The TLS server is an alternative to using SSH for protocol connections to remote hosts.

This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
David Steele
09fb9393f1 Write command configuration overrides explicitly.
If not written explicitly then it is impossible to distinguish the override from a NULL, which indicates no override.
2021-10-18 14:02:05 -04:00
David Steele
90f7f11a9f Add missing static keywords in test modules. 2021-10-18 12:22:48 -04:00
David Steele
4570c7e275 Allow error buffer to be resized for testing.
Some tests can generate very large error messages for diffs and they often get cut off before the end.

Also fix a test so it does not create too large a buffer on the stack.
2021-10-18 11:32:53 -04:00
David Steele
838ee3bd08 Increase some storage test timeouts.
32-bit Debian 9 is sometimes timing out on these tests so increase the timeouts to make the tests more reliable.
2021-10-18 11:05:53 -04:00
David Steele
6b9e19d423 Convert configuration optional rules to pack format.
The previous format was custom for configuration parsing and was not as expressive as the pack format. An immediate benefit is that commands with the same optional rules are merged.

Defaults are now represented correctly (not multiplied), which simplifies the option default functions used by help.
2021-10-16 12:35:47 -04:00
David Steele
360cff94e4 Update 32-bit test container to Debian 9.
Also rebalance PostgreSQL version integration tests.
2021-10-16 12:33:31 -04:00
David Steele
0e84c19a9f Remove allow range from pg-socket-path option.
The allow range was never processed because the string type does not allow ranges, but it is wasteful to have it in the parse rules.

It would be good if auto-generation errored on useless allow ranges, but that will need wait since it does not impact production.
2021-10-15 17:50:54 -04:00
David Steele
144469b977 Add const buffer functions to Pack type.
These allow packs to be created without allocating a buffer in the case that the buffer already exists or the data is in a global constant.

Also fix a rendering issue in hrnPackReadToStr().
2021-10-15 15:50:55 -04:00
David Steele
66bfd1327e Rename SSH connection control parameters in integration tests. 2021-10-13 19:48:41 -04:00
David Steele
447b24309d Update RHEL package URL. 2021-10-13 19:43:40 -04:00
David Steele
01b20724da Rename PostgreSQL pid file constants and tests. 2021-10-13 19:36:59 -04:00
David Steele
5701620408 Rename manifest file primary flag in tests. 2021-10-13 19:02:58 -04:00
David Steele
a44f9e373b Update Vagrantfile to Ubuntu 20.04. 2021-10-13 13:21:04 -04:00
David Steele
b16e827d69 Do not show output of PostgreSQL upgrade status check in user guide.
On some platforms the output may contain UTF-8 characters that the latex code is not prepared to handle.

Showing the command is much more important than showing the output, so no big loss.
2021-10-13 13:20:11 -04:00
David Steele
5e84645ac0 Update comments referring to the PostgreSQL primary. 2021-10-13 12:16:47 -04:00
David Steele
430efff98a Update documentation/links to main branch. 2021-10-13 12:01:53 -04:00
David Steele
1212668d5e Update contributing.xml with rendering changes from def7d513.
Also update help.xml path missed in f4e1babf.
2021-10-13 11:43:14 -04:00
David Steele
90c73183ea Add libc6-dbg required by updated valgrind to Vagrantfile/Dockerfile. 2021-10-13 09:37:03 -04:00
David Steele
c2d4552b73 Add debug options to code generation make in test.pl. 2021-10-13 08:51:58 -04:00
David Steele
bd91ebca75 Remove command overrides for output options.
The overrides are not needed since both commands require the same default and allow list.
2021-10-12 16:16:05 -04:00
David Steele
e8e346bc87 Remove command overrides for restore-only options.
The overrides are not needed since these options are only valid for one command.
2021-10-12 08:53:12 -04:00
David Steele
576b047634 Invert required in set option to simplify generated rules. 2021-10-11 16:25:36 -04:00
David Steele
980b777a4a Fix indentation. 2021-10-09 12:39:54 -04:00
David Steele
cc7f2eea90 Add assert in pckReadNext() to prevent reading into a field. 2021-10-09 12:37:25 -04:00
David Steele
610bfd736e Increase tolerance for 0ms sleep in common/time test. 2021-10-09 12:34:45 -04:00
David Steele
7ab8dcbe6e Read tag size in pckReadTagNext().
Rather than reading the size everywhere it is needed, get it when the tag is read, if it exists.

This simplifies logic around consuming the data when not needed. There are more use cases for this coming up.
2021-10-09 12:15:19 -04:00
David Steele
ed68792e76 Rename strNewN() to strNewZN().
Make the function name consistent with other functions that accept zero-terminated strings, e.g. strNewZ() and strCatZN().
2021-10-07 19:57:28 -04:00
David Steele
b7e17d80ea More efficient memory allocation for Strings and String Variants.
The vast majority of Strings are never modified so for most cases allocate memory for the string with the object. This results in one allocation in most cases instead of two. Use strNew() if strCat*() functions are needed.

Update varNewStr() in the same way since String Variants can never be modified. This results in one allocation in all cases instead of three. Also update varNewStrZ() to use STR() instead of strNewZ() to save two more allocations.
2021-10-07 19:43:28 -04:00
David Steele
208641ac7f Use constant string for user/group in performance/type test.
It is not safe to return strings created with STRDEF() from a function.
2021-10-07 18:50:56 -04:00
David Steele
74d3131830 More efficient generation of diff/incr backup label. 2021-10-07 14:58:11 -04:00
David Steele
498902e885
Allow "global" as a stanza prefix.
A stanza name like global_stanza was not allowed because the code was not selective enough about how a global section should be formatted.

Update the config parser to correctly recognize global sections.
2021-10-07 12:18:24 -04:00
David Steele
338102861f Improve instructions for rebuilding pgbackrest during stress testing. 2021-10-07 11:01:48 -04:00
David Steele
fb3f6928c9 Add configurable storage helpers to create repository storage.
Remove the hardcoded storage helpers from storageRepoGet() except for the the built-in Posix helper and the special remote helper.

The goal is to make storage driver development a bit easier by isolating as much of the code as possible into the driver module. This also makes coverage reporting much simpler for additional drivers since they do not need to provide coverage for storage/helper.

Consolidate the CIFS tests into the Posix tests since CIFS is just a special case of the Posix.

Test all storage features in the Posix test so that other storage driver tests do not need to provide coverage for storage/storage.

Remove some dead code in the storage/s3 test.
2021-10-06 19:27:04 -04:00
David Steele
cfd823355a Refactor S3 storage driver for additional auth methods.
Currently only two auth methods are supported and a lot of refactoring is required to add a third one.

Do the refactoring now to reduce noise in the commit that adds the third auth method.
2021-10-06 12:38:56 -04:00
David Steele
68c5f3eaf1
Allow link-map option to create new links.
Currently link-map only allows links that exist in the backup manifest to be remapped to a new destination.

Allow link-map to create a new link as long as a valid path/file from the backup is referenced.
2021-10-05 17:59:05 -04:00
David Steele
f2aeb30fc7 Add state to ProtocolClient.
This is currently only useful for debugging, but in the future the state may be used for resetting the protocol when something goes wrong.
2021-10-05 14:06:59 -04:00
David Steele
2c65fed80f Add missing asserts and move temp mem context block. 2021-10-05 12:29:16 -04:00
David Steele
6af827cbb1
Report original error and retries on local job failure.
The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync.

Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00
Stefan Fercot
34f7873432
Report backup file validation errors in backup.info.
Currently errors found during the backup are only available in text output when specifying --set.

Add a flag to backup.info that is available in both the text and json output when --set is not specified. This at least provides the basic info that an error was found in the cluster during the backup, though details are still only available as described above.
2021-10-04 13:45:53 -04:00
David Steele
57c6231546 Add arm64 testing on Cirrus CI.
These tests run in a container without permissions to mount tempfs, so add an option to ci.pl to not create tempfs. Also add some packages not in the base image.
2021-10-02 17:27:33 -04:00
David Steele
f1ed8f0e51 Sort WAL segment names when reporting duplicates.
Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output.

Found on arm64.
2021-10-02 16:29:31 -04:00
David Steele
71047a9d6d Use strncpy() to limit characters copied to optionName.
Valgrind complained about uninitialized values on arm64 when comparing the reset prefix, probably because "reset" ended up being larger than the option name: Conditional jump or move depends on uninitialised value(s) at cfgParseOption (parse.c:568).

Coverity complained because it could not verify the size of the string to be copied into optionName, probably because it does not understand the purpose of strSize(): You might overrun the 65-character fixed-size string "optionName" by copying the return value of "strZ" without checking the length.

Use strncpy() even though we have already checked the size and make sure the string is terminated. Keep the size check because searching for truncated option names is not a good idea.

This is not a production bug since the code has not been released yet.
2021-10-02 16:17:33 -04:00
David Steele
b792a14cd7 Use temp mem context when calling command handlers.
It is safer and more efficient to free memory after each handler completes.

The db command handlers use the server context so update them to use the top context.
2021-10-01 18:23:03 -04:00