1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

3563 Commits

Author SHA1 Message Date
David Steele
01b20724da Rename PostgreSQL pid file constants and tests. 2021-10-13 19:36:59 -04:00
David Steele
5701620408 Rename manifest file primary flag in tests. 2021-10-13 19:02:58 -04:00
David Steele
a44f9e373b Update Vagrantfile to Ubuntu 20.04. 2021-10-13 13:21:04 -04:00
David Steele
b16e827d69 Do not show output of PostgreSQL upgrade status check in user guide.
On some platforms the output may contain UTF-8 characters that the latex code is not prepared to handle.

Showing the command is much more important than showing the output, so no big loss.
2021-10-13 13:20:11 -04:00
David Steele
5e84645ac0 Update comments referring to the PostgreSQL primary. 2021-10-13 12:16:47 -04:00
David Steele
430efff98a Update documentation/links to main branch. 2021-10-13 12:01:53 -04:00
David Steele
1212668d5e Update contributing.xml with rendering changes from def7d513.
Also update help.xml path missed in f4e1babf.
2021-10-13 11:43:14 -04:00
David Steele
90c73183ea Add libc6-dbg required by updated valgrind to Vagrantfile/Dockerfile. 2021-10-13 09:37:03 -04:00
David Steele
c2d4552b73 Add debug options to code generation make in test.pl. 2021-10-13 08:51:58 -04:00
David Steele
bd91ebca75 Remove command overrides for output options.
The overrides are not needed since both commands require the same default and allow list.
2021-10-12 16:16:05 -04:00
David Steele
e8e346bc87 Remove command overrides for restore-only options.
The overrides are not needed since these options are only valid for one command.
2021-10-12 08:53:12 -04:00
David Steele
576b047634 Invert required in set option to simplify generated rules. 2021-10-11 16:25:36 -04:00
David Steele
980b777a4a Fix indentation. 2021-10-09 12:39:54 -04:00
David Steele
cc7f2eea90 Add assert in pckReadNext() to prevent reading into a field. 2021-10-09 12:37:25 -04:00
David Steele
610bfd736e Increase tolerance for 0ms sleep in common/time test. 2021-10-09 12:34:45 -04:00
David Steele
7ab8dcbe6e Read tag size in pckReadTagNext().
Rather than reading the size everywhere it is needed, get it when the tag is read, if it exists.

This simplifies logic around consuming the data when not needed. There are more use cases for this coming up.
2021-10-09 12:15:19 -04:00
David Steele
ed68792e76 Rename strNewN() to strNewZN().
Make the function name consistent with other functions that accept zero-terminated strings, e.g. strNewZ() and strCatZN().
2021-10-07 19:57:28 -04:00
David Steele
b7e17d80ea More efficient memory allocation for Strings and String Variants.
The vast majority of Strings are never modified so for most cases allocate memory for the string with the object. This results in one allocation in most cases instead of two. Use strNew() if strCat*() functions are needed.

Update varNewStr() in the same way since String Variants can never be modified. This results in one allocation in all cases instead of three. Also update varNewStrZ() to use STR() instead of strNewZ() to save two more allocations.
2021-10-07 19:43:28 -04:00
David Steele
208641ac7f Use constant string for user/group in performance/type test.
It is not safe to return strings created with STRDEF() from a function.
2021-10-07 18:50:56 -04:00
David Steele
74d3131830 More efficient generation of diff/incr backup label. 2021-10-07 14:58:11 -04:00
David Steele
498902e885
Allow "global" as a stanza prefix.
A stanza name like global_stanza was not allowed because the code was not selective enough about how a global section should be formatted.

Update the config parser to correctly recognize global sections.
2021-10-07 12:18:24 -04:00
David Steele
338102861f Improve instructions for rebuilding pgbackrest during stress testing. 2021-10-07 11:01:48 -04:00
David Steele
fb3f6928c9 Add configurable storage helpers to create repository storage.
Remove the hardcoded storage helpers from storageRepoGet() except for the the built-in Posix helper and the special remote helper.

The goal is to make storage driver development a bit easier by isolating as much of the code as possible into the driver module. This also makes coverage reporting much simpler for additional drivers since they do not need to provide coverage for storage/helper.

Consolidate the CIFS tests into the Posix tests since CIFS is just a special case of the Posix.

Test all storage features in the Posix test so that other storage driver tests do not need to provide coverage for storage/storage.

Remove some dead code in the storage/s3 test.
2021-10-06 19:27:04 -04:00
David Steele
cfd823355a Refactor S3 storage driver for additional auth methods.
Currently only two auth methods are supported and a lot of refactoring is required to add a third one.

Do the refactoring now to reduce noise in the commit that adds the third auth method.
2021-10-06 12:38:56 -04:00
David Steele
68c5f3eaf1
Allow link-map option to create new links.
Currently link-map only allows links that exist in the backup manifest to be remapped to a new destination.

Allow link-map to create a new link as long as a valid path/file from the backup is referenced.
2021-10-05 17:59:05 -04:00
David Steele
f2aeb30fc7 Add state to ProtocolClient.
This is currently only useful for debugging, but in the future the state may be used for resetting the protocol when something goes wrong.
2021-10-05 14:06:59 -04:00
David Steele
2c65fed80f Add missing asserts and move temp mem context block. 2021-10-05 12:29:16 -04:00
David Steele
6af827cbb1
Report original error and retries on local job failure.
The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync.

Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00
Stefan Fercot
34f7873432
Report backup file validation errors in backup.info.
Currently errors found during the backup are only available in text output when specifying --set.

Add a flag to backup.info that is available in both the text and json output when --set is not specified. This at least provides the basic info that an error was found in the cluster during the backup, though details are still only available as described above.
2021-10-04 13:45:53 -04:00
David Steele
57c6231546 Add arm64 testing on Cirrus CI.
These tests run in a container without permissions to mount tempfs, so add an option to ci.pl to not create tempfs. Also add some packages not in the base image.
2021-10-02 17:27:33 -04:00
David Steele
f1ed8f0e51 Sort WAL segment names when reporting duplicates.
Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output.

Found on arm64.
2021-10-02 16:29:31 -04:00
David Steele
71047a9d6d Use strncpy() to limit characters copied to optionName.
Valgrind complained about uninitialized values on arm64 when comparing the reset prefix, probably because "reset" ended up being larger than the option name: Conditional jump or move depends on uninitialised value(s) at cfgParseOption (parse.c:568).

Coverity complained because it could not verify the size of the string to be copied into optionName, probably because it does not understand the purpose of strSize(): You might overrun the 65-character fixed-size string "optionName" by copying the return value of "strZ" without checking the length.

Use strncpy() even though we have already checked the size and make sure the string is terminated. Keep the size check because searching for truncated option names is not a good idea.

This is not a production bug since the code has not been released yet.
2021-10-02 16:17:33 -04:00
David Steele
b792a14cd7 Use temp mem context when calling command handlers.
It is safer and more efficient to free memory after each handler completes.

The db command handlers use the server context so update them to use the top context.
2021-10-01 18:23:03 -04:00
David Steele
ae40ed6ec9 Add jobRetry parameter to HRN_CFG_LOAD().
Allow the default of 0 to be overridden to test retry behavior for commands.
2021-10-01 17:15:36 -04:00
David Steele
136d309dd4 Allow stack trace to be specified for errorInternalThrow().
This allows the stack trace to be set when an error is received by the protocol, rather than appending it to the message. Now these errors will look no different than any other error and the stack trace will be reported in the same way.

One immediate benefit is that test.pl --vm-out --log-level-test=debug will work for tests that check expect log results. Previously, the test would error at the first check because the stack trace included in the message would not match the expected log output.
2021-10-01 15:29:31 -04:00
David Steele
62f6fbe2a9 Update file mode in info/manifest test to 0600.
0400 is not a very realistic mode. It may have become the default due to copy-pasting.
2021-10-01 10:15:34 -04:00
David Steele
0690cb25a0 Remove repository format 6 notes.
The notes have been moved to a Github project.
2021-09-30 17:55:38 -04:00
David Steele
376362475e Move archive-header-check option to the archive reference section. 2021-09-30 16:15:45 -04:00
David Steele
cf1a57518f Refactor restoreManifestMap() to be driven by link-map.
This will allow new links to be added in a future commit. The current implementation is driven by the links that already exist in the manifest, which would make the new use case more complex to implement.

Also, add a more helpful error when a tablespace link is specified.
2021-09-30 14:29:49 -04:00
David Steele
d89a67776c Refactor restoreManifestMap() tests in the command/restore unit.
Add test titles, new tests, and rearrange.

Also manifestTargetFindDefault(), which will soon be used by core code in a refactoring commit.
2021-09-30 13:39:29 -04:00
David Steele
7a53ba7c7f Add note to comment for int64 typedef. 2021-09-30 13:28:14 -04:00
David Steele
815377cc60 Finalize catalog number for PostgreSQL 14 release. 2021-09-30 13:27:14 -04:00
Stefan Fercot
baf186bfb0 Fix comment typos. 2021-09-29 12:03:01 -04:00
David Steele
9e79f0e64b Add recovery start time to online backup restore log.
This helps give an idea of how much recovery needs to be done to reach the end of the WAL stream and is easier to read than the backup label.
2021-09-29 10:31:51 -04:00
David Steele
9346895f5b
Rename page checksum error to error list in info text output.
"error list" makes it clearer that other errors may be reported. For example, if checksum-page is true in the manifest but no checksum-page-error list is provided then the error is in alignment, i.e. the file size is not a multiple of the page size, with allowances made for a valid-looking partial page at the end of the file.

It is still not possible to differentiate between alignment and page checksum errors in the output but this will be addressed in a future commit.
2021-09-29 09:58:47 -04:00
David Steele
b7ef12a76f Add hints to standby replay timeout message. 2021-09-28 15:55:13 -04:00
David Steele
096829b3b2 Add repo-azure-uri-style option.
Azurite introduced a breaking change in 8f63964e to use automatically host-style URIs when the endpoint appears to be a multipart hostname.

This option allows the user to configure which style URI will be used, but changing the endpoint might cause breakage if Azurite decides to use a different style. Future changes to Azurite may also cause breakage.
2021-09-27 09:01:53 -04:00
David Steele
c8ea17c68f Convert page checksum filter result to a pack.
The pack is both more compact and more efficient than a variant.

Also aggregate the page error info in the main process rather than in the filter to allow additional LSN filtering, to be added in a future commit.
2021-09-24 17:40:31 -04:00
David Steele
ac1f6db4a2 Centralize and optimize tag stack management.
The push and pop code was duplicated in four places, so centralize the code into pckTagStackPop() and pckTagStackPush().

Also create a default bottom item for the stack to avoid allocating a list if there will only ever be the default container, which is very common. This avoids the extra time and memory to allocate a list.
2021-09-23 14:06:00 -04:00
David Steele
15e7ff10d3 Add Pack pseudo-type.
Rather than working directly with Buffer types, define a new Pack pseudo-type that represents a Buffer containing a pack. This makes it clearer that a pack is being stored and allows stronger typing.
2021-09-23 08:31:32 -04:00