1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3563 Commits

Author SHA1 Message Date
David Steele
dcb79ab8fb Decode JSON \u escaped characters.
ASCII may occasionally be encoded (e.g. &) to prevent ambiguity depending on where the JSON is located.

Only ASCII can be decoded. In general Unicode should not be encoded in JSON.
2021-02-19 07:32:40 -05:00
David Steele
c4243331de Silence chmod warning of empty path when removing unit test data. 2021-02-19 07:24:59 -05:00
David Steele
3837e61a75
Fix option warnings breaking async archive-get/archive-push.
Option warnings will cause the async process to fail because a warning is logged but stdout is closed so the process aborts.

This bug has existed for quite some time, but it was made worse by abb8ebe because now the async role can have different valid options than the default role. Previously at least a warning would be emitted before the async process died.

Fix this by only allowing warnings for the default role. Warnings were already suppressed for local and remote roles so the logic already exists.
2021-02-18 13:29:09 -05:00
David Steele
d7befd4189 Fix tests that ensure log levels are not set for local/remote roles.
These tests were broken because they were being gated by resetLogLevel. So they were not setting the log levels, but not because of the role setting. Because resetLogLevel was being checked last coverage testing indicated that the tests were working.

Fix the resetLogLevel parameter in the tests and move resetLogLevel to be tested first so coverage reporting works as expected. This isn't perfect but it is an improvement.
2021-02-16 16:28:19 -05:00
David Steele
d29855bd0b
Fix stack overflow in cipher passphrase generation.
The destination buffer on the stack was not large enough to contain the zero-terminating character.

Increase the buffer size and add an assertion to prevent regressions.

Found on arm64 running musl libc. Other architectures and glibc do not seem to be affected though it is clearly a bug.
2021-02-12 10:08:47 -05:00
David Steele
920c746adb Better exclusions for configure help.
Exclude known unused options rather than trying to include used options. This works better when new options are added.
2021-02-10 15:54:37 -05:00
David Steele
6a717e032f Set config path in configure script.
This allows the config path to be modified with a parameter to the configure script, though this commit does not do that.

Update the Perl code generator to allow literals so that defaults can be C defines rather than static strings.
2021-02-10 14:46:26 -05:00
Cynthia Shang
3408f1ee2e Enhance expire command multi-repo support.
The expire command has been enhanced to expire backups and archives from all configured repositories by default.

In addition, it will accept the --repo option to expire backups and archives only from the specified repository. Using the --repo options the --set option can also be refined further to the specified repo. If --set is provided but the --repo option has not, then all repositories will be searched and retention settings will be applied on each whether the backup set has been found or not.
2021-02-10 12:03:52 -05:00
David Steele
26cbebbda7 Use latex sloppypar to fix monospace wrapping in PDF rendering.
Monospaced identifiers could end up running over if latex was not able to find a place to break the line. Using sloppypar forces breaks so monospaced identifiers don't run over or get broken up.

Also add vspace to admonitions so they have some separation from the prior text.
2021-02-10 09:10:51 -05:00
David Steele
c7d7280fa8 Ensure test user has permissions before removing test files.
This allows files to be deleted even when tests have limited the permissions.
2021-02-08 16:36:38 -05:00
David Steele
b2bba678a0 Add missing linefeeds. 2021-02-08 16:28:16 -05:00
David Steele
b9f0070d91 Update config.guess and config.sub to latest versions. 2021-02-08 13:44:50 -05:00
David Steele
00f06065e7 Begin v2.33 development. 2021-02-08 13:18:22 -05:00
David Steele
aadc9e2fe6 v2.32: Repository Commands
Bug Fixes:

* Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.)

Features:

* Add repo-ls command. (Reviewed by Cynthia Shang, Stefan Fercot.)
* Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.)
* Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele, Michael Banck.)

Improvements:

* Improve archive-get performance. (Reviewed by Cynthia Shang.)
2021-02-08 09:08:16 -05:00
Cynthia Shang
d350d1cc21 Improve expire command documentation. 2021-02-05 11:48:07 -05:00
David Steele
b65c370346 Add repo-get command. 2021-02-05 10:39:03 -05:00
David Steele
218cd078a6 Add repo-ls command. 2021-02-05 10:07:43 -05:00
David Steele
73f61a1f60 Add missed comment. 2021-02-04 17:24:42 -05:00
David Steele
9154d73030 Add -g accidentally removed in 4e8d469f.
The tests all run fine without debug info but gdb and valgrind are a lot less useful without it.
2021-02-02 17:05:55 -05:00
Stefan Fercot
4b46115345
Add archive-mode-check option.
This option disallows the PostgreSQL archive_mode=always setting and disabling it allows the setting.
2021-02-02 13:43:14 -05:00
David Steele
101bf5d114 Log configuration parameters in config test harness.
This makes it easier to find the current configuration in tests.
2021-02-01 14:50:22 -05:00
David Steele
9baf5d7da7 Set dry-run before updating options in config test harness.
This is the same order that is used in cfgLoad() so apply it here as well.

Also remove vestigial HRN_FEATURE_STORAGE define.
2021-02-01 14:42:22 -05:00
David Steele
67d444b9e8 Add bufEmpty().
This seems more readable than bufUsed() == 0, just like 7d6c0319.
2021-02-01 09:22:01 -05:00
David Steele
7d6c0319f0 Add lstEmpty(), strLstEmpty(), and varLstEmpty().
This seems more readable than lst*Size() == 0.

Hopefully this will also eliminate usage of lst*Size() > 0/lst*Size() != 0 variants for the inverse.
2021-01-29 14:27:56 -05:00
David Steele
5281e31422 Add configurable error handlers.
The stackTrace and memContext error handlers were hard-coded which made testing the error module in isolation impossible.

Making the error handlers configurable also makes adding new ones in the future easier.
2021-01-27 17:25:13 -05:00
David Steele
8e9f04cc32 Add HRN_INTEST_* define to indicate when a test is being run.
This is useful for initialization that needs to be done for the test and all subsequent tests.

Use the new defines to implement initialization for sockets and statistics.
2021-01-27 16:54:41 -05:00
Cynthia Shang
d5b919e657
Update expire command log messages with repo prefix.
In preparation for multi-repo support, a repo tag is added in this commit to the expire command log and error messages. This change also affects the expect logs and the user-guide. The format of the tag is "repoX:" where X is the repo key used in the configuration.

Until multi-repo support has been completed, this tag will always be "repo1:".
2021-01-27 16:33:01 -05:00
David Steele
5d34bf3f38 Move cvtDoubleToStr() to strNewDbl().
This is a more logical location and it reduces the dependencies required to compile the common/convert module.
2021-01-27 11:50:10 -05:00
David Steele
87eb081a8f Make unit test builds incremental based on coverage in prior tests.
When building tests only include files covered by the current test or by prior tests. This increases performance (less compilation and linking) and also helps detect cross-dependencies in the code. Since there are currently cross-dependencies the depend option is used to document them and allow compilation. The idea is to resolve them incrementally over time.

Add the harness option to include harness modules when the minimum requirements for compilation are met.

Add the feature option to indicate which features are now available in the harness (based on source modules already tested). This allows conditional compilation in harness modules when some features are not yet available.
2021-01-27 10:57:42 -05:00
David Steele
73a8e91868 Disable DEBUG_TEST_TRACE when NDEBUG defined.
The underlying functions used by the FUNCTION_TEST*() macros will not be able when NDEBUG is defined so disable them.

It would be better to change #ifndef NDEBUG to #ifdef DEBUG_TEST_TRACE where needed to bring these into alignment but that will be a larger effort best undertaken separately.
2021-01-27 10:29:23 -05:00
David Steele
088df03355 Fix indentation in .cirrus.yml. 2021-01-26 19:59:14 -05:00
David Steele
14f2b41bda Fix .editorconfig indent for yaml files. 2021-01-26 19:58:28 -05:00
David Steele
59e57c7231 Remove storageLocal() dependency from common/parse module.
storageLocal() requires the storage helper which is a huge dependency for this module.

Pass in a storage object instead.
2021-01-26 19:55:20 -05:00
David Steele
468507843f Replace storageLocal() with storageTest in postgres/interface test. 2021-01-26 19:34:39 -05:00
David Steele
77823af53d Add test in common/error module.
This is required for coverage when the common/error module is run with just the source files required to make it run, rather than all source files as we do now.

Likely something in the harness is providing coverage, but cover it explicitly so the coverage won't be lost if the harness changes.
2021-01-26 19:31:48 -05:00
David Steele
85ad1aa2be Remove version comment left over from Perl. 2021-01-26 12:47:52 -05:00
David Steele
456a300bb7 Remove too-verbose braces in switch statements.
The original intention was to enclose complex code in braces but somehow braces got propagated almost everywhere.

Document the standard for braces in switch statements and update the code to reflect the standard.
2021-01-26 12:10:24 -05:00
David Steele
f669da7dcc Use minio latest in documentation and integration tests.
At one time Minio had stability problems with latest but that appears to be resolved for the last year or so.

Use latest so we'll know if something breaks since Minio is frequently used in production.
2021-01-26 11:25:29 -05:00
Cynthia Shang
2e60b93709
Add backup verification to internal verify command.
This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered.

Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Cynthia Shang
5d48dd2fb3 Use explicit characters instead of Posix class in restore expression.
It is not clear how portable/supported the Posix character classes are. This way seems simpler and more portable.

Updated from 5c98157b.
2021-01-25 11:33:41 -05:00
Cynthia Shang
e251ec574a Add note about removing configuration to stanza-delete documentation. 2021-01-25 11:14:28 -05:00
Cynthia Shang
00fac1c0d1 Improve info command text output and --set handling.
The info command provides total sizes for files in the backup on the database as well as the repository. The text output and associated user documentation has been updated to provide more clarity regarding the sizes being displayed.

In addition, the info command is updated to allow a user to optionally specify the repository when requesting a specific backup set. In this case, the text output will reflect the status of the stanza, the cipher types and archive min/max over all the repositories instead of a single repository when the repo option is specified.
2021-01-25 09:19:05 -05:00
David Steele
8e48308b0b Add XML::Checker::Parser to MacOS Perl modules. 2021-01-24 18:30:14 -05:00
David Steele
ee71eb28c4 Add .editorconfig to help enforce project style. 2021-01-24 16:27:00 -05:00
David Steele
4e8d469f4d Use configure to generate Makefile variables for unit tests.
The unit test Makefile generation was a hodge-podge of constants and rules based on distros/versions that easily got out of date and did not work on an unknown system. All of this dates from the mixed Perl/C unit test implementation.

Instead use configure to generate most of the important Makefile variables, which allows the unit tests to run on multiple platforms, e.g. MacOS and FreeBSD.

There is plenty of work to be done here and not all the unit tests work on MacOS and FreeBSD for various reasons.

As a POC update the MacOS and FreeBSD tests on Cirrus-CI to run a few command unit tests.
2021-01-24 16:24:14 -05:00
David Steele
ef2dc6d3f4 Add chmod to make file removal after tests more reliable.
MacOS does not allow files to be removed recursively unless the owner has write and execute permissions on all the directories.

Some tests leave the permissions in a bad state so fix them up before trying to delete.
2021-01-24 15:48:32 -05:00
David Steele
04e84da0ef Allow the make command to be configured for test.pl. 2021-01-24 15:35:40 -05:00
David Steele
aafd2f528d Cast constants explicitly in command/backupCommon test.
Clang was not clear that these constants should be uint64_t.

Found on MacOS M1.
2021-01-24 15:29:10 -05:00
David Steele
d922cb6b8e Add wheel as possible group name in storage/posix test.
Some platforms use wheel as the group for symlinks instead of root.
2021-01-24 15:25:02 -05:00
David Steele
88e54ee6c7 More reliable log expect in common/log test.
The exact message is platform dependent so get the platform error to use in the expect.

It doesn't matter what the message is as long as there is an error and it is logged.
2021-01-24 15:22:29 -05:00