1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1916 Commits

Author SHA1 Message Date
David Steele
8e9f04cc32 Add HRN_INTEST_* define to indicate when a test is being run.
This is useful for initialization that needs to be done for the test and all subsequent tests.

Use the new defines to implement initialization for sockets and statistics.
2021-01-27 16:54:41 -05:00
Cynthia Shang
d5b919e657
Update expire command log messages with repo prefix.
In preparation for multi-repo support, a repo tag is added in this commit to the expire command log and error messages. This change also affects the expect logs and the user-guide. The format of the tag is "repoX:" where X is the repo key used in the configuration.

Until multi-repo support has been completed, this tag will always be "repo1:".
2021-01-27 16:33:01 -05:00
David Steele
5d34bf3f38 Move cvtDoubleToStr() to strNewDbl().
This is a more logical location and it reduces the dependencies required to compile the common/convert module.
2021-01-27 11:50:10 -05:00
David Steele
87eb081a8f Make unit test builds incremental based on coverage in prior tests.
When building tests only include files covered by the current test or by prior tests. This increases performance (less compilation and linking) and also helps detect cross-dependencies in the code. Since there are currently cross-dependencies the depend option is used to document them and allow compilation. The idea is to resolve them incrementally over time.

Add the harness option to include harness modules when the minimum requirements for compilation are met.

Add the feature option to indicate which features are now available in the harness (based on source modules already tested). This allows conditional compilation in harness modules when some features are not yet available.
2021-01-27 10:57:42 -05:00
David Steele
59e57c7231 Remove storageLocal() dependency from common/parse module.
storageLocal() requires the storage helper which is a huge dependency for this module.

Pass in a storage object instead.
2021-01-26 19:55:20 -05:00
David Steele
468507843f Replace storageLocal() with storageTest in postgres/interface test. 2021-01-26 19:34:39 -05:00
David Steele
77823af53d Add test in common/error module.
This is required for coverage when the common/error module is run with just the source files required to make it run, rather than all source files as we do now.

Likely something in the harness is providing coverage, but cover it explicitly so the coverage won't be lost if the harness changes.
2021-01-26 19:31:48 -05:00
David Steele
456a300bb7 Remove too-verbose braces in switch statements.
The original intention was to enclose complex code in braces but somehow braces got propagated almost everywhere.

Document the standard for braces in switch statements and update the code to reflect the standard.
2021-01-26 12:10:24 -05:00
David Steele
f669da7dcc Use minio latest in documentation and integration tests.
At one time Minio had stability problems with latest but that appears to be resolved for the last year or so.

Use latest so we'll know if something breaks since Minio is frequently used in production.
2021-01-26 11:25:29 -05:00
Cynthia Shang
2e60b93709
Add backup verification to internal verify command.
This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered.

Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Cynthia Shang
00fac1c0d1 Improve info command text output and --set handling.
The info command provides total sizes for files in the backup on the database as well as the repository. The text output and associated user documentation has been updated to provide more clarity regarding the sizes being displayed.

In addition, the info command is updated to allow a user to optionally specify the repository when requesting a specific backup set. In this case, the text output will reflect the status of the stanza, the cipher types and archive min/max over all the repositories instead of a single repository when the repo option is specified.
2021-01-25 09:19:05 -05:00
David Steele
8e48308b0b Add XML::Checker::Parser to MacOS Perl modules. 2021-01-24 18:30:14 -05:00
David Steele
4e8d469f4d Use configure to generate Makefile variables for unit tests.
The unit test Makefile generation was a hodge-podge of constants and rules based on distros/versions that easily got out of date and did not work on an unknown system. All of this dates from the mixed Perl/C unit test implementation.

Instead use configure to generate most of the important Makefile variables, which allows the unit tests to run on multiple platforms, e.g. MacOS and FreeBSD.

There is plenty of work to be done here and not all the unit tests work on MacOS and FreeBSD for various reasons.

As a POC update the MacOS and FreeBSD tests on Cirrus-CI to run a few command unit tests.
2021-01-24 16:24:14 -05:00
David Steele
ef2dc6d3f4 Add chmod to make file removal after tests more reliable.
MacOS does not allow files to be removed recursively unless the owner has write and execute permissions on all the directories.

Some tests leave the permissions in a bad state so fix them up before trying to delete.
2021-01-24 15:48:32 -05:00
David Steele
04e84da0ef Allow the make command to be configured for test.pl. 2021-01-24 15:35:40 -05:00
David Steele
aafd2f528d Cast constants explicitly in command/backupCommon test.
Clang was not clear that these constants should be uint64_t.

Found on MacOS M1.
2021-01-24 15:29:10 -05:00
David Steele
d922cb6b8e Add wheel as possible group name in storage/posix test.
Some platforms use wheel as the group for symlinks instead of root.
2021-01-24 15:25:02 -05:00
David Steele
88e54ee6c7 More reliable log expect in common/log test.
The exact message is platform dependent so get the platform error to use in the expect.

It doesn't matter what the message is as long as there is an error and it is logged.
2021-01-24 15:22:29 -05:00
David Steele
cbccae05b8 Skip lz4 in performance/storage test when it is not present. 2021-01-24 15:18:02 -05:00
David Steele
fda105ebd1 Add casts to performance/storage test for 32-bit architectures. 2021-01-24 15:15:50 -05:00
David Steele
d2057c53bd Use YAML::Any module instead of YAML::XS in Perl.
YAML::XS requires libyaml so it not as portable as pure Perl versions of YAML.

Instead of using YAML:PP just use the general YAML::Any module which uses whatever is installed. We are not concerned about performance for YAML so whatever works is fine.
2021-01-24 15:06:38 -05:00
David Steele
ea0dc8c2cb Add header required for test harness to compile on FreeBSD. 2021-01-24 13:39:13 -05:00
David Steele
4a83ca4f78 Remove semicolons from TEST_RESULT*() macros.
These prevented the macros from being used in single line statements (e.g. if-else).

Also fix some missing semicolons in macro invocations.
2021-01-24 08:55:13 -05:00
David Steele
5cb9f166ec Add stderr to unit test error messages.
Messages on stderr were being lost due to the error suppression used to customize the error message.

Also update the formatting to be more informative and concise.
2021-01-24 08:23:59 -05:00
David Steele
f95850c546 Fix logical -> bitwise boolean operator in backup unit test.
This unset more than the storageFeatureCompress flag but the test was not affected.

Found on MacOS M1.
2021-01-24 08:12:31 -05:00
David Steele
a7d32259cf Avoid NULL pointer arithmetic in MemContext unit test.
Similar to b23a2a0b. In this case add 1 to avoid a NULL pointer.

Found on MacOS M1.
2021-01-24 08:05:31 -05:00
David Steele
fe8ef3197e Add MacOS error messages to RegExp unit test. 2021-01-24 07:58:07 -05:00
David Steele
483695cac6 Don't pass --ignore-missing-args to rsync on MacOS.
MacOS has a very old version of rsync that does not support this option.

Rather than require a newer version of rsync exclude the option since the plan is to remove the requirement for it.
2021-01-22 13:16:48 -05:00
David Steele
708c3e9135 Move version check against release notes to release.pl.
This is a more appropriate place for the check and means test.pl can avoid loading any XML files if --no-gen is specified.

The XML::Checker::Parser module originally selected for XML in Perl is not very portable so the requirement reduces the number of platforms where tests can be run.
2021-01-22 12:52:30 -05:00
David Steele
b23a2a0baf Use uintptr_t in ASSERT_ALLOC_VALID() to avoid pointer arithmetic.
Clang justifiably complains about pointer arithmetic on a known NULL value during testing. We know this is fine but use uintptr_t to silence the warnings.

Found on MacOS M1.
2021-01-22 10:48:22 -05:00
David Steele
3f72eca909 Remove useless casts of vsnprintf() return value.
The return value is not checked because we are happy with a truncated result in this case, which is guaranteed by passing the buffer size.

Found on MacOS M1.
2021-01-21 17:26:12 -05:00
David Steele
6f529155b6 Remove useless assignment. 2021-01-21 17:17:03 -05:00
Cynthia Shang
f32eb9b94e
Partial multi-repository implementation.
Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands.

Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations.

Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
David Steele
d3d484b27c Use standard environment variable format in hrnCfgEnv*().
The option portion was not being capitalized or replacing - with _.

The parser does not care, but in cases where we have mixed hrnCfgEnv*()/setenv() calls the env variable might not get cleared, which can lead to funny test results.
2021-01-21 11:59:19 -05:00
David Steele
4e56948128 Compensate for numeric auto conversion in newer Perls. 2021-01-19 12:07:05 -05:00
David Steele
d9fae09848 Remove errant todo comment.
The test is not a duplicate of the one above because it adds the archive-async option.
2021-01-19 11:59:47 -05:00
David Steele
a1365b26d4 Remove duplicate mkdir for /tmp/pgbackrest in Vagrantfile. 2021-01-18 23:41:09 -05:00
David Steele
8322dfd7af Add pkg-config package to Vagrantfile.
Newer Ubuntu versions do not install this package by default.
2021-01-18 23:39:14 -05:00
David Steele
49582c13cf Use test lock path in archive-get test.
The default lock path should fail since the test VM gives ownership of /tmp to root.

For some reason this was not working as expected under u18 but it fails under u20.
2021-01-18 23:05:55 -05:00
David Steele
065b5f93ae Improve test coverage list handling.
All unit tests now require full coverage so the "full" keyword is obsolete and has been removed.

The covered code modules are simply listed, with only "no code" modules annotated.
2021-01-15 10:56:51 -05:00
David Steele
a8fb285756
Improve archive-get performance.
Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get.

Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process.

Cache results of storageList() when looking for multiple files to reduce storage I/O.

Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
David Steele
c2c702c09d Add co7 package to support llvm.
This is required for new package versions.

Also remove the obsolete 9.2 package and update the supported versions list.
2021-01-13 17:32:42 -05:00
David Steele
a57e0be395 Add duplicate WAL test to synchronous archive-get. 2021-01-13 13:01:40 -05:00
David Steele
22fd223fc3 Improve logging in archive-get command.
Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though).

Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful.

Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
David Steele
aae6f06bcf Add tests for the archive-get command.
Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
David Steele
aeee83044d
Fix resume after partial delete of backup by prior resume.
If files other than backup.manifest.copy were left in a backup path by a prior resume then the next resume would skip the backup rather than removing it. Since the backup path still existed, it would be found during backup label generation and cause an error if it appeared to be later than the new backup label. This occurred if the skipped backup was full.

The error was only likely on object stores such as S3 because of the order of file deletion. Posix file systems delete from the bottom up because directories containing files cannot be deleted. Object stores do not have directories so files are deleted in whatever order they are provided by the list command. However, the issue can be reproduced on a Posix file system by manually deleting backup.manifest.copy from a resumable backup path.

Fix the issue by removing the resumable backup if it has no manifest files. Also add a new warning message for this condition.

Note that this issue could be resolved by running expire or a new full backup.
2021-01-12 12:38:32 -05:00
David Steele
96fd678662
Add job-retry and job-retry-interval options.
These options specify the number of local worker job retries and the retry interval after one immediate retry.

There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests.

Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally.

These options are currently internal.
2021-01-11 15:15:25 -05:00
David Steele
f35d69c1c7 Refactor common/archiveGet unit test.
The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor.

Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
David Steele
8567b7f733 Make archive-get locality error generate a global.error file.
Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
David Steele
7d3d6ecbe1 Add storage test macros for common operations.
The macros should make it much easier to write complex tests, especially when compression and encryption are involved.

Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
David Steele
97c260bb56 Add TEST_RESULT_STRLST_*() unit test macros.
This avoids the need for strLstJoin() when testing lists.

Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
David Steele
1ef36b4032 Add clarifications to unit test string diff messages. 2021-01-08 10:54:17 -05:00
David Steele
17e29eb1bd Improve formatting of unit test titles.
Add separation and some visual cues to help identify the start of a test.

Also add a counter which can be used to search for a specific test, which is useful if there is a lot of debug output to search through.
2021-01-08 10:45:26 -05:00
David Steele
dc0284412b Add test paths for pg, repo, and spool.
These may be used as standard locations for these paths in tests.
2021-01-08 10:35:26 -05:00
David Steele
6bdbcdc7fe Create TEST_PATH define in test.c.
This allows paths to be constructed without using sprintf(), which makes the tests simpler and faster.
2021-01-08 10:32:57 -05:00
David Steele
ca9c1707f0 Remove forks from command/archive-get test.
These were required to deal with the legacy Perl code being unable to load new options between tests.

The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
David Steele
6e7a3eb383 Remove archive-timeout from test in mock/archive.
No timeout is expected here but the small timeout prevents errors from being thrown.

This is not a bug since the error would be thrown on the next archive-get call but it does make the tests harder to debug when there is an error.

It is not clear why there was a timeout here at all. It is likely cruft from a prior test or a copy/paste error.
2021-01-05 18:11:28 -05:00
Cynthia Shang
656e711296
Remove duplicate tests from command/infoTest.
Tests that are duplicated are being removed from the info command unit tests. Specifically tests where the only thing different was whether a lock was held or not which affects only the status display. Removing these tests will reduce churn in the upcoming multi-repo support.
2021-01-05 15:14:16 -05:00
David Steele
af17bc4b6d Update test to work with different collations.
The data returned by the protocol has not been sorted yet so it is vulnerable to differences in collation.

Multiple records are not needed for this test so limit it to one path to solve this issue.
2020-12-31 15:34:07 -05:00
David Steele
108038292c Audit options valid for expire command. 2020-12-31 12:13:20 -05:00
David Steele
0acfcb669e Audit options valid for start/stop commands. 2020-12-31 11:10:48 -05:00
David Steele
09fdde359c Limit pg option validity and make it command-line only.
The pg option only has one current usage, to let the backup local know which pg index it should copy files from.

There are other possible uses for this option, but they need thought, tests, and documentation.
2020-12-31 10:08:58 -05:00
David Steele
951cfa9e90 Remove repo option.
This option was added in advance of the multi-repo functionality but it has no purpose and it is not clear what the validity rules should be.

The option will be added back when multi-repo functionality is committed.
2020-12-31 08:12:35 -05:00
Cynthia Shang
cc90163233
Add empty archive array to info command JSON when stanza is missing.
There is an inconsistency when the JSON is output for the case when a stanza is requested and it does not exist in the repo. This was the only case where the archive array was not added to the JSON. Adding it will simplify the upcoming multi-repo support code.

Also, a redundant test was removed rather than updating it for this case.
2020-12-30 16:17:56 -05:00
David Steele
9bf7dbf6a2 Do not pass pg-local/repo-local to a remote process.
This was a hack to prevent the remote from loading host settings, which is now handled by option validity for command roles.

These options are still useful so don't remove them, but do leave them internal for now.
2020-12-30 16:03:49 -05:00
David Steele
abb8ebe58b
Limit option validity by command role.
Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here.

The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.
2020-12-29 15:49:37 -05:00
David Steele
26e81785d9 Update unit tests for stricter option validity.
Some tests used options in contexts that are currently valid but are not correct usage, i.e. usage of internal options for the default role.

Update these tests in advance of the option validity becoming stricter.
2020-12-28 15:22:21 -05:00
David Steele
23f5712d02
Allow option validity to be determined by command role.
Validity by command was not granular enough so numerous options needed be marked internal so users would not stumble across them. Options were also needlessly being passed to roles that had no use for them.

Introduce per-role validity lists that depend on what roles are valid per command. Also add a check to ensure that only valid roles are used with a command.

This commit adds the functionality but does not introduce any new behavior, i.e. all options are valid for all roles that the command is valid for. A subsequent commit will introduce the new role restrictions to make the changes easier to audit.
2020-12-28 09:43:23 -05:00
David Steele
9e9e7c4a0d Move all parse-related rules to parse module.
Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl.

Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement.

Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
David Steele
f520ecc89a Move help data from define.auto.c/config.auto.c to a pack.
The help data can be represented more compactly in a pack and this separates data needed for help from data needed for parsing, freeing each to have a more appropriate representation.
2020-12-16 15:59:36 -05:00
David Steele
558ff1e555 Always advance id when pack field is NULL.
This was done in the internal versions but not the user-facing function. That meant the field had to be explicitly read after determining it was NULL, which is wasteful.

Since there is only one behavior now, remove pckReadDefaultNull() and move the logic to pckReadNullInternal().
2020-12-16 09:59:48 -05:00
David Steele
39963f6aa5 Remove cfgDefOptionIndexTotal().
This function was only used in one place, which was better served by cfgOptionGroupIdxTotal().
2020-12-14 14:37:23 -05:00
David Steele
7f66ba5762 Remove deprecated option from command/archive-push unit test. 2020-12-10 17:58:59 -05:00
David Steele
d01669aa58 Move most tests to Github Actions.
Testing on Travis-CI has been getting slower (from ~18 minutes to 3-6 hours) and the travis-ci.org service will be terminated at the end of the year. Moving to travis-ci.com is an option but the quotas are too low for our purposes.

Instead use Github Actions, which does not currently have quotas, and runs our current tests with just a few tweaks.

This still leaves multi-architecture tests on Travis-CI but we may be able to run those and stay within the new quotas.

Also fix a minor bug in restoreTest.c exposed by Github Actions using a different name for the user and group.
2020-12-09 15:19:01 -05:00
David Steele
8361a97482
Add pack type.
The pack type is an architecture-independent format for serializing data compactly, inspired by ProtocolBuffers and Avro.

Also add ioReadSmall(), which is optimized for small binary reads, similar to ioReadLineParam().
2020-12-09 12:05:14 -05:00
David Steele
87996558d2
Replace double type with time in config module.
The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand.

Also remove Variant doubles since they are no longer used.

Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
David Steele
e116b535e6 v2.31: Minor Bug Fixes and Improvements
Bug Fixes:

* Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)
* Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)

Features:

* Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
* Add repo-azure-endpoint option. (Reviewed by Cynthia Shang, Brian Peterson. Suggested by Brian Peterson.)
* Add pg-database option. (Reviewed by Cynthia Shang.)

Improvements:

* Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)
* Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang, Oscar.)
* Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)

Documentation Improvements:

* Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.
2020-12-07 09:55:00 -05:00
David Steele
31becf05b7 Add RHEL/CentOS 8 documentation.
Update RHEL/CentOS 7 to cover the versions that were previously covered by RHEL/CentOS 6.

Since RHEL/CentOS 7/8 work the same update the documentation logic and labels to reflect this compatibility.
2020-12-04 10:59:57 -05:00
David Steele
b0ea337965 Add pg-database option.
In some rare cases there is no postgres database so this option may be used to specify an alternate database.
2020-12-02 22:42:50 -05:00
David Steele
1913ecee63 Fix incorrect usage of TEST_RESULT_DOUBLE().
These work fine but it's better to use the correct macro.
2020-12-02 22:42:05 -05:00
David Steele
d4211d3aaf Add retries to PostgreSQL sleep when starting a backup.
Inaccuracies in sleep time or clock skew might make a single sleep insufficient to reach the next second.

Add a few retries to make the process more reliable but still avoid an infinite loop if something is seriously wrong.
2020-12-02 22:41:14 -05:00
David Steele
ec9f23d31f Remove CentOS 6 from tests and documentation.
CentOS6 EOL'd and the mirrors were swiftly deleted, leading to failures in tests and documentation.

Remove CentOS 6 for now to get builds going again with the intention to replace it in the near future with CentOS 8.
2020-12-02 16:23:05 -05:00
David Steele
ffc50719d9 Improve error message when PQgetCancel() returns NULL.
There is not a lot to be done in this case since it looks like PostgreSQL disconnected while the query was running, but at least improve the error message and remove the assert, which indicates a coding error.
2020-12-01 15:15:35 -05:00
Stefan Fercot
5488de8b6a
Report page checksum errors in info command text output.
This feature currently only works for text output. JSON output is planned for the future.
2020-11-25 12:14:03 -05:00
David Steele
117f03eba1 Prepare configuration module for multi-repository support.
Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum).

Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage.

Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption.

Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy.

Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn.

Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related.

Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
David Steele
7fda83b31e
Allow multiple remote locks from the same main process.
Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed.

The exec-id is also useful for associating remote logs with main logs for debugging purposes.
2020-11-23 12:41:54 -05:00
David Steele
77198dce97 Update Vagrant box version. 2020-11-22 16:26:30 -05:00
Stefan Fercot
191b8ec18b
Create standby.signal only on PostgreSQL 12 when restore type is standby.
When restore type standby is provided, the recovery.signal isn't needed and may lead to some confusion (see #1236).

Lately, when using pg_basebackup --write-recovery-conf, only the standby.signal file is created. This change would then align with that behaviour.
2020-11-19 16:57:19 -05:00
David Steele
62d9f23961 Rename hrnCfgEnvId*() to hrnCfgEnvKey*().
The hrnCfg*() functions were renamed before commit but these got missed.
2020-11-17 10:48:40 -05:00
David Steele
dd708e6d27 Fix override of user option reset.
If a user reset an option such as pg-default on the command-line then an override in the code would not take effect.

Ignore a reset when the code explicitly sets an option to prevent this.
2020-11-09 16:37:05 -05:00
David Steele
6acd76b60c Fix test title in storage/remote test. 2020-11-09 16:28:53 -05:00
David Steele
63c12d2541 Fix spacing and typos in backup/manifest modules. 2020-11-09 16:26:43 -05:00
David Steele
8cf47f82e4 Log warnings in archive-push async log.
These warnings were only being reported to PostgreSQL on the console. Now they are also recorded in the async log increasing the chance that they will be seen.

This also improves coverage by requiring a warning during async processing to have a test case, which has been added.
2020-11-09 16:10:59 -05:00
David Steele
d25e965c47 Use harnessCfgLoad() in config/load unit tests.
The tests were originally written by loading values directly into the configuration before the parser was available.

Update to use harnessCfgLoad() to simplify the tests and make them compatible with upcoming config changes.
2020-10-27 13:29:58 -04:00
Stefan Fercot
abe9d90c89
Improve info command output when a stanza is specified but missing.
Return a path missing error when a stanza is specified for the info command but the stanza does not exist in the repository.

Previously [] was returned, which is still the case if no stanza is specified and the repository does not exist.
2020-10-27 08:34:18 -04:00
David Steele
770b65de80
Improve performance of large file lists in backup/restore commands.
lstRemoveIdx(list, 0) resulted in the entire list being moved down to the first position which could take a long time for big lists. This is a common pattern in backup/restore when processing file queues.

Instead simply move the list pointer up when first item is removed. Then on insert check if there is space at the beginning when there is no longer space at the end and do the move then. This way if a list is built and then drained without any new inserts then no move is required.
2020-10-26 12:18:45 -04:00
David Steele
d452e9cc38
Use zero-based indexes when referring to option indexes.
There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa.

Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user.

Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
David Steele
80a770ed2e Clarify intended usage of harnessCfgLoadRaw(). 2020-10-24 11:16:06 -04:00
David Steele
76cfd8ca70
Allow [, #, and space as the first character in database names.
iniLoad() was trimming lines which meant that a leading space would not pass checksum validation when a manifest was reloaded. Remove the trims since files we write should never contain extraneous spaces. This further diverges the format for the functions that read conf files (e.g. pgbackrest.conf) and those that read info (e.g. manifest) files.

While we are at it also allow [ and # as initial characters. # was reserved for comments but we never put comments into info files. [ denotes a section but we can get around this by never allowing arrays as values in info files, so if a line ends in ] it must be a section. This is currently the case but enforce it by adding an assert to info/info.c.
2020-10-24 11:07:07 -04:00
David Steele
c573ef2814 Add indexed option > 1 to config/exec unit test. 2020-10-20 16:14:03 -04:00
David Steele
176cf0bf60 Use harnessCfgLoadRaw() in command/command and common/exit unit tests.
The tests were originally written by loading values directly into the configuration before the parser was available.

Update to use harnessCfgLoadRaw() to simplify the tests and make them compatible with upcoming config changes.

Note that some unreachable conditions were removed since they could not be reached via a parsed config, only by munging values directly into the config. cfgOptionTest(optionId) was removed because a non-default value must always be set. cfgOptionValid(cfgOptLogTimestamp) was removed because it is true for all commands except for cfgCmdNone, which is checked with an assert.
2020-10-20 14:54:28 -04:00
David Steele
41789d70d1 Remove cfgOptionId() and replace it with cfgParseOption().
cfgOptionId() did not recognize deprecated options which made the help command throw errors when they were specified on the command line. cfgParseOption() will correctly identify deprecated options.

cfgParseOption() can also be used in cfgParse() to reduce code duplication when parsing info out of the option value returned by optionFind().

Finally, code the option key index separately in parse.auto.c. For now they are simply added back together but future code will need them separated.
2020-10-20 11:24:26 -04:00
David Steele
6414ae9707 Remove ConfigDefineCommand enum.
This has always been equivalent to the ConfigCommand enum so it just adds complexity.

It was created for symmetry with ConfigDefineOption, which will also be removed soon.
2020-10-19 18:17:47 -04:00
David Steele
ba29e24684 Fix repo/pg-local option being overwritten.
This option would get overwritten when the option values were moved to index 1.

Set the repo/pg-local option after options are moved to fix this.
2020-10-19 17:17:47 -04:00
David Steele
2d38d2fc82 Reset additional options in real/all integration test.
Currently indexes above 1 do not have dependencies checked, so this doesn't error.

In a future commit we will enable those checks and this will error if it is not fixed.
2020-10-19 17:06:52 -04:00
David Steele
996387ea93 Use harnessCfgLoadRaw() in command/help unit test.
This correctly set defaults from cfgLoadUpdateOption(), e.g. repo-host-cmd.
2020-10-19 16:27:52 -04:00
David Steele
303b9b17e7 Remove extraneous calls to cfgOptionValid() in unit tests. 2020-10-19 14:19:28 -04:00
David Steele
7d069a2b91 Remove indexed option constants.
These constants don't scale well as the index total is increased for an option.

The core code rarely uses these options and they are easily replaced with cfgOptionName().

The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
Stefan Fercot
86275c4f85
Expire history files.
WAL timeline history files were not being expired because they were small and generally not very plentiful.

However, in some cases large numbers of history files may be generated so it makes sense to remove useless history files to keep things tidy.

The history file for the oldest retained timeline is kept for debugging purposes even though it is not used for recovery.
2020-10-16 07:42:03 -04:00
David Steele
4b8dfd34c0 Assert when buffer used is greater than size limit.
099281c5 reduced used but this could lead to surprising behavior so assert instead.

Call bufUsedSet() before bufLimitSet() to avoid the assertion.
2020-10-08 17:09:21 -04:00
David Steele
e0f09687e4
Add option groups.
Group related options together so operations (e.g. valid, test, index total) can be performed on all options in the group.

Previously, options at the top of the hierarchy of the related options were used to do these tests. This was prone to error as option relationships changed and it was not always clear which option (or options) should be used.
2020-10-08 10:52:19 -04:00
David Steele
099281c510 Reduce buffer used when size limit set.
If the size is reduced then the amount used should be reduced as well.

This is a bug but does not affect current use cases.
2020-10-07 18:02:05 -04:00
David Steele
39aca3269c Add bufSizeAlloc(). 2020-10-07 16:34:13 -04:00
David Steele
98f6a1cffd Remove unused cfgDefOptionPrefix() function and data.
This function was made obsolete when Perl was removed in f0ef73db.
2020-10-07 12:15:50 -04:00
David Steele
9377d05072
Add repo-azure-endpoint option.
This option allows alternate endpoints (e.g. Azure Government) to be configured.
2020-10-06 17:15:48 -04:00
David Steele
9229d6c866 v2.30: PostgreSQL 13 Support
Bug Fixes:

* Error with hints when backup user cannot read pg_settings. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Mohamed Insaf K.)

Features:

* PostgreSQL 13 support. (Reviewed by Cynthia Shang.)

Improvements:

* Improve PostgreSQL version identification. (Reviewed by Cynthia Shang, Stephen Frost.)
* Improve working directory error message. (Reviewed by Stefan Fercot.)
* Add hint about starting the stanza when WAL segment not found. (Contributed by David Christensen. Reviewed by David Steele.)
* Add hint for protocol version mismatch. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)

Documentation Improvements:

* Add note that pgBackRest versions must match when running remotely. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)
* Move info command text to the reference and link to user guide. (Reviewed by Cynthia Shang. Suggested by Christophe Courtois.)
* Update yum repository path for CentOS/RHEL user guide. (Contributed by Heath Lord. Reviewed by David Steele.)
2020-10-05 11:51:45 -04:00
David Steele
31962377d9
Add hint and documentation for protocol version mismatches.
Update the documentation to explicitly state that versions must match across hosts when running remotely.

Add a hint to the protocol version mismatch error to help the user identify the problem.
2020-09-25 10:30:29 -04:00
David Steele
b096a25b49 Update test containers for PostgreSQL 13.
Add older PostgreSQL versions to the u18 container that were not available before.

This also updates all minor versions for prior versions of PostgreSQL.
2020-09-24 11:19:51 -04:00
Cynthia Shang
ad79932ba5
Add internal verify command.
Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet).

The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
David Steele
927d9adbee
Improve PostgreSQL version identification.
Previously, catalog versions were fixed for all versions which made maintaining the catalog versions during PostgreSQL beta and release candidate cycles very painful. A version of pgBackRest which was functionally compatible was rendered useless by a catalog version bump in PostgreSQL.

Instead use only the control version to identify a PostgreSQL version when possible. Some older versions require a catalog version to positively identify a PostgreSQL version, so include them when required.

Since the catalog number is required to work with tablespaces it will need to be stored. There's already a copy of it in backup.info so use that (even though we have been ignoring it in the C versions).
2020-09-18 16:55:26 -04:00
David Steele
4cd61152f5 Update PostgreSQL 13 test catalog versions missed in 6bb111c1.
These values are not used by the Perl integration tests so maybe it would be better to remove them, but for now just update since they should not be changing again for PG13.
2020-09-17 12:39:30 -04:00
David Steele
9db3143973
Error with hints when backup user cannot read pg_settings.
This condition used to give a not-very-clear error which we have been intending to improve. But in the meantime the changes in fbff299 resulted in a segfault for this condition instead because the data_directory was assumed to be non-NULL.

Fix this by explicitly throwing an error with hints when any row in pg_settings cannot be selected.
2020-09-17 10:35:27 -04:00
David Steele
c71609879b Fix incorrect control/catalog versions in command/stanza test.
These were not caught because backup info load automatically corrects them.
2020-09-16 09:33:37 -04:00
David Steele
8dce7bbb60
Ignore backup_manifest in PG13.
This file is created by pg_basebackup so might be in the data directory if the cluster was restored from a pg_basebackup backup. Also exclude backup_manifest.tmp since it is possible to find that in the backup directory.
2020-09-14 10:15:40 -04:00
David Steele
fc77c51182
Improve working directory error message.
Improve the wording of the error message and add a hint to make it clearer what is wrong and how the user can fix it.

Also change the assert to a regular error since this is not an internal error.
2020-09-11 10:10:25 -04:00
David Steele
dac182c06e Improve alignment in TEST_ERROR output.
Align the error types and messages so they are easier to compare.
2020-09-11 08:23:26 -04:00
Cynthia Shang
ecb112f802 Add infoArchiveMove() and infoBackupMove(). 2020-09-08 13:02:13 -04:00
Cynthia Shang
4239f8a81a Add manifestFree(). 2020-09-08 12:47:35 -04:00
Cynthia Shang
b8efb13bcb Move archiveIdComparator() to archive/common module. 2020-09-08 12:28:56 -04:00
David Christensen
9fd31913a8 Add hint about starting the stanza when WAL segment not found.
If a stop command has been issued the check command fails due to archiving timing out.

Provide a hint to document this situation and point the user in the proper direction.
2020-09-03 07:49:49 -04:00
David Steele
41118e1b68
Allow ProtocolParallel to complete with no jobs.
If the callback never returned any jobs then protocolParallelDone() would never be true. The reason is that the done state was being set in protocolParallelResult(), which never gets called if there are no results.

Calling protocolParallelResult() doesn't make much sense in this case so instead move the done logic to protocolParallelDone().

For current usage of ProtocolParallel we ensure there are jobs before processing so this is not a live issue, but the new behavior is required for future development.
2020-09-02 11:03:49 -04:00
David Steele
1df7d3eee5 v2.29: Auto S3 Credentials on AWS
Bug Fixes:

* Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. (Reviewed by Cynthia Shang. Reported by argdenis.)
* Fix issue with = character in file or database names. (Reviewed by Bastian Wegge, Cynthia Shang. Reported by Brad Nicholson, Bastian Wegge.)

Features:

* Automatically retrieve temporary S3 credentials on AWS instances. (Contributed by David Steele, Stephen Frost. Reviewed by Cynthia Shang, David Youatt, Aleš Zelený, Jeanette Bromage.)
* Add archive-mode option to disable archiving on restore. (Reviewed by Stephen Frost. Suggested by Stephen Frost.)

Improvements:

* PostgreSQL 13 beta3 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
* Asynchronous list/remove for S3/Azure storage. (Reviewed by Cynthia Shang, Stephen Frost.)
* Improve memory usage of unlogged relation detection in manifest build. (Reviewed by Cynthia Shang, Stephen Frost, Brad Nicholson, Oscar. Suggested by Oscar, Brad Nicholson.)
* Proactively close file descriptors after forking async process. (Reviewed by Stephen Frost, Cynthia Shang.)
* Delay backup remote connection close until after archive check. (Contributed by Floris van Nee. Reviewed by David Steele.)
* Improve detailed error output. (Reviewed by Cynthia Shang.)
* Improve TLS error reporting. (Reviewed by Cynthia Shang, Stephen Frost.)

Documentation Bug Fixes:

* Add none to compress-type option reference and fix example. (Reported by Ugo Bellavance, Don Seiler.)
* Add missing azure type in repo-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)
* Fix typo in repo-cipher-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)

Documentation Improvements:

* Clarify that expire must be run regularly when expire-auto is disabled. (Reviewed by Douglas J Hunley. Suggested by Douglas J Hunley.)
2020-08-31 07:28:22 -04:00
David Steele
8c2960fab3
Add archive-mode option to disable archiving on restore.
When restoring a cluster that will be promoted but is not intended to be the new primary, it is important to disable archiving to avoid polluting the repository with useless WAL. This option makes disabling archiving a bit easier.
2020-08-25 15:05:41 -04:00
David Steele
851f2e814e
Automatically retrieve temporary S3 credentials on AWS instances.
Automatically retrieve the role and temporary credentials for S3 when the AWS instance is associated with an IAM role. Credentials are automatically updated when they are <= 5 minutes from expiring.

Basic configuration is to set repo1-s3-key-type=auto. repo1-s3-role can be used to set a specific role, otherwise it will be retrieved automatically.
2020-08-25 10:38:49 -04:00
David Steele
1d2b8ae7bb
Improve detailed error output.
Add more info (command, version, options) to asserts, and errors when debug logging is enabled. This won't cover all cases but might mean we get more info in some circumstances.
2020-08-24 14:51:08 -04:00
David Steele
7fba1f0bbe Consolidate variables into structs in common/stackTrace module.
This makes it easier to determine which variables are used in which parts of the module and reduces the chance of name conflicts while unit testing.
2020-08-21 16:12:44 -04:00
David Steele
0e8e5881d8 Remove invalid call to stackTraceTestFileLineSet().
When testing the common/stack-trace module it is important not to call this test function since the trace stack is empty and it will cause a buffer under run.

Instead use a macro that is only defined under the correct circumstances and add an assert() to catch future regressions.
2020-08-21 11:43:26 -04:00
David Steele
25ee913932 Rearrange members in StackTraceData struct to save memory.
This arrangement is more efficient on 64-bit systems due to alignment. There is no difference on 32-bit systems.
2020-08-20 15:53:01 -04:00
David Steele
859b8a50fd Remove unused parameter from cmdBegin(). 2020-08-20 14:16:36 -04:00
David Steele
fccca0d716 Refactor option logging into a general-purpose function. 2020-08-20 14:11:40 -04:00
David Steele
959f77cd6a
Add general-purpose statistics collector.
Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome.

Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually.

No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible.

The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
David Steele
de0f8c2654
Add user-agent to HTTP requests. 2020-08-18 10:01:24 -04:00
David Steele
d935af5e59 Update *((Type *)lstGet()) pattern to *(Type *)lstGet().
This pattern has extra parens and was only used in a few places so replace it with the general *(Type *)lstGet() pattern.
2020-08-18 07:26:02 -04:00
David Steele
fbee6ec170
Add support for HTTP/1.0.
HTTP/1.0 connections are closed by default after a single response. Other than that, treat 1.0 the same as 1.1.

HTTP/1.0 allows different date formats that we can't parse but for now, at least, we don't need any date headers from 1.0 requests.
2020-08-14 13:11:33 -04:00
David Steele
6bb111c170 PostgreSQL 13 beta3 support. 2020-08-14 10:08:25 -04:00
David Steele
65858d12dd Improve server test harness to support multiple protocols/connections.
The prior implementation only supported a single connection on TLS. This is not flexible enough for complex testing scenarios which might require multiple simultaneous connections on different protocols.

Allow multiple simultaneous connections and add plain sockets as a protocol option. Rename the functions used for server scripting to hrnServerScript*() to make it clear they are related. Improve error messages when less input is received by the server than expected.

Also, do a bit of cleanup and add more comments.
2020-08-10 16:16:11 -04:00
David Steele
7fdbd94e39
Implement IoClient/IoSession interfaces for SocketClient/SocketSession.
Following up on 111d33c, implement the new interfaces for socket client/session. Now HTTP objects can be used over TLS or plain sockets.

This required adding ioSessionFd() and ioSessionRole() to provide the functionality of sckSessionFd() and sckSessionType(). sckClientHost() and sckClientPort don't make sense in a generic interface so they were replaced with ioSessionName().
2020-08-10 16:03:38 -04:00
David Steele
8b34f854f3 Simplify S3 configuration tests and add security token tests.
Rather than calling storageS3New() directly, create the storage by loading a configuration and calling repoStorageGet(). This is a better end-to-end test and cuts down on a lot of redundant tests.

Add tests that include security tokens in error messages to ensure they are redacted.
2020-08-08 15:52:33 -04:00
David Steele
4d22d6eeca
Move file descriptor read/write ready into IoRead/IoWrite.
Move sckSessionReadyRead()/Write() into the IoRead/IoWrite interfaces. This is a more logical place for them and the alternative would be to add them to the IoSession interface, which does not seem like a good idea.

This is mostly a refactor, but a big change is the select() logic in fdRead.c has been replaced by ioReadReady(). This was duplicated code that was being used by our protocol but not TLS. Since we have not had any problems with requiring poll() in the field this seems like a good time to remove our dependence on select().

Also, IoFdWrite now requires a timeout so update where required, mostly in the tests.
2020-08-08 11:23:37 -04:00
David Steele
111d33c123
Add IoClient and IoSession interfaces.
These interfaces allow the HttpClient and HttpSession objects to work with protocols other than TLS, .e.g. plain sockets. This is necessary to allow standard HTTP -- right now only HTTPS is allowed, i.e. HTTP over TLS.

For now only TlsClient and TlsSession have been converted to the new interfaces. SocketClient and SocketSession will also need to be converted but first sckSessionReadyRead() and sckSessionReadyWrite() need to be moved into the IoRead and IoWrite interfaces, since they are not a good fit for IoSession.
2020-08-08 10:39:39 -04:00
David Steele
9b7fd1a894 Improve error message in HARNESS_FORK_END().
Also include expected exit status to make debugging easier.
2020-08-08 10:02:04 -04:00
David Steele
cde2c756ea Rename handle to fd.
Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd.

There were also plenty of places fd was used so now all uses are consistent.

The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
David Steele
a1d7d50223 Fix macro spacing. 2020-08-05 17:30:38 -04:00
David Steele
d5edc3fe37 Remove unused port struct member.
This does not appear to have been used in quite some time and the tests are equally useless because they don't prove the correct port was passed to httpClientNew().
2020-08-05 15:24:13 -04:00
David Steele
d74c8f4b58 Add memContextFreeing().
This can be used to determine if functions are running in a callback from memContextFree(). In this case, data in child mem contexts is not valid.
2020-08-05 14:07:04 -04:00
David Steele
847e61ce21
Improve TLS error reporting.
Before 9f2d647 TLS errors included additional details in at least some cases. After 9f2d647 a connection to an HTTP server threw `TLS error [1]` instead of `unable to negotiate TLS connection: [336031996] unknown protocol`.

Bring back the detailed messages to make debugging TLS errors easier. Since the error routine is now generic the `unable to negotiate TLS connection context` is not available so the error looks like `TLS error [1:336031996] unknown protocol`.
2020-08-04 15:15:24 -04:00
David Steele
e81533bbab
Improve memory usage of unlogged relation detection in manifest build.
This loop was using a lot of memory without freeing it at intervals.

Rewrite to use char arrays when possible to reduce memory that needs to be allocated and freed.
2020-08-04 10:16:51 -04:00
David Steele
5a9856c2f9 Add functions for Zigzag encoding/decoding.
Zigzag encoding places the sign bit in the least significant bit so that -1 is encoded as 1, 1 as 2, etc. This moves as many bits as possible into the low order bits which is good for other types of encoding, e.g. base-128.

See https://en.wikipedia.org/wiki/Variable-length_quantity#Zigzag_encoding.
2020-08-01 09:42:03 -04:00
David Steele
caef2ba482 Accept NULL filenames in hrnStorageInfoListCallback().
This allows it to be used with StorageInfo objects returned from storageInfo(), which do not contain a filename.
2020-07-31 16:35:58 -04:00
David Steele
a8e47c38c6 Add lstGetLast() and lstRemoveLast().
When a list is being treated as a stack it is useful to get/remove the last (top) item.
2020-07-31 16:27:57 -04:00
David Steele
bfb489a82d Add file name to make performance/storage test more realistic.
Also add timing information.
2020-07-31 16:18:56 -04:00
David Steele
9f1e7d88da Remove special protocol encoding for StorageType.
It seems like overkill to encode this when other enums (e.g. StorageInfoLevel) are passed as integers.

Instead note that StorageType values should not be changed and remove the special encoding.
2020-07-31 16:13:05 -04:00
David Steele
ac72e1f193
Preserve Variant parsed from JSON in iniLoad().
The fix for = characters in info files (039d314) added JSON validation but discarded the resulting Variant which means the JSON is being parsed twice. This nearly doubles the time to load a manifest since a lot of complex JSON is involved.

Time to load a million file manifest:
Before 039d314: 7.8s
039d314: 15.5s
This patch: 7.5s

To fix this regression return the Variant in the callback so the caller does not have to parse it again. The new code appears slightly more efficient overall, probably because there are fewer operations against Strings.
2020-07-30 10:59:50 -04:00
David Steele
3e9dce0d76 Rename strPtr()/strPtrNull() to strZ()/strZNull().
We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern.

As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
David Steele
039d314438
Fix issue with = character in file or database names.
The manifest uses the = character as the key/value separator so = characters in the key cause parsing errors and lead to an error or segfault.

Since the value must be valid JSON we can keep checking the value on the right side of the = and stop building the key when the value is valid. It's a bit hackish but it does seem to do the job without breaking the manifest format.

Unsurprisingly this makes parsing about 50% slower but it's still more than fast enough. Parsing 10 million key/values takes about 6.5s for the old code and 10s for the new code. Since the value is used as JSON downstream we can reclaim most of this time by just passing the JSON value rather than making the callback reparse it. We'll save that for another commit, though.
2020-07-28 14:00:23 -04:00
David Steele
63a93db6fd
Suppress errors when closing local/remote processes.
Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened.

Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
David Steele
a260d4a53b Add zstd to CentOS/RHEL 6 test container.
Zstd is now required by the upstream yum package.
2020-07-28 08:09:10 -04:00
David Steele
8e776cc537 Add manifestNewBuild() to performance/type test.
Use a test storage driver to allow manifestNewBuild() to be run against a test cluster at any scale without having to write files to disk.

Simplify the test by using the output of manifestNewBuild() to feed manifestSave() and manifestNewLoad().

Also add manifest size to the output.
2020-07-25 11:26:14 -04:00
David Steele
f9d923ca3b Add memContextSize().
Calculates the memory used by the context and all child contexts.

This is primarily useful for debugging but it is not conditional on DEBUG because it is useful for profile/performance tests.
2020-07-25 11:06:25 -04:00
David Steele
216a61d936 Move dummy storage driver to test harness.
The dummy driver is the basis for creating test storage drivers so it makes sense to locate it in the harness where all tests can access it.
2020-07-25 08:44:41 -04:00
David Steele
cbf5dd19c2 Fix non-JSON ini values in tests.
A number of tests used invalid JSON values where an error was expected or the value would be ignored.

Update these tests to use valid JSON values so all values in the file can be validated even if they are not used.
2020-07-24 11:05:09 -04:00
David Steele
9d8f6d9fc2 Error in jsonToVar() when input not entirely consumed.
Something like 3="string" would return an Int64 variant and ignore the invalid portion after the integer. Other JSON interface functions have this check but it was forgotten here.

There are no current issues because of this but we want to be able to validate arbitrary JSON strings and this function was not working correctly for that usage.
2020-07-24 10:47:51 -04:00
Cynthia Shang
78ef442a18
Add storage parameter to pgWalFromFile(). 2020-07-21 16:28:05 -04:00
David Steele
90ffde8881 Remove unused constructor strLstNewParam().
This function is not used in the core code so remove it and update the test where it was used.

There may eventually be a need for a strLstNewP() function but it doesn't seem worth the code churn until there is an actual requirement.
2020-07-20 18:09:32 -04:00
David Steele
1783e0490a Remove lstNew() constructor in favor of lstNewP().
The old constructor was left around to reduce code churn during the migration but it just makes the code harder to read and search.

Remove the old constructor and rename all remaining instances to lstNewP(), which by default has the same semantics.
2020-07-20 15:22:33 -04:00
David Steele
3f0b41eb9c Add support for testing on 64-bit big-endian architectures.
In particular add support for s390x but we hope this will work for other 64-bit big-endian architectures.

Run basic unit tests on Travis CI for 390x.
2020-07-20 09:59:16 -04:00
David Steele
55277357b8 Reduce reliance on static checksums in unit tests.
Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures.

Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
David Steele
5a4b91f90a v2.28: Azure Repository Storage
Bug Fixes:

* Fix restore --force acting like --force --delta. This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue. (Reviewed by Cynthia Shang.)

Features:

* Azure support for repository storage. (Reviewed by Cynthia Shang, Don Seiler.)
* Add expire-auto option. This allows automatic expiration after a successful backup to be disabled. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)

Improvements:

* Asynchronous S3 multipart upload. (Reviewed by Stephen Frost.)
* Automatic retry for backup, restore, archive-get, and archive-push. (Reviewed by Cynthia Shang.)
* Disable query parallelism in PostgreSQL sessions used for backup control. (Reviewed by Stefan Fercot.)
* PostgreSQL 13 beta2 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
* Improve handling of invalid HTTP response status. (Reviewed by Cynthia Shang.)
* Improve error when pg1-path option missing for archive-get command. (Reviewed by Cynthia Shang.)
* Add hint when checksum delta is enabled after a timeline switch. (Reviewed by Matt Bunter, Cynthia Shang.)
* Use PostgreSQL instead of postmaster where appropriate. (Reviewed by Cynthia Shang.)

Documentation Bug Fixes:

* Fix incorrect example for repo-retention-full-type option. (Reported by Höseyin Sönmez.)
* Remove internal commands from HTML and man command references. (Reported by Cynthia Shang.)

Documentation Improvements:

* Update PostgreSQL versions used to build user guides. Also add version ranges to indicate that a user guide is accurate for a range of PostgreSQL versions even if it was built for a specific version. (Reviewed by Stephen Frost.)
* Update FAQ for expiring a specific backup set. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Update FAQ to clarify default PITR behavior. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2020-07-20 08:57:22 -04:00
David Steele
24d2c5b277
Remove real/all integration tests now covered by unit tests.
Remove all check and stanza-* tests except for the ones that are intended to succeed. The successful tests show that the queries run with expected results against each version of PG which should also validate queries for the failure tests in the unit tests.

Also remove the tests for --no-online backups since they don't require a database and are well tested in the unit tests.
2020-07-16 13:57:14 -04:00
Stefan Fercot
047d85c263
Automatically determine cipher passphrase in repo-get command.
The prior code was only able to use the main passphrase automatically and expected sub passphrases to be specified for each operation. This was fine for testing but hardly sufficient for a user-facing feature.

Update the code to determine which passphrase to use for any file in the repository and error when an invalid file or location is selected.

The repo-get command is still internal for now, but with this improvement it should be ready to be made public.
2020-07-16 12:24:03 -04:00
David Steele
aa4e13b665 Move encrypted files as raw in integration tests.
The encryption key should not be changed when moving a file so no need to decrypt/encrypt.
2020-07-16 11:27:14 -04:00
David Steele
88b0f6245d Run non version specific real/tests on the expect version.
There are a few non version specific tests that need to be run in integration because we can't get coverage in the unit tests.

To save some time we'll only run those tests against the same version we use for expect testing.
2020-07-15 13:19:16 -04:00
David Steele
574f36c9d2 Rename httpRequest() to httpRequestResponse() and fix comment. 2020-07-14 15:14:41 -04:00
David Steele
620a8d17cf
Automatic retry for backup, restore, archive-get, and archive-push.
If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors.

Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
David Steele
91c7adc834 Allow redactions for HTTP queries.
The Azure storage driver exposes secrets in the query when using SAS authorization. These secrets can show up during logging or when an error occurs.

Allow redaction of queries to prevent secrets from being exposed in logs and errors.
2020-07-14 13:09:48 -04:00
Stefan Fercot
d3dd32a031
Add expire-auto option.
This allows automatic expiration after a successful backup to be disabled.
2020-07-14 08:12:25 -04:00
David Steele
d5df3974b5 Read segment size from WAL headers.
This allows validation of the WAL segment size for PostgreSQL versions <= 10.
2020-07-09 17:32:36 -04:00
David Steele
2f7823c627
Add shared access signature (SAS) authorization for Azure.
A shared access signature (SAS) provides granular, delegated access to resources in a storage account. This is often preferable to using a shared key which provides more access and is a greater security risk if compromised.
2020-07-09 14:46:48 -04:00
David Steele
511e5db5bf Improve buffer size limit implementation.
Rework size limits so that this->size is always the current size no matter how much is allocated.

Most importantly, this removes the conditional in bufSize(), which makes it a better candidate for inlining.
2020-07-09 11:16:45 -04:00
David Steele
15502f5b4b Remove bufNewUseC().
This was used in the Perl LibC interface to wrap Perl-allocated buffers but is no longer needed since LibC was removed.
2020-07-09 07:16:15 -04:00
David Steele
dd9e14b628 Add pgLsnFromWalSegment().
Provides the reverse operation for pgLsnToWalSegment().
2020-07-08 12:25:39 -04:00
David Steele
a27ff7c335 Remove dead test code that should have been removed in 3f4371d7. 2020-07-07 08:24:08 -04:00
David Steele
682ac656f5
Fix restore --force acting like --force --delta.
This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue.
2020-07-06 15:03:24 -04:00
David Steele
cf284fbe8a Add httpUriDecode(), httpQueryNewStr(), and httpQueryMerge().
httpUriDecode() reverses the encoding in httpUriEncode().

httpQueryNewStr() creates a new HttpQuery by parsing a query string.

httpQueryMerge() merges the contents of one query into another query.
2020-07-06 07:48:12 -04:00
David Steele
3f4371d7a2 Azure support for repository storage.
Azure and Azure-compatible object stores can now be used for repository storage.

Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
David Steele
c2dea180fb Remove redundant storage type constants.
These constants predate the C storage drivers which now provide their own constants.
2020-06-26 16:50:29 -04:00
David Steele
96adf8e513 PostgreSQL 13 beta2 support.
There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.

Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2020-06-26 07:44:56 -04:00
David Steele
974cc10b90 Minor improvements to storage/s3 unit test. 2020-06-26 06:46:25 -04:00
David Steele
ea04ec7b3f Disable query parallelism in PostgreSQL sessions used for backup control.
There is no need to have parallelism enabled in a backup control session. In particular, 9.6 marks pg_stop_backup() as parallel-safe but an error will be thrown if pg_stop_backup() is run in a worker.
2020-06-25 08:02:48 -04:00
David Steele
ce98e326e1 Replace HRNPQ_MACRO_OPEN_92() test macro with HRNPQ_MACRO_OPEN_GE_92(). 2020-06-24 18:40:19 -04:00
David Steele
f55cb386d4 Fix versions passed to HRNPQ_MACRO_OPEN_GE_92() test macro.
These were not noticed because currently 9.3 and 9.6 behave the same on open.
2020-06-24 18:33:20 -04:00
David Steele
c5892d1291
Asynchronous S3 multipart upload.
When uploading large files the upload is split into multiple parts which are assembled at the end to create the final file. Previously we waited until each part was acknowledged before starting on the processing (i.e. compression, etc.) of the next part.

Now, the request for each part is sent while processing continues and the response is read just before sending the request for the next part. This asynchronous method allows us to continue processing while the S3 server formulates a response.

Testing from outside AWS in a high-bandwidth, low-latency environment showed a 35% improvement in the upload time of 1GB files. The time spent waiting for multipart notifications was reduced by ~300% (this measurement included the final part which is not uploaded asynchronously).

There are still some possible improvements: 1) the creation of the multipart id could be made asynchronous when it looks like the upload will need to be multipart (this may incur cost if the upload turns out not to be multipart). 2) allow more than one async request (this will use more memory).

A fair amount of refactoring was required to make the HTTP responses asynchronous. This may seem like overkill but having well-defined request, response, and session objects will also be advantageous for the upcoming HTTP server functionality.

Another advantage is that the lifecycle of an HttpSession is better defined. We only want to reuse sessions that complete the request/response cycle successfully, otherwise we consider the session to be in a bad state and would prefer to start clean with a new one. Previously, this required complex notifications to mark a session as "successfully done". Now, ownership of the session is passed to the request and then the response and only returned to the client after a successful response. If an error occurs anywhere along the way the session will be automatically closed by the object destructor when the request/response object is freed (depending on which one currently owns the session).
2020-06-24 13:44:00 -04:00
David Steele
45d9b03136
Add strCatZ().
strCat() did not follow our convention of appending Z to functions that accept zero-terminated strings rather than String objects.

Add strCatZ() to accept zero-terminated strings and update strCat() to accept String objects.

Use LF_STR where appropriate but don't use other String constants because they do not improve readability.
2020-06-24 12:09:24 -04:00
David Steele
dab00e2010 Remove expect logs obsoleted in a3e5e66f.
These expect logs are no longer used but are not automatically removed by test.pl.
2020-06-24 07:45:00 -04:00
David Steele
a3e5e66f05 Simplify test matrix for real/all tests.
Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added.

Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro.

Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it.

Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6.

The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them.

The following is an example of the new test log (note longer runtime of each test):

module=real, test=all, run=1, pg-version=10 (106.91s)
module=real, test=all, run=1, pg-version=9.5 (151.09s)
module=real, test=all, run=1, pg-version=9.2 (123.11s)
module=real, test=all, run=1, pg-version=9.1 (129s)

vs. the old test log (sub-second tests were skipped entirely):

module=real, test=all, run=2, pg-version=10 (0.31s)
module=real, test=all, run=3, pg-version=10 (0.26s)
module=real, test=all, run=4, pg-version=10 (60.39s)
module=real, test=all, run=1, pg-version=10 (69.12s)
module=real, test=all, run=6, pg-version=10 (34s)
module=real, test=all, run=5, pg-version=10 (42.75s)
module=real, test=all, run=2, pg-version=9.5 (0.21s)
module=real, test=all, run=3, pg-version=9.5 (0.21s)
module=real, test=all, run=4, pg-version=9.5 (0.21s)
module=real, test=all, run=5, pg-version=9.5 (0.26s)
module=real, test=all, run=6, pg-version=9.5 (0.21s)
module=real, test=all, run=1, pg-version=9.2 (72.78s)
module=real, test=all, run=2, pg-version=9.2 (0.26s)
module=real, test=all, run=3, pg-version=9.2 (0.31s)
module=real, test=all, run=4, pg-version=9.2 (0.21s)
module=real, test=all, run=5, pg-version=9.2 (0.21s)
module=real, test=all, run=6, pg-version=9.2 (0.21s)
module=real, test=all, run=1, pg-version=9.5 (88.41s)
module=real, test=all, run=2, pg-version=9.1 (0.21s)
module=real, test=all, run=3, pg-version=9.1 (0.26s)
module=real, test=all, run=4, pg-version=9.1 (0.21s)
module=real, test=all, run=5, pg-version=9.1 (0.31s)
module=real, test=all, run=6, pg-version=9.1 (0.26s)
module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 13:44:29 -04:00
David Steele
d560c1bf19 Ignore "unsupported frontend protocol" error on Centos/RHEL 6.
The unsupported version error is showing up on older versions of PostgreSQL (e.g. 9.1, 9.2) on RHEL6 when setting up a standby with streaming replication. The error occurs when a client does not properly send a version number and it's not clear why it is happening here, but it does not appear to have anything to do with pgBackRest and only affects RHEL6, i.e. 9.1 and 9.2 do not show this error on other distros.

For now ignore the error since RHEL6 is nearly EOL.
2020-06-23 12:42:46 -04:00
David Steele
1aedc75b03 Rename http/Http to HTTP in comments and messages.
HTTP is an acronym so it should be capitalized. Coding conventions dictate otherwise for function and type names but that should not have been propagated to comments and messages.
2020-06-21 11:47:41 -04:00
David Steele
911384d9b9 Add httpDateFromTime().
Also rename httpLastModifiedToTime() to httpDateToTime() since the RFC-2822 date format used by HTTP is used in all Date headers.
2020-06-21 11:07:18 -04:00
David Steele
fbff29957c
Inline strPtr() to increase profiling accuracy.
strPtr() is called more than any other function and during profiling (with or without optimization) it can end up using a disproportionate amount of the total runtime. Even though it is fast, the profiler has a minimum resolution for each function call so strPtr() will often end up towards the top of the list even though the real runtime is quite small.

Instead, inline strPtr() and indicate to gcc that it should be inlined even for non-optimized builds, since that's how profiles are usually generated.

To make strPtr() smaller require "this" to be non-NULL and add another function, strPtrNull(), to deal with the few cases where we need NULL handling.

As a bonus this makes the executable about 1% smaller even when compared to a prior optimized build which would inline some percentage of strPtr() calls.
2020-06-18 13:13:55 -04:00
David Steele
3d74ec1190
Use PostgreSQL instead of postmaster where appropriate.
Using postmaster in messages was not very helpful since users rarely interact directly with the postmaster. Using PostgreSQL instead seems clearer.
2020-06-17 15:14:59 -04:00
David Steele
417818dcca Add --no-coverage-report to test.pl to disable report generation.
There is no sense in generating detailed coverage reports in CI environments where they will never be seen. It takes time and format differences in some older versions can cause problems in the report generation code.

Note that missing coverage will still be reported on stdout and the test will fail.
2020-06-17 15:07:30 -04:00
David Steele
ea984c4d3e Update TEST_RESULT_PTR() to TEST_RESULT_STR() where appropriate.
These were missed in d41eea68 when the functionality of TEST_RESULT_STR() was changed. Using TEST_RESULT_STR() instead of TEST_RESULT_PTR() is more type-safe and clearer.

Add a comment to make it clear that TEST_RESULT_PTR() should be used only when a better alternative is not available.
2020-06-17 09:46:09 -04:00
David Steele
0680cfc8dc Rename most instances of master to primary in tests.
This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702).

Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 14:06:38 -04:00
David Steele
11c192f30e
Add hint when checksum delta is enabled after a timeline switch.
This warning is normal when restoring a backup or promoting a standby so add a hint to make that clear.
2020-06-16 13:20:01 -04:00
David Steele
6fe60a2428
Improve behavior of the repo-ls command.
* Exclude linefeed when there is no output to avoid a blank line.
* Honor filter when adding . path or listing a single file.
2020-06-11 13:17:35 -04:00
David Steele
237ba54d20
Fix expression when recursion enabled in storageInfoListP().
Expressions only worked at the first level of recursion because the expression was also being applied to paths so the path had to match the filter in order to recurse.

This is not considered a bug since it does not affect any existing code paths, but it is required for the general-purpose repo-ls command.
2020-06-11 11:48:42 -04:00
David Steele
da4f15663b Improve error when pg1-path option missing for archive-get command.
The assert thrown was not as descriptive as a proper option missing error.
2020-06-10 11:41:08 -04:00
David Steele
d0e08a537d Better error when closing an already closed session in TLS test harness. 2020-06-05 15:08:39 -04:00
David Steele
ae15aced99 Update RHEL package to add logrotate script. 2020-06-01 12:24:19 -04:00
David Steele
fe829af4ec Remove exclamations from test data.
Three exclamations are commonly used to mark areas of the code that need attention before commit so having them in a test is distracting.
2020-05-28 10:27:45 -04:00
David Steele
3b5f76b434
Improve handling of invalid HTTP response status.
A truncated HTTP response status could lead to an an unfriendly error message, which would be retried, but could be confusing if the error was persistent and required debugging.

Improve the error handling overall to catch more error cases explicitly and respond better to edge cases.

Also update the terminology in comments to align with the RFC. Variable and function names were not changed because a refactor is intended for HTTP response and it doesn't seem worth the additional code churn.
2020-05-27 15:13:55 -04:00
David Steele
d05090ab7b Remove Debian package patch now that it has been merged upstream. 2020-05-27 09:22:30 -04:00
David Steele
b27f9e886b Refactor TLS server test harness for ease of use.
The prior harness required a separate function to contain the server behavior but this made keeping the client/server code in sync very difficult and in general meant test writing took longer.

Now, commands to define server behavior are inline with the client code, which should greatly simplify test writing.
2020-05-26 09:16:57 -04:00
David Steele
d8214e0d78 v2.27: Expiration Improvements and Compression Drivers
Bug Fixes:

* Fix issue checking if file links are contained in path links. (Reviewed by Cynthia Shang. Reported by Christophe Cavallié.)
* Allow pg-path1 to be optional for synchronous archive-push. (Reviewed by Cynthia Shang. Reported by Jerome Peng.)
* The expire command now checks if a stop file is present. (Fixed by Cynthia Shang. Reviewed by David Steele.)
* Handle missing reason phrase in HTTP response. (Reviewed by Cynthia Shang. Reported by Tenuun.)
* Increase buffer size for lz4 compression flush. (Reviewed by Cynthia Shang. Reported by Eric Radman.)
* Ignore pg-host* and repo-host* options for the remote command. (Reviewed by Cynthia Shang. Reported by Pavel Suderevsky.)
* Fix possibly missing pg1-* options for the remote command. (Reviewed by Cynthia Shang. Reported by Andrew L'Ecuyer.)

Features:

* Time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. (Contributed by Cynthia Shang, Pierre Ducroquet. Reviewed by David Steele.)
* Ad hoc backup expiration. Allow the user to remove a specified backup regardless of retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Zstandard compression support. Note that setting compress-type=zst will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
* bzip2 compression support. Note that setting compress-type=bz2 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Contributed by Stephen Frost. Reviewed by David Steele, Cynthia Shang.)
* Add backup/expire running status to the info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)

Improvements:

* Expire WAL archive only when repo-retention-archive threshold is met. WAL prior to the first full backup was previously expired after the first full backup. Now it is preserved according to retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
* Add local MD5 implementation so S3 works when FIPS is enabled. (Reviewed by Cynthia Shang, Stephen Frost. Suggested by Brian Almeida, John Kelley.)
* PostgreSQL 13 beta1 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. (Reviewed by Cynthia Shang.)
* Reduce buffer-size default to 1MiB. (Reviewed by Stephen Frost.)
* Throw user-friendly error if expire is not run on repository host. (Contributed by Cynthia Shang. Reviewed by David Steele.)
2020-05-26 08:11:50 -04:00
David Steele
20d8c76b6c
Ignore pg-host* and repo-host* options for the remote command.
The purpose of the remote command is to get access to local resources, so a remote should never start another remote. However, this could happen if there were host settings on the remote host, which ended badly with lock errors, loops, etc.

Add pg-local and repo-local options to indicate that the resource is local even if there are host settings.

Note that for the time being these options are internal and not intended for general usage. However, this is likely the direction needed to allow for more symmetric and manageable configurations.
2020-05-22 13:51:26 -04:00
David Steele
ae75ffc173
Fix possibly missing pg1-* options for the remote command.
Some pg1-* options are required by the remote so if they are not provided in the remote's configuration file then it may cause a configuration error, depending on the operation. This currently only applies to the pg1-path option.

This is still an issue for repo-* options but the same solution cannot be applied because some repo-* options are secure and cannot be passed on the command-line.
2020-05-21 16:09:23 -04:00
David Steele
ec7b7c5a3e
PostgreSQL 13 beta1 support.
There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.

Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2020-05-21 13:46:16 -04:00
David Steele
f15d6104d2
Add local MD5 implementation so S3 works when FIPS is enabled.
S3 requires the Content-MD5 header for many requests but MD5 is not available via OpenSSL when FIPS is enabled because it is considered to be insecure.

Even though our usage does not present any security risks a local M5 implementation is required to circumvent the over-broad FIPS restriction.

Vendorize the MD5 implementation found at https://openwall.info/wiki/people/solar/software/public-domain-source-code/md5 and add full coverage for the module in the common/crypto unit tests.
2020-05-20 14:56:13 -04:00
David Steele
ea9147e2e0 Reduce buffer-size default to 1MiB.
The prior default was determined by benchmarking the Perl code prior to the 1.0 release. In general buffer allocation was more expensive in Perl so large buffers gave the best performance. This was due to multiple buffer allocations for each filter in an IO operation.

The C code allocates fixed buffers for each IO operation so the cost for buffer allocation is lower than Perl. That being the case it made sense to benchmark the C code to determine the optimal buffer default.

The performance/storage tests were used to measure the performance of a variety of filters. 1GiB of data was processed by each filter 10 times and the results of the tests were averaged.

While most buffer sizes gave similar performance, 1MiB appeared to perform the best overall. Of course, different architectures are likely to yield different results but this seems like a sensible default. The buffer-size option may still need to be manually configured to give optimal results.

Raw test data for reference:

4MB buffer (prior default)

copy time 1807ms, avg time 180ms, avg throughput: 5942MB/s
md5 time 14200ms, avg time 1420ms, avg throughput: 756MB/s
sha1 time 11431ms, avg time 1143ms, avg throughput: 939MB/s
sha256 time 23463ms, avg time 2346ms, avg throughput: 457MB/s
gzip -6 time 381199ms, avg time 38119ms, avg throughput: 28MB/s
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s

1MB buffer (new default)

copy time 1760ms, avg time 176ms, avg throughput: 6100MB/s
md5 time 13739ms, avg time 1373ms, avg throughput: 781MB/s
sha1 time 11025ms, avg time 1102ms, avg throughput: 973MB/s
sha256 time 22539ms, avg time 2253ms, avg throughput: 476MB/s
gzip -6 time 372995ms, avg time 37299ms, avg throughput: 28MB/s
lz4 -1 time 15118ms, avg time 1511ms, avg throughput: 710MB/s

512K buffer

copy time 1782ms, avg time 178ms, avg throughput: 6025MB/s
md5 time 13724ms, avg time 1372ms, avg throughput: 782MB/s
sha1 time 10959ms, avg time 1095ms, avg throughput: 979MB/s
sha256 time 22982ms, avg time 2298ms, avg throughput: 467MB/s
gzip -6 time 378120ms, avg time 37812ms, avg throughput: 28MB/s
lz4 -1 time 15484ms, avg time 1548ms, avg throughput: 693MB/s

256K buffer

copy time 1805ms, avg time 180ms, avg throughput: 5948MB/s
md5 time 13706ms, avg time 1370ms, avg throughput: 783MB/s
sha1 time 11074ms, avg time 1107ms, avg throughput: 969MB/s
sha256 time 22588ms, avg time 2258ms, avg throughput: 475MB/s
gzip -6 time 372645ms, avg time 37264ms, avg throughput: 28MB/s
lz4 -1 time 16346ms, avg time 1634ms, avg throughput: 656MB/s
2020-05-19 16:58:49 -04:00
David Steele
f773d909be Improve storage filter performance tests.
Improve the accuracy of the calculations in several areas with better integer expressions.

Make the input buffer size configurable. Previously it was always 1mb, i.e. block size.

Use a macro for output results to reduce code duplication.
2020-05-19 14:35:20 -04:00
David Steele
a3d9d9a387 Handle missing reason phrase in HTTP response.
Reason phrases (e.g. OK) are optional in HTTP 1.1 but the space after the status code is not. When the reason phrase was missing the required space was trimmed along with the trailing CR leading to a format error.

Rework the logic to preserve the space and allow empty reason phrases.

Found while testing against the Backblaze S3-compatible API.
2020-05-19 08:20:33 -04:00
David Steele
688ec2a8f5 Use an extension to denote vendorized code.
Vendorized code is copied from another project when a library is not available and a git subproject won't work. Currently all the vendorized code is copied from PostgreSQL but it makes sense to have a more general mechanism for indicating vendorized code.

The .vendor extension will be used to denote vendorized code in the same way that .auto is used to denote auto-generated code.
2020-05-18 19:11:26 -04:00
David Steele
a329afd3be Add MD5 hash filter to performance tests. 2020-05-18 19:02:11 -04:00
David Steele
92c036b966 Add code count rule for valgrind suppression missed in 6be5ea33.
6be5ea33 changed valgrind suppression file naming but failed to update the code count rules.
2020-05-18 18:09:41 -04:00
David Steele
ea485e916b Add jq to tools installed by Vagrantfile. 2020-05-14 18:45:23 -04:00
David Steele
4cbd1f1e7e Fix incorrect whitespace. 2020-05-13 14:27:28 -04:00
David Steele
b5dd14e6f3 Make storage type more generic in the integration tests.
Rather than bS3 use strStorage which can indicate more than two storage types.

For the moment there are still only two storage types but this change is required before more can be added.
2020-05-12 18:55:20 -04:00
David Steele
9639a2c15f Add missing do...while loop to harness macro. 2020-05-12 13:30:46 -04:00
David Steele
86855e271d Fix subtle timing issue in command/expire tests.
cdebfb09 added relative times to backup.into but a subtle issue was introduced that would cause the tests to fail if the time acquired by cmdExpire() was exactly the same as timeNow used to format backup.info. cmdExpire() was working correctly given the inputs, but the tests did not run predictably.

This was found while running the tests with --no-valgrind --no-coverage which allows them to run a lot faster, thus exposing the timing issue.
2020-05-09 12:12:29 -04:00
David Steele
22d260ad53 Allow more tests to run outside of containers.
These tests required sudo to achieve complete coverage.

Add a new coverage exception, vm_covered, that applies to code that can only be covered in a container. When the test is run outside of a container code sections that require a container will be excluded with TEST_CONTAINER_REQUIRED and the coverage exception will be added to prevent a coverage error.

This does require marking up the core code with vm_covered, which in some modules (e.g. common/io/tls/client) can be extensive. It's possible that some of these tests can be rewritten to be less dependent on sudo but no attempt was made to do that here.

Only allow coverage summaries in a vm since coverage summaries outside a vm will not be complete, which was true even before this commit.
2020-05-09 09:17:33 -04:00
Stephen Frost
b4fc1804a8
Minor updates for bzip2 compression after more review.
Update error types throw by bzip2 to be more consistent with gzip.

Update the bzip2 and gzip error default to be AssertError as that's the more common case in both, and add a 'break;' to the default clause -- we don't intend to be just falling through those case statements, even if the default is the last, we should be explicit about that.

Clean up some tabs that snuck in, rename a variable to be more clear, and add some comments.
2020-05-08 16:27:54 -04:00
Cynthia Shang
cdebfb09e0
Add time-based retention for full backups.
The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days.

The new option will default to 'count' and therefore will not affect current installations. Setting repo-retention-full-type to 'time' will allow the user to use a time period, in days, to indicate full backup retention. Using this method, a full backup can be expired only if the time the backup completed is older than the number of days set with repo-retention-full (calculated from the moment the 'expire' command is run) and at least one full backup meets the retention period. If archive retention has not been configured, then the default settings will expire archives that are prior to the oldest retained full backup. For example, if there are three full backups ending in times that are 25 days old (F1), 20 days old (F2) and 10 days old (F3), then if the full retention period is 15 days, then only F1 will be expired; F2 will be retained because F1 is not at least 15 days old.
2020-05-08 15:25:03 -04:00
David Steele
e873ad6da0 Update Minio version to 2020-05-06T23-23-25Z in tests/documentation.
This release fixes the issue we submitted regarding an unquoted eTag: https://github.com/minio/minio/issues/9517
2020-05-07 17:26:46 -04:00
David Steele
faabf1227d Update Fedora container to Fedora 32.
This allows unit testing on gcc 10.

Also fix an incorrect enum in the config/config unit test that was caught by the new compiler.
2020-05-07 11:06:56 -04:00
David Steele
f8509ab76c Don't allow sudo to disable core dumps in test containers.
Newer versions of sudo output this message to stderr when run in a container:

sudo: setrlimit(RLIMIT_CORE): Operation not permitted

See https://github.com/sudo-project/sudo/issues/42 for details.

A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.
2020-05-07 07:38:28 -04:00
David Steele
6be5ea3388 Suppress Valgrind errors on a per-VM basis.
There are a number of Valgrind errors on Ubuntu 12.04 which do not happen on newer distro versions. However, suppressions for these errors have masked legitimate issues in subsequent code.

Instead, make suppressions VM specific so errors in other VMs are not masked.
2020-05-06 18:24:48 -04:00
David Steele
8aede3353c Always use 127.0.0.1 on TLS tests outside of containers.
Resolving localhost can vary based on the local network configuration so it is safer to just use a static IP.

This was found while testing on Travis-CI arm64.
2020-05-06 14:49:03 -04:00
Stephen Frost
a021c9fe05
Add bzip2 compression support.
bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.

bzip2 is currently available on all supported platforms.
2020-05-05 16:49:01 -04:00
David Steele
98f30ef222 Move PostgreSQL 9.4 real integration tests to Ubuntu 18.04.
PostgreSQL 9.4 packages for RHEL 6 were dropped from yum.p.o.
2020-05-05 15:00:13 -04:00
David Steele
99405cbb15 Replace booleans with enums in compressType parameters.
This was an oversight in 438b957f which added multiple compression type support. The booleans were interpreted as none and gz which works fine for the CompressType enum until the position of gz or none changes.
2020-05-05 13:23:36 -04:00