1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

2306 Commits

Author SHA1 Message Date
David Steele
6b355806cc v2.41: Backup Annotations
Bug Fixes:

* Fix incorrect time expiration being used for non-default repositories. (Reviewed by Stefan Fercot. Reported by Adam Brusselback.)
* Fix issue when listing directories recursively with a filter. (Reviewed by Stephen Frost. Reported by Efremov Egor.)

Features:

* Backup key/value annotations. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Adam Berlin.)

Improvements:

* Support --set in JSON output for info command. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Anton Kurochkin.)
* Update archive.info timestamps after a successful backup. (Reviewed by Stefan Fercot. Suggested by Alex Richman.)
* Move standby timeline check after checkpoint. (Reviewed by Stefan Fercot, Keith Fiske. Suggested by Keith Fiske.)
* Improve warning message on backup resume. (Suggested by Cynthia Shang.)

Documentation Improvements:

* Add absolute path for kill in pgbackrest.service. (Suggested by Don Seiler.)
2022-09-19 10:08:10 -07:00
David Steele
c39c9f220e
Fix issue when listing directories recursively with a filter.
While recursing and filtering, if the last entry in a directory was another directory containing entries then the parent list would get freed too early, causing a double free error or segfault.

Fix by ensuring that the completed list is at the top of the stack before freeing it. This will defer freeing parent lists until the contents of paths have been processed.
2022-09-15 12:00:44 -07:00
David Steele
240cd755d1 Add mem context test missing from 0f7b6a33.
A coverage exception was added during development but was not removed before commit.

Remove the exception and add a test for coverage.
2022-09-14 10:06:06 -07:00
David Steele
8fb61a809d
Add FN_INLINE_ALWAYS macro.
Eliminate a lot of useless repetition for a commonly used pattern.
2022-09-08 18:36:03 -06:00
David Steele
3b5df1e089
Update archive.info timestamps after a successful backup.
Lifecycle policies can cause the archive.info file and its copy to be removed since they are only updated on a stanza-upgrade. Update the timestamps after a successful backup to prevent this.

This does not mean that lifecycle policies should be used as a replacement for expiration. However, in some cases there may be policies in place that are out of admin control. If the lifecycle expiration is less than pgbackrest expiration then corruption of the earliest backup will occur at the very least and there might be other corruption which would make the repo unrecoverable.
2022-09-02 10:03:18 -05:00
David Steele
0f7b6a3344
Skip mem context cleanup in CATCH_FATAL() block.
An error that gets raised all the way to the top TRY block might need to free a lot of resources and any of these callbacks could throw an error and mask the original error. In fact this is pretty likely since we are already in an error state. For example, the Db object will try to close the remote db connection, but if the protocol is in a bad state it will not be able to do so.

Solve this, for now, by not freeing memory or calling callbacks in the CATCH_FATAL() block. This gives us a better chance if being able to report the error without encountering another error first.

For the most part, we don't need to worry about freeing resources (file handles, TLS contexts, etc.) if the program is going to exit immediately. However, it is important to attempt to terminate all active protocol connections, which is done by protocolFree() in main() since the protocol objects live in the top context.

Another way to handle this would be to implement an error stack and that is probably something we will do in the future. But, in the case of a segfault the original error would still be lost. Yet another option would be to still do cleanup but defer it until after the CATCH_FATAL() block.
2022-08-31 12:44:55 -04:00
David Steele
eda7f81ee4
Fix incorrect time expiration being used for non-default repositories.
If a repo is not specified for the expire command then the lowest repo becomes the default. The repo-retention-full value for time was being retrieved from the default rather than a specific repo which led to an incorrect expiration being applied.

Get the value from the specific repo and add a test.

It would be better if the default repo could not be queried in this case but it is not clear how to do that since the repo option is valid for expire (unlike, e.g., archive-push).
2022-08-31 10:01:12 -04:00
David Steele
f1cb848591 Fix comment typos in command/expire unit test. 2022-08-30 18:04:32 -04:00
Stefan Fercot
db75ffd270
Support --set in JSON output for info command.
Allow detailed information (e.g. error list, tablespace list) in JSON output that is already available in text output with the --set option.
2022-08-25 10:12:22 -04:00
Stefan Fercot
381fd0a5a4
Backup key/value annotations.
Allow key/value annotations to be added with the backup command and added/modified/removed with the new annotate command.

Annotations can be viewed with the info command in text mode when --set is specified and are always included in JSON output.
2022-08-24 10:52:33 -04:00
David Steele
b05d31f531
Allow upload chunk size to be configured for object stores.
There are performance benefits to increasing the upload chunk size as long as the tradeoff with additional memory consumption is acceptable.

Make the chunk size configurable for S3, GCS, and Azure, but don't attempt to do any validation of the chunk size beyond some sane limits. The defaults remain as is for each storage type to avoid any unintentional regressions.
2022-08-24 09:33:26 -04:00
David Steele
ff1188f92d Allow quote tag in command-line help.
The quote tag will be used in an upcoming commit.
2022-08-22 13:51:05 -04:00
David Steele
82786da154
Do not allow CATCH() to catch a fatal error.
Catching individual fatal errors was only used in testing so the tests have been updated to use other errors instead. CATCH_FATAL() is now the only way to catch fatal errors.

This simplifies the logic a bit for upcoming changes to error handling and cleanup.

Also fix an issue where passing errorMessage() directly to THROW*() would attempt to copy the message buffer instead of preserving it, which is undefined behavior. Since there were no instances of this behavior before this commit, this was not a live bug.
2022-08-16 16:15:48 -04:00
David Steele
02665a5894 Update Minio test/documentation container version. 2022-08-02 11:05:31 -04:00
David Steele
eb287b18c8 Add profiling, performance, and optimization to C test harness.
All unit and performance tests are now built by the C harness.

Remove all unit/performance test build code from Perl.

Remove code from C harness that is no longer used. This code was included so the C harness could be run separately, but that is no longer needed with this full integration.
2022-07-29 10:31:36 -04:00
David Steele
1e83f2a022 Add coverage to C test harness.
Coverage reports are still generated in Perl, but all the settings have been added to the C harness to generate raw coverage data.
2022-07-28 14:53:48 -04:00
David Steele
c99ea54f17 Integrate C test harness with Perl test harness.
The C test harness is used for unit tests from the Perl harness where possible. Currently, unit tests can be run in the C harness when --no-coverage is specified and --profile is not specified.

C harness tests work on meson 0.45.

The C harness runs with valgrind by default. Valgrind can be disabled with --no-valgrind.

Also rebuild containers to add meson and update the documentation so that meson builds will work (even though we don't do them yet).
2022-07-27 10:32:32 -04:00
David Steele
79d9884141 Move release build check to src/meson.build.
This allows the C harness to perform release builds which are required for some tests.
2022-07-25 09:39:54 -04:00
David Steele
c267ba51b1
Move standby timeline check after checkpoint.
The standby timeline check was being performed using pg_control data loaded before the backup started. If the backup was started immediately after a promotion the standby might not have executed a checkpoint and written the new timeline to pg_control.

Instead perform the timeline check after the checkpoint is executed. This should ensure that the new timeline is in pg_control.
2022-07-22 16:24:55 -04:00
David Steele
cbbe93f592 Improve warning message on backup resume.
The prior warning made it sound as if some action was required on the part of the user.

The new message should make it clearer that this action will be performed by pgBackRest.
2022-07-22 14:45:38 -04:00
David Steele
4c47cc5d47 Remove Debian package patch now that it has been merged upstream. 2022-07-22 09:24:29 -04:00
David Steele
19d9941367 More C test harness improvements and CI.
Build pgbackrest binary and auto-generated code automatically.

Remove --module option and allow modules to run by parameter. This is less verbose and multiple modules can be run at a time.

Allow filtering of modules. Multiple tests can be passed as parameters and if the module ends in / it will be used as a prefix filter. For example, common/ will run all the common modules.

If a test errors the remaining tests will still run but the test process will eventually exit with an error.

CI tests are included but unit tests remain on the development branch.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off) or containerReq (e.g. protocol/protocol) keywords.

Building the C test harness has been simplified:

meson -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug test/build/none pgbackrest
ninja -C test/build/none test/src/test-pgbackrest

To run all modules:

test/build/none/test/src/test-pgbackrest test

Just the common/error module:

test/build/none/test/src/test-pgbackrest test common/error

All info modules:

test/build/none/test/src/test-pgbackrest test info/
2022-07-21 20:10:51 -04:00
David Steele
edfcf1652c Test Dockerfile improvements.
Add tzdata package so timezone tests in command/restore work correctly.

Mark default git path as safe. This is a security fix that is not applicable in this environment, but must be set.

Also remove package cleanup, which is inconvenient when new packages need to be installed. It makes sense for containers that will be downloaded from Dockerhub but not so much for a locally-maintained container.
2022-07-21 19:27:58 -04:00
David Steele
5e5b04be37 Fix common/lock test creating "750" path.
This was clearly an attempt to set the mode when creating a directory, but it never worked and instead created a "750" directory in the current working directory.

Detected when running in an environment where the current working directory was read-only.
2022-07-21 18:19:47 -04:00
David Steele
f9bbafbf3f C test harness improvements.
Add harness depends when present.

Include libyaml in all test builds.

Fix mode on paths before trying to remove and set test path with mode 770 to match the Perl test harness.

With these changes all unit tests run except those that specify the define (e.g. common/assert-off), binReq (e.g. command/archive-get), or containerReq (e.g. protocol/protocol) keywords.

Builds and code generation need to be done in advance. The following commands are required for setup:

meson setup -Dwerror=true -Dfatal-errors=true -Dbuildtype=debug build pgbackrest
ninja -C build test/src/test-pgbackrest
build/src/build-code help pgbackrest
build/src/build-code postgres pgbackrest

Now tests can be run, e.g.:

build/test/src/test-pgbackrest --module=postgres/interface
2022-07-20 19:02:14 -04:00
David Steele
c625f05a13 Unify code builder binaries into a single binary.
Creating new binaries was convenient at first but has now become a maintenance issue.

Solve this by combining that into a single binary that takes an additional parameter to indicate which code should be built.

Also clean up path handling to make it easier to build code from the command line.
2022-07-20 17:45:39 -04:00
David Steele
7eb5d67985 Add module harness and shim support to the C test harness.
This allows running unit tests up to common/ini, excluding common/debug-off and common/assert-off.
2022-07-19 18:03:39 -04:00
David Steele
8fdeed7807 Copy repository links as files for testing.
This makes the test code a bit simpler where we are listing a path but not following links.

Links in the repository can be used for testing but should never be committed to the main branch.
2022-07-19 17:57:13 -04:00
David Steele
eb77579008 Fix comment typo. 2022-07-19 17:25:07 -04:00
David Steele
1ff531090b v2.40: OpenSSL 3 Support
NOTE TO PACKAGERS: An experimental meson build has been added but packagers should continue to use the autoconf/make build for the foreseeable future.

Improvements:

* OpenSSL 3 support. (Reviewed by Stephen Frost.)
* Create snapshot when listing contents of a path. (Reviewed by John Morris, Stephen Frost.)
* Force target-timeline=current when restore type=immediate. (Reviewed by Stephen Frost.)
* Truncate files during delta restore when they are larger than expected. (Reviewed by Stephen Frost.)
* Disable incremental manifest save when resume=n. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Set backup percent complete to zero before copy start. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Use S3 IsTruncated flag to determine list continuation. (Reviewed by John Morris, Soulou. Suggested by Christian Montagne.)

Documentation Bug Fixes:

* Skip internal options in the configuration reference. (Reported by Francisco Miguel Biete.)

Documentation Improvements:

* Add link to PostgreSQL configuration in repository host section. (Reviewed by Stefan Fercot. Suggested by Julien Cigar.)

Test Suite Improvements:

* Add experimental Meson build. (Reviewed by Eli Schwartz, Sam Bassaly.)
* Allow any path to be passed to the --test-path option. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
* Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
2022-07-18 09:32:30 -04:00
David Steele
364af1635d
Force target-timeline=current when restore type=immediate.
Explicitly set target timeline to "current" when type=immediate and PostgreSQL >= 12. We do this because type=immediate means there won't be any actual attempt to change timelines, but if we leave the target timeline as the default of "latest" then PostgreSQL might fail to restore because it can't reach the "latest" timeline in the repository from this backup.

This is really a PostgreSQL bug and will hopefully be addressed there, but we'll handle it here for older versions, at least until they aren't really seen in the wild any longer.

PostgreSQL < 12 defaults to "current" (but does not accept "current" as a parameter) so no need set it explicitly.
2022-07-14 08:26:03 -04:00
David Steele
75623d4583
Create snapshot when listing contents of a path.
Previously a callback was used to list path contents and if no sort was specified then a snapshot was not required. When deleting files from the path some filesystems could omit files that still existed, which meant the path could not be removed.

Filter . out of lists in the Posix driver since this special entry was only used by test code (and filtered everywhere in the core code).

Also remove callbacks from the storage interface and replace with an iterator that should be easier to use and guarantees efficient use of the snapshots.
2022-07-08 17:21:39 -04:00
David Steele
0eccbc8bf4 Meson builds work on version 0.45.
v0.45 ships with Ubuntu 18.04, which is currently the oldest distro we support. We may never do a Meson release on Ubuntu 18.04 but this allows us to start running unit tests with Meson in the meantime.

Some more granular options are not available so we use buildtype in more places.

The check for a in-tree autoconf/make build had to be removed since the filesystem APIs are not available.

Finally, alias_target was removed. This means that full paths must be used for build targets, which does not seem too bad. For instance, test/src/test-pgbackrest must now be used as a build target instead of simple test-pgbackrest.
2022-07-06 18:17:52 -04:00
David Steele
72960bbf17 Rename strTrunc() to strTruncIdx() and add strTrunc().
strTrunc() is now shorthand for truncating to index 0. This is convenient when a string is being reused.
2022-07-06 09:55:32 -04:00
David Steele
1c0bf0b15d Reorder null user/group checks in remote protocol and add tests.
Coverage for these checks was dependent on the order the files were read from disk, which made the tests fragile.

Rearrange the checks and add a test that won't depend on order.
2022-07-05 14:28:40 -04:00
David Steele
845d82f682
Use S3 IsTruncated flag to determine list continuation.
Previously we were just checking for the existence of NextContinuationToken, which the S3 documentation indicates will not be present when the list is not truncated. However, recent versions of Scality send a blank NextContinuationToken when IsTruncated is false. Sending the blank continuation token back causes Scality to send another blank continuation token and an infinite loop occurs.

Instead use IsTruncated (which is required to be present) to determine whether NextContinuationToken should be present. Error if NextContinuationToken is then missing or empty, since an empty token caused an infinite loop with the Scality server (which arguably should have errored when passed an empty token).
2022-07-01 20:32:10 -04:00
David Steele
61ca9b58a0 Replace hrnStorageInfoListCallback() with TEST_STORAGE_LIST() in tests.
The TEST_STORAGE_LIST() macro is more robust and hides the callback mechanism from the caller.

Add features to TEST_STORAGE_LIST() that hrnStorageInfoListCallback() had.

Update tests to use the abbreviated type output (e.g. path/) generated by TEST_STORAGE_LIST().
2022-06-28 14:15:08 -04:00
David Steele
f863fc9888 Add experimental unit test harness written in C.
Having the test harness in C will allow us to remove duplicated Perl code and test on systems where Perl support is not present.

Custom harnesses and shims are currently not implemented, which means only the following tests in the common module will run: error, stack-trace, type-convert, assert-on, mem-context, time, encode, type-object, type-string, type-list, type-buffer, type-variant, reg-exp, log.

The experimental test harness is being committed with partial functionality so it can be used in Windows development. The remaining features will follow as needed.
2022-06-23 12:20:56 -04:00
David Steele
9fd85a104a Disable meson for all but debug builds.
The meson builds are still experimental so for now the configure/make build process is preferred for release builds. This message should help prevent any automated build systems from picking up meson instead.
2022-06-21 09:50:38 -04:00
David Steele
5ecae90f02 Use constants rather than replacements when possible in test.c.
Some of the replacements that were being done already existed as constants, so use the constants instead.

Also fix a minor formatting error introduced when testAdd() was renamed to hrnAdd().
2022-06-17 16:31:48 -04:00
David Steele
dab1e4b6c6 Add strReplace().
Allows substrings to be replaced with another string.
2022-06-17 16:27:31 -04:00
David Steele
fb9acc1c93 Add higher level YAML functions.
These functions make parsing YAML simpler.
2022-06-17 16:17:02 -04:00
David Steele
55bcb933ee Move protocol module from common to command.
This module has dependencies on command/command so it does not make sense for it to be in the common module. Also move protocolFree() to main() since this is a very large dependency.

Adjust the tests so command/exit can be tested later. This is a bit messy but will get adjusted as we improve the test harness.
2022-06-17 11:17:52 -04:00
Stefan Fercot
eb72a80b47 Fix continuation character spacing. 2022-06-15 09:06:25 -04:00
David Steele
04f5ef25fa Add valgrind to Dockerfile missed in a16cf5ea. 2022-06-09 17:18:32 -04:00
David Steele
79443bea36 Move bldEnum() to the build/common/render module.
This function will be useful for other build/test modules so extern it.

Also skip the first upper-case when no prefix is provided.
2022-06-09 16:55:07 -04:00
David Steele
a16cf5eac7 Update CI to use Ubuntu 22.04 and Fedora 36.
Both have newer gcc and OpenSSL 3.

Fedora 36 runs horribly slow with valgrind enabled so run the valgrind tests on Ubuntu 22.04. Fedora 36 has a newer gcc so it is still worth testing on.
2022-06-06 16:32:20 -04:00
David Steele
08242ee6ac
OpenSSL 3 support.
There are two changes:

* Suppress deprecation warnings so we can build with -Werror and -Wfatal-errors. At some point we'll need to migrate to the new APIs but there does not seem to be a good reason to support two sets of code right now.

* Update the handling for unexpected EOF to handle EOF or error. The error code for EOF has changed and become harder to identify, but we probably don't care whether it is an error or EOF.
2022-06-06 14:47:47 -04:00
David Steele
f92ce674f7
Automatically create PostgreSQL version interfaces.
Maintaining the version interfaces was complicated by the fact that each interface needed to be in separate compilation unit to avoid type conflicts. This also meant that various build/test files needed to be updated to add the new interfaces.

Solve these problems by auto-generating all the interfaces into a single file. This is made possible by parsing defines and types out of the header files and creating macros to rename the types. At the end of the version interface everything is undef'd. Another benefit is that the auto-generated interfaces can be static and included directly into postgres/interface.c.

Since some code generation is now always required for tests, change --no-gen to --min-gen in test.pl.

It would also make sense to auto-generate the version defines in postgres/version.h, but that will be left for a future commit.
2022-06-06 13:52:56 -04:00
David Steele
b8fc20d5b8
Add experimental Meson build.
Meson is a new build system that offers simpler syntax and superior performance to autoconf/make. In addition, Windows is supported natively.

The Meson build appears complete, but currently is used only for auto-generation of code and the host build of pgbackrest. Some container upgrades will be required before Meson can be used for container builds.

Also patch the Debian package to force autoconf/make rather than Meson.
2022-06-03 14:13:56 -04:00
David Steele
148956aed8 Remove useless command/check unit test.
This test was a placeholder and did not provide any coverage, but it did give inconsistent results on different shell versions.
2022-06-01 10:13:57 -04:00
Andrey Sokolov
29b2a54fcc
Allow any path to be passed to the --test-path option.
A hard-coded path prevented this from working correctly.
2022-05-31 17:28:58 -04:00
David Steele
2c38c9a56f Skip stopping PostgreSQL 9.1 in real/all integration test.
Stopping the cluster has started consistently running out of memory on PostgreSQL 9.1. This seems to have happened after pulling in new packages at some point so it might be build related.

Stopping the cluster is not critical for 9.1 so skip it.
2022-05-31 16:43:18 -04:00
David Steele
2feaaeaac8 Add .inc extension to C files included in other C files.
These files were never intended to be compiled on their own so the .c extension was a bit misleading. In particular Meson does not like .c files that are not intended to be compiled independently.

Leave header files as is since they are already protected against being included more than once and are never expected to be compiled.
2022-05-31 16:06:41 -04:00
Reid Thompson
60d70fa66f
Disable incremental manifest save when resume=n.
The manifest is saved on a regular basis during a backup so a failed backup can be resumed. For backups that the user has configured/invoked as not resumable, skip the incremental save of the manifest.
2022-05-25 18:18:37 -04:00
David Steele
8c2b3a044f Use StringId for type in cryptoHash() and cryptoHmacOne().
This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.
2022-05-25 15:27:53 -04:00
David Steele
7ec51e7e62
Truncate files during delta restore when they are larger than expected.
Previously the behavior was to download the file from the repository when it was not exactly the same size in PGDATA. However, it may just be that the file was extended and the contents are the same up to the file size recorded in the manifest. This could also be very valuable for files that are always append only, like logs.

Change info.size to file->size in one place. Both are technically correct but file->size makes more sense.

Use the new fileName variable in a few existing places.

Also adjust some existing comments to make them clearer.
2022-05-24 16:39:35 -04:00
David Steele
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
David Steele
c7a66ac1af
Improve memory usage of mem contexts.
Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible.

This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case.

Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed.

Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree().

Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
David Steele
83af3f1b7a Add additional detail to warnings when delta checksum is auto-enabled.
Hopefully this will help with debugging when it is not clear why delta checksum is being enabled.
2022-05-18 08:48:48 -04:00
David Steele
4dcc9df222 Add labels in parse.auto.c to make diffs easier to read.
Because there is a lot of repetition in this file, changes can look very jumbled with existing data in a diff. Also, if can be hard to tell what is being modified if the diff does not show enough lines before and after.

This change adds labels to the end of the line to localize the diff and make it easier to see what has been changed. Also, remove some linefeeds and make separators more consistent.

The change to parse.auto.c will be committed separately so it can be ignored in history/blame.
2022-05-16 09:14:46 -04:00
David Steele
901e829f6d v2.39: Verify and File Bundling
Bug Fixes:

* Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)
* Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.)

Features:

* Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)
* Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)
* PostgreSQL 15 support. (Reviewed by Stefan Fercot.)
* Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)
* Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)

Improvements:

* Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)
* Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)
* Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)
* Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)
* Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
* Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)
* Add user:group to lock permission error. (Reviewed by Reid Thompson.)

Documentation Bug Fixes:

* Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)
* Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)
* Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)

Documentation Improvements:

* Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)
* Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)
* Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
2022-05-16 08:46:24 -04:00
David Steele
19dd015d58
Fix issues in improved path validation for repo-* commands.
If the user requested the exact repo path then strSub() would be passed an invalid start value leading to an assertion:

$ pgbackrest --stanza=test repo-ls /var/lib/pgbackrest
ASSERT: [025]: start <= this->pub.size (on dev builds)
ASSERT: [025]: string size must be <= 1073741824 bytes (on prod builds)

Fix this by checking if the requested path exactly equals the repo path and returning an empty relative path in this case.

Another issue was that invalid subpaths were not detected if they started with the repo path. For example, /var/lib/pgbackrestsub would not generate an error if the repo path was /var/lib/pgbackrest. Fix this by explictly checking for a / between the repo path and the subpath. This also requires special handling when the repo path is /.

This is not a live bug since the issues were found in an unreleased feature introduced in 5ae84d5.
2022-05-13 09:41:53 -04:00
David Steele
024500782e Reduce encrypted WAL segments sizes in command tests.
The encrypted archive-push and repo tests were running very slowly on 32-bit with Valgrind enabled. This appears to be an issue with a newer version of Valgrind, but it has been going on long enough that bisecting does not seem to be worthwhile.

Reduce the size of the encrypted test segments where possible to improve overall test performance.
2022-05-12 17:02:08 -04:00
David Steele
7d9b2e267c Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.
Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.
2022-05-12 11:57:12 -04:00
David Steele
3d8ee552fe Move some PostgreSQL integration tests to rh7.
This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.
2022-05-11 15:59:37 -04:00
Reid Thompson
a913113fda
Add option type descriptions.
This cuts down on repetition of the size descriptions and adds basic descriptions for the other option types.
2022-05-11 15:23:41 -04:00
David Steele
8ee85bc605 Fix "that that" typos. 2022-05-11 08:46:23 -04:00
David Steele
de816a0f57
Remove integration expect log testing.
Integration expect log testing was originally used as a rough-and-ready way to make sure that certain code paths were being executed before the unit tests existed. Now that we have 100% unit test coverage (with expect log testing) the value of the integration expect tests seems minimal at best.

But they do cause numerous issues:

- Maintenance of the expect code and replacements that are required to keep logs reproducible.
- Even a trivial change can cause massive churn in the expect logs, e.g. d9088b2. These changes should be minutely audited but since the expect logs have little value now it is seldom worth the effort.
- The OS version used to do expect testing (RHEL7) can only be used to test one version of PostgreSQL. This makes it hard to balance the PostgreSQL version testing between OS versions.
- When a commit affects expect logs it is not clear (especially for new developers) how to regenerate them and our contributing guide is silent on the issue.

The goal is to migrate the integration tests to C and expect testing is not part of that plan. It seems best to get rid of them now.
2022-05-10 13:18:26 -04:00
David Steele
3a40394472 Remove obsolete test in common/memContext.
Once upon a time the allocation array was allocated up front so this test was required for the top context, which did not allocate up front.

Now allocations are done on demand so this case is covered for every context that does not allocate memory.
2022-05-10 11:18:12 -04:00
Reid Thompson
6b98b3534e Fix typo. 2022-05-10 06:52:56 -04:00
David Steele
b4c1ca7b80 Split 32-bit CI tests.
This helps rebalance some of the tests that are running long, i.e. d9 and u20.

I would be better to move more PostgreSQL versions to d9, but the base VM does not contain more versions. New minor versions will be out later in the week so that seems a better time to be rebuilding containers.
2022-05-09 14:19:05 -04:00
David Steele
39dddbb6bc Add limited CI for ppc64le/s390x using emulation.
The emulation is so slow that running all the unit tests would be too expensive, but this at least shows that the build works and some of the more complex tests run. In particular, it is good to test on one big-endian architecture to be sure that checksums are correct.

Update checksums in the tests where they had gotten out of date since the last time we were testing on s390x. Also use a different test in command/archivePushTest to show the name of the file when a checksum does not match to aid in debugging.

The command/archive-push test was updated but not included because there is also a permissions issue, which looks to be the same as what we see on MacOS/FreedBSD. Hopefully we'll be able to fix all of those at the same time.
2022-05-09 12:48:19 -04:00
David Steele
ef4c4ab852
Use variable instead of function to track FINALLY() state.
The function worked fine, but Coverity was unable to determine that the finally block was run, which led to false positives about unfreed memory.

Using a boolean in the block makes it clear to Coverity that the finally block will always be run no matter what else happens.

We'll depend on the compiler to optimize away the boolean if it is not used in a finally block. The cost of the boolean is fairly low in comparison to everything else being done in these macros, so it does not seem worth having a separate block even if the compiler is not able to eliminate the boolean.

This reverts most of 9a271e9 that fixed a bug caused by c5b5b58, which was also attempting to help Coverity understand FINALLY() blocks.
2022-05-09 10:39:43 -04:00
David Steele
46b7b72874 Add hint when unable to find the primary cluster.
If all available clusters are in recovery, pgBackRest will not be able to find a primary for the backup.
2022-05-06 18:23:36 -04:00
David Steele
53bfdbc01e Remove useless test in config/parse unit test.
Since the packSize field is 7 bits, it could never fail the check for > 127.

The compiler will catch any packs that are larger than 7 bits and then the pack size will need to be adjusted. For now just adjust the comment to reflect what the test does and give a clearer indication of what to do when a pack grows too large.
2022-05-06 16:02:44 -04:00
David Steele
77311a9af7 Fix indentation.
gcc11 complains about this indentation being misleading.
2022-05-06 15:38:03 -04:00
David Steele
efbcd975c4 Update Docker test image for Debian 9.
A change invalidated the current image which has been causing the d9 test to run longer.
2022-05-06 13:56:58 -04:00
David Steele
68a410779a Add zNewFmt().
This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
Reid Thompson
65d22e4325
Add verify output and verbose options.
These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
David Steele
e70c71049e Use uint8_t for optionResolveOrder.
This saves a bit of space and should not affect processing speed.

On MacOS (clang) this unexpectedly reduces the size of the binary by 16kiB but on Linux (gcc) there are no savings at all.
2022-05-06 07:49:23 -04:00
David Steele
808f7bf11c Replace strNewFmt() with TEST_ERROR_FMT() in command/archive-push module.
This test was likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:14:13 -04:00
David Steele
efe0a39a75 Use TEST_ERROR_FMT() rather than strNewFmt() in common/lock module.
These tests were likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:01:02 -04:00
David Steele
876f3bbd1c Remove COLON_STR and separator parameter from cfgParseCommandRoleName().
The separator parameter in cfgParseCommandRoleName() was useless since it was always set to : and COLON_STR did not provide any clarity its the single other usage.
2022-05-05 18:15:05 -04:00
David Steele
5f8c9cd66a
Add ClockError for unexpected clock skew and timezone changes.
A distinct result code should help debugging of clock skew and timezone issues.
2022-05-05 10:19:11 -04:00
David Steele
b6bfd9f99d
Strip extensions from history manifest before showing in error message.
In cases where clock skew or timezone issues are preventing backup label generation the user could see an error like this:

new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I.manifest.gz'

This will happen if the most recent label is drawn from the history. It is cleaner (and probably less confusing) to strip off the extensions so the user sees:

new backup label '20220504-152308F' is not later than latest backup label '20220504-222042F_20220504-222141I'
2022-05-05 09:20:49 -04:00
David Steele
ef672c74ad
Prevent memContextFree() from freeing memory needed by callbacks.
The order of callbacks and frees meant that memory needed during a callback (for logging in all known cases) might end up being freed before a callback needed it.

Requiring callbacks and logging to check the validity of their allocations is pretty risky and it is not clear that all possible cases have been accounted for.

Instead recursively execute all the callbacks first and then come back and recursively free the context. This is safer and it removes the need to check if a context is freeing so a simple active flag (in debug builds) will do. The caller no longer needs this information at all so remove memContextFreeing() and objMemContextFreeing().
2022-05-04 14:53:05 -04:00
Reid Thompson
d9088b2e2b
Show backup percent complete in info output.
In the JSON output the percent complete is storage as an integer of the percent complete * 100. So, before display it should be converted to double and divided by 100, or split using integer mod and div.

Note that percent complete will only be displayed on the host where the backup was executed. Remote hosts will show a backup/expire running with no percent complete.
2022-05-04 12:52:05 -04:00
David Steele
20782c88bc
PostgreSQL 15 support.
PostgreSQL 15 drops support for exclusive backup and renames the start/stop backup commands.

This is based on the pgdg-testing repo since beta1 has not been released yet, but it seems unlikely that breaking changes will be made at this point. beta1 should be tagged just before our next release so we'll retest before the release.
2022-05-04 11:55:59 -04:00
David Steele
692fe496bd
Remove dependency on pg_database.datlastsysoid.
This column has been removed in PostgreSQL 15. Rather than add a lot of special handling, it seems better just to update all versions to not depend on this column.

Add centralized functions to identify the type of database (i.e. system or user) by name and use FirstNormalObjectId when a name is not available.

The new query in the db module will still return the prior result for PostgreSQL <= 15, which will be stored in the manifest. This is important to preserve behavior when downgrading pgBackRest. There are no concerns here for PostgreSQL 15 since older versions of pgBackRest won't be able to restore backups for PostgreSQL 15 anyway.
2022-05-04 08:22:45 -04:00
David Steele
9a271e925c
Fix error thrown from FINALLY() causing an infinite loop.
Any error thrown resets execution to the last setjmp(), which means that parts of the try block need to make sure they don't get run again. FINALLY() was not doing this so if it threw an error it would end up back in the FINALLY() block, where the error would likely be thrown again, causing an infinite loop.

Fix this by tracking the state of FINALLY() and only running it once. This requires cleaning the error stack like CATCH*() and clearing the error like TRY_END() depending on the order of execution.
2022-05-03 14:34:05 -04:00
David Steele
b89c568b5f Fix obsolete variable naming. 2022-05-03 10:50:48 -04:00
David Steele
9629908694
Error on all lock failures except another process holding the lock.
The archive-get/archive-push commands would not error for, .e.g permissions errors, when attempting to get a lock before launching the async process. Since the async process was not launched there would be no error status file and the user would get a generic failure message. Also, there would be no async log.

Refactor lockAcquireFile() to throw an error when failOnNoLock = false unless the file is locked by another process. This seems to be the original intent of this parameter and there may have been a mistake when porting from Perl. In any case it looks wrong enough to be considered a bug.
2022-05-03 10:13:32 -04:00
David Steele
eb435becb3 Exclude mem context name from production builds.
The mem context name is used to produce clearer debug errors but it has no purpose in production builds.

Also remove memContextName() and access the struct directly since the name is only used within the common/memContext module.

Note that a few errors that were thrown in production builds (and required the name) are now only thrown in debug builds. In practice we have not seen these errors in production builds due to extensive coverage so it does not seem worth modifying the error to work without the context name.

This saves some memory, which is worthwhile, but the goal is to refactor Strings and Variants to have their own mem contexts and this change will prevent them from using more memory than they are now, along with other changes that will be coming later.
2022-05-02 15:17:34 -04:00
David Steele
0055fa40fe Add user:group to lock permission error.
This will help debug permissions errors when the lock file cannot be created.
2022-05-02 09:45:57 -04:00
David Steele
03c71aa606 Add hint to check the log on archive-get/archive-push async error.
If this error is thrown rather than a specific error returned from the async process, it means the async process is unable to write the status files for some reason and the only way to get the error is out of the async log.

This hint includes the exact async log path and name to make finding errors easier.
2022-05-02 08:49:13 -04:00
David Steele
4872a3f121 Improvements to test harness memory debugging.
Only set -DDEBUG_MEM for the modules currently being tested rather than globally.

Also run tests in a temp mem context. Running in the top context can confuse memory accounting when a new context is created in the top context.
2022-04-28 12:33:39 -04:00
David Steele
90f939b36f Fix leaks in common/io unit test.
These leaks make it harder to detect leaks in the core code, so fix them.
2022-04-28 12:31:59 -04:00
David Steele
8047e97e31 Fix leaked String and Variant in harnessPqScriptRun(). 2022-04-28 12:17:33 -04:00
David Steele
083c93eaa3 Reuse Strings in iniLoad().
Reuse the section/key/value Strings by truncating them instead of creating a new one every time.

Also add an error for empty sections. This function is only used for loading info files (not config files), which should never contain an empty section.
2022-04-28 10:11:15 -04:00