1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

4127 Commits

Author SHA1 Message Date
David Steele
8c2b3a044f Use StringId for type in cryptoHash() and cryptoHmacOne().
This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.
2022-05-25 15:27:53 -04:00
Reid Thompson
38ad2838d5
Set backup percent complete to zero before copy start.
Waiting to write percent complete until the first file completed resulted in a period of time where the backup was running without status available to the user.

Remedy this by initializing percent complete to zero when the backup is ready to start copying files.
2022-05-25 14:20:19 -04:00
David Steele
7b1935c710 Remove unused errors.
Most of these were probably never ported from Perl to C and others became obsolete over time.

Fix one error that was the wrong type.

Also fix/improve some comments.
2022-05-25 10:23:43 -04:00
David Steele
f6f2f2e2a3 Add missing static keywords.
Add static keyword to local variables where missing.
2022-05-25 09:42:49 -04:00
David Steele
7ec51e7e62
Truncate files during delta restore when they are larger than expected.
Previously the behavior was to download the file from the repository when it was not exactly the same size in PGDATA. However, it may just be that the file was extended and the contents are the same up to the file size recorded in the manifest. This could also be very valuable for files that are always append only, like logs.

Change info.size to file->size in one place. Both are technically correct but file->size makes more sense.

Use the new fileName variable in a few existing places.

Also adjust some existing comments to make them clearer.
2022-05-24 16:39:35 -04:00
David Steele
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
David Steele
69adb990dc Use storagePathP() instead of cfgOptionStr() to get base repo path.
cfgOptionStr() may not have the correct value if the repo is remote.

Use storagePathP() instead since it can ask the remote for the correct value when required.
2022-05-19 12:25:58 -04:00
David Steele
c7a66ac1af
Improve memory usage of mem contexts.
Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible.

This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case.

Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed.

Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree().

Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
David Steele
83af3f1b7a Add additional detail to warnings when delta checksum is auto-enabled.
Hopefully this will help with debugging when it is not clear why delta checksum is being enabled.
2022-05-18 08:48:48 -04:00
David Steele
5dfd00bb6c Fix RHEL container build for documentation.
For some reason /lib/systemd/system/sysinit.target.wants no longer exists in the rockylinux:8 container.

Create this directory explicitly in case it does not exist.
2022-05-18 08:18:34 -04:00
David Steele
5360f2ec0a Fix comment indentation. 2022-05-16 10:50:07 -04:00
David Steele
b598f49ded Udpate parse.auto.c with labels from 4dcc9df2.
Committed separately so it can be ignored in history/blame.
2022-05-16 09:17:00 -04:00
David Steele
4dcc9df222 Add labels in parse.auto.c to make diffs easier to read.
Because there is a lot of repetition in this file, changes can look very jumbled with existing data in a diff. Also, if can be hard to tell what is being modified if the diff does not show enough lines before and after.

This change adds labels to the end of the line to localize the diff and make it easier to see what has been changed. Also, remove some linefeeds and make separators more consistent.

The change to parse.auto.c will be committed separately so it can be ignored in history/blame.
2022-05-16 09:14:46 -04:00
David Steele
f5023a769d Update config.guess to latest version. 2022-05-16 08:54:19 -04:00
David Steele
243eef1e52 Begin v2.40 development. 2022-05-16 08:51:37 -04:00
David Steele
901e829f6d v2.39: Verify and File Bundling
Bug Fixes:

* Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)
* Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.)

Features:

* Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)
* Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)
* PostgreSQL 15 support. (Reviewed by Stefan Fercot.)
* Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)
* Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)

Improvements:

* Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)
* Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)
* Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)
* Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)
* Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
* Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)
* Add user:group to lock permission error. (Reviewed by Reid Thompson.)

Documentation Bug Fixes:

* Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)
* Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)
* Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)

Documentation Improvements:

* Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)
* Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)
* Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
2022-05-16 08:46:24 -04:00
David Steele
19dd015d58
Fix issues in improved path validation for repo-* commands.
If the user requested the exact repo path then strSub() would be passed an invalid start value leading to an assertion:

$ pgbackrest --stanza=test repo-ls /var/lib/pgbackrest
ASSERT: [025]: start <= this->pub.size (on dev builds)
ASSERT: [025]: string size must be <= 1073741824 bytes (on prod builds)

Fix this by checking if the requested path exactly equals the repo path and returning an empty relative path in this case.

Another issue was that invalid subpaths were not detected if they started with the repo path. For example, /var/lib/pgbackrestsub would not generate an error if the repo path was /var/lib/pgbackrest. Fix this by explictly checking for a / between the repo path and the subpath. This also requires special handling when the repo path is /.

This is not a live bug since the issues were found in an unreleased feature introduced in 5ae84d5.
2022-05-13 09:41:53 -04:00
David Steele
024500782e Reduce encrypted WAL segments sizes in command tests.
The encrypted archive-push and repo tests were running very slowly on 32-bit with Valgrind enabled. This appears to be an issue with a newer version of Valgrind, but it has been going on long enough that bisecting does not seem to be worthwhile.

Reduce the size of the encrypted test segments where possible to improve overall test performance.
2022-05-12 17:02:08 -04:00
David Steele
7d9b2e267c Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.
Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.
2022-05-12 11:57:12 -04:00
David Steele
24f138b067 Remove excessive const usage in build module.
This was an experiment that attempted to create immutable structs (at least without casting). It turned out to be a bit burdensome and required unsafe-looking casting in some cases.
2022-05-11 16:20:41 -04:00
David Steele
3d8ee552fe Move some PostgreSQL integration tests to rh7.
This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.
2022-05-11 15:59:37 -04:00
Reid Thompson
a913113fda
Add option type descriptions.
This cuts down on repetition of the size descriptions and adds basic descriptions for the other option types.
2022-05-11 15:23:41 -04:00
David Steele
5fbea6da81 Add br tag for documentation.
This tag allows for a simple linefeed in a p tag instead of being forced to start a new paragraph.
2022-05-11 10:39:31 -04:00
David Steele
8ee85bc605 Fix "that that" typos. 2022-05-11 08:46:23 -04:00
Reid Thompson
c4f7edef2b Fix typos in help. 2022-05-11 08:42:46 -04:00
David Christensen
50d409a812
Add FAQ about backup types and restore speed.
Based on several questions/misunderstandings, provide clarification about the backup type only affecting the backup action, and not the restore.
2022-05-10 14:17:05 -04:00
David Steele
de816a0f57
Remove integration expect log testing.
Integration expect log testing was originally used as a rough-and-ready way to make sure that certain code paths were being executed before the unit tests existed. Now that we have 100% unit test coverage (with expect log testing) the value of the integration expect tests seems minimal at best.

But they do cause numerous issues:

- Maintenance of the expect code and replacements that are required to keep logs reproducible.
- Even a trivial change can cause massive churn in the expect logs, e.g. d9088b2. These changes should be minutely audited but since the expect logs have little value now it is seldom worth the effort.
- The OS version used to do expect testing (RHEL7) can only be used to test one version of PostgreSQL. This makes it hard to balance the PostgreSQL version testing between OS versions.
- When a commit affects expect logs it is not clear (especially for new developers) how to regenerate them and our contributing guide is silent on the issue.

The goal is to migrate the integration tests to C and expect testing is not part of that plan. It seems best to get rid of them now.
2022-05-10 13:18:26 -04:00
David Steele
3a40394472 Remove obsolete test in common/memContext.
Once upon a time the allocation array was allocated up front so this test was required for the top context, which did not allocate up front.

Now allocations are done on demand so this case is covered for every context that does not allocate memory.
2022-05-10 11:18:12 -04:00
Reid Thompson
6b98b3534e Fix typo. 2022-05-10 06:52:56 -04:00
David Christensen
cc5b061489
Document required base branch for pull requests.
Be explicit when submitting a PR about which branch to use as the base.
2022-05-09 18:07:11 -04:00
David Steele
b4c1ca7b80 Split 32-bit CI tests.
This helps rebalance some of the tests that are running long, i.e. d9 and u20.

I would be better to move more PostgreSQL versions to d9, but the base VM does not contain more versions. New minor versions will be out later in the week so that seems a better time to be rebuilding containers.
2022-05-09 14:19:05 -04:00
David Steele
39dddbb6bc Add limited CI for ppc64le/s390x using emulation.
The emulation is so slow that running all the unit tests would be too expensive, but this at least shows that the build works and some of the more complex tests run. In particular, it is good to test on one big-endian architecture to be sure that checksums are correct.

Update checksums in the tests where they had gotten out of date since the last time we were testing on s390x. Also use a different test in command/archivePushTest to show the name of the file when a checksum does not match to aid in debugging.

The command/archive-push test was updated but not included because there is also a permissions issue, which looks to be the same as what we see on MacOS/FreedBSD. Hopefully we'll be able to fix all of those at the same time.
2022-05-09 12:48:19 -04:00
David Steele
eefa0b161a
Simplify messaging around supported versions in the documentation.
The version ranges given in the user guides caused confusion. For example, because the user guide for RHEL specified PostgreSQL 9.6-11, users questioned whether pgBackRest worked for PostgreSQL 12 on RHEL.

Remove these ranges and add more explanatory text to the introduction to try and make it clearer how the user guides work and which versions are covered (basically all of them).
2022-05-09 11:59:08 -04:00
David Steele
ef4c4ab852
Use variable instead of function to track FINALLY() state.
The function worked fine, but Coverity was unable to determine that the finally block was run, which led to false positives about unfreed memory.

Using a boolean in the block makes it clear to Coverity that the finally block will always be run no matter what else happens.

We'll depend on the compiler to optimize away the boolean if it is not used in a finally block. The cost of the boolean is fairly low in comparison to everything else being done in these macros, so it does not seem worth having a separate block even if the compiler is not able to eliminate the boolean.

This reverts most of 9a271e9 that fixed a bug caused by c5b5b58, which was also attempting to help Coverity understand FINALLY() blocks.
2022-05-09 10:39:43 -04:00
David Steele
e8c40a24df Remove unnecessary TRY() block in common/regExp module.
This code was written before MEM_CONTEXT_TEMP*() was available, which is a better solution.
2022-05-09 09:56:19 -04:00
David Steele
4d8c36715d Remove legacy Travis-CI configuration.
Travis-CI is now strictly a paid service. Multiple attempts to use their "free" service have failed due to lack of community credit and general issues with their plugin.

Remove the configuration so it does not appear we are testing on Travis-CI.
2022-05-06 19:44:46 -04:00
David Steele
46b7b72874 Add hint when unable to find the primary cluster.
If all available clusters are in recovery, pgBackRest will not be able to find a primary for the backup.
2022-05-06 18:23:36 -04:00
David Steele
53bfdbc01e Remove useless test in config/parse unit test.
Since the packSize field is 7 bits, it could never fail the check for > 127.

The compiler will catch any packs that are larger than 7 bits and then the pack size will need to be adjusted. For now just adjust the comment to reflect what the test does and give a clearer indication of what to do when a pack grows too large.
2022-05-06 16:02:44 -04:00
David Steele
77311a9af7 Fix indentation.
gcc11 complains about this indentation being misleading.
2022-05-06 15:38:03 -04:00
David Steele
efbcd975c4 Update Docker test image for Debian 9.
A change invalidated the current image which has been causing the d9 test to run longer.
2022-05-06 13:56:58 -04:00
David Steele
68a410779a Add zNewFmt().
This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
David Steele
475e7c692d Clean up dividers in the documentation.
Dividers were used in some files, but not others, and some had section names (which are hard to maintain) and others did not.

Try to make this more consistent by putting a divider on front of every section, variable block, and wherever else seems appropriate.
2022-05-06 12:11:04 -04:00
David Steele
356bc27bf2 Remove key dividers in help.xml.
The idea was to make this file easier to browse and edit, but in fact it is much easier to just search for the command/option needed.

The dividers were never applied consistently and at some point we decided to get rid of the comments because they were hard to keep updated. The result was a mix of styles which did nobody any favors.
2022-05-06 11:41:28 -04:00
Reid Thompson
65d22e4325
Add verify output and verbose options.
These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
David Steele
f405fc6ae2
Backup file bundling documentation.
Make the feature user visible and add documentation to the user guide.
2022-05-06 10:21:20 -04:00
Reid Thompson
4cc0d46d60 Fix comment wrapping. 2022-05-06 09:34:39 -04:00
David Steele
e70c71049e Use uint8_t for optionResolveOrder.
This saves a bit of space and should not affect processing speed.

On MacOS (clang) this unexpectedly reduces the size of the binary by 16kiB but on Linux (gcc) there are no savings at all.
2022-05-06 07:49:23 -04:00
David Steele
808f7bf11c Replace strNewFmt() with TEST_ERROR_FMT() in command/archive-push module.
This test was likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:14:13 -04:00
David Steele
efe0a39a75 Use TEST_ERROR_FMT() rather than strNewFmt() in common/lock module.
These tests were likely written before TEST_ERROR_FMT() existed.
2022-05-05 20:01:02 -04:00
David Steele
5089a26633 Convert strNewFmt() to THROW_FMT() in config/parse module.
It's not clear why strNewFmt() was used here, but there is no need for it.
2022-05-05 18:35:00 -04:00