1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4206 Commits

Author SHA1 Message Date
Sam Bassaly
29d2f0f9fd
Add cast to handle compilers (e.g. MSVC) that coerce to signed int.
MSVC changes the sign in this case, presumable because of the subtraction. Cast so that MSVC does not also trigger a mixed sign warning.
2022-06-22 09:52:30 -04:00
David Steele
9fd85a104a Disable meson for all but debug builds.
The meson builds are still experimental so for now the configure/make build process is preferred for release builds. This message should help prevent any automated build systems from picking up meson instead.
2022-06-21 09:50:38 -04:00
David Steele
665da12ae7 Update meson.build include comment to be more general.
Eventually multiple subdirs will be included so update the comment to reflect that.
2022-06-17 16:41:48 -04:00
David Steele
5ecae90f02 Use constants rather than replacements when possible in test.c.
Some of the replacements that were being done already existed as constants, so use the constants instead.

Also fix a minor formatting error introduced when testAdd() was renamed to hrnAdd().
2022-06-17 16:31:48 -04:00
David Steele
dab1e4b6c6 Add strReplace().
Allows substrings to be replaced with another string.
2022-06-17 16:27:31 -04:00
David Steele
fb9acc1c93 Add higher level YAML functions.
These functions make parsing YAML simpler.
2022-06-17 16:17:02 -04:00
David Steele
55bcb933ee Move protocol module from common to command.
This module has dependencies on command/command so it does not make sense for it to be in the common module. Also move protocolFree() to main() since this is a very large dependency.

Adjust the tests so command/exit can be tested later. This is a bit messy but will get adjusted as we improve the test harness.
2022-06-17 11:17:52 -04:00
Stefan Fercot
eb72a80b47 Fix continuation character spacing. 2022-06-15 09:06:25 -04:00
Alexey Gordeev
716bba5800
Fix hard-coded WAL size assertion in walSegmentNext().
PG_WAL_SEGMENT_SIZE_DEFAULT is used to compare and check WAL size on pre-11 installations. However, there is a hard-coded assertion in walSegmentNext() which doesn't respect PG_WAL_SEGMENT_SIZE_DEFAULT.

Update the assertion to use PG_WAL_SEGMENT_SIZE_DEFAULT.
2022-06-14 08:13:22 -04:00
David Steele
282edda661 Remove storageLocal() dependency from common/lock module.
The storage/helper module is a very heavy dependency to introduce in the common module. Creating Posix storage objects is cheap so just do that instead.
2022-06-09 17:42:00 -04:00
David Steele
04f5ef25fa Add valgrind to Dockerfile missed in a16cf5ea. 2022-06-09 17:18:32 -04:00
David Steele
79443bea36 Move bldEnum() to the build/common/render module.
This function will be useful for other build/test modules so extern it.

Also skip the first upper-case when no prefix is provided.
2022-06-09 16:55:07 -04:00
Andrey Sokolov
1a00ab1003
Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG.
If DEBUG is not defined then the ASSERT() macro expands to nothing. In this case the timeBegin variable is never used and a compilation error occurs.

This test should work without DEBUG defined so use CHECK() instead of ASSERT().
2022-06-09 07:34:11 -04:00
Sam Bassaly
0dabf88e9d
Add FN_NO_RETURN macro.
Change all instances of __attribute__((__noreturn__)) to a macro in meson.build / build.auto.h.in.

As compiler attributes written in the form of __attribute__ are not supported by MSVC, this is one of several commits to make the code-base more robust and allow using MSVC-style attributes later.
2022-06-08 17:43:23 -04:00
David Steele
8babd558bc Add missing build.auto.h includes.
These are required for proper building but were probably not noticed before since they are not part of the core code.
2022-06-08 07:26:15 -04:00
David Steele
a16cf5eac7 Update CI to use Ubuntu 22.04 and Fedora 36.
Both have newer gcc and OpenSSL 3.

Fedora 36 runs horribly slow with valgrind enabled so run the valgrind tests on Ubuntu 22.04. Fedora 36 has a newer gcc so it is still worth testing on.
2022-06-06 16:32:20 -04:00
David Steele
08242ee6ac
OpenSSL 3 support.
There are two changes:

* Suppress deprecation warnings so we can build with -Werror and -Wfatal-errors. At some point we'll need to migrate to the new APIs but there does not seem to be a good reason to support two sets of code right now.

* Update the handling for unexpected EOF to handle EOF or error. The error code for EOF has changed and become harder to identify, but we probably don't care whether it is an error or EOF.
2022-06-06 14:47:47 -04:00
David Steele
f92ce674f7
Automatically create PostgreSQL version interfaces.
Maintaining the version interfaces was complicated by the fact that each interface needed to be in separate compilation unit to avoid type conflicts. This also meant that various build/test files needed to be updated to add the new interfaces.

Solve these problems by auto-generating all the interfaces into a single file. This is made possible by parsing defines and types out of the header files and creating macros to rename the types. At the end of the version interface everything is undef'd. Another benefit is that the auto-generated interfaces can be static and included directly into postgres/interface.c.

Since some code generation is now always required for tests, change --no-gen to --min-gen in test.pl.

It would also make sense to auto-generate the version defines in postgres/version.h, but that will be left for a future commit.
2022-06-06 13:52:56 -04:00
David Steele
b8fc20d5b8
Add experimental Meson build.
Meson is a new build system that offers simpler syntax and superior performance to autoconf/make. In addition, Windows is supported natively.

The Meson build appears complete, but currently is used only for auto-generation of code and the host build of pgbackrest. Some container upgrades will be required before Meson can be used for container builds.

Also patch the Debian package to force autoconf/make rather than Meson.
2022-06-03 14:13:56 -04:00
David Steele
148956aed8 Remove useless command/check unit test.
This test was a placeholder and did not provide any coverage, but it did give inconsistent results on different shell versions.
2022-06-01 10:13:57 -04:00
Andrey Sokolov
29b2a54fcc
Allow any path to be passed to the --test-path option.
A hard-coded path prevented this from working correctly.
2022-05-31 17:28:58 -04:00
David Steele
be354c4890 Update CodeQL to version 2.
Version 1 will be deprecated on December 12, so upgrade now to avoid nasty surprises.
2022-05-31 16:52:26 -04:00
David Steele
2c38c9a56f Skip stopping PostgreSQL 9.1 in real/all integration test.
Stopping the cluster has started consistently running out of memory on PostgreSQL 9.1. This seems to have happened after pulling in new packages at some point so it might be build related.

Stopping the cluster is not critical for 9.1 so skip it.
2022-05-31 16:43:18 -04:00
David Steele
2feaaeaac8 Add .inc extension to C files included in other C files.
These files were never intended to be compiled on their own so the .c extension was a bit misleading. In particular Meson does not like .c files that are not intended to be compiled independently.

Leave header files as is since they are already protected against being included more than once and are never expected to be compiled.
2022-05-31 16:06:41 -04:00
David Steele
cb891fa2d4 Run remaining Github Actions CI on Ubuntu 20.04.
The s390x/ppc64le tests are already running on 20.04.
2022-05-31 13:02:51 -04:00
David Steele
2643050be0 Skip internal options in the configuration reference.
Most internal options were being skipped, but not in the case where an option was marked internal for a specific command.

The command-line help was not affected by this issue.
2022-05-31 12:36:21 -04:00
David Steele
f0aae6ceab Remote repo options should not be valid for the backup command.
Some of the remote repo options were gated by repo-local, but the rest relied on repo-host-cmd.

Remove backup from the repo-host-cmd option since none of the dependent options are valid for backup.
2022-05-31 12:26:57 -04:00
David Steele
a902c7808d Make all repo-* options visible for stanza create/update/delete.
31c7824a allowed these commands to run remotely but neglected to remove some internal flags, which prevented all the repo-* options from being visible in the documentation.
2022-05-31 12:24:21 -04:00
Reid Thompson
60d70fa66f
Disable incremental manifest save when resume=n.
The manifest is saved on a regular basis during a backup so a failed backup can be resumed. For backups that the user has configured/invoked as not resumable, skip the incremental save of the manifest.
2022-05-25 18:18:37 -04:00
David Steele
8c2b3a044f Use StringId for type in cryptoHash() and cryptoHmacOne().
This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.
2022-05-25 15:27:53 -04:00
Reid Thompson
38ad2838d5
Set backup percent complete to zero before copy start.
Waiting to write percent complete until the first file completed resulted in a period of time where the backup was running without status available to the user.

Remedy this by initializing percent complete to zero when the backup is ready to start copying files.
2022-05-25 14:20:19 -04:00
David Steele
7b1935c710 Remove unused errors.
Most of these were probably never ported from Perl to C and others became obsolete over time.

Fix one error that was the wrong type.

Also fix/improve some comments.
2022-05-25 10:23:43 -04:00
David Steele
f6f2f2e2a3 Add missing static keywords.
Add static keyword to local variables where missing.
2022-05-25 09:42:49 -04:00
David Steele
7ec51e7e62
Truncate files during delta restore when they are larger than expected.
Previously the behavior was to download the file from the repository when it was not exactly the same size in PGDATA. However, it may just be that the file was extended and the contents are the same up to the file size recorded in the manifest. This could also be very valuable for files that are always append only, like logs.

Change info.size to file->size in one place. Both are technically correct but file->size makes more sense.

Use the new fileName variable in a few existing places.

Also adjust some existing comments to make them clearer.
2022-05-24 16:39:35 -04:00
David Steele
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
David Steele
69adb990dc Use storagePathP() instead of cfgOptionStr() to get base repo path.
cfgOptionStr() may not have the correct value if the repo is remote.

Use storagePathP() instead since it can ask the remote for the correct value when required.
2022-05-19 12:25:58 -04:00
David Steele
c7a66ac1af
Improve memory usage of mem contexts.
Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible.

This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case.

Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed.

Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree().

Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
David Steele
83af3f1b7a Add additional detail to warnings when delta checksum is auto-enabled.
Hopefully this will help with debugging when it is not clear why delta checksum is being enabled.
2022-05-18 08:48:48 -04:00
David Steele
5dfd00bb6c Fix RHEL container build for documentation.
For some reason /lib/systemd/system/sysinit.target.wants no longer exists in the rockylinux:8 container.

Create this directory explicitly in case it does not exist.
2022-05-18 08:18:34 -04:00
David Steele
5360f2ec0a Fix comment indentation. 2022-05-16 10:50:07 -04:00
David Steele
b598f49ded Udpate parse.auto.c with labels from 4dcc9df2.
Committed separately so it can be ignored in history/blame.
2022-05-16 09:17:00 -04:00
David Steele
4dcc9df222 Add labels in parse.auto.c to make diffs easier to read.
Because there is a lot of repetition in this file, changes can look very jumbled with existing data in a diff. Also, if can be hard to tell what is being modified if the diff does not show enough lines before and after.

This change adds labels to the end of the line to localize the diff and make it easier to see what has been changed. Also, remove some linefeeds and make separators more consistent.

The change to parse.auto.c will be committed separately so it can be ignored in history/blame.
2022-05-16 09:14:46 -04:00
David Steele
f5023a769d Update config.guess to latest version. 2022-05-16 08:54:19 -04:00
David Steele
243eef1e52 Begin v2.40 development. 2022-05-16 08:51:37 -04:00
David Steele
901e829f6d v2.39: Verify and File Bundling
Bug Fixes:

* Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)
* Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.)

Features:

* Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)
* Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)
* PostgreSQL 15 support. (Reviewed by Stefan Fercot.)
* Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)
* Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)

Improvements:

* Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)
* Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)
* Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)
* Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)
* Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
* Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)
* Add user:group to lock permission error. (Reviewed by Reid Thompson.)

Documentation Bug Fixes:

* Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)
* Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)
* Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)

Documentation Improvements:

* Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)
* Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)
* Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)
* Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
2022-05-16 08:46:24 -04:00
David Steele
19dd015d58
Fix issues in improved path validation for repo-* commands.
If the user requested the exact repo path then strSub() would be passed an invalid start value leading to an assertion:

$ pgbackrest --stanza=test repo-ls /var/lib/pgbackrest
ASSERT: [025]: start <= this->pub.size (on dev builds)
ASSERT: [025]: string size must be <= 1073741824 bytes (on prod builds)

Fix this by checking if the requested path exactly equals the repo path and returning an empty relative path in this case.

Another issue was that invalid subpaths were not detected if they started with the repo path. For example, /var/lib/pgbackrestsub would not generate an error if the repo path was /var/lib/pgbackrest. Fix this by explictly checking for a / between the repo path and the subpath. This also requires special handling when the repo path is /.

This is not a live bug since the issues were found in an unreleased feature introduced in 5ae84d5.
2022-05-13 09:41:53 -04:00
David Steele
024500782e Reduce encrypted WAL segments sizes in command tests.
The encrypted archive-push and repo tests were running very slowly on 32-bit with Valgrind enabled. This appears to be an issue with a newer version of Valgrind, but it has been going on long enough that bisecting does not seem to be worthwhile.

Reduce the size of the encrypted test segments where possible to improve overall test performance.
2022-05-12 17:02:08 -04:00
David Steele
7d9b2e267c Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.
Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.
2022-05-12 11:57:12 -04:00
David Steele
24f138b067 Remove excessive const usage in build module.
This was an experiment that attempted to create immutable structs (at least without casting). It turned out to be a bit burdensome and required unsafe-looking casting in some cases.
2022-05-11 16:20:41 -04:00
David Steele
3d8ee552fe Move some PostgreSQL integration tests to rh7.
This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.
2022-05-11 15:59:37 -04:00