1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

4268 Commits

Author SHA1 Message Date
David Steele
f9ac53db92 Fix typo. 2022-07-08 14:38:14 -04:00
David Steele
74a4ac801d
Add link to PostgreSQL configuration in repository host section.
This should make the documentation clearer when starting from this section.
2022-07-08 11:13:55 -04:00
David Steele
0eccbc8bf4 Meson builds work on version 0.45.
v0.45 ships with Ubuntu 18.04, which is currently the oldest distro we support. We may never do a Meson release on Ubuntu 18.04 but this allows us to start running unit tests with Meson in the meantime.

Some more granular options are not available so we use buildtype in more places.

The check for a in-tree autoconf/make build had to be removed since the filesystem APIs are not available.

Finally, alias_target was removed. This means that full paths must be used for build targets, which does not seem too bad. For instance, test/src/test-pgbackrest must now be used as a build target instead of simple test-pgbackrest.
2022-07-06 18:17:52 -04:00
David Steele
72960bbf17 Rename strTrunc() to strTruncIdx() and add strTrunc().
strTrunc() is now shorthand for truncating to index 0. This is convenient when a string is being reused.
2022-07-06 09:55:32 -04:00
David Steele
cd25cec147 Add comment to indicate that type is in a different info level. 2022-07-05 14:34:31 -04:00
David Steele
1c0bf0b15d Reorder null user/group checks in remote protocol and add tests.
Coverage for these checks was dependent on the order the files were read from disk, which made the tests fragile.

Rearrange the checks and add a test that won't depend on order.
2022-07-05 14:28:40 -04:00
David Steele
326d152a14 Update contributor name. 2022-07-05 06:58:19 -04:00
David Steele
845d82f682
Use S3 IsTruncated flag to determine list continuation.
Previously we were just checking for the existence of NextContinuationToken, which the S3 documentation indicates will not be present when the list is not truncated. However, recent versions of Scality send a blank NextContinuationToken when IsTruncated is false. Sending the blank continuation token back causes Scality to send another blank continuation token and an infinite loop occurs.

Instead use IsTruncated (which is required to be present) to determine whether NextContinuationToken should be present. Error if NextContinuationToken is then missing or empty, since an empty token caused an infinite loop with the Scality server (which arguably should have errored when passed an empty token).
2022-07-01 20:32:10 -04:00
David Steele
61ca9b58a0 Replace hrnStorageInfoListCallback() with TEST_STORAGE_LIST() in tests.
The TEST_STORAGE_LIST() macro is more robust and hides the callback mechanism from the caller.

Add features to TEST_STORAGE_LIST() that hrnStorageInfoListCallback() had.

Update tests to use the abbreviated type output (e.g. path/) generated by TEST_STORAGE_LIST().
2022-06-28 14:15:08 -04:00
David Steele
59f148bf6e Generate -fmacro-prefix-map to strip relative path from __FILE__.
This provides reproducible builds and minimizes the file path in debug messages, just like an in-tree make build.

For test source, prefix with test/ in case there are any module name collisions.
2022-06-26 08:42:43 -04:00
David Steele
f863fc9888 Add experimental unit test harness written in C.
Having the test harness in C will allow us to remove duplicated Perl code and test on systems where Perl support is not present.

Custom harnesses and shims are currently not implemented, which means only the following tests in the common module will run: error, stack-trace, type-convert, assert-on, mem-context, time, encode, type-object, type-string, type-list, type-buffer, type-variant, reg-exp, log.

The experimental test harness is being committed with partial functionality so it can be used in Windows development. The remaining features will follow as needed.
2022-06-23 12:20:56 -04:00
David Steele
b7a1b3ec2c Used DEBUG instead of NDEBUG in common/stackTrace module.
In some testing cases these might not be in sync, which causes unpredictable behavior.
2022-06-23 10:57:58 -04:00
Sam Bassaly
29d2f0f9fd
Add cast to handle compilers (e.g. MSVC) that coerce to signed int.
MSVC changes the sign in this case, presumable because of the subtraction. Cast so that MSVC does not also trigger a mixed sign warning.
2022-06-22 09:52:30 -04:00
David Steele
9fd85a104a Disable meson for all but debug builds.
The meson builds are still experimental so for now the configure/make build process is preferred for release builds. This message should help prevent any automated build systems from picking up meson instead.
2022-06-21 09:50:38 -04:00
David Steele
665da12ae7 Update meson.build include comment to be more general.
Eventually multiple subdirs will be included so update the comment to reflect that.
2022-06-17 16:41:48 -04:00
David Steele
5ecae90f02 Use constants rather than replacements when possible in test.c.
Some of the replacements that were being done already existed as constants, so use the constants instead.

Also fix a minor formatting error introduced when testAdd() was renamed to hrnAdd().
2022-06-17 16:31:48 -04:00
David Steele
dab1e4b6c6 Add strReplace().
Allows substrings to be replaced with another string.
2022-06-17 16:27:31 -04:00
David Steele
fb9acc1c93 Add higher level YAML functions.
These functions make parsing YAML simpler.
2022-06-17 16:17:02 -04:00
David Steele
55bcb933ee Move protocol module from common to command.
This module has dependencies on command/command so it does not make sense for it to be in the common module. Also move protocolFree() to main() since this is a very large dependency.

Adjust the tests so command/exit can be tested later. This is a bit messy but will get adjusted as we improve the test harness.
2022-06-17 11:17:52 -04:00
Stefan Fercot
eb72a80b47 Fix continuation character spacing. 2022-06-15 09:06:25 -04:00
Alexey Gordeev
716bba5800
Fix hard-coded WAL size assertion in walSegmentNext().
PG_WAL_SEGMENT_SIZE_DEFAULT is used to compare and check WAL size on pre-11 installations. However, there is a hard-coded assertion in walSegmentNext() which doesn't respect PG_WAL_SEGMENT_SIZE_DEFAULT.

Update the assertion to use PG_WAL_SEGMENT_SIZE_DEFAULT.
2022-06-14 08:13:22 -04:00
David Steele
282edda661 Remove storageLocal() dependency from common/lock module.
The storage/helper module is a very heavy dependency to introduce in the common module. Creating Posix storage objects is cheap so just do that instead.
2022-06-09 17:42:00 -04:00
David Steele
04f5ef25fa Add valgrind to Dockerfile missed in a16cf5ea. 2022-06-09 17:18:32 -04:00
David Steele
79443bea36 Move bldEnum() to the build/common/render module.
This function will be useful for other build/test modules so extern it.

Also skip the first upper-case when no prefix is provided.
2022-06-09 16:55:07 -04:00
Andrey Sokolov
1a00ab1003
Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG.
If DEBUG is not defined then the ASSERT() macro expands to nothing. In this case the timeBegin variable is never used and a compilation error occurs.

This test should work without DEBUG defined so use CHECK() instead of ASSERT().
2022-06-09 07:34:11 -04:00
Sam Bassaly
0dabf88e9d
Add FN_NO_RETURN macro.
Change all instances of __attribute__((__noreturn__)) to a macro in meson.build / build.auto.h.in.

As compiler attributes written in the form of __attribute__ are not supported by MSVC, this is one of several commits to make the code-base more robust and allow using MSVC-style attributes later.
2022-06-08 17:43:23 -04:00
David Steele
8babd558bc Add missing build.auto.h includes.
These are required for proper building but were probably not noticed before since they are not part of the core code.
2022-06-08 07:26:15 -04:00
David Steele
a16cf5eac7 Update CI to use Ubuntu 22.04 and Fedora 36.
Both have newer gcc and OpenSSL 3.

Fedora 36 runs horribly slow with valgrind enabled so run the valgrind tests on Ubuntu 22.04. Fedora 36 has a newer gcc so it is still worth testing on.
2022-06-06 16:32:20 -04:00
David Steele
08242ee6ac
OpenSSL 3 support.
There are two changes:

* Suppress deprecation warnings so we can build with -Werror and -Wfatal-errors. At some point we'll need to migrate to the new APIs but there does not seem to be a good reason to support two sets of code right now.

* Update the handling for unexpected EOF to handle EOF or error. The error code for EOF has changed and become harder to identify, but we probably don't care whether it is an error or EOF.
2022-06-06 14:47:47 -04:00
David Steele
f92ce674f7
Automatically create PostgreSQL version interfaces.
Maintaining the version interfaces was complicated by the fact that each interface needed to be in separate compilation unit to avoid type conflicts. This also meant that various build/test files needed to be updated to add the new interfaces.

Solve these problems by auto-generating all the interfaces into a single file. This is made possible by parsing defines and types out of the header files and creating macros to rename the types. At the end of the version interface everything is undef'd. Another benefit is that the auto-generated interfaces can be static and included directly into postgres/interface.c.

Since some code generation is now always required for tests, change --no-gen to --min-gen in test.pl.

It would also make sense to auto-generate the version defines in postgres/version.h, but that will be left for a future commit.
2022-06-06 13:52:56 -04:00
David Steele
b8fc20d5b8
Add experimental Meson build.
Meson is a new build system that offers simpler syntax and superior performance to autoconf/make. In addition, Windows is supported natively.

The Meson build appears complete, but currently is used only for auto-generation of code and the host build of pgbackrest. Some container upgrades will be required before Meson can be used for container builds.

Also patch the Debian package to force autoconf/make rather than Meson.
2022-06-03 14:13:56 -04:00
David Steele
148956aed8 Remove useless command/check unit test.
This test was a placeholder and did not provide any coverage, but it did give inconsistent results on different shell versions.
2022-06-01 10:13:57 -04:00
Andrey Sokolov
29b2a54fcc
Allow any path to be passed to the --test-path option.
A hard-coded path prevented this from working correctly.
2022-05-31 17:28:58 -04:00
David Steele
be354c4890 Update CodeQL to version 2.
Version 1 will be deprecated on December 12, so upgrade now to avoid nasty surprises.
2022-05-31 16:52:26 -04:00
David Steele
2c38c9a56f Skip stopping PostgreSQL 9.1 in real/all integration test.
Stopping the cluster has started consistently running out of memory on PostgreSQL 9.1. This seems to have happened after pulling in new packages at some point so it might be build related.

Stopping the cluster is not critical for 9.1 so skip it.
2022-05-31 16:43:18 -04:00
David Steele
2feaaeaac8 Add .inc extension to C files included in other C files.
These files were never intended to be compiled on their own so the .c extension was a bit misleading. In particular Meson does not like .c files that are not intended to be compiled independently.

Leave header files as is since they are already protected against being included more than once and are never expected to be compiled.
2022-05-31 16:06:41 -04:00
David Steele
cb891fa2d4 Run remaining Github Actions CI on Ubuntu 20.04.
The s390x/ppc64le tests are already running on 20.04.
2022-05-31 13:02:51 -04:00
David Steele
2643050be0 Skip internal options in the configuration reference.
Most internal options were being skipped, but not in the case where an option was marked internal for a specific command.

The command-line help was not affected by this issue.
2022-05-31 12:36:21 -04:00
David Steele
f0aae6ceab Remote repo options should not be valid for the backup command.
Some of the remote repo options were gated by repo-local, but the rest relied on repo-host-cmd.

Remove backup from the repo-host-cmd option since none of the dependent options are valid for backup.
2022-05-31 12:26:57 -04:00
David Steele
a902c7808d Make all repo-* options visible for stanza create/update/delete.
31c7824a allowed these commands to run remotely but neglected to remove some internal flags, which prevented all the repo-* options from being visible in the documentation.
2022-05-31 12:24:21 -04:00
Reid Thompson
60d70fa66f
Disable incremental manifest save when resume=n.
The manifest is saved on a regular basis during a backup so a failed backup can be resumed. For backups that the user has configured/invoked as not resumable, skip the incremental save of the manifest.
2022-05-25 18:18:37 -04:00
David Steele
8c2b3a044f Use StringId for type in cryptoHash() and cryptoHmacOne().
This brings hash types in line with cipher types (i.e. a StringId enum) and allows removal of some Strings.
2022-05-25 15:27:53 -04:00
Reid Thompson
38ad2838d5
Set backup percent complete to zero before copy start.
Waiting to write percent complete until the first file completed resulted in a period of time where the backup was running without status available to the user.

Remedy this by initializing percent complete to zero when the backup is ready to start copying files.
2022-05-25 14:20:19 -04:00
David Steele
7b1935c710 Remove unused errors.
Most of these were probably never ported from Perl to C and others became obsolete over time.

Fix one error that was the wrong type.

Also fix/improve some comments.
2022-05-25 10:23:43 -04:00
David Steele
f6f2f2e2a3 Add missing static keywords.
Add static keyword to local variables where missing.
2022-05-25 09:42:49 -04:00
David Steele
7ec51e7e62
Truncate files during delta restore when they are larger than expected.
Previously the behavior was to download the file from the repository when it was not exactly the same size in PGDATA. However, it may just be that the file was extended and the contents are the same up to the file size recorded in the manifest. This could also be very valuable for files that are always append only, like logs.

Change info.size to file->size in one place. Both are technically correct but file->size makes more sense.

Use the new fileName variable in a few existing places.

Also adjust some existing comments to make them clearer.
2022-05-24 16:39:35 -04:00
David Steele
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
David Steele
69adb990dc Use storagePathP() instead of cfgOptionStr() to get base repo path.
cfgOptionStr() may not have the correct value if the repo is remote.

Use storagePathP() instead since it can ask the remote for the correct value when required.
2022-05-19 12:25:58 -04:00
David Steele
c7a66ac1af
Improve memory usage of mem contexts.
Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible.

This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case.

Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed.

Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree().

Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
David Steele
83af3f1b7a Add additional detail to warnings when delta checksum is auto-enabled.
Hopefully this will help with debugging when it is not clear why delta checksum is being enabled.
2022-05-18 08:48:48 -04:00