1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

805 Commits

Author SHA1 Message Date
Andrey Sokolov
29b2a54fcc
Allow any path to be passed to the --test-path option.
A hard-coded path prevented this from working correctly.
2022-05-31 17:28:58 -04:00
David Steele
2c38c9a56f Skip stopping PostgreSQL 9.1 in real/all integration test.
Stopping the cluster has started consistently running out of memory on PostgreSQL 9.1. This seems to have happened after pulling in new packages at some point so it might be build related.

Stopping the cluster is not critical for 9.1 so skip it.
2022-05-31 16:43:18 -04:00
David Steele
2feaaeaac8 Add .inc extension to C files included in other C files.
These files were never intended to be compiled on their own so the .c extension was a bit misleading. In particular Meson does not like .c files that are not intended to be compiled independently.

Leave header files as is since they are already protected against being included more than once and are never expected to be compiled.
2022-05-31 16:06:41 -04:00
David Steele
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
David Steele
7d9b2e267c Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.
Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.
2022-05-12 11:57:12 -04:00
David Steele
3d8ee552fe Move some PostgreSQL integration tests to rh7.
This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.
2022-05-11 15:59:37 -04:00
David Steele
8ee85bc605 Fix "that that" typos. 2022-05-11 08:46:23 -04:00
David Steele
de816a0f57
Remove integration expect log testing.
Integration expect log testing was originally used as a rough-and-ready way to make sure that certain code paths were being executed before the unit tests existed. Now that we have 100% unit test coverage (with expect log testing) the value of the integration expect tests seems minimal at best.

But they do cause numerous issues:

- Maintenance of the expect code and replacements that are required to keep logs reproducible.
- Even a trivial change can cause massive churn in the expect logs, e.g. d9088b2. These changes should be minutely audited but since the expect logs have little value now it is seldom worth the effort.
- The OS version used to do expect testing (RHEL7) can only be used to test one version of PostgreSQL. This makes it hard to balance the PostgreSQL version testing between OS versions.
- When a commit affects expect logs it is not clear (especially for new developers) how to regenerate them and our contributing guide is silent on the issue.

The goal is to migrate the integration tests to C and expect testing is not part of that plan. It seems best to get rid of them now.
2022-05-10 13:18:26 -04:00
Reid Thompson
6b98b3534e Fix typo. 2022-05-10 06:52:56 -04:00
David Steele
b4c1ca7b80 Split 32-bit CI tests.
This helps rebalance some of the tests that are running long, i.e. d9 and u20.

I would be better to move more PostgreSQL versions to d9, but the base VM does not contain more versions. New minor versions will be out later in the week so that seems a better time to be rebuilding containers.
2022-05-09 14:19:05 -04:00
David Steele
20782c88bc
PostgreSQL 15 support.
PostgreSQL 15 drops support for exclusive backup and renames the start/stop backup commands.

This is based on the pgdg-testing repo since beta1 has not been released yet, but it seems unlikely that breaking changes will be made at this point. beta1 should be tagged just before our next release so we'll retest before the release.
2022-05-04 11:55:59 -04:00
David Steele
4872a3f121 Improvements to test harness memory debugging.
Only set -DDEBUG_MEM for the modules currently being tested rather than globally.

Also run tests in a temp mem context. Running in the top context can confuse memory accounting when a new context is created in the top context.
2022-04-28 12:33:39 -04:00
David Steele
5dba0d6e9b Set option-archive-copy flag in backup.manifest to false when offline.
In offline mode the pg_wal directory is copied, but that is not the same as archive-copy, which copies the exact set of WAL required from the archive.

This flag is purely for informational purposes so there is no live bug here, but the prior behavior was certainly misleading.
2022-04-05 18:42:19 -04:00
David Steele
54b4187527 Show Docker output when building containers if --log-level=detail.
This helps with debugging and monitoring container builds.
2022-04-05 13:14:42 -04:00
David Steele
f60ec5055a Cleanup output to stderr in unit tests.
The unit tests were ignoring stderr but nothing being output there was important. Now a test will fail if there is anything on stderr.

This makes it easier to work with -fsanitize, which outputs to stderr.
2022-03-24 18:43:43 -06:00
David Steele
14016a86e7 Check that sha1 checksum is not empty in manifestFileUpdate().
The manifest test module was setting a blank value here and causing a stack overflow because memcpy() is used instead of strcpy().

This was really just a test issue but add an assert just in case the same were to happen in production code.

Also update a bogus checksum in the integration tests to the correct length to avoid running afoul of the assert.

Found with -fsanitize=address.
2022-03-24 13:13:35 -06:00
David Steele
7afaac0a3d
Allow repo-hardlink option to be changed after full backup.
This rule was added because there were not sufficient tests to demonstrate that the repo-hardlink option could be changed in a backup set.

Remove the restriction and add/update tests to show that it works.

This is necessary now because bundling requires that hardlinking be disabled. Rather than add code complexity, it seems better just to address this limitation.
2022-03-22 08:35:34 -06:00
David Steele
2c96327e65 Remove extraneous double spaces in code and comments. 2022-03-15 17:55:48 -06:00
David Steele
3f66f42ef9
Rename bundle-* options to repo-bundle-*.
It seems best for these to be repo options so they can be configured per repo, rather than globally.

All clarify usage for repo-bundle-size and repo-bundle-limit.
2022-03-14 17:49:52 -06:00
David Steele
9eec98c613
Retry on page checksum validation failure during backup.
Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL.

Also remove tests made redundant by the test merge in b4897077.
2022-02-23 12:05:53 -06:00
David Steele
34d649579e
Bundle files in the repository during backup.
Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest.

Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented.

Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups.

Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example.

Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling.

One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
David Steele
b1da4e84e8 Revert Minio to prior release.
The most recent release of Minio has broken CI builds but there is no logging to indicate what is wrong.

For now, just use the prior release to get CI builds working again. This kind if breakage is not uncommon for Minio but they usually resolve it in the next release.
2022-02-02 14:39:39 -06:00
David Steele
cf5b3a302f Fix language in rh7 test container for aarch64.
The /etc/profile.d/lang.sh script was causing issues but it does not exist on amd64, so it seems the easiest thing was to remove it.

Fix how 32-bit VMs are determined now that another 64-bit architecture has been added.

And remove some obsolete VM hashes.
2022-01-26 13:22:31 -06:00
David Steele
4a73a02863
Simplify manifest defaults.
Manifest defaults for user, group, and mode were previously generated by scanning the data to find the most common values. This was very accurate but slow and complicated. It could also lead to surprising changes in the manifest when a default value suddenly changed.

Instead, use the $PGDATA path to generate defaults. In the vast majority of cases the same user/group should own all the path/files and the default file mode is easily derived from the path mode. There may be some edge cases where this generates larger manifests, but in general it reduces time and complexity when saving the manifest.

Remove the MCV code since it is longer longer used.
2022-01-21 15:22:48 -05:00
David Steele
b0db4b8ff0 Simplify base path mode in mock/all integration tests.
Change the mode back to 0700 earlier to reduce churn in the expect logs.

This will be especially important in a future commit that gets the defaults exclusively from the base path.
2022-01-21 08:52:51 -05:00
David Steele
8c062e1af8
Remove primary flag from manifest.
This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role.

Since backup is the only code concerned with the primary flag, move the code into the backup module.

We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
David Steele
16559d9e42 Use the PG_FILE_POSTMTRPID constant where appropriate.
Do the same in Perl with the MANIFEST_FILE_POSTMTRPID constant.
2022-01-20 08:41:05 -05:00
David Steele
bb4b30ddd3
Remove support for PostgreSQL 8.3/8.4.
There is no evidence that users need 8.3/8.4 anymore but it does cost us in terms of development and testing, especially now that we have a number of new backup/restore features planned.

It seems to make sense to remove this support now. If there are users who need to use/migrate from these versions they can use an older version of pgBackRest.
2022-01-06 15:34:04 -05:00
David Steele
9c76056dd0 Add error type and message to CHECK() macro.
A CHECK() worked exactly like ASSERT() except that it was compiled into production code. However, over time many checks have been added that should not throw AssertError, which should be reserved for probable coding errors.

Allow the error code to be specified so other error types can be thrown. Also add a human-readable message since many of these could be seen by users even when there is no coding error.

Update coverage exceptions for CHECK() to match ASSERT() since all conditions will never be covered.
2021-11-30 16:21:15 -05:00
David Steele
2d963ce947 Rename server-start command to server. 2021-11-18 17:23:11 -05:00
David Steele
43cfa9cef7 Revive archive performance test.
This test was lost due to a syntax issue in a58635ac.

Update the test to use system() to better mimic what postgres does and add logging so pgBackRest timing can be determined.
2021-11-10 12:14:41 -05:00
David Steele
c32e000ab9 Use Rocky Linux for documentation builds instead of CentOS.
Since CentOS 8 will be EOL at the end of the year it makes sense to do this now. The centos:8 image is still used in documentation.xml because changes there require manual testing, which will need to be done at a later date. The changes are not user-facing, however, and can be done at any time.

Also update CentOS references to RHEL since that is what we are emulating for testing purposes.
2021-10-28 15:15:49 -04:00
David Steele
7fb99c59c8 Use externed instead of extern'd in comments.
This is mostly to revert some comment changes in b11ab9f7 that will break the ppc64le patch, but at the same time keep the spelling consistent in all comments and documentation.

Also revert some space changes for the same reason.
2021-10-26 07:46:48 -04:00
David Steele
653ffcf8d9 Adjustments for new breaking change in Azurite.
Azurite released another breaking change (see fbd018cd, 096829b3, c38d6926, and Azurite issue 1039) so make adjustments as needed to documentation and tests.

Also remove some dead code that hid the repo-storage-host option and was made obsolete by all these changes.
2021-10-25 15:42:28 -04:00
David Steele
5dfdd6dd5b Add -Werror -Wfatal-errors -g flags to configure --enable-test.
These flags are used for all tests but it was not possible to add them to configure before the change in 046d6643. This is especially important for adhoc tests to ensure the flags are not forgotten.

Remove the flags from test make commands where they were being applied.

There is no change for production builds.
2021-10-19 12:45:20 -04:00
David Steele
ccc255d3e0 Add TLS Server.
The TLS server is an alternative to using SSH for protocol connections to remote hosts.

This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
David Steele
4570c7e275 Allow error buffer to be resized for testing.
Some tests can generate very large error messages for diffs and they often get cut off before the end.

Also fix a test so it does not create too large a buffer on the stack.
2021-10-18 11:32:53 -04:00
David Steele
360cff94e4 Update 32-bit test container to Debian 9.
Also rebalance PostgreSQL version integration tests.
2021-10-16 12:33:31 -04:00
David Steele
66bfd1327e Rename SSH connection control parameters in integration tests. 2021-10-13 19:48:41 -04:00
David Steele
01b20724da Rename PostgreSQL pid file constants and tests. 2021-10-13 19:36:59 -04:00
David Steele
5701620408 Rename manifest file primary flag in tests. 2021-10-13 19:02:58 -04:00
David Steele
5e84645ac0 Update comments referring to the PostgreSQL primary. 2021-10-13 12:16:47 -04:00
David Steele
815377cc60 Finalize catalog number for PostgreSQL 14 release. 2021-09-30 13:27:14 -04:00
David Steele
096829b3b2 Add repo-azure-uri-style option.
Azurite introduced a breaking change in 8f63964e to use automatically host-style URIs when the endpoint appears to be a multipart hostname.

This option allows the user to configure which style URI will be used, but changing the endpoint might cause breakage if Azurite decides to use a different style. Future changes to Azurite may also cause breakage.
2021-09-27 09:01:53 -04:00
David Steele
802373cb9d Limit valgrind error output to the first error.
Generally the first error is the only important error. The rest simply lead to a lot of scrolling.
2021-09-21 10:16:16 -04:00
David Steele
c38d6926d6 Revert Azurite version for testing to 3.14.0.
3.14.2 is causing breakage in the documentation. There is no obvious cause so for now just revert to the last working version.
2021-09-09 08:48:45 -04:00
David Steele
1afea449e2 Build newer valgrind and move CA cert install in test containers.
The newer version of valgrind helps with some arm64 issues that have been fixed since the architecture has become more popular. Also add the valgrind builds to the Vagrantfile and Dockerfile.

Move the CA cert install from the base container to the test container. This means the CA cert can be changed without rebuilding all the base containers.
2021-09-02 13:26:21 -04:00
David Steele
02b06aa495
Increase max index allowed for pg/repo options to 256.
The prior limitations were based on using getopt_long() to parse command-line options, which required a static list of allowed options. Setting index max too high bloated the binary unacceptably. 45a4e80 replaced the functionality of getopt_long() but the static list remained.

Improve cfgParseOption() to use available option data and remove the need for a static list. This also allows the option deprecations to be represented more compactly.

Index max is still capped at 256 because a large enough index could cause parseOptionIdxValue() to run out of memory since it allocates a static list based on the highest index found. If that function were improved with a map of found index values then index max could be set to UINT64_MAX.

Note that deprecations no longer set an index max or define whether reset is valid. These were space-saving measures which are no longer required. This means that indexed deprecated options will also be valid up to 256 and always allow reset, but it doesn't seem worth additional code to limit this behavior.

cfgParseOptionId() is no longer needed because calling cfgParseOption() with .ignoreMissingIndex = true duplicates the functionality of cfgParseOptionId(). This leads to some simplification in the help code.
2021-08-31 12:09:50 -04:00
David Steele
ad5e063990 Update containers for new PostgreSQL releases.
Also add a package that is required for older Debian versions.
2021-08-19 17:25:57 -04:00
David Steele
eb98b8d2db Fix typo. 2021-07-21 13:19:09 -04:00