1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Commit Graph

97 Commits

Author SHA1 Message Date
6f562fba60 Migrate coverage testing to C and remove dependency on lcov.
lcov does not seem to be very well maintained and is often not compatible with the version of gcc it ships with until a few months after a new distro is released. In any case, lcov is that not useful for us because it generates reports on all coverage while we are mainly interested in missing coverage during development.

Instead use the JSON output generated by gcov to generate our minimal coverage report and metrics for the documentation.

There are some slight differences in the metrics. The difference in the common module was due to a bug in the old code -- build/common was being added into common as well as being reported separately. The source of the two additional branches in the backup module is unknown but almost certainly down to how exclusions are processed with regular expressions. Since there is additional coverage rather than coverage missing this seems fine.

Since this was pretty much a rewrite it was also a good time to migrate to C.
2024-05-31 14:52:07 +10:00
28ad1badd9 Update Fedora test image to Fedora 40. 2024-05-18 18:55:53 +10:00
4ea4e3f380 Update Fedora test image to Fedora 38.
This should have been done in 434938e3 but somehow it didn't happen.

Fedora 38 requires 2048 bit keys so update the VM builds to use them. Update the documentation to use 2048 bit keys. This is not technically required by this commit but it makes sense to do it now.

Also update the key location for the yum.p.o repository.

Lastly, shuffle test PostgreSQL versions since PostgreSQL 11 is not longer available in the yum.p.o repository.
2024-04-23 22:34:32 +10:00
fb22f04555 PostgreSQL 17 Support.
Add catalog version and WAL magic for PostgreSQL 17.
2024-04-18 10:56:24 +10:00
dcf0781987 Remove support for PostgreSQL 9.3.
Per our policy to support five EOL versions of PostgreSQL, 9.3 is no longer supported by pgBackRest.

Remove all logic associated with 9.3 and update the tests.
2023-11-09 12:59:12 -03:00
434938e32b Update Fedora test image to Fedora 38.
This adds testing for the latest tool versions, e.g. gcc.
2023-06-22 18:23:06 +02:00
87087fac23 SFTP support for repository storage. 2023-05-13 19:16:16 +03:00
1bd5530a59 Remove double spaces from comments and documentation.
Double spaces have fallen out of favor in recent years because they no longer contribute to readability.

We have been using single spaces and editing related paragraphs for some time, but now it seems best to update the remaining instances to avoid churn in unrelated commits and to make it clearer what spacing contributors should use.
2023-05-02 12:57:12 +03:00
3fc3690dd7 PostgreSQL 16 Support.
Add catalog version and WAL magic for PostgreSQL 16.

The GUC to force parallel mode has be renamed so update that in the tests.
2023-04-27 10:30:50 +03:00
ff98636e41 Update 32-bit CI host to Debian 10 from Ubuntu 18.04.
Ubuntu 18.04 will be EOL before the next release, so update to the oldest available Debian version.

Also fix one incorrect return value type, a test cast, and adjust some test timeouts.
2023-04-06 08:22:14 +04:00
c826429123 Improve libbacktrace implementation.
The libbacktrace feature has not been working since the move to meson because libbacktrace detection was not added to the meson build. Add libbacktrace to meson and improve the feature so that it can be compiled into release builds.

The prior implementation fetched line numbers with each stack trace push. Not only was this slow but it missed any functions that were not being tracked on our stack.

Instead just examine the backtrace when an error happens and merge it with the info we have on our stack. If the backtrace is not available then the output remains as before.

Also remove --backtrace from test.pl since the library is now auto-detected.

Leave this library out of the production build for now to give it a little time to shake out in testing.
2023-01-11 11:19:26 +07:00
77c721eb63 Remove support for PostgreSQL 9.0/9.1/9.2.
Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest.

Remove all logic associated with 9.0/9.1/9.2 and update the tests.

Document the new support policy.

Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
3ad588443b Update test.pl --psql-bin option to match command-line help.
The option to specify the path to psql was shown in the command-line help as --psql-bin but the option was actually named --pgsql-bin.

Rename to match the help so they are consistent.
2022-11-14 12:47:27 +08:00
a16cf5eac7 Update CI to use Ubuntu 22.04 and Fedora 36.
Both have newer gcc and OpenSSL 3.

Fedora 36 runs horribly slow with valgrind enabled so run the valgrind tests on Ubuntu 22.04. Fedora 36 has a newer gcc so it is still worth testing on.
2022-06-06 16:32:20 -04:00
c98baab6b5 New CI container builds for PostgreSQL 15 beta1 and minor releases.
Remove VM_OS_REPO since it is no longer required.

Rebalance PostgreSQL versions for more efficient test times.

Always print version of PostgreSQL when testing. This helps verify that new minor releases are being used.
2022-05-19 18:24:09 -04:00
7d9b2e267c Move 32-bit CI testing from Debian 9 to Ubuntu 18.04.
Debian 9 will be EOL before our next release, so move 32-bit testing to Ubuntu 18.04, which is older than Debian 10.
2022-05-12 11:57:12 -04:00
3d8ee552fe Move some PostgreSQL integration tests to rh7.
This makes the u20 test run faster. Since u20 is currently the longest running test, the time to complete all tests is also improved.
2022-05-11 15:59:37 -04:00
de816a0f57 Remove integration expect log testing.
Integration expect log testing was originally used as a rough-and-ready way to make sure that certain code paths were being executed before the unit tests existed. Now that we have 100% unit test coverage (with expect log testing) the value of the integration expect tests seems minimal at best.

But they do cause numerous issues:

- Maintenance of the expect code and replacements that are required to keep logs reproducible.
- Even a trivial change can cause massive churn in the expect logs, e.g. d9088b2. These changes should be minutely audited but since the expect logs have little value now it is seldom worth the effort.
- The OS version used to do expect testing (RHEL7) can only be used to test one version of PostgreSQL. This makes it hard to balance the PostgreSQL version testing between OS versions.
- When a commit affects expect logs it is not clear (especially for new developers) how to regenerate them and our contributing guide is silent on the issue.

The goal is to migrate the integration tests to C and expect testing is not part of that plan. It seems best to get rid of them now.
2022-05-10 13:18:26 -04:00
b4c1ca7b80 Split 32-bit CI tests.
This helps rebalance some of the tests that are running long, i.e. d9 and u20.

I would be better to move more PostgreSQL versions to d9, but the base VM does not contain more versions. New minor versions will be out later in the week so that seems a better time to be rebuilding containers.
2022-05-09 14:19:05 -04:00
20782c88bc PostgreSQL 15 support.
PostgreSQL 15 drops support for exclusive backup and renames the start/stop backup commands.

This is based on the pgdg-testing repo since beta1 has not been released yet, but it seems unlikely that breaking changes will be made at this point. beta1 should be tagged just before our next release so we'll retest before the release.
2022-05-04 11:55:59 -04:00
cf5b3a302f Fix language in rh7 test container for aarch64.
The /etc/profile.d/lang.sh script was causing issues but it does not exist on amd64, so it seems the easiest thing was to remove it.

Fix how 32-bit VMs are determined now that another 64-bit architecture has been added.

And remove some obsolete VM hashes.
2022-01-26 13:22:31 -06:00
bb4b30ddd3 Remove support for PostgreSQL 8.3/8.4.
There is no evidence that users need 8.3/8.4 anymore but it does cost us in terms of development and testing, especially now that we have a number of new backup/restore features planned.

It seems to make sense to remove this support now. If there are users who need to use/migrate from these versions they can use an older version of pgBackRest.
2022-01-06 15:34:04 -05:00
c32e000ab9 Use Rocky Linux for documentation builds instead of CentOS.
Since CentOS 8 will be EOL at the end of the year it makes sense to do this now. The centos:8 image is still used in documentation.xml because changes there require manual testing, which will need to be done at a later date. The changes are not user-facing, however, and can be done at any time.

Also update CentOS references to RHEL since that is what we are emulating for testing purposes.
2021-10-28 15:15:49 -04:00
360cff94e4 Update 32-bit test container to Debian 9.
Also rebalance PostgreSQL version integration tests.
2021-10-16 12:33:31 -04:00
66bfd1327e Rename SSH connection control parameters in integration tests. 2021-10-13 19:48:41 -04:00
6397d73535 Update test container OS versions.
Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial.
Update Ubuntu 18.04 to 20.04.
Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up.

Add --no-install-recommends to apt-get commands to save a bit of time and space.

Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small.

Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
2021-07-18 17:29:20 -04:00
81602f1593 Remove test.pl --vm-host option.
This option was once used to optimize bin builds but has been defunct for some time.
2021-07-17 11:09:53 -04:00
2452c4d5a4 Add PostgreSQL 14 support.
There are no code changes from PostgreSQL 13 so simply add the new version.

Add CATALOG_VERSION_NO_MAX to allow the catalog version to "float" during the PostgreSQL beta/rc period so new pgBackRest versions are not required when the catalog version changes.

Update the integration tests to handle new PostgreSQL startup messages.
2021-05-24 17:17:03 -04:00
baddec1e9a Basic multi-architecture support for test containers.
The tests worked fine on multiple architectures, but would only run "bare metal", i.e. tests that required containers could not be run.

Enable basic multi-architecture support by allowing containers to be built using whatever architecture the host supports. Also allow cached containers to be defined for multiple architectures in container.yaml.

Add a Dockerfile which can be used as a container for other containers to provide a consistent development environment.

The primary goal is to allow development on Mac M1 but other architectures should find these improvements useful.
2021-05-03 16:31:27 -04:00
6c005afd5b Add Ubuntu 20.04 test VM definition. 2021-04-08 14:38:20 -04:00
c2c702c09d Add co7 package to support llvm.
This is required for new package versions.

Also remove the obsolete 9.2 package and update the supported versions list.
2021-01-13 17:32:42 -05:00
d01669aa58 Move most tests to Github Actions.
Testing on Travis-CI has been getting slower (from ~18 minutes to 3-6 hours) and the travis-ci.org service will be terminated at the end of the year. Moving to travis-ci.com is an option but the quotas are too low for our purposes.

Instead use Github Actions, which does not currently have quotas, and runs our current tests with just a few tweaks.

This still leaves multi-architecture tests on Travis-CI but we may be able to run those and stay within the new quotas.

Also fix a minor bug in restoreTest.c exposed by Github Actions using a different name for the user and group.
2020-12-09 15:19:01 -05:00
31becf05b7 Add RHEL/CentOS 8 documentation.
Update RHEL/CentOS 7 to cover the versions that were previously covered by RHEL/CentOS 6.

Since RHEL/CentOS 7/8 work the same update the documentation logic and labels to reflect this compatibility.
2020-12-04 10:59:57 -05:00
ec9f23d31f Remove CentOS 6 from tests and documentation.
CentOS6 EOL'd and the mirrors were swiftly deleted, leading to failures in tests and documentation.

Remove CentOS 6 for now to get builds going again with the intention to replace it in the near future with CentOS 8.
2020-12-02 16:23:05 -05:00
b096a25b49 Update test containers for PostgreSQL 13.
Add older PostgreSQL versions to the u18 container that were not available before.

This also updates all minor versions for prior versions of PostgreSQL.
2020-09-24 11:19:51 -04:00
a260d4a53b Add zstd to CentOS/RHEL 6 test container.
Zstd is now required by the upstream yum package.
2020-07-28 08:09:10 -04:00
ec7b7c5a3e PostgreSQL 13 beta1 support.
There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.

Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2020-05-21 13:46:16 -04:00
faabf1227d Update Fedora container to Fedora 32.
This allows unit testing on gcc 10.

Also fix an incorrect enum in the config/config unit test that was caught by the new compiler.
2020-05-07 11:06:56 -04:00
98f30ef222 Move PostgreSQL 9.4 real integration tests to Ubuntu 18.04.
PostgreSQL 9.4 packages for RHEL 6 were dropped from yum.p.o.
2020-05-05 15:00:13 -04:00
47aa765375 Add Zstandard compression support.
Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

Zstandard version >= 1.0 is required, which is generally only available on newer distributions.
2020-05-04 15:25:27 -04:00
1aaaa94253 Remove Ubuntu 19.04 container definition.
Ubuntu 19.04 is no longer supported.
2020-05-04 14:02:25 -04:00
39f5f3a0b4 Remove PostgreSQL 9.4 for Fedora 30 dropped from yum.p.o. 2020-05-04 13:12:52 -04:00
cfab67a775 Enable coverage testing on Fedora 30.
Now that coverage testing works reliably with gcc9 it makes sense to enable it for CI.
2020-03-20 13:49:23 -04:00
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
c279a00279 Add lz4 compression support.
LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.

Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 14:45:27 -04:00
79cfd3aebf Remove LibC.
This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.

The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.

The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.

Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.

Update test.pl and related code to remove LibC builds.

Ding, dong, LibC is dead.
2020-03-09 17:41:59 -04:00
a4b9440d35 Only install specific lcov version when required.
Installing lcov 1.14 everywhere turned out to be a problem just as using 1.13 on Ubuntu 19.04 was.

Since we primarily use Ubuntu 18.04 for coverage testing and reporting, we definitely want to make sure that works.  So, revert to using the default packaged lcov except when specified otherwise in VmTest.pm.

PostgreSQL minor version releases are also included since all containers have been rebuilt.
2019-11-22 19:25:49 -05:00
8b682b75d2 Allow mock integration tests for all VM types.
Previously the mock integration tests would be skipped for VMs other than the standard four used in CI.  Now VMs outside the standard four will run the same tests as VM4 (currently U18).
2019-11-02 10:35:48 +01:00
e06db21e35 Error when specified vm is invalid. 2019-10-17 14:00:18 +02:00