1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

789 Commits

Author SHA1 Message Date
David Steele
01b8e2258f
Improve archive-push command fault tolerance.
3b8f0ef missed some cases that could cause archive-push to fail:

* Checking archive info.
* Checking to see if a WAL segment already exists.

These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
Cynthia Shang
31c7824a4d
Allow stanza-* commands to be run remotely.
The stanza-create, stanza-upgrade and stanza-delete were required to be run on the repository host. When there was only one repository allowed this was not a problem.

However, with the introduction of multiple repository support, this becomes more of a burden to the user, therefore the stanza-create, stanza-upgrade and stanza-delete commands have been improved to allow for them to be run remotely.
2021-03-10 08:10:46 -05:00
David Steele
fe4ba455ed Move configuration definition to src/build/config/config.yaml.
Moving to YAML allows the configuration data to be read by C programs.

Also go back to using YAML::XS since it is the only implementation that has proper boolean support.
2021-03-08 16:01:05 -05:00
David Steele
1dbb3bf50b
Multiple repository support.
Up to four repositories may be configured. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.

Some commands, e.g. stanza-create/stanza-update, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.

Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.

The archive-push command will always push WAL to the archive in all configured repositories but backups will need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
For single repository configurations there should be no change in behavior.
2021-03-08 13:31:13 -05:00
David Steele
088662d986
GCS support for repository storage.
GCS and GCS-compatible object stores can now be used for repository storage.
2021-03-05 12:13:51 -05:00
David Steele
d1aa765a9d
Consolidate less commonly used repository storage options.
The following options are renamed as specified:

repo1-azure-ca-file -> repo1-storage-ca-file
repo1-azure-ca-path -> repo1-storage-ca-path
repo1-azure-host -> repo1-storage-host
repo1-azure-port -> repo1-storage-port
repo1-azure-verify-tls -> repo1-storage-verify-tls
repo1-s3-ca-file -> repo1-storage-ca-file
repo1-s3-ca-path -> repo1-storage-ca-path
repo1-s3-host -> repo1-storage-host
repo1-s3-port -> repo1-storage-port
repo1-s3-verify-tls -> repo1-storage-verify-tls

The old option names (e.g. repo1-s3-port) will continue to work for repo1, but repo2, etc. will require the new names.
2021-03-02 13:51:40 -05:00
Cynthia Shang
13dc8e68d7 Make --repo optional for backup command.
If there are multiple repos and the --repo option is not specified then backup will automatically select the highest priority repo.
2021-02-26 14:49:50 -05:00
Cynthia Shang
0ddc0380ff Remove restore default repo from integration tests.
The default is now to scan all repos so update the integration tests to reflect that.
2021-02-24 11:32:13 -05:00
David Steele
bec3e20b2c Add archive-get command multi-repo support.
Repositories will be searched in order for the requested archive file.

Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
David Steele
9154d73030 Add -g accidentally removed in 4e8d469f.
The tests all run fine without debug info but gdb and valgrind are a lot less useful without it.
2021-02-02 17:05:55 -05:00
David Steele
8e9f04cc32 Add HRN_INTEST_* define to indicate when a test is being run.
This is useful for initialization that needs to be done for the test and all subsequent tests.

Use the new defines to implement initialization for sockets and statistics.
2021-01-27 16:54:41 -05:00
David Steele
87eb081a8f Make unit test builds incremental based on coverage in prior tests.
When building tests only include files covered by the current test or by prior tests. This increases performance (less compilation and linking) and also helps detect cross-dependencies in the code. Since there are currently cross-dependencies the depend option is used to document them and allow compilation. The idea is to resolve them incrementally over time.

Add the harness option to include harness modules when the minimum requirements for compilation are met.

Add the feature option to indicate which features are now available in the harness (based on source modules already tested). This allows conditional compilation in harness modules when some features are not yet available.
2021-01-27 10:57:42 -05:00
David Steele
f669da7dcc Use minio latest in documentation and integration tests.
At one time Minio had stability problems with latest but that appears to be resolved for the last year or so.

Use latest so we'll know if something breaks since Minio is frequently used in production.
2021-01-26 11:25:29 -05:00
David Steele
4e8d469f4d Use configure to generate Makefile variables for unit tests.
The unit test Makefile generation was a hodge-podge of constants and rules based on distros/versions that easily got out of date and did not work on an unknown system. All of this dates from the mixed Perl/C unit test implementation.

Instead use configure to generate most of the important Makefile variables, which allows the unit tests to run on multiple platforms, e.g. MacOS and FreeBSD.

There is plenty of work to be done here and not all the unit tests work on MacOS and FreeBSD for various reasons.

As a POC update the MacOS and FreeBSD tests on Cirrus-CI to run a few command unit tests.
2021-01-24 16:24:14 -05:00
David Steele
ef2dc6d3f4 Add chmod to make file removal after tests more reliable.
MacOS does not allow files to be removed recursively unless the owner has write and execute permissions on all the directories.

Some tests leave the permissions in a bad state so fix them up before trying to delete.
2021-01-24 15:48:32 -05:00
David Steele
04e84da0ef Allow the make command to be configured for test.pl. 2021-01-24 15:35:40 -05:00
David Steele
d2057c53bd Use YAML::Any module instead of YAML::XS in Perl.
YAML::XS requires libyaml so it not as portable as pure Perl versions of YAML.

Instead of using YAML:PP just use the general YAML::Any module which uses whatever is installed. We are not concerned about performance for YAML so whatever works is fine.
2021-01-24 15:06:38 -05:00
David Steele
5cb9f166ec Add stderr to unit test error messages.
Messages on stderr were being lost due to the error suppression used to customize the error message.

Also update the formatting to be more informative and concise.
2021-01-24 08:23:59 -05:00
Cynthia Shang
f32eb9b94e
Partial multi-repository implementation.
Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands.

Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations.

Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
David Steele
4e56948128 Compensate for numeric auto conversion in newer Perls. 2021-01-19 12:07:05 -05:00
David Steele
065b5f93ae Improve test coverage list handling.
All unit tests now require full coverage so the "full" keyword is obsolete and has been removed.

The covered code modules are simply listed, with only "no code" modules annotated.
2021-01-15 10:56:51 -05:00
David Steele
c2c702c09d Add co7 package to support llvm.
This is required for new package versions.

Also remove the obsolete 9.2 package and update the supported versions list.
2021-01-13 17:32:42 -05:00
David Steele
96fd678662
Add job-retry and job-retry-interval options.
These options specify the number of local worker job retries and the retry interval after one immediate retry.

There is some value in allowing retries to be specified by the user but for the most part these options are for suppressing retries during testing, which can save a lot of time. The bug introduced in d1d25c7 and fixed in 8b86d5e also suggests it is better not to use retries in tests.

Remove the default delayed retries for archive-get/archive-push, leaving only the immediate retry. These commands are retried by PostgreSQL so it doesn't make sense to do too many retries internally.

These options are currently internal.
2021-01-11 15:15:25 -05:00
David Steele
6e7a3eb383 Remove archive-timeout from test in mock/archive.
No timeout is expected here but the small timeout prevents errors from being thrown.

This is not a bug since the error would be thrown on the next archive-get call but it does make the tests harder to debug when there is an error.

It is not clear why there was a timeout here at all. It is likely cruft from a prior test or a copy/paste error.
2021-01-05 18:11:28 -05:00
David Steele
d01669aa58 Move most tests to Github Actions.
Testing on Travis-CI has been getting slower (from ~18 minutes to 3-6 hours) and the travis-ci.org service will be terminated at the end of the year. Moving to travis-ci.com is an option but the quotas are too low for our purposes.

Instead use Github Actions, which does not currently have quotas, and runs our current tests with just a few tweaks.

This still leaves multi-architecture tests on Travis-CI but we may be able to run those and stay within the new quotas.

Also fix a minor bug in restoreTest.c exposed by Github Actions using a different name for the user and group.
2020-12-09 15:19:01 -05:00
David Steele
e116b535e6 v2.31: Minor Bug Fixes and Improvements
Bug Fixes:

* Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)
* Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)

Features:

* Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)
* Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
* Add repo-azure-endpoint option. (Reviewed by Cynthia Shang, Brian Peterson. Suggested by Brian Peterson.)
* Add pg-database option. (Reviewed by Cynthia Shang.)

Improvements:

* Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)
* Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang, Oscar.)
* Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)

Documentation Improvements:

* Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.
2020-12-07 09:55:00 -05:00
David Steele
31becf05b7 Add RHEL/CentOS 8 documentation.
Update RHEL/CentOS 7 to cover the versions that were previously covered by RHEL/CentOS 6.

Since RHEL/CentOS 7/8 work the same update the documentation logic and labels to reflect this compatibility.
2020-12-04 10:59:57 -05:00
David Steele
ec9f23d31f Remove CentOS 6 from tests and documentation.
CentOS6 EOL'd and the mirrors were swiftly deleted, leading to failures in tests and documentation.

Remove CentOS 6 for now to get builds going again with the intention to replace it in the near future with CentOS 8.
2020-12-02 16:23:05 -05:00
David Steele
7fda83b31e
Allow multiple remote locks from the same main process.
Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed.

The exec-id is also useful for associating remote logs with main logs for debugging purposes.
2020-11-23 12:41:54 -05:00
David Steele
2d38d2fc82 Reset additional options in real/all integration test.
Currently indexes above 1 do not have dependencies checked, so this doesn't error.

In a future commit we will enable those checks and this will error if it is not fixed.
2020-10-19 17:06:52 -04:00
David Steele
b096a25b49 Update test containers for PostgreSQL 13.
Add older PostgreSQL versions to the u18 container that were not available before.

This also updates all minor versions for prior versions of PostgreSQL.
2020-09-24 11:19:51 -04:00
David Steele
4cd61152f5 Update PostgreSQL 13 test catalog versions missed in 6bb111c1.
These values are not used by the Perl integration tests so maybe it would be better to remove them, but for now just update since they should not be changing again for PG13.
2020-09-17 12:39:30 -04:00
David Steele
959f77cd6a
Add general-purpose statistics collector.
Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome.

Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually.

No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible.

The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
David Steele
e81533bbab
Improve memory usage of unlogged relation detection in manifest build.
This loop was using a lot of memory without freeing it at intervals.

Rewrite to use char arrays when possible to reduce memory that needs to be allocated and freed.
2020-08-04 10:16:51 -04:00
David Steele
a260d4a53b Add zstd to CentOS/RHEL 6 test container.
Zstd is now required by the upstream yum package.
2020-07-28 08:09:10 -04:00
David Steele
24d2c5b277
Remove real/all integration tests now covered by unit tests.
Remove all check and stanza-* tests except for the ones that are intended to succeed. The successful tests show that the queries run with expected results against each version of PG which should also validate queries for the failure tests in the unit tests.

Also remove the tests for --no-online backups since they don't require a database and are well tested in the unit tests.
2020-07-16 13:57:14 -04:00
David Steele
aa4e13b665 Move encrypted files as raw in integration tests.
The encryption key should not be changed when moving a file so no need to decrypt/encrypt.
2020-07-16 11:27:14 -04:00
David Steele
88b0f6245d Run non version specific real/tests on the expect version.
There are a few non version specific tests that need to be run in integration because we can't get coverage in the unit tests.

To save some time we'll only run those tests against the same version we use for expect testing.
2020-07-15 13:19:16 -04:00
Stefan Fercot
d3dd32a031
Add expire-auto option.
This allows automatic expiration after a successful backup to be disabled.
2020-07-14 08:12:25 -04:00
David Steele
d5df3974b5 Read segment size from WAL headers.
This allows validation of the WAL segment size for PostgreSQL versions <= 10.
2020-07-09 17:32:36 -04:00
David Steele
682ac656f5
Fix restore --force acting like --force --delta.
This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue.
2020-07-06 15:03:24 -04:00
David Steele
3f4371d7a2 Azure support for repository storage.
Azure and Azure-compatible object stores can now be used for repository storage.

Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
David Steele
ea04ec7b3f Disable query parallelism in PostgreSQL sessions used for backup control.
There is no need to have parallelism enabled in a backup control session. In particular, 9.6 marks pg_stop_backup() as parallel-safe but an error will be thrown if pg_stop_backup() is run in a worker.
2020-06-25 08:02:48 -04:00
David Steele
a3e5e66f05 Simplify test matrix for real/all tests.
Test matrices were previously simplified for the mock/* tests (e.g. d4410611, d489eb87) but not for real/all since the rules for which tests would run with which options was extremely complex. This only got more complex when new compression formats were added.

Because the loop-generated matrix was so large, mosts tests were skipped for most option combinations following arcane logic which was nearly impossible to decipher even when reading the code, and completely impossible from the test.pl interface. As a consequence, important tests got excluded. For example, backup from standby was excluded for most versions of PostgreSQL because it was only run once per distro, against the latest version to be included in that distro.

Simplify the tests by having a single run per PostgreSQL version and vary test parameters according to the capabilities of each version and the underlying distro. So, ZST testing is based on whether the distro supports ZST. Every test is run for each set of parameters based on the capabilities of the PostgreSQL version, e.g. backup from standby is not attempted on versions that don't support it.

Note that since more tests are running the overall time to run the mock/all tests has increased by about 20-25%. Some time may be saved my removing tests that are adequately covered by unit tests but that should the subject of another commit. Another option would be to limit some non version-specific tests to a single, well defined version of PostgreSQL, .e.g the version that is run by expect tests, currently 9.6.

The motivation for this refactor is that new storage drivers are coming and the loop-generated test matrix simply was not up to the task of adding them.

The following is an example of the new test log (note longer runtime of each test):

module=real, test=all, run=1, pg-version=10 (106.91s)
module=real, test=all, run=1, pg-version=9.5 (151.09s)
module=real, test=all, run=1, pg-version=9.2 (123.11s)
module=real, test=all, run=1, pg-version=9.1 (129s)

vs. the old test log (sub-second tests were skipped entirely):

module=real, test=all, run=2, pg-version=10 (0.31s)
module=real, test=all, run=3, pg-version=10 (0.26s)
module=real, test=all, run=4, pg-version=10 (60.39s)
module=real, test=all, run=1, pg-version=10 (69.12s)
module=real, test=all, run=6, pg-version=10 (34s)
module=real, test=all, run=5, pg-version=10 (42.75s)
module=real, test=all, run=2, pg-version=9.5 (0.21s)
module=real, test=all, run=3, pg-version=9.5 (0.21s)
module=real, test=all, run=4, pg-version=9.5 (0.21s)
module=real, test=all, run=5, pg-version=9.5 (0.26s)
module=real, test=all, run=6, pg-version=9.5 (0.21s)
module=real, test=all, run=1, pg-version=9.2 (72.78s)
module=real, test=all, run=2, pg-version=9.2 (0.26s)
module=real, test=all, run=3, pg-version=9.2 (0.31s)
module=real, test=all, run=4, pg-version=9.2 (0.21s)
module=real, test=all, run=5, pg-version=9.2 (0.21s)
module=real, test=all, run=6, pg-version=9.2 (0.21s)
module=real, test=all, run=1, pg-version=9.5 (88.41s)
module=real, test=all, run=2, pg-version=9.1 (0.21s)
module=real, test=all, run=3, pg-version=9.1 (0.26s)
module=real, test=all, run=4, pg-version=9.1 (0.21s)
module=real, test=all, run=5, pg-version=9.1 (0.31s)
module=real, test=all, run=6, pg-version=9.1 (0.26s)
module=real, test=all, run=1, pg-version=9.1 (72.4s)
2020-06-23 13:44:29 -04:00
David Steele
d560c1bf19 Ignore "unsupported frontend protocol" error on Centos/RHEL 6.
The unsupported version error is showing up on older versions of PostgreSQL (e.g. 9.1, 9.2) on RHEL6 when setting up a standby with streaming replication. The error occurs when a client does not properly send a version number and it's not clear why it is happening here, but it does not appear to have anything to do with pgBackRest and only affects RHEL6, i.e. 9.1 and 9.2 do not show this error on other distros.

For now ignore the error since RHEL6 is nearly EOL.
2020-06-23 12:42:46 -04:00
David Steele
3d74ec1190
Use PostgreSQL instead of postmaster where appropriate.
Using postmaster in messages was not very helpful since users rarely interact directly with the postmaster. Using PostgreSQL instead seems clearer.
2020-06-17 15:14:59 -04:00
David Steele
417818dcca Add --no-coverage-report to test.pl to disable report generation.
There is no sense in generating detailed coverage reports in CI environments where they will never be seen. It takes time and format differences in some older versions can cause problems in the report generation code.

Note that missing coverage will still be reported on stdout and the test will fail.
2020-06-17 15:07:30 -04:00
David Steele
0680cfc8dc Rename most instances of master to primary in tests.
This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702).

Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 14:06:38 -04:00
David Steele
ec7b7c5a3e
PostgreSQL 13 beta1 support.
There don't appear to be any behavioral changes since PostgreSQL 12 and all the tests pass.

Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2020-05-21 13:46:16 -04:00
David Steele
688ec2a8f5 Use an extension to denote vendorized code.
Vendorized code is copied from another project when a library is not available and a git subproject won't work. Currently all the vendorized code is copied from PostgreSQL but it makes sense to have a more general mechanism for indicating vendorized code.

The .vendor extension will be used to denote vendorized code in the same way that .auto is used to denote auto-generated code.
2020-05-18 19:11:26 -04:00
David Steele
92c036b966 Add code count rule for valgrind suppression missed in 6be5ea33.
6be5ea33 changed valgrind suppression file naming but failed to update the code count rules.
2020-05-18 18:09:41 -04:00
David Steele
b5dd14e6f3 Make storage type more generic in the integration tests.
Rather than bS3 use strStorage which can indicate more than two storage types.

For the moment there are still only two storage types but this change is required before more can be added.
2020-05-12 18:55:20 -04:00
David Steele
22d260ad53 Allow more tests to run outside of containers.
These tests required sudo to achieve complete coverage.

Add a new coverage exception, vm_covered, that applies to code that can only be covered in a container. When the test is run outside of a container code sections that require a container will be excluded with TEST_CONTAINER_REQUIRED and the coverage exception will be added to prevent a coverage error.

This does require marking up the core code with vm_covered, which in some modules (e.g. common/io/tls/client) can be extensive. It's possible that some of these tests can be rewritten to be less dependent on sudo but no attempt was made to do that here.

Only allow coverage summaries in a vm since coverage summaries outside a vm will not be complete, which was true even before this commit.
2020-05-09 09:17:33 -04:00
David Steele
e873ad6da0 Update Minio version to 2020-05-06T23-23-25Z in tests/documentation.
This release fixes the issue we submitted regarding an unquoted eTag: https://github.com/minio/minio/issues/9517
2020-05-07 17:26:46 -04:00
David Steele
faabf1227d Update Fedora container to Fedora 32.
This allows unit testing on gcc 10.

Also fix an incorrect enum in the config/config unit test that was caught by the new compiler.
2020-05-07 11:06:56 -04:00
David Steele
f8509ab76c Don't allow sudo to disable core dumps in test containers.
Newer versions of sudo output this message to stderr when run in a container:

sudo: setrlimit(RLIMIT_CORE): Operation not permitted

See https://github.com/sudo-project/sudo/issues/42 for details.

A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.
2020-05-07 07:38:28 -04:00
David Steele
6be5ea3388 Suppress Valgrind errors on a per-VM basis.
There are a number of Valgrind errors on Ubuntu 12.04 which do not happen on newer distro versions. However, suppressions for these errors have masked legitimate issues in subsequent code.

Instead, make suppressions VM specific so errors in other VMs are not masked.
2020-05-06 18:24:48 -04:00
Stephen Frost
a021c9fe05
Add bzip2 compression support.
bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.

bzip2 is currently available on all supported platforms.
2020-05-05 16:49:01 -04:00
David Steele
98f30ef222 Move PostgreSQL 9.4 real integration tests to Ubuntu 18.04.
PostgreSQL 9.4 packages for RHEL 6 were dropped from yum.p.o.
2020-05-05 15:00:13 -04:00
David Steele
47aa765375 Add Zstandard compression support.
Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

Zstandard version >= 1.0 is required, which is generally only available on newer distributions.
2020-05-04 15:25:27 -04:00
David Steele
1aaaa94253 Remove Ubuntu 19.04 container definition.
Ubuntu 19.04 is no longer supported.
2020-05-04 14:02:25 -04:00
David Steele
39f5f3a0b4 Remove PostgreSQL 9.4 for Fedora 30 dropped from yum.p.o. 2020-05-04 13:12:52 -04:00
David Steele
1d45282b97 Add missing spaces between while keyword and condition.
Our convention is to have a space here but some were missed.
2020-05-01 09:31:50 -04:00
Cynthia Shang
c5241e5007 Expire WAL archive only when repo-retention-archive threshold is met.
Previously when retention-archive was set (either by the user or by default), archives prior to the archive-start of the oldest remaining full backup (after backup expiration occurred) would be expired even though the retention-archive threshold had not been met. For example, if there were 1 full backup remaining after backup expiration and the retention-archive was set to 2 and retention-archive-type=full, then archives prior to the archive-start of the remaining full backup would still be removed even though retention-archive required 2 full backups remaining before archives should be expired.

The thought was to keep the archive directory clean and since the full backup did not require prior archives, it was safe to delete them. However, this has caused problems for some users in the past (because they needed the WAL for other purposes) and with the new adhoc and time-based retention features, it was decided that the archives should remain until the threshold was met. The archives will eventually be removed and if having them causes space issues, the expire command and the retention-archive can always be run and adjusted.
2020-04-29 08:06:49 -04:00
Cynthia Shang
1c1a710460 Add --set option to the expire command.
The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
2020-04-27 14:00:36 -04:00
David Steele
1c319e051b Remove stray copy of LibCAuto.pm.
This was removed in 79cfd3ae but accidentally added back in 2fa69af8, probably due to an incorrectly resolved merge conflict.
2020-04-20 11:49:11 -04:00
David Steele
76b88a3cd5 Add UNCONSTIFY() macro.
Allows casting const-ness away from an expression, but doesn't allow changing the type. Enforcement of the latter currently only works for gcc-like compilers.

Note that it is not safe to cast const-ness away if the result will ever be modified (it would be undefined behavior). Doing so can cause compiler mis-optimizations or runtime crashes (by modifying read-only memory). It is only safe to use when the result will not be modified, but API design or language restrictions prevent you from declaring that (e.g. because a function returns both const and non-const variables).

Note that this only works in function scope, not for global variables (it would be nice, but not trivial, to improve that).

UNCONSTIFY() requires static assert which is a feature in its own right.
2020-04-02 16:58:38 -04:00
David Steele
8989118cc6 Add SocketClient object.
This functionality was embedded into TlsClient but that was starting to get unwieldy.

Add SocketClient to contain all socket-related client functionality.
2020-03-31 12:43:29 -04:00
David Steele
50cf7370ee Add --no-performance to test.pl to suppress performance tests.
Performance tests do not need to be run on all platforms. Using vm=none to run performance tests seems best ... for performance.
2020-03-26 20:36:09 -04:00
David Steele
8af0462c5d Fix race condition in real/all integration tests.
If the tests are running quickly then the time target might end up the same as the end time of the prior full backup. That means restore auto-select will not pick it as a candidate and restore the last backup instead causing the restore compare to fail.

So, sleep one second.
2020-03-26 15:30:59 -04:00
David Steele
e63fdfbfd2 Debug and optimize flag cleanup for unit tests.
Decisions about when to optimize or enable debug code were spread out in too many places making it hard to keep them consistent.

Centralize the logic as much as possible to make it easier to maintain.
2020-03-26 11:16:35 -04:00
Cynthia Shang
eb7f7dd5ca Fix backup-prior for diff backups in mock/expire integration test.
In the ExpireEnvTest.pm backupCreate() function, backup-prior was incorrectly set for diff backups to the previous backup regardless of what backup type the previous backup was. This did not cause any issues in the Mock Expire tests before because it was not being checked. However, in order to reduce churn in the expect logs for a new feature where the backup-prior is utilized, this is being fixed so that the full backup is always used as backup-prior.
2020-03-25 15:47:03 -04:00
David Steele
f9c86b11a5 More improvements to custom coverage report.
* Fix a few issues with file names being truncated introduced in 787d3fd6.

* Use function line info from the lcov file to calculate which lines to show for uncovered functions.  This is more accurate than what we were doing before and function comment headers are now excluded which reduces clutter in the report.
2020-03-23 12:17:34 -04:00
David Steele
8c76ea2d93 Fix space lost in d70ca259. 2020-03-22 14:18:16 -04:00
David Steele
3ec630f5b2 Allow suppression of times in testing for reproducibility.
Timestamps, timings, etc. cause a lot of log churn when included in documentation.
2020-03-22 10:12:29 -04:00
David Steele
cfab67a775 Enable coverage testing on Fedora 30.
Now that coverage testing works reliably with gcc9 it makes sense to enable it for CI.
2020-03-20 13:49:23 -04:00
David Steele
782c9f89f4 Remove old coverage data before starting new test.
The old coverage data has been recorded so it is no longer needed. In newer versions of gcc leaving this file around can lead to an error when writing profile data after forking off to a non-pgbackrest binary (which we do in some unit tests).
2020-03-20 13:43:08 -04:00
David Steele
787d3fd67b Improve custom coverage report.
* Show all uncovered branch parts even when there are more than two parts per branch. This is the way gcc9 reports coverage so it needs to work even if it doesn't make as much sense as the old way.

* Show covered branches in functions where coverage is missing.  Showing just the uncovered branches can be confusing because it's not always clear how the coverage relates to the code.  By showing all branch coverage (+ or -) this correspondence is made easier.
2020-03-20 12:54:29 -04:00
David Steele
8af8029006 Fix lcov report when test module missing coverage.
We don't report branch coverage on test modules (e.g. test/src/module/common/errorTest.c) but the code that excluded branch coverage from the test module would also exclude it from all core modules if the test module was included in the lcov report due to lack of function/line coverage.

Adjust the coverage code to only exclude branches during the extraction of test module coverage.
2020-03-20 09:16:38 -04:00
David Steele
f6e9bb0819 Remove obsolete -O2 option for Fedora 30 unit test builds.
For some reason gcc9 would not do -O0 builds in combination with one of the options that libperl required.  Now that libperl is gone this exception is no longer required.
2020-03-19 19:30:09 -04:00
David Steele
d677b07081 Move coverage code to CoverageTest module.
This code needs some work, which will be easier if it is all in one place.
2020-03-19 12:07:51 -04:00
David Steele
9a47b88da3 Add links to custom coverage report.
When multiple files were missing coverage it could be hard to locate the coverage report for a specific file.

Add links for uncovered files to make this easier.

Also move table titles out of the table so they are valid html.
2020-03-16 20:02:36 -04:00
Cynthia Shang
2fa69af8da Add --dry-run option to the expire command.
Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything.
2020-03-16 13:56:52 -04:00
David Steele
4328bc1ac6 Move raw coverage results to test/result/raw path.
These results were stored in the vagrant path along with a full copy of src.

Instead store the raw coverage data in test/result/raw and change source references to the files that already exist in [test-path]/repo.
2020-03-16 08:41:32 -04:00
David Steele
d702249507 Build binaries in the test path rather than the vagrant path.
It makes more sense to build in the test path since many developers won't have a vagrant path. Anyway, it's better not to modify the vagrant path since it belongs to vagrant.

Instead of installing the binary just mount it into the container from where it was built. This saves a bit of time and space.
2020-03-15 10:09:27 -04:00
David Steele
959dce569b Update code classification and remove XS definition. 2020-03-14 18:30:24 -04:00
David Steele
213cc6e8be Move docker files to test/result. 2020-03-14 15:40:37 -04:00
David Steele
75ff25f17f Move profile results to test/result. 2020-03-14 14:50:36 -04:00
David Steele
0ba8062f5f Get package source files dynamically during package build.
The prior method was to build a special container to hold these files which meant they would get stale on development systems.  On CI the container was always rebuilt so failures would be seen there even when dev seemed to be working.

Instead get the package source when the package is built to ensure it is as up-to-date as possible.

This change was prompted by failures on the Ubuntu 12.04 container while getting the package source, probably due to an ancient version of git.  Package builds are no longer supported on that platform with the addition of lz4 compression so it didn't seem worth fixing.
2020-03-12 08:48:45 -04:00
David Steele
4a5bd002c0 Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.
The primary source for project info is now src/version.h.

The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 17:57:02 -04:00
David Steele
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
c279a00279 Add lz4 compression support.
LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.

Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 14:45:27 -04:00
David Steele
79cfd3aebf Remove LibC.
This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.

The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.

The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.

Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.

Update test.pl and related code to remove LibC builds.

Ding, dong, LibC is dead.
2020-03-09 17:41:59 -04:00
David Steele
3c4f91b319 Remove Perl unit tests made obsolete in 434cd832.
These were replaced by C unit tests but not all the unit test setup code was removed in the Perl module.
2020-03-09 13:35:26 -04:00
David Steele
438b957f9c Add infrastructure for multiple compression type support.
Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function.

Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format).

Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types.

Additional compression types will be added in future commits.
2020-03-06 14:41:03 -05:00
David Steele
02aa03d1a2 Remove obsolete methods in pgBackRest::Storage::Storage module.
All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible.

First remove storageDb() and use storageTest() instead.  Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC.  Now the only storage using the LibC interface is storageRepo().

Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed.  Same for owner().

Remove pathSync() because syncs are not required in the tests.  No test data is reused after a crash.

Path create/exists functions should never be explicitly performed on a repo so remove those.  File exists can be implemented by calling info() instead.

Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete.

Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used.  That also means there are no filters returning results so remove all the result code.

Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 14:10:09 -05:00
David Steele
00647c7109 Remove Perl Db module and LibC dependencies.
This was mostly dead code except the DB_BACKUP_ADVISORY_LOCK constant, moved to the real/all test module, and the function that pulls info from pg_control, moved to ExpireEnvTest.pm.
2020-03-06 07:21:17 -05:00
David Steele
2e0fe25650 Remove dependency on LibC hash filter.
Perl provides Digest::SHA for hashing so there is no need to expose this via LibC anymore.
2020-03-05 18:34:59 -05:00
David Steele
eb4347f20b Use static checksums in mock/all integration tests.
Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests.

We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 13:56:20 -05:00