1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

365 Commits

Author SHA1 Message Date
David Steele
688ec2a8f5 Use an extension to denote vendorized code.
Vendorized code is copied from another project when a library is not available and a git subproject won't work. Currently all the vendorized code is copied from PostgreSQL but it makes sense to have a more general mechanism for indicating vendorized code.

The .vendor extension will be used to denote vendorized code in the same way that .auto is used to denote auto-generated code.
2020-05-18 19:11:26 -04:00
David Steele
92c036b966 Add code count rule for valgrind suppression missed in 6be5ea33.
6be5ea33 changed valgrind suppression file naming but failed to update the code count rules.
2020-05-18 18:09:41 -04:00
David Steele
22d260ad53 Allow more tests to run outside of containers.
These tests required sudo to achieve complete coverage.

Add a new coverage exception, vm_covered, that applies to code that can only be covered in a container. When the test is run outside of a container code sections that require a container will be excluded with TEST_CONTAINER_REQUIRED and the coverage exception will be added to prevent a coverage error.

This does require marking up the core code with vm_covered, which in some modules (e.g. common/io/tls/client) can be extensive. It's possible that some of these tests can be rewritten to be less dependent on sudo but no attempt was made to do that here.

Only allow coverage summaries in a vm since coverage summaries outside a vm will not be complete, which was true even before this commit.
2020-05-09 09:17:33 -04:00
David Steele
faabf1227d Update Fedora container to Fedora 32.
This allows unit testing on gcc 10.

Also fix an incorrect enum in the config/config unit test that was caught by the new compiler.
2020-05-07 11:06:56 -04:00
David Steele
f8509ab76c Don't allow sudo to disable core dumps in test containers.
Newer versions of sudo output this message to stderr when run in a container:

sudo: setrlimit(RLIMIT_CORE): Operation not permitted

See https://github.com/sudo-project/sudo/issues/42 for details.

A simple workaround is to prevent sudo from disabling core dumps. This seems safe enough because if sudo is segfaulting then core files are the least of our worries.
2020-05-07 07:38:28 -04:00
David Steele
6be5ea3388 Suppress Valgrind errors on a per-VM basis.
There are a number of Valgrind errors on Ubuntu 12.04 which do not happen on newer distro versions. However, suppressions for these errors have masked legitimate issues in subsequent code.

Instead, make suppressions VM specific so errors in other VMs are not masked.
2020-05-06 18:24:48 -04:00
Stephen Frost
a021c9fe05
Add bzip2 compression support.
bzip2 is a widely available, high-quality data compressor. It typically compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical compressors), while being around twice as fast at compression and six times faster at decompression.

bzip2 is currently available on all supported platforms.
2020-05-05 16:49:01 -04:00
David Steele
98f30ef222 Move PostgreSQL 9.4 real integration tests to Ubuntu 18.04.
PostgreSQL 9.4 packages for RHEL 6 were dropped from yum.p.o.
2020-05-05 15:00:13 -04:00
David Steele
47aa765375 Add Zstandard compression support.
Zstandard is a fast lossless compression algorithm targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

Zstandard version >= 1.0 is required, which is generally only available on newer distributions.
2020-05-04 15:25:27 -04:00
David Steele
1aaaa94253 Remove Ubuntu 19.04 container definition.
Ubuntu 19.04 is no longer supported.
2020-05-04 14:02:25 -04:00
David Steele
39f5f3a0b4 Remove PostgreSQL 9.4 for Fedora 30 dropped from yum.p.o. 2020-05-04 13:12:52 -04:00
David Steele
1d45282b97 Add missing spaces between while keyword and condition.
Our convention is to have a space here but some were missed.
2020-05-01 09:31:50 -04:00
David Steele
76b88a3cd5 Add UNCONSTIFY() macro.
Allows casting const-ness away from an expression, but doesn't allow changing the type. Enforcement of the latter currently only works for gcc-like compilers.

Note that it is not safe to cast const-ness away if the result will ever be modified (it would be undefined behavior). Doing so can cause compiler mis-optimizations or runtime crashes (by modifying read-only memory). It is only safe to use when the result will not be modified, but API design or language restrictions prevent you from declaring that (e.g. because a function returns both const and non-const variables).

Note that this only works in function scope, not for global variables (it would be nice, but not trivial, to improve that).

UNCONSTIFY() requires static assert which is a feature in its own right.
2020-04-02 16:58:38 -04:00
David Steele
8989118cc6 Add SocketClient object.
This functionality was embedded into TlsClient but that was starting to get unwieldy.

Add SocketClient to contain all socket-related client functionality.
2020-03-31 12:43:29 -04:00
David Steele
50cf7370ee Add --no-performance to test.pl to suppress performance tests.
Performance tests do not need to be run on all platforms. Using vm=none to run performance tests seems best ... for performance.
2020-03-26 20:36:09 -04:00
David Steele
e63fdfbfd2 Debug and optimize flag cleanup for unit tests.
Decisions about when to optimize or enable debug code were spread out in too many places making it hard to keep them consistent.

Centralize the logic as much as possible to make it easier to maintain.
2020-03-26 11:16:35 -04:00
David Steele
f9c86b11a5 More improvements to custom coverage report.
* Fix a few issues with file names being truncated introduced in 787d3fd6.

* Use function line info from the lcov file to calculate which lines to show for uncovered functions.  This is more accurate than what we were doing before and function comment headers are now excluded which reduces clutter in the report.
2020-03-23 12:17:34 -04:00
David Steele
8c76ea2d93 Fix space lost in d70ca259. 2020-03-22 14:18:16 -04:00
David Steele
3ec630f5b2 Allow suppression of times in testing for reproducibility.
Timestamps, timings, etc. cause a lot of log churn when included in documentation.
2020-03-22 10:12:29 -04:00
David Steele
cfab67a775 Enable coverage testing on Fedora 30.
Now that coverage testing works reliably with gcc9 it makes sense to enable it for CI.
2020-03-20 13:49:23 -04:00
David Steele
782c9f89f4 Remove old coverage data before starting new test.
The old coverage data has been recorded so it is no longer needed. In newer versions of gcc leaving this file around can lead to an error when writing profile data after forking off to a non-pgbackrest binary (which we do in some unit tests).
2020-03-20 13:43:08 -04:00
David Steele
787d3fd67b Improve custom coverage report.
* Show all uncovered branch parts even when there are more than two parts per branch. This is the way gcc9 reports coverage so it needs to work even if it doesn't make as much sense as the old way.

* Show covered branches in functions where coverage is missing.  Showing just the uncovered branches can be confusing because it's not always clear how the coverage relates to the code.  By showing all branch coverage (+ or -) this correspondence is made easier.
2020-03-20 12:54:29 -04:00
David Steele
8af8029006 Fix lcov report when test module missing coverage.
We don't report branch coverage on test modules (e.g. test/src/module/common/errorTest.c) but the code that excluded branch coverage from the test module would also exclude it from all core modules if the test module was included in the lcov report due to lack of function/line coverage.

Adjust the coverage code to only exclude branches during the extraction of test module coverage.
2020-03-20 09:16:38 -04:00
David Steele
f6e9bb0819 Remove obsolete -O2 option for Fedora 30 unit test builds.
For some reason gcc9 would not do -O0 builds in combination with one of the options that libperl required.  Now that libperl is gone this exception is no longer required.
2020-03-19 19:30:09 -04:00
David Steele
d677b07081 Move coverage code to CoverageTest module.
This code needs some work, which will be easier if it is all in one place.
2020-03-19 12:07:51 -04:00
David Steele
9a47b88da3 Add links to custom coverage report.
When multiple files were missing coverage it could be hard to locate the coverage report for a specific file.

Add links for uncovered files to make this easier.

Also move table titles out of the table so they are valid html.
2020-03-16 20:02:36 -04:00
David Steele
4328bc1ac6 Move raw coverage results to test/result/raw path.
These results were stored in the vagrant path along with a full copy of src.

Instead store the raw coverage data in test/result/raw and change source references to the files that already exist in [test-path]/repo.
2020-03-16 08:41:32 -04:00
David Steele
d702249507 Build binaries in the test path rather than the vagrant path.
It makes more sense to build in the test path since many developers won't have a vagrant path. Anyway, it's better not to modify the vagrant path since it belongs to vagrant.

Instead of installing the binary just mount it into the container from where it was built. This saves a bit of time and space.
2020-03-15 10:09:27 -04:00
David Steele
959dce569b Update code classification and remove XS definition. 2020-03-14 18:30:24 -04:00
David Steele
213cc6e8be Move docker files to test/result. 2020-03-14 15:40:37 -04:00
David Steele
75ff25f17f Move profile results to test/result. 2020-03-14 14:50:36 -04:00
David Steele
0ba8062f5f Get package source files dynamically during package build.
The prior method was to build a special container to hold these files which meant they would get stale on development systems.  On CI the container was always rebuilt so failures would be seen there even when dev seemed to be working.

Instead get the package source when the package is built to ensure it is as up-to-date as possible.

This change was prompted by failures on the Ubuntu 12.04 container while getting the package source, probably due to an ancient version of git.  Package builds are no longer supported on that platform with the addition of lz4 compression so it didn't seem worth fixing.
2020-03-12 08:48:45 -04:00
David Steele
4a5bd002c0 Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo.
The primary source for project info is now src/version.h.

The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 17:57:02 -04:00
David Steele
731b862e6f Rename BackRestDoc Perl module to pgBackRestDoc.
This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523.

More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 15:41:56 -04:00
David Steele
36d4ab9bff Move Perl modules out of lib directory.
This directory was once the home of the production Perl code but since f0ef73db this is no longer true.

Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well).

The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 15:12:44 -04:00
David Steele
c279a00279 Add lz4 compression support.
LZ4 compresses data faster than gzip but at a lower ratio.  This can be a good tradeoff in certain scenarios.

Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 14:45:27 -04:00
David Steele
79cfd3aebf Remove LibC.
This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests.  This is expensive code to maintain just for testing.

The main dependency was the interface to storage, no matter where it was located, e.g. S3.  Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line.

The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary.  Replace these with hard-coded strings in most places and create new constants for commonly used values.

Remove all auto-generated Perl code.  This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm.  This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired.

Update test.pl and related code to remove LibC builds.

Ding, dong, LibC is dead.
2020-03-09 17:41:59 -04:00
David Steele
02aa03d1a2 Remove obsolete methods in pgBackRest::Storage::Storage module.
All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible.

First remove storageDb() and use storageTest() instead.  Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC.  Now the only storage using the LibC interface is storageRepo().

Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed.  Same for owner().

Remove pathSync() because syncs are not required in the tests.  No test data is reused after a crash.

Path create/exists functions should never be explicitly performed on a repo so remove those.  File exists can be implemented by calling info() instead.

Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete.

Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used.  That also means there are no filters returning results so remove all the result code.

Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 14:10:09 -05:00
David Steele
e2c304d473 Prevent defunct processes in asynchronous archive commands.
The main improvement is a double-fork to prevent zombie processes if the parent process exits after the (child) async process. This is a real possibility since the parent process sticks around to monitor the results of the async process.

In the first fork, ignore SIGCHLD in the very unlikely case that the async process exits before the first fork. This is probably only possible if the async process exits immediately, perhaps due to a chdir() failure. Set SIGCHLD back to default in the async process so waitpid() will work as expected.

Also update the comment on chdir() to more accurately reflect what is happening.

Finally, add a test in certain debug builds to ensure the first fork exits very quickly. This only works when valgrind is not in use because valgrind makes forking so slow that it is hard to tell if the async process performed work or not (in the case that the second fork goes missing and the async process is a direct child).
2020-02-12 12:17:23 -07:00
Cynthia Shang
856980ae99 Auto-select backup set on restore when time target is specified.
Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used.

Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 14:38:05 -07:00
David Steele
90abc3cf17 Use pkg-config instead of xml2-config for libxml2 build options.
pkg-config is a generic way to get build options rather than relying on a package-specific utility.

XML2_CONFIG can be used to override this utility for systems that do not ship pkg-config.
2020-01-24 10:08:05 -07:00
David Steele
94842ccece Fix comment. 2020-01-21 11:59:25 -07:00
David Steele
03d434c7e1 Remove RHEL package patch now that it has been merged upstream.
Also revert 731ffcfb and update ContainerTest.pm for upstream changes.
2020-01-21 11:57:59 -07:00
David Steele
e81629b442 Reclassify Perl and LibC code as test/harness.
These were still being included in the core totals but they are no longer used by core.
2020-01-15 13:53:30 -07:00
David Steele
7a1871c341 Fix test log message to match pg-version parameter name.
It was confusing that this part of the log message did not match the parameter name, which made reproducing test failures from CI a little harder.
2020-01-08 09:54:44 -07:00
David Steele
74c3842595 Remove errant tabs and fix spacing. 2019-12-19 16:25:46 -05:00
David Steele
977ec2e307 Integration test improvements for disk and memory efficiency.
Set log-level-file=off when more that one test will run.  In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test.  This improves performance pretty dramatically since trace-level logging is expensive.  If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option.

Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel.

Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 15:23:07 -05:00
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
b031dbbcf8 Allow timezones to be explicitly set for testing.
The TZ environment variable was not reliably pushed down to the test processes.

Instead pass TZ via a command line parameter and set explicitly in the test process.
2019-12-11 22:11:04 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00