1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

555 Commits

Author SHA1 Message Date
David Steele
3c4f91b319 Remove Perl unit tests made obsolete in 434cd832.
These were replaced by C unit tests but not all the unit test setup code was removed in the Perl module.
2020-03-09 13:35:26 -04:00
David Steele
438b957f9c Add infrastructure for multiple compression type support.
Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function.

Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format).

Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types.

Additional compression types will be added in future commits.
2020-03-06 14:41:03 -05:00
David Steele
02aa03d1a2 Remove obsolete methods in pgBackRest::Storage::Storage module.
All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible.

First remove storageDb() and use storageTest() instead.  Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC.  Now the only storage using the LibC interface is storageRepo().

Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed.  Same for owner().

Remove pathSync() because syncs are not required in the tests.  No test data is reused after a crash.

Path create/exists functions should never be explicitly performed on a repo so remove those.  File exists can be implemented by calling info() instead.

Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete.

Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used.  That also means there are no filters returning results so remove all the result code.

Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 14:10:09 -05:00
David Steele
00647c7109 Remove Perl Db module and LibC dependencies.
This was mostly dead code except the DB_BACKUP_ADVISORY_LOCK constant, moved to the real/all test module, and the function that pulls info from pg_control, moved to ExpireEnvTest.pm.
2020-03-06 07:21:17 -05:00
David Steele
2e0fe25650 Remove dependency on LibC hash filter.
Perl provides Digest::SHA for hashing so there is no need to expose this via LibC anymore.
2020-03-05 18:34:59 -05:00
David Steele
eb4347f20b Use static checksums in mock/all integration tests.
Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests.

We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 13:56:20 -05:00
David Steele
4ab8943ca8 Use PG_PAGE_SIZE_DEFAULT constant instead of pageSize variable.
Page size is passed around a lot but in fact it can only have one value, PG_PAGE_SIZE_DEFAULT, which is checked when pg_control is loaded. There may be an argument for supporting multiple page sizes in the future but for now just use the constant to simplify the code.

There is also a significant performance benefit.  Because pageSize was being used in pageChecksumBlock() the main loop was neither unrolled nor vectorized (-funroll-loops -ftree-vectorize) as it is now with a constant loop boundary.
2020-03-05 09:14:27 -05:00
David Steele
91f321fb86 Rename old page*() functions to conform to new conventions.
The general convention now is to prefix PostgreSQL functions with "pg".
2020-03-04 14:24:40 -05:00
Cynthia Shang
ceb050e950 Fix flapping test in real/all module.
The restore test function was passing strBackup to the restoreCompare function but when the restore is expected to pick a backup based on a timestamp, then strBackup may not be the one chosen.

Modified the code so that strBackupExpected is set based on the parameters passed to the function and this is then passed to restoreCompare.
2020-02-28 14:50:50 -05:00
David Steele
e2c304d473 Prevent defunct processes in asynchronous archive commands.
The main improvement is a double-fork to prevent zombie processes if the parent process exits after the (child) async process. This is a real possibility since the parent process sticks around to monitor the results of the async process.

In the first fork, ignore SIGCHLD in the very unlikely case that the async process exits before the first fork. This is probably only possible if the async process exits immediately, perhaps due to a chdir() failure. Set SIGCHLD back to default in the async process so waitpid() will work as expected.

Also update the comment on chdir() to more accurately reflect what is happening.

Finally, add a test in certain debug builds to ensure the first fork exits very quickly. This only works when valgrind is not in use because valgrind makes forking so slow that it is hard to tell if the async process performed work or not (in the case that the second fork goes missing and the async process is a direct child).
2020-02-12 12:17:23 -07:00
Cynthia Shang
856980ae99 Auto-select backup set on restore when time target is specified.
Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used.

Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 14:38:05 -07:00
David Steele
90abc3cf17 Use pkg-config instead of xml2-config for libxml2 build options.
pkg-config is a generic way to get build options rather than relying on a package-specific utility.

XML2_CONFIG can be used to override this utility for systems that do not ship pkg-config.
2020-01-24 10:08:05 -07:00
David Steele
94842ccece Fix comment. 2020-01-21 11:59:25 -07:00
David Steele
03d434c7e1 Remove RHEL package patch now that it has been merged upstream.
Also revert 731ffcfb and update ContainerTest.pm for upstream changes.
2020-01-21 11:57:59 -07:00
David Steele
e81629b442 Reclassify Perl and LibC code as test/harness.
These were still being included in the core totals but they are no longer used by core.
2020-01-15 13:53:30 -07:00
David Steele
7a1871c341 Fix test log message to match pg-version parameter name.
It was confusing that this part of the log message did not match the parameter name, which made reproducing test failures from CI a little harder.
2020-01-08 09:54:44 -07:00
David Steele
33e328abbf Remove unused LibC code.
The code was made obsolete by the migration to C.
2019-12-28 18:30:32 -07:00
David Steele
74c3842595 Remove errant tabs and fix spacing. 2019-12-19 16:25:46 -05:00
David Steele
620386f034 Remove integration tests that are now covered in the unit tests.
Most of these tests are just checking that errors are thrown when required.  These are well covered in various unit tests.

The "cannot resume" tests are also well covered in the backup unit tests.

Finally, config warnings are well covered in the config unit tests.

There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-17 20:14:45 -05:00
David Steele
977ec2e307 Integration test improvements for disk and memory efficiency.
Set log-level-file=off when more that one test will run.  In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test.  This improves performance pretty dramatically since trace-level logging is expensive.  If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option.

Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel.

Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 15:23:07 -05:00
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
1f2ce45e6b The backup command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C except as noted below.

A backup can now be initiated from a linked directory.  The link will not be stored in the manifest or recreated on restore.  If a link or directory does not already exist in the restore location then a directory will be created.

The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew.  This has never been an issue in the field that we know of, but we found it in testing.

For online backups all times are fetched from the PostgreSQL primary host (before only copy start was).  This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting.

Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not.  This was a long-standing bug in the Perl code.

Resume will now work even if hardlink settings have been changed.

Reviewed by Cynthia Shang.
2019-12-13 17:14:26 -05:00
David Steele
b031dbbcf8 Allow timezones to be explicitly set for testing.
The TZ environment variable was not reliably pushed down to the test processes.

Instead pass TZ via a command line parameter and set explicitly in the test process.
2019-12-11 22:11:04 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
David Steele
d7d663c2b9 Make buildPutDiffers() work with empty files.
If the file was empty the timestamp was updated.  If the file is empty and there is no content then file should not be saved.
2019-12-10 13:02:36 -05:00
David Steele
e632c60525 Fix backup labels in mock/all resume integration tests.
These were not getting updated to match the directory name when the manifests were copied.

The Perl code didn't care but the C code expects labels to be set correctly.
2019-12-06 11:48:41 -05:00
David Steele
8dfe0e48e2 Use more general error code when tablespace linked into PGDATA.
The specific error code was not that useful since we also test the error message which contains details of the link error.
2019-12-02 10:49:25 -05:00
David Steele
fc291b6f28 Reduce the scope of mock/all exclusion tests.
Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-01 17:47:47 -05:00
David Steele
686b6f91da Set archive-check option in manifest correctly when offline.
Archive check does not run when in offline backup mode but the option was set to true in the manifest.  It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 08:27:21 -05:00
David Steele
158e439689 Remove obsolete Perl archive code.
This should have been removed in a1c13a50 but was missed.
2019-11-26 17:16:45 -05:00
David Steele
82df7e6f3b Update integration tests in real/all that use test points.
Test points are not supported by the new C code so these will be replaced with unit tests.

The fact that the tests still pass even when the changes aren't made mid-backup (except application_name) shows how weak they were in the first place.

Even so, this does represent a regression in (soon to be be removed) Perl coverage.
2019-11-26 11:32:12 -05:00
David Steele
8800f32ad9 Remove exclusions once they have been tested in mock/all.
The exclusions no longer have any effect after a restore and just add noise to the expect log.
2019-11-25 08:35:26 -05:00
David Steele
9856fef586 Update integration tests in mock/all that use test points.
Test points will not be available in the C code so update these tests as best as possible without using them.

This represents a loss of coverage for the Perl code (soon to be removed) which will be made up in the C code with unit tests.
2019-11-25 07:48:52 -05:00
David Steele
3cd45a7411 Remove start/stop --force integration tests in mock/all.
These tests require test points which are not being implemented in the C code.

This functionality is fully tested in the command/control unit tests so integration tests are no longer required.
2019-11-25 07:45:58 -05:00
David Steele
01aefc563d Update Perl page checksum expression.
This expression determines which files contain page checksums but it was also including the directory above the relation directories.  In a real PostgreSQL installation this not a problem because these directories don't contain any files.

However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no.

Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 07:37:09 -05:00
David Steele
a4b9440d35 Only install specific lcov version when required.
Installing lcov 1.14 everywhere turned out to be a problem just as using 1.13 on Ubuntu 19.04 was.

Since we primarily use Ubuntu 18.04 for coverage testing and reporting, we definitely want to make sure that works.  So, revert to using the default packaged lcov except when specified otherwise in VmTest.pm.

PostgreSQL minor version releases are also included since all containers have been rebuilt.
2019-11-22 19:25:49 -05:00
David Steele
c524ec4f95 Remove obsolete integration tests from mock/all.
The protocol timeout tests have been superceded by unit tests.

The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto.  It didn't hurt anything but it did add 200ms to each test where it appeared.

Catalog and control version tests were redundant.  The database version and system id tests covered the important code paths and the C code gets these values from a lookup table.

Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 16:06:27 -05:00
David Steele
2d10293d04 v2.19: C Migrations and Bug Fixes
Bug Fixes:

* Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
* Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)

Improvements:

* JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang.)
* The check command is implemented entirely in C. (Contributed by Cynthia Shang.)

Documentation Improvements:

* Document how to contribute to pgBackRest. (Contributed by Cynthia Shang.)
* Document maximum version for auto-stop option. (Contributed by Brad Nicholson.)

Test Suite Improvements:

* Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
* Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
* Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
2019-11-12 15:51:28 -05:00
David Steele
4317178633 Update MinIO to newest release.
We had some problems with newer versions so had held off on updating.  Those problems appear to have been resolved.

In addition, the --compat flag is no longer required.  Prior versions of MinIO required all parts of a multi-part upload (except the last) to be of equal size.  The --compat flag was introduced to restore the default S3 behavior.  Now --compat is only required when ETag is being used for MD5 verification, which we don't do.
2019-11-08 17:56:34 -05:00
David Steele
8b682b75d2 Allow mock integration tests for all VM types.
Previously the mock integration tests would be skipped for VMs other than the standard four used in CI.  Now VMs outside the standard four will run the same tests as VM4 (currently U18).
2019-11-02 10:35:48 +01:00
David Steele
e06db21e35 Error when specified vm is invalid. 2019-10-17 14:00:18 +02:00
David Steele
fa6a54bb45 Update last tests that required sudo.
All tests should now run in a sudo-less environment.
2019-10-16 17:05:24 +02:00
David Steele
48bd9e22f1 C test harness refactor.
Consolidate setting configuration into hrnInit() and rename other functions for consistency.

Split out internal functions into a new header.
2019-10-16 15:48:33 +02:00
David Steele
a2fa1d04b0 Update container images to PostgreSQL 12 GA. 2019-10-12 11:26:13 -04:00
David Steele
397a41e0f9 Add Ubuntu 19.04 container definition. 2019-10-12 11:24:55 -04:00
David Steele
93656db186 Update lcov to 1.14.
1.13 is not compatible with gcc 8 which is what ships with newer distributions.  Build from source to get a more recent version.

1.13 is not compatible with gcc 9 so we'll need to address that at a later date.
2019-10-12 11:24:21 -04:00
David Steele
11c7c8fabb Remove pgbackrest test user.
This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories.  The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user.

Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user.  This allows us to remove most sudos, except where they are explicitly needed in tests.

While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 09:45:18 -04:00
David Steele
6f0e7f00af Fix recovery test failing in PostgreSQL 12.0.
This test was not creating recovery.signal when testing with --type=preserve.  The preserve recovery type only keeps existing files and does not create any.

RC1 was just ignoring recovery.signal and going right into recovery.  Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.
2019-10-12 09:26:19 -04:00
Cynthia Shang
2972580566 Remove info expect tests from mock/all and mock/stanza.
These tests are redundant now that we have full coverage in the unit tests are are not worth maintaining anymore.
2019-10-11 12:38:03 -04:00
David Steele
6db4e59a66 Allow tests that use ports to run in parallel.
Set the test index in the C unit test code so it can assign port numbers that won't conflict between tests.
2019-10-10 16:13:43 -04:00