1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

675 Commits

Author SHA1 Message Date
David Steele
f0ef73db70 pgBackRest is now pure C.
Remove embedded Perl from the distributed binary.  This includes code, configure, Makefile, and packages.  The distributed binary is now pure C.

Remove storagePathEnforceSet() from the C Storage object which allowed Perl to write outside of the storage base directory.  Update mock/all and real/all integration tests to use storageLocal() where they were violating this rule.

Remove "c" option that allowed the remote to tell if it was being called from C or Perl.

Code to convert options to JSON for passing to Perl (perl/config.c) has been moved to LibC since it is still required for Perl integration tests.

Update build and installation instructions in the user guide.

Remove all Perl unit tests.

Remove obsolete Perl code.  In particular this included all the Perl protocol code which required modifications to the Perl storage, manifest, and db objects that are still required for integration testing but only run locally.  Any remaining Perl code is required for testing, documentation, or code generation.

Rename perlReq to binReq in define.yaml to indicate that the binary is required for a test.  This had been the actual meaning for quite some time but the key was never renamed.
2019-12-13 17:55:41 -05:00
David Steele
1f2ce45e6b The backup command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C except as noted below.

A backup can now be initiated from a linked directory.  The link will not be stored in the manifest or recreated on restore.  If a link or directory does not already exist in the restore location then a directory will be created.

The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew.  This has never been an issue in the field that we know of, but we found it in testing.

For online backups all times are fetched from the PostgreSQL primary host (before only copy start was).  This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting.

Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not.  This was a long-standing bug in the Perl code.

Resume will now work even if hardlink settings have been changed.

Reviewed by Cynthia Shang.
2019-12-13 17:14:26 -05:00
David Steele
b031dbbcf8 Allow timezones to be explicitly set for testing.
The TZ environment variable was not reliably pushed down to the test processes.

Instead pass TZ via a command line parameter and set explicitly in the test process.
2019-12-11 22:11:04 -05:00
David Steele
d0ba8ff58c Remove test point infrastructure.
82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code.

Since tests points are no longer used remove the infrastructure.

Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 13:16:47 -05:00
David Steele
d7d663c2b9 Make buildPutDiffers() work with empty files.
If the file was empty the timestamp was updated.  If the file is empty and there is no content then file should not be saved.
2019-12-10 13:02:36 -05:00
David Steele
e632c60525 Fix backup labels in mock/all resume integration tests.
These were not getting updated to match the directory name when the manifests were copied.

The Perl code didn't care but the C code expects labels to be set correctly.
2019-12-06 11:48:41 -05:00
David Steele
8dfe0e48e2 Use more general error code when tablespace linked into PGDATA.
The specific error code was not that useful since we also test the error message which contains details of the link error.
2019-12-02 10:49:25 -05:00
David Steele
fc291b6f28 Reduce the scope of mock/all exclusion tests.
Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-01 17:47:47 -05:00
David Steele
686b6f91da Set archive-check option in manifest correctly when offline.
Archive check does not run when in offline backup mode but the option was set to true in the manifest.  It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 08:27:21 -05:00
David Steele
158e439689 Remove obsolete Perl archive code.
This should have been removed in a1c13a50 but was missed.
2019-11-26 17:16:45 -05:00
David Steele
82df7e6f3b Update integration tests in real/all that use test points.
Test points are not supported by the new C code so these will be replaced with unit tests.

The fact that the tests still pass even when the changes aren't made mid-backup (except application_name) shows how weak they were in the first place.

Even so, this does represent a regression in (soon to be be removed) Perl coverage.
2019-11-26 11:32:12 -05:00
David Steele
8800f32ad9 Remove exclusions once they have been tested in mock/all.
The exclusions no longer have any effect after a restore and just add noise to the expect log.
2019-11-25 08:35:26 -05:00
David Steele
9856fef586 Update integration tests in mock/all that use test points.
Test points will not be available in the C code so update these tests as best as possible without using them.

This represents a loss of coverage for the Perl code (soon to be removed) which will be made up in the C code with unit tests.
2019-11-25 07:48:52 -05:00
David Steele
3cd45a7411 Remove start/stop --force integration tests in mock/all.
These tests require test points which are not being implemented in the C code.

This functionality is fully tested in the command/control unit tests so integration tests are no longer required.
2019-11-25 07:45:58 -05:00
David Steele
01aefc563d Update Perl page checksum expression.
This expression determines which files contain page checksums but it was also including the directory above the relation directories.  In a real PostgreSQL installation this not a problem because these directories don't contain any files.

However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no.

Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 07:37:09 -05:00
David Steele
a4b9440d35 Only install specific lcov version when required.
Installing lcov 1.14 everywhere turned out to be a problem just as using 1.13 on Ubuntu 19.04 was.

Since we primarily use Ubuntu 18.04 for coverage testing and reporting, we definitely want to make sure that works.  So, revert to using the default packaged lcov except when specified otherwise in VmTest.pm.

PostgreSQL minor version releases are also included since all containers have been rebuilt.
2019-11-22 19:25:49 -05:00
David Steele
c524ec4f95 Remove obsolete integration tests from mock/all.
The protocol timeout tests have been superceded by unit tests.

The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto.  It didn't hurt anything but it did add 200ms to each test where it appeared.

Catalog and control version tests were redundant.  The database version and system id tests covered the important code paths and the C code gets these values from a lookup table.

Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 16:06:27 -05:00
David Steele
2d10293d04 v2.19: C Migrations and Bug Fixes
Bug Fixes:

* Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
* Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)

Improvements:

* JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang.)
* The check command is implemented entirely in C. (Contributed by Cynthia Shang.)

Documentation Improvements:

* Document how to contribute to pgBackRest. (Contributed by Cynthia Shang.)
* Document maximum version for auto-stop option. (Contributed by Brad Nicholson.)

Test Suite Improvements:

* Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
* Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
* Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
2019-11-12 15:51:28 -05:00
David Steele
4317178633 Update MinIO to newest release.
We had some problems with newer versions so had held off on updating.  Those problems appear to have been resolved.

In addition, the --compat flag is no longer required.  Prior versions of MinIO required all parts of a multi-part upload (except the last) to be of equal size.  The --compat flag was introduced to restore the default S3 behavior.  Now --compat is only required when ETag is being used for MD5 verification, which we don't do.
2019-11-08 17:56:34 -05:00
David Steele
8b682b75d2 Allow mock integration tests for all VM types.
Previously the mock integration tests would be skipped for VMs other than the standard four used in CI.  Now VMs outside the standard four will run the same tests as VM4 (currently U18).
2019-11-02 10:35:48 +01:00
David Steele
e06db21e35 Error when specified vm is invalid. 2019-10-17 14:00:18 +02:00
David Steele
fa6a54bb45 Update last tests that required sudo.
All tests should now run in a sudo-less environment.
2019-10-16 17:05:24 +02:00
David Steele
48bd9e22f1 C test harness refactor.
Consolidate setting configuration into hrnInit() and rename other functions for consistency.

Split out internal functions into a new header.
2019-10-16 15:48:33 +02:00
David Steele
a2fa1d04b0 Update container images to PostgreSQL 12 GA. 2019-10-12 11:26:13 -04:00
David Steele
397a41e0f9 Add Ubuntu 19.04 container definition. 2019-10-12 11:24:55 -04:00
David Steele
93656db186 Update lcov to 1.14.
1.13 is not compatible with gcc 8 which is what ships with newer distributions.  Build from source to get a more recent version.

1.13 is not compatible with gcc 9 so we'll need to address that at a later date.
2019-10-12 11:24:21 -04:00
David Steele
11c7c8fabb Remove pgbackrest test user.
This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories.  The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user.

Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user.  This allows us to remove most sudos, except where they are explicitly needed in tests.

While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 09:45:18 -04:00
David Steele
6f0e7f00af Fix recovery test failing in PostgreSQL 12.0.
This test was not creating recovery.signal when testing with --type=preserve.  The preserve recovery type only keeps existing files and does not create any.

RC1 was just ignoring recovery.signal and going right into recovery.  Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.
2019-10-12 09:26:19 -04:00
Cynthia Shang
2972580566 Remove info expect tests from mock/all and mock/stanza.
These tests are redundant now that we have full coverage in the unit tests are are not worth maintaining anymore.
2019-10-11 12:38:03 -04:00
David Steele
6db4e59a66 Allow tests that use ports to run in parallel.
Set the test index in the C unit test code so it can assign port numbers that won't conflict between tests.
2019-10-10 16:13:43 -04:00
David Steele
13fcbb24e9 Fix container test path being used when --vm=none.
Suggested by Stephen Frost.
2019-10-10 15:09:11 -04:00
David Steele
9a3ba649e1 Remove code to generate .travis.yml.
Most of the logic has been moved to test/travis.pl so there wasn't much purpose to this code anymore.
2019-10-10 11:25:59 -04:00
David Steele
7f369006b5 Add gcc 9 support.
A number of tests have been updated and Fedora 30 has been added to the test suite so the unit tests can run on gcc 9.

Stop running unit tests on co6/7 since we appear to have ample unit test coverage.
2019-10-09 15:03:03 -04:00
David Steele
528f4c4347 Remove dependency on aws cli for testing.
This tool was only being used it a few places but was a pretty large dependency.

Rework the forceStorageMove() code using our storage layer and replace one aws cli cp with a storage put.

Also, remove the Dockerfile that was once used to build the Scality S3 test container.
2019-10-09 14:38:24 -04:00
David Steele
61c4f64895 Be smarter about which packages are loaded for testing.
Now that our tests are more diversified it makes sense to load only the packages that are needed for each test.

Move the package loads from .travis.yaml to test/travis.pl where we have more control over what is loaded.
2019-10-08 18:56:55 -04:00
Cynthia Shang
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
David Steele
45881c74ae Allow most unit tests to run outside of a container.
Three major changes were required to get this working:

1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.

1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.

2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.

Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.

Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
David Steele
29e132f5e9 PostgreSQL 12 support.
Recovery settings are now written into postgresql.auto.conf instead of recovery.conf.  Existing recovery_target* settings will be commented out to help avoid conflicts.

A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed.

recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
David Steele
f1ba428fb0 Add performance test capability in C with scaling.
Scaling allows the starting values to be increased from the command-line without code changes.

Also suppress valgrind and assertions when running performance testing.  Optimization is left at -O0 because we should not be depending on compiler optimizations to make our code performant, and it makes profiling more informative.
2019-09-28 14:02:12 -04:00
David Steele
004ff99a2d Identify Perl performance test by appending -perl.
This is intended to differentiate the upcoming C performance tests from the Perl performance tests that will eventually be migrated.
2019-09-28 13:17:21 -04:00
David Steele
afc483ef86 Clarify which timeline should be used for timeline integration test. 2019-09-27 13:37:59 -04:00
David Steele
d82102d6ef Add explicit promotes to recovery integration tests.
PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead.  Set target_action explicitly so all versions will behave the same way.

This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 13:04:36 -04:00
David Steele
833d0da0d9 Store recovery file name in integration when testing preserve recovery.
This makes the test a little more maintainable and is friendly with the changes needed for PostgreSQL 12.
2019-09-27 12:29:33 -04:00
David Steele
80eb561caf Add missing PostgreSQL 11 control/WAL versions in Perl tests.
These values don't seem to be used for testing but better to be tidy.
2019-09-27 09:45:11 -04:00
David Steele
d6a6d93a04 Add PostgreSQL 12 to u18 container.
This does not add PostgresQL 12 support; it simply adds PostgreSQL 12 to the u18 container for development and testing.
2019-09-27 09:35:59 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
451ae397be The restore command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C.

There is one important behavioral change with regard to how file permissions are handled.  The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user.  This usually just led to errors and frustration.

The C code works like this:

If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.

If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.

Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
David Steele
e968acbdd7 Fix outdated comment.
This was probably missed when a new test was added and the timeline was updated.
2019-09-24 16:55:11 -04:00
David Steele
d3a7055ee5 Only enable test.pl --debug-test-trace option when --debug also enabled.
The other way makes no sense and leads to compile errors since --debug-test-trace requires some code that is only enabled by --debug.
2019-09-23 15:15:04 -04:00
Cynthia Shang
56bf9d0566 Update HINT messages to conform to new standard detailed in CODING.md. 2019-09-14 12:21:08 -04:00
David Steele
92365fb801 Disable missing-field-initializers warnings in unit testing.
This warning gives very unpredictable results between compiler versions and seems unrealistic since most of our structs are zeroed for initialization.

This warning has been disabled in the Makefile for a long time.
2019-09-12 15:55:18 -04:00
David Steele
f809d2f008 Ignore apt-get update errors in Travis CI.
Broken vendor packages have been causing builds to break due to an error on apt-get update.

Ignore errors and proceed directory to apt-get install.  It's possible that we'll try to reference an expired package version and get an error anyway, but that seems better than a guaranteed hard error.
2019-09-12 15:16:42 -04:00
David Steele
dca5b63f97 Move documentation job first for Travis CI.
Since this job has been running long recently this should improved overall performance when multiple commits are queued up.
2019-09-10 13:06:44 -04:00
David Steele
f8d0574759 Increase process timeout and emit occasional warnings.
Travis will timeout after 10 minutes with no output.  Emit a warning every 5 minutes to keep Travis alive and increase the total timeout to 20 minutes.

Documentation builds have been timing out a lot recently so hopefully this will help.
2019-09-10 12:29:36 -04:00
David Steele
ce2bf29998 v2.17: C Migrations and Bug Fixes
Bug Fixes:

* Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)
* Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)

Improvements:

* The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)

Documentation Bug Fixes:

* Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)

Documentation Improvements:

* Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)
2019-09-03 16:39:32 -04:00
David Steele
3a28b68b8b Disable S3 and encryption on u18 integration tests for mock/all/1.
This test is commonly used for sanity checking but the combination of S3 and encryption makes it hard to use and encourages temporary changes to make it usable.

Acknowledge this and disable S3 and encryption for this test and move them to mock/all/2.
2019-09-02 19:06:12 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
22aa532be1 Add storage tests for files beginning with dots.
Prevent a regression of the issue fixed in f88012ce by adding some tests.
2019-08-26 11:37:21 -04:00
David Steele
01c2669b97 Fix exclusions for special files.
Prior to 2.16 the Perl manifest code would skip any file that began with a dot.  This was not intentional but it allowed PostgreSQL socket files to be located in the data directory.  The new C code in 2.16 did not have this unintentional exclusion so socket files in the data directory caused errors.

Worse, the file type error was being thrown before the exclusion check so there was really no way around the issue except to move the socket files out of the data directory.

Special file types (e.g. socket, pipe) will now be automatically skipped and a warning logged to notify the user of the exclusion.  The warning can be suppressed with an explicit --exclude.

Reported by CluelessTechnologist, Janis Puris, Rachid Broum.
2019-08-23 07:47:54 -04:00
David Steele
2862f480cd Add special file type to storageInfo().
There's not much we can do with special files, but higher level logic can at least exclude them gracefully rather than throwing a hard error.
2019-08-23 07:24:25 -04:00
David Steele
f88012cef3 Fix regexp to ignore ./.. directories in the Posix driver.
In versions <= 2.15 the old regexp caused any file or directory beginning with . to be ignored during a backup.  This has caused behavioral differences in 2.16 because the new C code correctly excludes ./.. directories.

This Perl code is only used for testing now, but it should still match the output of the C functions.
2019-08-22 10:18:34 -04:00
David Steele
c002a2ce2f Move info file checksum to the end of the file.
Putting the checksum at the beginning of the file made it impossible to stream the file out when saving.  The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning.

Instead place the checksum at the end.  This does not break the existing Perl or C code since the read is not order dependent.

There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient.

Reviewed by Cynthia Shang.
2019-08-21 19:45:48 -04:00
Cynthia Shang
c733319063 The stanza-create/update/delete commands are implemented entirely in C.
Contributed by Cynthia Shang.
2019-08-21 16:26:28 -04:00
David Steele
3df075bf40 Fix test writing "null" into manifest files.
"null" is not allowed in the manifest format (null values should be missing instead) but Perl was treating the invalid values written by this test as if they were missing.

Update the test code to remove the values rather than setting them to "null".
2019-08-18 15:29:18 -04:00
David Steele
f8b0676fd6 Allow modules to be included for testing without requiring coverage.
Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested.  The include directive allows this.

Update modules that had previously been added to coverage that only need to be included.
2019-07-25 20:15:06 -04:00
David Steele
d8ca0e5c5b Add Perl interface to C PgQuery object.
This validates that all current queries work with the new interface and removes the dependency on DBD::Pg.
2019-07-25 17:05:39 -04:00
David Steele
415542b4a3 Add PostgreSQL query client.
This direct interface to libpq allows simple queries to be run against PostgreSQL and supports timeouts.

Testing is performed using a shim that can use scripted responses to test all aspects of the client code.  The shim will be very useful for testing backup scenarios on complex topologies.

Reviewed by Cynthia Shang.
2019-07-25 14:50:02 -04:00
David Steele
59f135340d The local command for backup is implemented entirely in C.
The local process is now entirely migrated to C.  Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C.

Contributed by David Steele, Cynthia Shang.
2019-07-25 14:34:16 -04:00
David Steele
e10577d0b0 Fix incorrect offline upper bound for ignoring page checksum errors.
For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX.  This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode.

Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 09:13:56 -04:00
David Steele
27b3246e85 Exclude more build files from rsync between tests.
Files (especially build.auto.h) were being removed and forcing a full build between separate invocations of test.pl.

This affected ad-hoc testing at the command-line, not a full test run in CI.
2019-07-08 08:29:25 -04:00
David Steele
5e1ed2e8a5 Remove clang static analysis.
This analysis never produced anything but false positives (var might be NULL) but took over a minute per test run and added 600MB to the test container.
2019-07-05 18:34:15 -04:00
David Steele
488fb67294 Force PostgreSQL versions to string for newer versions of JSON:PP.
Since 2.91 JSON::PP has a bias for saving variables that look like numbers as numbers even if they were declared as strings.

Force versions to strings where needed by appending ''.

Update the json-pp-perl package on Ubuntu 18.04 to 2.97 to provide test coverage.
2019-07-05 17:25:01 -04:00
David Steele
9836578520 Remove perl critic and coverage.
No new Perl code is being developed, so these tools are just taking up time and making migrations to newer platforms harder.  There are only a few Perl tests remaining with full coverage so the coverage tool does not warn of loss of coverage in most cases.

Remove both tools and associated libraries.
2019-07-05 16:55:17 -04:00
David Steele
1708f1d151 Use minio for integration testing.
ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing.  Replace it with minio which starts quickly and ships as a single executable or a tiny container.

Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly.

This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-02 22:20:35 -04:00
David Steele
4bffa0c5bb Add test function to create the S3 bucket instead of using aws cli.
Eventually the idea is to remove the dependency on aws cli since Python is a big install.
2019-06-26 15:02:30 -04:00
David Steele
4815752ccc Add Perl interface to C storage layer.
Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.

The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 08:24:58 -04:00
Cynthia Shang
b498188f01 Error on db history mismatch when expiring.
Amend commit 434cd832 to error when the db history in archive.info and backup.info do not match.

The Perl code would attempt to reconcile the history by matching on system id and version but we are not planning to migrate that code to C.  It's possible that there are users with mismatches but if so they should have been getting errors from info for the last six months.  It's easy enough to manually fix these files if there are any mismatches in the field.

Contributed by Cynthia Shang.
2019-06-24 11:59:44 -04:00
David Steele
434cd83285 The expire command is implemented entirely in C.
This implementation duplicates the functionality of the Perl code but does so with different logic and includes full unit tests.

Along the way at least one bug was fixed, see issue #748.

Contributed by Cynthia Shang.
2019-06-18 15:19:20 -04:00
David Steele
f88bee7b33 TLS/HTTP statistics log replacements.
These statistics can change with any code update so they cause a lot of churn in the expect logs.
2019-06-18 10:13:28 -04:00
David Steele
0a96a2895d Add storage layer for tests and documentation.
The tests and documentation have been using the core storage layer but soon that will depend entirely on the C library, creating a bootstrap problem (i.e. the storage layer will be needed to build the C library).

Create a simplified Posix storage layer to be used by documentation and the parts of the test code that build and execute the actual tests.  The actual tests will still use the core storage driver so they can interact with any type of storage.
2019-06-17 09:16:44 -04:00
David Steele
9ba95e993b Use retries to wait for test S3 server to start.
The prior method of tailing the docker log no longer seems reliable.  Instead, keep retrying the make bucket command until it works and show the error if it times out.
2019-06-13 17:58:33 -04:00
David Steele
6ff3325c77 Enforce requiring repo-cipher-pass at config parse time.
This was not enforced at parse time because repo1-cipher-type could be passed on the command-line even in cases where encryption was not needed by the subprocess.

Filter repo-cipher-type so it is never passed on the command line.  If the subprocess does not have access to the passphrase then knowing the encryption type is useless anyway.
2019-06-05 11:43:17 -04:00
David Steele
92e04ea9f4 Remove per-stanza repo cache clear during testing.
This was not being used and is not supported by the equivalent C code.
2019-06-04 10:34:19 -04:00
David Steele
12bca3c43e Add CPPFLAGS to compile rules.
This should silence the last of the Debian package warnings.
2019-06-01 09:28:31 -04:00
David Steele
a2ec1253e9 Add code classification exclusion missed in 3e1b06ac. 2019-05-30 10:44:35 -04:00
David Steele
3e1b06acaa Use minio as local S3 emulator in documentation.
The documentation was relying on a ScalityS3 container built for testing which wasn't very transparent.  Instead, use the stock minio container and configure it in the documentation.

Also, install certificates and CA so that TLS verification can be enabled.
2019-05-27 07:37:20 -04:00
David Steele
86482c7db9 Reduce log level for all expect tests to detail.
The C code is designed to be efficient rather than deterministic at the debug log level.  As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level.

Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level.  Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-22 18:23:44 -04:00
David Steele
e3fe3434b4 Rename repo-s3-verify-ssl option to repo-s3-verify-tls.
The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure).

The old name will continue to be accepted.
2019-05-21 10:14:41 -04:00
Cynthia Shang
19d8358cba Update mock/expire module test matrix so expect tests output.
Also add an error message to prevent regression.

Contributed by Cynthia Shang.
2019-05-16 09:53:55 -04:00
Cynthia Shang
18d4cb5741 Bypass database checks when stanza-delete issued with force.
Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted.

Contributed by Cynthia Shang.
Suggested by Roman.
2019-05-15 13:14:58 -04:00
David Steele
5c1d4bcd0d Automate coverage summary report generation.
This report replaces the lcov report that was generated manually for each release.

The lcov report was overly verbose just to say that we have virtually 100% coverage.
2019-05-15 13:04:56 -04:00
David Steele
5bba72b874 Remove -Wswitch-enum compiler option.
The -Wswitch option included in -Wall provides the same level of coverage and allows enum options to be grouped into default.
2019-05-15 12:55:08 -04:00
David Steele
87f36e814e Improve macros and coverage rules that were hiding missing coverage.
The branch coverage exclusion rules were overly broad and included functions that ended in a capital letter, which disabled all coverage for the statement.  Improve matching so that all characters in the name must be upper-case for a match.

Some macros with internal branches accepted parameters that might contain conditionals.  This made it impossible to tell which branches belonged to which, and in any case an overzealous exclusion rule was ignoring all branches in such cases.  Add the DEBUG_COVERAGE flag to build a modified version of the macros without any internal branches to be used for coverage testing.  In most cases, the branches were optimizations (like checking logWill()) that improve production performance but are not needed for testing.  In other cases, a parameter needed to be added to the underlying function to handle the branch during coverage testing.

Also tweak the coverage rules so that macros without conditionals are automatically excluded from branch coverage as long as they are not themselves a parameter.

Finally, update tests and code where missing coverage was exposed by these changes.  Some code was updated to remove existing coverage exclusions when it was a simple change.
2019-05-11 14:51:51 -04:00
David Steele
cb00030ee3 Remove dead code missed in 1b486847.
This commit removed all Perl references to spool storage but some stuff was left behind.
2019-05-08 18:58:07 -04:00
David Steele
8c712d89eb Improve type safety of interfaces and drivers.
The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations.  Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been.

Use void pointers instead so that no casts are required.  Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer.

Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type.  This cuts down on accessor methods and the vast majority of those functions were not being used.  Move functions that are still required to .intern.h.

Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
David Steele
28359eea83 Update code count rules missed in 027c2638. 2019-05-02 16:33:23 -04:00
David Steele
027c263871 Add configure script for improved multi-platform support.
Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile.

Each C file must include "build.auto.conf" before all other includes and defines.  This is enforced by test.pl for includes, but it won't detect incorrect define ordering.

Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
David Steele
3505559a80 Update test containers with PostgreSQL minor releases and liblz4.
Update RHEL repos that have changed upstream.  Remove PostgreSQL 9.3 since the RHEL6/7 packages have disappeared.

Remove PostgreSQL versions from U12 that are still getting minor updates so the container does not need to be rebuilt.

LZ4 is included for future development, but this seems like a good time to add it to the containers.
2019-04-24 13:23:32 -04:00
David Steele
1ae8a6a716 Add build-max option to set max build processes.
Currently this controls make processes via -j.
2019-04-23 20:52:03 -04:00
David Steele
c11c936366 Reduce ScalityS3 processes since only two are needed. 2019-04-23 20:19:31 -04:00