1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1179 Commits

Author SHA1 Message Date
David Steele
48bd9e22f1 C test harness refactor.
Consolidate setting configuration into hrnInit() and rename other functions for consistency.

Split out internal functions into a new header.
2019-10-16 15:48:33 +02:00
David Steele
b4aeb217e6 Allow parameters to be passed to travis.pl.
This makes configuring tests easier.

Also add a parameter for tests that require sudo.  This should be retired at some point but some tests still require it.
2019-10-15 17:19:42 +02:00
David Steele
f3b2189659 Remove package build sudo into the container.
By running this in the container we no longer need sudo on the host system for package builds.
2019-10-15 13:27:03 +02:00
David Steele
67dde73727 Run tests in tmpfs.
This will likely improve performance, but it also makes the filesystem consistent between platforms.

A number of tests were failing on shiftfs, which was the default for arm64 on Travis.
2019-10-14 11:51:14 +02:00
David Steele
64c6102a15 Update packages required for Travis-CI builds.
These packages are expected on the arm64 build even though we are using the same os image as amd64.  It appears the arm64 image is slimmer.
2019-10-12 14:47:01 -04:00
David Steele
35eef2b867 Use a lower user id for posix storage tests.
arm64 was not happy with the old user id, so use something smaller.
2019-10-12 14:16:22 -04:00
David Steele
827e95944a Use < 0 and > 0 for strCmp() tests.
Using -1 and 1 was a bit sloppy since the spec only guarantees that the values will be < 0 and > 0.

Found on arm64 where the values were -64 and 64.
2019-10-12 13:52:45 -04:00
David Steele
a2fa1d04b0 Update container images to PostgreSQL 12 GA. 2019-10-12 11:26:13 -04:00
David Steele
397a41e0f9 Add Ubuntu 19.04 container definition. 2019-10-12 11:24:55 -04:00
David Steele
93656db186 Update lcov to 1.14.
1.13 is not compatible with gcc 8 which is what ships with newer distributions.  Build from source to get a more recent version.

1.13 is not compatible with gcc 9 so we'll need to address that at a later date.
2019-10-12 11:24:21 -04:00
David Steele
11c7c8fabb Remove pgbackrest test user.
This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories.  The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user.

Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user.  This allows us to remove most sudos, except where they are explicitly needed in tests.

While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 09:45:18 -04:00
David Steele
6f0e7f00af Fix recovery test failing in PostgreSQL 12.0.
This test was not creating recovery.signal when testing with --type=preserve.  The preserve recovery type only keeps existing files and does not create any.

RC1 was just ignoring recovery.signal and going right into recovery.  Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.
2019-10-12 09:26:19 -04:00
Cynthia Shang
db1dc4f275 Remove pretty-printing from jsonFromKv() and jsonFromVar().
Now that pretty-printing has been removed from the info command it no longer has a purpose, so remove it.
2019-10-11 13:03:52 -04:00
Cynthia Shang
d90b2724f8 JSON output from the info command is no longer pretty-printed.
Monitoring systems can more easily ingest the JSON without linefeeds.

External tools such as jq can be used to pretty-print if desired.
2019-10-11 12:56:03 -04:00
Cynthia Shang
2972580566 Remove info expect tests from mock/all and mock/stanza.
These tests are redundant now that we have full coverage in the unit tests are are not worth maintaining anymore.
2019-10-11 12:38:03 -04:00
David Steele
642ce003c8 Don't autogenerate embedded libc code by default.
This is only needed when new code is added to the Perl C library, which is becoming rare as the migration progresses.

Also, the code will vary slightly based on the Perl version used for generation so for normal users it is just noise.

Suggested by Stephen Frost.
2019-10-11 11:32:51 -04:00
David Steele
bcd3e4953a Make perl/exec test container required.
This test fails in some cases when --vm=none but it's not worth investigating since this code will be going away soon.
2019-10-10 22:10:20 -04:00
David Steele
e3d87ebace Fix mismatched timezone in expect test.
Also run the --vm-none tests in a non-UTC timezone to prevent regressions.

Suggested by Stephen Frost.
2019-10-10 19:43:42 -04:00
David Steele
6db4e59a66 Allow tests that use ports to run in parallel.
Set the test index in the C unit test code so it can assign port numbers that won't conflict between tests.
2019-10-10 16:13:43 -04:00
David Steele
13fcbb24e9 Fix container test path being used when --vm=none.
Suggested by Stephen Frost.
2019-10-10 15:09:11 -04:00
David Steele
9a3ba649e1 Remove code to generate .travis.yml.
Most of the logic has been moved to test/travis.pl so there wasn't much purpose to this code anymore.
2019-10-10 11:25:59 -04:00
David Steele
696e6a7c44 Don't require sudo to run tests with --vm=none.
Run these tests without sudo privileges on Travis to prevent regressions.
2019-10-10 11:21:09 -04:00
David Steele
7f369006b5 Add gcc 9 support.
A number of tests have been updated and Fedora 30 has been added to the test suite so the unit tests can run on gcc 9.

Stop running unit tests on co6/7 since we appear to have ample unit test coverage.
2019-10-09 15:03:03 -04:00
David Steele
528f4c4347 Remove dependency on aws cli for testing.
This tool was only being used it a few places but was a pretty large dependency.

Rework the forceStorageMove() code using our storage layer and replace one aws cli cp with a storage put.

Also, remove the Dockerfile that was once used to build the Scality S3 test container.
2019-10-09 14:38:24 -04:00
David Steele
61c4f64895 Be smarter about which packages are loaded for testing.
Now that our tests are more diversified it makes sense to load only the packages that are needed for each test.

Move the package loads from .travis.yaml to test/travis.pl where we have more control over what is loaded.
2019-10-08 18:56:55 -04:00
Cynthia Shang
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
Cynthia Shang
ecae5e34e5 Update expire command to use C backup.info reconstruct.
This was still being done in Perl until the C Manifest object was available.
2019-10-08 17:30:33 -04:00
Cynthia Shang
4e4d1f414a Add infoBackupLoadFileReconstruct() to InfoBackup object.
Check the backup.info file against the backup path.  Add any backups that are missing and remove any backups that no longer exist.

It's important to run this before backup or expire to be sure we are using the most up-to-date list of backups.
2019-10-08 16:04:27 -04:00
David Steele
b2825b82c7 Add missing header file. 2019-10-08 15:47:47 -04:00
Cynthia Shang
6d8d0eeba7 Add pgBackRest version to Info and Manifest objects.
This was not being exposed previously because it is primarily informational, but now it is needed to reconstruct the backup.info file.
2019-10-08 15:37:08 -04:00
Cynthia Shang
38b72eded4 Document how to contribute to pgBackRest.
There's a lot more to be done here, but this is a good start.
2019-10-08 15:27:17 -04:00
David Steele
45881c74ae Allow most unit tests to run outside of a container.
Three major changes were required to get this working:

1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.

1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.

2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.

Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.

Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
David Steele
5394893e33 Remove pgPath parameter from pgControlFromFile().
In practice this function is always used with storagePg*() so pgPath is known.
2019-10-03 11:14:22 -04:00
David Steele
29e132f5e9 PostgreSQL 12 support.
Recovery settings are now written into postgresql.auto.conf instead of recovery.conf.  Existing recovery_target* settings will be commented out to help avoid conflicts.

A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed.

recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
Cynthia Shang
f96c54c4ba Add info command set option for detailed text output.
The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations.

This information is not included in the JSON output because it requires reading the manifest which is too IO intensive to do for all manifests.  We plan to include this information for JSON in a future release.
2019-09-30 12:39:38 -04:00
David Steele
a58635ac02 Move C performance tests out of unit tests.
Performance tests were being done in unit tests until there was a better place to put them.

Now there is, so move them there.
2019-09-28 14:24:27 -04:00
David Steele
f1ba428fb0 Add performance test capability in C with scaling.
Scaling allows the starting values to be increased from the command-line without code changes.

Also suppress valgrind and assertions when running performance testing.  Optimization is left at -O0 because we should not be depending on compiler optimizations to make our code performant, and it makes profiling more informative.
2019-09-28 14:02:12 -04:00
David Steele
004ff99a2d Identify Perl performance test by appending -perl.
This is intended to differentiate the upcoming C performance tests from the Perl performance tests that will eventually be migrated.
2019-09-28 13:17:21 -04:00
David Steele
cb62bebadf Use bsearch() on sorted lists rather than an iterative method.
bsearch() is far more efficient than an iterative approach except in the most trivial cases.

For now insert will reset the sort order to none and the list will need to be resorted before bsearch() can be used.  This is necessary because item pointers are not stable after a sort, i.e. they can move around.  Until lists are stable it's not a good idea to surprise the caller by mixing up their pointers on insert.
2019-09-28 10:08:20 -04:00
David Steele
d3d2a7cd86 Add line number and fix spacing in TEST_LOG*() macros. 2019-09-28 09:57:06 -04:00
David Steele
afc483ef86 Clarify which timeline should be used for timeline integration test. 2019-09-27 13:37:59 -04:00
David Steele
d82102d6ef Add explicit promotes to recovery integration tests.
PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead.  Set target_action explicitly so all versions will behave the same way.

This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 13:04:36 -04:00
David Steele
833d0da0d9 Store recovery file name in integration when testing preserve recovery.
This makes the test a little more maintainable and is friendly with the changes needed for PostgreSQL 12.
2019-09-27 12:29:33 -04:00
David Steele
80eb561caf Add missing PostgreSQL 11 control/WAL versions in Perl tests.
These values don't seem to be used for testing but better to be tidy.
2019-09-27 09:45:11 -04:00
David Steele
d6a6d93a04 Add PostgreSQL 12 to u18 container.
This does not add PostgresQL 12 support; it simply adds PostgreSQL 12 to the u18 container for development and testing.
2019-09-27 09:35:59 -04:00
David Steele
03a7bda511 Refactor recovery file generation.
Separate the generation of recovery values and formatting them into recovery.conf format.  This is generally a good idea, but also makes the code ready to deal with a different recovery file in PostgreSQL 12.

Also move the recovery file logic out of cmdRestore() into restoreRecoveryWrite().
2019-09-27 09:19:12 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
451ae397be The restore command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C.

There is one important behavioral change with regard to how file permissions are handled.  The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user.  This usually just led to errors and frustration.

The C code works like this:

If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.

If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.

Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
David Steele
e968acbdd7 Fix outdated comment.
This was probably missed when a new test was added and the timeline was updated.
2019-09-24 16:55:11 -04:00
David Steele
71349c89ae Add TEST_TITLE() macro.
This macro displays a title for each test.  A test frequently has multiple parts and it was hard to tell which subparts went together.  We used ad hoc indentation to do this.

Anything that is a not a title is automatically indented so manually indenting is not longer needed.  This should make the tests and the test output easier to read.
2019-09-23 17:56:17 -04:00