1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2304 Commits

Author SHA1 Message Date
David Steele
e06db21e35 Error when specified vm is invalid. 2019-10-17 14:00:18 +02:00
David Steele
a52faf83a5 Disable code generation on dry-run. 2019-10-17 11:56:45 +02:00
David Steele
fa6a54bb45 Update last tests that required sudo.
All tests should now run in a sudo-less environment.
2019-10-16 17:05:24 +02:00
David Steele
48bd9e22f1 C test harness refactor.
Consolidate setting configuration into hrnInit() and rename other functions for consistency.

Split out internal functions into a new header.
2019-10-16 15:48:33 +02:00
David Steele
b4aeb217e6 Allow parameters to be passed to travis.pl.
This makes configuring tests easier.

Also add a parameter for tests that require sudo.  This should be retired at some point but some tests still require it.
2019-10-15 17:19:42 +02:00
David Steele
f3b2189659 Remove package build sudo into the container.
By running this in the container we no longer need sudo on the host system for package builds.
2019-10-15 13:27:03 +02:00
David Steele
67dde73727 Run tests in tmpfs.
This will likely improve performance, but it also makes the filesystem consistent between platforms.

A number of tests were failing on shiftfs, which was the default for arm64 on Travis.
2019-10-14 11:51:14 +02:00
David Steele
b1dc5a6d26 Add disk/mem info for Travis.
This is helpful for determining the virtual machine resources.
2019-10-13 14:02:52 -04:00
David Steele
64c6102a15 Update packages required for Travis-CI builds.
These packages are expected on the arm64 build even though we are using the same os image as amd64.  It appears the arm64 image is slimmer.
2019-10-12 14:47:01 -04:00
David Steele
35eef2b867 Use a lower user id for posix storage tests.
arm64 was not happy with the old user id, so use something smaller.
2019-10-12 14:16:22 -04:00
David Steele
827e95944a Use < 0 and > 0 for strCmp() tests.
Using -1 and 1 was a bit sloppy since the spec only guarantees that the values will be < 0 and > 0.

Found on arm64 where the values were -64 and 64.
2019-10-12 13:52:45 -04:00
David Steele
a2fa1d04b0 Update container images to PostgreSQL 12 GA. 2019-10-12 11:26:13 -04:00
David Steele
397a41e0f9 Add Ubuntu 19.04 container definition. 2019-10-12 11:24:55 -04:00
David Steele
93656db186 Update lcov to 1.14.
1.13 is not compatible with gcc 8 which is what ships with newer distributions.  Build from source to get a more recent version.

1.13 is not compatible with gcc 9 so we'll need to address that at a later date.
2019-10-12 11:24:21 -04:00
David Steele
11c7c8fabb Remove pgbackrest test user.
This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories.  The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user.

Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user.  This allows us to remove most sudos, except where they are explicitly needed in tests.

While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 09:45:18 -04:00
David Steele
6f0e7f00af Fix recovery test failing in PostgreSQL 12.0.
This test was not creating recovery.signal when testing with --type=preserve.  The preserve recovery type only keeps existing files and does not create any.

RC1 was just ignoring recovery.signal and going right into recovery.  Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.
2019-10-12 09:26:19 -04:00
David Steele
59a4a0c1b1 Add a warning about jq rounding numbers to the documentation. 2019-10-11 13:14:47 -04:00
Cynthia Shang
db1dc4f275 Remove pretty-printing from jsonFromKv() and jsonFromVar().
Now that pretty-printing has been removed from the info command it no longer has a purpose, so remove it.
2019-10-11 13:03:52 -04:00
Cynthia Shang
d90b2724f8 JSON output from the info command is no longer pretty-printed.
Monitoring systems can more easily ingest the JSON without linefeeds.

External tools such as jq can be used to pretty-print if desired.
2019-10-11 12:56:03 -04:00
Cynthia Shang
2972580566 Remove info expect tests from mock/all and mock/stanza.
These tests are redundant now that we have full coverage in the unit tests are are not worth maintaining anymore.
2019-10-11 12:38:03 -04:00
David Steele
e4408c7dd3 Refactor --pre option in documentation.
Mark all pre commands as skip so they won't be run again after the container is built.

Ensure that pre commands added to the container are run as the container user if they are not intended to run as root.
2019-10-11 12:28:47 -04:00
David Steele
642ce003c8 Don't autogenerate embedded libc code by default.
This is only needed when new code is added to the Perl C library, which is becoming rare as the migration progresses.

Also, the code will vary slightly based on the Perl version used for generation so for normal users it is just noise.

Suggested by Stephen Frost.
2019-10-11 11:32:51 -04:00
David Steele
bcd3e4953a Make perl/exec test container required.
This test fails in some cases when --vm=none but it's not worth investigating since this code will be going away soon.
2019-10-10 22:10:20 -04:00
David Steele
e3d87ebace Fix mismatched timezone in expect test.
Also run the --vm-none tests in a non-UTC timezone to prevent regressions.

Suggested by Stephen Frost.
2019-10-10 19:43:42 -04:00
David Steele
6db4e59a66 Allow tests that use ports to run in parallel.
Set the test index in the C unit test code so it can assign port numbers that won't conflict between tests.
2019-10-10 16:13:43 -04:00
David Steele
13fcbb24e9 Fix container test path being used when --vm=none.
Suggested by Stephen Frost.
2019-10-10 15:09:11 -04:00
David Steele
e732720d85 Disable docker service for --vm=none test.
Starting this service wastes time and this will help catch any regressions where containers are being built when they should not be.
2019-10-10 11:28:37 -04:00
David Steele
9a3ba649e1 Remove code to generate .travis.yml.
Most of the logic has been moved to test/travis.pl so there wasn't much purpose to this code anymore.
2019-10-10 11:25:59 -04:00
David Steele
696e6a7c44 Don't require sudo to run tests with --vm=none.
Run these tests without sudo privileges on Travis to prevent regressions.
2019-10-10 11:21:09 -04:00
David Steele
50d1d0a4e8 Move release note to correct section. 2019-10-10 09:28:50 -04:00
David Steele
47e0ba2502 Fix remote timeout in delta restore.
When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout.

Add keep-alives to prevent remote timeout.

Reported by James Sewell, Jens Wilke.
2019-10-10 09:27:46 -04:00
Cynthia Shang
82baf1e521 Fix typo. 2019-10-10 08:33:40 -04:00
David Steele
7f369006b5 Add gcc 9 support.
A number of tests have been updated and Fedora 30 has been added to the test suite so the unit tests can run on gcc 9.

Stop running unit tests on co6/7 since we appear to have ample unit test coverage.
2019-10-09 15:03:03 -04:00
David Steele
528f4c4347 Remove dependency on aws cli for testing.
This tool was only being used it a few places but was a pretty large dependency.

Rework the forceStorageMove() code using our storage layer and replace one aws cli cp with a storage put.

Also, remove the Dockerfile that was once used to build the Scality S3 test container.
2019-10-09 14:38:24 -04:00
David Steele
ac870b42de Refactor error logic to make sure Db object gets freed immediately.
Because db can be reset to NULL on an error in the try block we need nested try blocks to ensure that db is non-NULL and can be freed on an error after being created.

This is not a production issue because the db will be freed when the temp mem context is freed, but it does affect reproducibility in the tests and is a bit tidier.
2019-10-09 08:43:30 -04:00
David Steele
61c4f64895 Be smarter about which packages are loaded for testing.
Now that our tests are more diversified it makes sense to load only the packages that are needed for each test.

Move the package loads from .travis.yaml to test/travis.pl where we have more control over what is loaded.
2019-10-08 18:56:55 -04:00
Cynthia Shang
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
Cynthia Shang
ecae5e34e5 Update expire command to use C backup.info reconstruct.
This was still being done in Perl until the C Manifest object was available.
2019-10-08 17:30:33 -04:00
Cynthia Shang
4e4d1f414a Add infoBackupLoadFileReconstruct() to InfoBackup object.
Check the backup.info file against the backup path.  Add any backups that are missing and remove any backups that no longer exist.

It's important to run this before backup or expire to be sure we are using the most up-to-date list of backups.
2019-10-08 16:04:27 -04:00
David Steele
b2825b82c7 Add missing header file. 2019-10-08 15:47:47 -04:00
Cynthia Shang
6d8d0eeba7 Add pgBackRest version to Info and Manifest objects.
This was not being exposed previously because it is primarily informational, but now it is needed to reconstruct the backup.info file.
2019-10-08 15:37:08 -04:00
Cynthia Shang
38b72eded4 Document how to contribute to pgBackRest.
There's a lot more to be done here, but this is a good start.
2019-10-08 15:27:17 -04:00
Cynthia Shang
a4152a0ea1 Update release instructions with a reminder to install latex.
Latex is no longer installed by default in the Vagrantfile.
2019-10-08 12:29:42 -04:00
David Steele
45881c74ae Allow most unit tests to run outside of a container.
Three major changes were required to get this working:

1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.

1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.

2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.

Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.

Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
David Steele
77b0c6c993 Add missing spaces. 2019-10-08 10:35:47 -04:00
David Steele
5394893e33 Remove pgPath parameter from pgControlFromFile().
In practice this function is always used with storagePg*() so pgPath is known.
2019-10-03 11:14:22 -04:00
Brad Nicholson
887ec97e07 Document maximum version for auto-stop option. 2019-10-03 06:58:52 -04:00
David Steele
008ec6b05c Begin v2.19 development. 2019-10-01 17:04:26 -04:00
David Steele
4629237d6a v2.18: PostgreSQL 12 Support
Features:

* PostgreSQL 12 support.
* Add info command set option for detailed text output. The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations. (Contributed by Cynthia Shang. Suggested by Stephen Frost, ejberdecia.)
* Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf for PostgreSQL < 12 and creates standby.signal for PostgreSQL ≥ 12, creating a common interface between PostgreSQL versions. (Reviewed by Cynthia Shang.)

Improvements:

* The restore command is implemented entirely in C. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Document the relationship between db-timeout and protocol-timeout. (Contributed by Cynthia Shang. Suggested by James Chanco Jr.)
* Add documentation clarifications regarding standby repositories. (Contributed by Cynthia Shang.)
* Add FAQ for time-based Point-in-Time Recovery. (Contributed by Cynthia Shang.)
2019-10-01 16:02:07 -04:00
David Steele
29e132f5e9 PostgreSQL 12 support.
Recovery settings are now written into postgresql.auto.conf instead of recovery.conf.  Existing recovery_target* settings will be commented out to help avoid conflicts.

A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed.

recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00