1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2829 Commits

Author SHA1 Message Date
David Steele
13fcbb24e9 Fix container test path being used when --vm=none.
Suggested by Stephen Frost.
2019-10-10 15:09:11 -04:00
David Steele
e732720d85 Disable docker service for --vm=none test.
Starting this service wastes time and this will help catch any regressions where containers are being built when they should not be.
2019-10-10 11:28:37 -04:00
David Steele
9a3ba649e1 Remove code to generate .travis.yml.
Most of the logic has been moved to test/travis.pl so there wasn't much purpose to this code anymore.
2019-10-10 11:25:59 -04:00
David Steele
696e6a7c44 Don't require sudo to run tests with --vm=none.
Run these tests without sudo privileges on Travis to prevent regressions.
2019-10-10 11:21:09 -04:00
David Steele
50d1d0a4e8 Move release note to correct section. 2019-10-10 09:28:50 -04:00
David Steele
47e0ba2502 Fix remote timeout in delta restore.
When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout.

Add keep-alives to prevent remote timeout.

Reported by James Sewell, Jens Wilke.
2019-10-10 09:27:46 -04:00
Cynthia Shang
82baf1e521 Fix typo. 2019-10-10 08:33:40 -04:00
David Steele
7f369006b5 Add gcc 9 support.
A number of tests have been updated and Fedora 30 has been added to the test suite so the unit tests can run on gcc 9.

Stop running unit tests on co6/7 since we appear to have ample unit test coverage.
2019-10-09 15:03:03 -04:00
David Steele
528f4c4347 Remove dependency on aws cli for testing.
This tool was only being used it a few places but was a pretty large dependency.

Rework the forceStorageMove() code using our storage layer and replace one aws cli cp with a storage put.

Also, remove the Dockerfile that was once used to build the Scality S3 test container.
2019-10-09 14:38:24 -04:00
David Steele
ac870b42de Refactor error logic to make sure Db object gets freed immediately.
Because db can be reset to NULL on an error in the try block we need nested try blocks to ensure that db is non-NULL and can be freed on an error after being created.

This is not a production issue because the db will be freed when the temp mem context is freed, but it does affect reproducibility in the tests and is a bit tidier.
2019-10-09 08:43:30 -04:00
David Steele
61c4f64895 Be smarter about which packages are loaded for testing.
Now that our tests are more diversified it makes sense to load only the packages that are needed for each test.

Move the package loads from .travis.yaml to test/travis.pl where we have more control over what is loaded.
2019-10-08 18:56:55 -04:00
Cynthia Shang
a1c13a50dd The check command is implemented entirely in C.
Note that building the manifest on each host has been temporarily removed.

This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-08 18:04:09 -04:00
Cynthia Shang
ecae5e34e5 Update expire command to use C backup.info reconstruct.
This was still being done in Perl until the C Manifest object was available.
2019-10-08 17:30:33 -04:00
Cynthia Shang
4e4d1f414a Add infoBackupLoadFileReconstruct() to InfoBackup object.
Check the backup.info file against the backup path.  Add any backups that are missing and remove any backups that no longer exist.

It's important to run this before backup or expire to be sure we are using the most up-to-date list of backups.
2019-10-08 16:04:27 -04:00
David Steele
b2825b82c7 Add missing header file. 2019-10-08 15:47:47 -04:00
Cynthia Shang
6d8d0eeba7 Add pgBackRest version to Info and Manifest objects.
This was not being exposed previously because it is primarily informational, but now it is needed to reconstruct the backup.info file.
2019-10-08 15:37:08 -04:00
Cynthia Shang
38b72eded4 Document how to contribute to pgBackRest.
There's a lot more to be done here, but this is a good start.
2019-10-08 15:27:17 -04:00
Cynthia Shang
a4152a0ea1 Update release instructions with a reminder to install latex.
Latex is no longer installed by default in the Vagrantfile.
2019-10-08 12:29:42 -04:00
David Steele
45881c74ae Allow most unit tests to run outside of a container.
Three major changes were required to get this working:

1) Provide the path to pgbackrest in the build directory when running outside a container.  Tests in a container will continue to install and run against /usr/bin/pgbackrest.

1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path.  Also set a per-test log-path while we are at it.

2) Use localhost instead of a custom host for TLS test connections.  Tests in containers will continue to update /etc/hosts and use the custom host.

Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing.

Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time.  Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
David Steele
77b0c6c993 Add missing spaces. 2019-10-08 10:35:47 -04:00
David Steele
5394893e33 Remove pgPath parameter from pgControlFromFile().
In practice this function is always used with storagePg*() so pgPath is known.
2019-10-03 11:14:22 -04:00
Brad Nicholson
887ec97e07 Document maximum version for auto-stop option. 2019-10-03 06:58:52 -04:00
David Steele
008ec6b05c Begin v2.19 development. 2019-10-01 17:04:26 -04:00
David Steele
4629237d6a v2.18: PostgreSQL 12 Support
Features:

* PostgreSQL 12 support.
* Add info command set option for detailed text output. The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations. (Contributed by Cynthia Shang. Suggested by Stephen Frost, ejberdecia.)
* Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf for PostgreSQL < 12 and creates standby.signal for PostgreSQL ≥ 12, creating a common interface between PostgreSQL versions. (Reviewed by Cynthia Shang.)

Improvements:

* The restore command is implemented entirely in C. (Reviewed by Cynthia Shang.)

Documentation Improvements:

* Document the relationship between db-timeout and protocol-timeout. (Contributed by Cynthia Shang. Suggested by James Chanco Jr.)
* Add documentation clarifications regarding standby repositories. (Contributed by Cynthia Shang.)
* Add FAQ for time-based Point-in-Time Recovery. (Contributed by Cynthia Shang.)
2019-10-01 16:02:07 -04:00
David Steele
29e132f5e9 PostgreSQL 12 support.
Recovery settings are now written into postgresql.auto.conf instead of recovery.conf.  Existing recovery_target* settings will be commented out to help avoid conflicts.

A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed.

recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 13:20:43 -04:00
Cynthia Shang
6be7e6fde5 Fix misplaced String declarations. 2019-10-01 10:25:08 -04:00
David Steele
309ae66e2f Remove unneeded static declarations and use sizeof() where appropriate. 2019-10-01 08:47:56 -04:00
Cynthia Shang
f96c54c4ba Add info command set option for detailed text output.
The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations.

This information is not included in the JSON output because it requires reading the manifest which is too IO intensive to do for all manifests.  We plan to include this information for JSON in a future release.
2019-09-30 12:39:38 -04:00
David Steele
33ec5a3aac Sort lists where it might help performance.
Most of these lists should be quite small with the exception of the list in get.c, but it doesn't cost much to sort them and may help in corner cases we have not thought of.
2019-09-29 09:05:55 -04:00
David Steele
a58635ac02 Move C performance tests out of unit tests.
Performance tests were being done in unit tests until there was a better place to put them.

Now there is, so move them there.
2019-09-28 14:24:27 -04:00
David Steele
f1ba428fb0 Add performance test capability in C with scaling.
Scaling allows the starting values to be increased from the command-line without code changes.

Also suppress valgrind and assertions when running performance testing.  Optimization is left at -O0 because we should not be depending on compiler optimizations to make our code performant, and it makes profiling more informative.
2019-09-28 14:02:12 -04:00
David Steele
004ff99a2d Identify Perl performance test by appending -perl.
This is intended to differentiate the upcoming C performance tests from the Perl performance tests that will eventually be migrated.
2019-09-28 13:17:21 -04:00
David Steele
cb62bebadf Use bsearch() on sorted lists rather than an iterative method.
bsearch() is far more efficient than an iterative approach except in the most trivial cases.

For now insert will reset the sort order to none and the list will need to be resorted before bsearch() can be used.  This is necessary because item pointers are not stable after a sort, i.e. they can move around.  Until lists are stable it's not a good idea to surprise the caller by mixing up their pointers on insert.
2019-09-28 10:08:20 -04:00
David Steele
d3d2a7cd86 Add line number and fix spacing in TEST_LOG*() macros. 2019-09-28 09:57:06 -04:00
David Steele
afc483ef86 Clarify which timeline should be used for timeline integration test. 2019-09-27 13:37:59 -04:00
David Steele
0a4122c77d Add reviewer. 2019-09-27 13:34:59 -04:00
Cynthia Shang
6355451870 Fix old option name in documentation. 2019-09-27 13:33:14 -04:00
David Steele
d82102d6ef Add explicit promotes to recovery integration tests.
PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead.  Set target_action explicitly so all versions will behave the same way.

This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 13:04:36 -04:00
David Steele
833d0da0d9 Store recovery file name in integration when testing preserve recovery.
This makes the test a little more maintainable and is friendly with the changes needed for PostgreSQL 12.
2019-09-27 12:29:33 -04:00
David Steele
80eb561caf Add missing PostgreSQL 11 control/WAL versions in Perl tests.
These values don't seem to be used for testing but better to be tidy.
2019-09-27 09:45:11 -04:00
David Steele
d6a6d93a04 Add PostgreSQL 12 to u18 container.
This does not add PostgresQL 12 support; it simply adds PostgreSQL 12 to the u18 container for development and testing.
2019-09-27 09:35:59 -04:00
David Steele
03a7bda511 Refactor recovery file generation.
Separate the generation of recovery values and formatting them into recovery.conf format.  This is generally a good idea, but also makes the code ready to deal with a different recovery file in PostgreSQL 12.

Also move the recovery file logic out of cmdRestore() into restoreRecoveryWrite().
2019-09-27 09:19:12 -04:00
David Steele
cf1e96e827 Refactor handling of ignored files in restore delta cleanup.
Replace the boolean logic with a simple list that will be checked to see if the file should be ignored.
2019-09-27 08:50:33 -04:00
David Steele
c41fb575fb Add standby restore type.
This restore type automatically adds standby_mode=on to recovery.conf.

This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal.

The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 17:39:45 -04:00
David Steele
8979336819 Comment out unprivileged user documentation.
We haven't had the time to complete this documentation and it has suffered bit rot.

This prevents us from building the docs on PostgreSQL >= 11 so just comment it all out until it can be updated.
2019-09-26 16:02:39 -04:00
David Steele
451ae397be The restore command is implemented entirely in C.
For the most part this is a direct migration of the Perl code into C.

There is one important behavioral change with regard to how file permissions are handled.  The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user.  This usually just led to errors and frustration.

The C code works like this:

If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.

If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.

Reviewed by Cynthia Shang.
2019-09-26 07:52:02 -04:00
Cynthia Shang
92e48c856a Add missing asserts. 2019-09-26 07:27:07 -04:00
David Steele
f2781bdad4 Allow documentation to be built without encryption.
This is mainly useful for testing, where an encrypted repo is a real nuisance.
2019-09-25 16:38:13 -04:00
David Steele
e968acbdd7 Fix outdated comment.
This was probably missed when a new test was added and the timeline was updated.
2019-09-24 16:55:11 -04:00
David Steele
6af64df9b6 Thoughts on repository version 6. 2019-09-23 19:26:24 -04:00