1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2418 Commits

Author SHA1 Message Date
David Steele
48e8942e86 Allow trailing / for relative paths in strPathAbsolute().
The trailing / does nothing but is nevertheless valid syntax.
2019-11-15 08:53:15 -05:00
David Steele
83ab272171 Update varNew*() calls to Variant constant macros.
The constants created by the macros are faster and use less memory.
2019-11-14 16:50:58 -05:00
David Steele
3b879c2cb3 Filter logged command options based on the command definition.
Previously, options were being filtered based on what was currently valid.  For chained commands (e.g. backup then expire) some options may be valid for the first command but not the second.

Filter based on the command definition rather than what is currently valid to avoid logging options that are not valid for subsequent commands.  This reduces the number of options logged and will hopefully help avoid confusion and expect log churn.
2019-11-14 16:48:41 -05:00
Cynthia Shang
c5b76d213b Modify InfoBackupData struct to use time_t for backup start/stop times.
The uint64_t types worked but were not consistent with how timestamps are handled in other parts of the code.
2019-11-12 17:05:09 -05:00
David Steele
6f65dde8d3 Begin v2.20 development. 2019-11-12 15:56:28 -05:00
David Steele
2d10293d04 v2.19: C Migrations and Bug Fixes
Bug Fixes:

* Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
* Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)

Improvements:

* JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang.)
* The check command is implemented entirely in C. (Contributed by Cynthia Shang.)

Documentation Improvements:

* Document how to contribute to pgBackRest. (Contributed by Cynthia Shang.)
* Document maximum version for auto-stop option. (Contributed by Brad Nicholson.)

Test Suite Improvements:

* Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
* Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
* Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
2019-11-12 15:51:28 -05:00
David Steele
a44c5d0315 Add Strings for STORAGE_REPO_ARCHIVE and STORAGE_REPO_BACKUP.
These constants are used often enough that they deserve to have String constants rather than repeatedly calling STRDEF().
2019-11-12 13:12:07 -05:00
David Steele
10c8eeaf6c Fix handling of repeated HTTP headers.
When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior.

Reported by donicrosby.
2019-11-08 18:58:45 -05:00
David Steele
4317178633 Update MinIO to newest release.
We had some problems with newer versions so had held off on updating.  Those problems appear to have been resolved.

In addition, the --compat flag is no longer required.  Prior versions of MinIO required all parts of a multi-part upload (except the last) to be of equal size.  The --compat flag was introduced to restore the default S3 behavior.  Now --compat is only required when ETag is being used for MD5 verification, which we don't do.
2019-11-08 17:56:34 -05:00
David Steele
edcc7306a3 Add TIME parameter debug type.
Previously we were using int64_t to debug time_t but this may not be right depending on how the compiler represents time_t, e.g. it could be a float.

Since a mismatch would have caused a compiler error we are not worried that this has actually happened, and anyway the worst case is that the debug log would be wonky.

The primary benefit, aside from correctness, is that it makes choosing a parameter debug type for time_t obvious.
2019-11-08 09:46:00 -05:00
David Steele
eca00e0be0 Add building a development environment to contributing documentation.
This documentation shows how to build a development environment on Ubuntu 19.04 and should work for other Debian-based distros.

Note that this document is not included in automated testing due to some unresolved issues with Docker in Docker on Travis CI.  We'll address this in the future when we add contributing documentation to the website.
2019-11-07 13:11:01 -05:00
David Steele
8b682b75d2 Allow mock integration tests for all VM types.
Previously the mock integration tests would be skipped for VMs other than the standard four used in CI.  Now VMs outside the standard four will run the same tests as VM4 (currently U18).
2019-11-02 10:35:48 +01:00
David Steele
b3e5d88304 Add markdown renderer for execute-list block. 2019-11-02 09:59:39 +01:00
David Steele
7168e07440 Use getcwd() to construct path when WAL path is relative.
Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places.  PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
David Steele
e06db21e35 Error when specified vm is invalid. 2019-10-17 14:00:18 +02:00
David Steele
a52faf83a5 Disable code generation on dry-run. 2019-10-17 11:56:45 +02:00
David Steele
fa6a54bb45 Update last tests that required sudo.
All tests should now run in a sudo-less environment.
2019-10-16 17:05:24 +02:00
David Steele
48bd9e22f1 C test harness refactor.
Consolidate setting configuration into hrnInit() and rename other functions for consistency.

Split out internal functions into a new header.
2019-10-16 15:48:33 +02:00
David Steele
b4aeb217e6 Allow parameters to be passed to travis.pl.
This makes configuring tests easier.

Also add a parameter for tests that require sudo.  This should be retired at some point but some tests still require it.
2019-10-15 17:19:42 +02:00
David Steele
f3b2189659 Remove package build sudo into the container.
By running this in the container we no longer need sudo on the host system for package builds.
2019-10-15 13:27:03 +02:00
David Steele
67dde73727 Run tests in tmpfs.
This will likely improve performance, but it also makes the filesystem consistent between platforms.

A number of tests were failing on shiftfs, which was the default for arm64 on Travis.
2019-10-14 11:51:14 +02:00
David Steele
b1dc5a6d26 Add disk/mem info for Travis.
This is helpful for determining the virtual machine resources.
2019-10-13 14:02:52 -04:00
David Steele
64c6102a15 Update packages required for Travis-CI builds.
These packages are expected on the arm64 build even though we are using the same os image as amd64.  It appears the arm64 image is slimmer.
2019-10-12 14:47:01 -04:00
David Steele
35eef2b867 Use a lower user id for posix storage tests.
arm64 was not happy with the old user id, so use something smaller.
2019-10-12 14:16:22 -04:00
David Steele
827e95944a Use < 0 and > 0 for strCmp() tests.
Using -1 and 1 was a bit sloppy since the spec only guarantees that the values will be < 0 and > 0.

Found on arm64 where the values were -64 and 64.
2019-10-12 13:52:45 -04:00
David Steele
a2fa1d04b0 Update container images to PostgreSQL 12 GA. 2019-10-12 11:26:13 -04:00
David Steele
397a41e0f9 Add Ubuntu 19.04 container definition. 2019-10-12 11:24:55 -04:00
David Steele
93656db186 Update lcov to 1.14.
1.13 is not compatible with gcc 8 which is what ships with newer distributions.  Build from source to get a more recent version.

1.13 is not compatible with gcc 9 so we'll need to address that at a later date.
2019-10-12 11:24:21 -04:00
David Steele
11c7c8fabb Remove pgbackrest test user.
This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories.  The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user.

Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user.  This allows us to remove most sudos, except where they are explicitly needed in tests.

While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 09:45:18 -04:00
David Steele
6f0e7f00af Fix recovery test failing in PostgreSQL 12.0.
This test was not creating recovery.signal when testing with --type=preserve.  The preserve recovery type only keeps existing files and does not create any.

RC1 was just ignoring recovery.signal and going right into recovery.  Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.
2019-10-12 09:26:19 -04:00
David Steele
59a4a0c1b1 Add a warning about jq rounding numbers to the documentation. 2019-10-11 13:14:47 -04:00
Cynthia Shang
db1dc4f275 Remove pretty-printing from jsonFromKv() and jsonFromVar().
Now that pretty-printing has been removed from the info command it no longer has a purpose, so remove it.
2019-10-11 13:03:52 -04:00
Cynthia Shang
d90b2724f8 JSON output from the info command is no longer pretty-printed.
Monitoring systems can more easily ingest the JSON without linefeeds.

External tools such as jq can be used to pretty-print if desired.
2019-10-11 12:56:03 -04:00
Cynthia Shang
2972580566 Remove info expect tests from mock/all and mock/stanza.
These tests are redundant now that we have full coverage in the unit tests are are not worth maintaining anymore.
2019-10-11 12:38:03 -04:00
David Steele
e4408c7dd3 Refactor --pre option in documentation.
Mark all pre commands as skip so they won't be run again after the container is built.

Ensure that pre commands added to the container are run as the container user if they are not intended to run as root.
2019-10-11 12:28:47 -04:00
David Steele
642ce003c8 Don't autogenerate embedded libc code by default.
This is only needed when new code is added to the Perl C library, which is becoming rare as the migration progresses.

Also, the code will vary slightly based on the Perl version used for generation so for normal users it is just noise.

Suggested by Stephen Frost.
2019-10-11 11:32:51 -04:00
David Steele
bcd3e4953a Make perl/exec test container required.
This test fails in some cases when --vm=none but it's not worth investigating since this code will be going away soon.
2019-10-10 22:10:20 -04:00
David Steele
e3d87ebace Fix mismatched timezone in expect test.
Also run the --vm-none tests in a non-UTC timezone to prevent regressions.

Suggested by Stephen Frost.
2019-10-10 19:43:42 -04:00
David Steele
6db4e59a66 Allow tests that use ports to run in parallel.
Set the test index in the C unit test code so it can assign port numbers that won't conflict between tests.
2019-10-10 16:13:43 -04:00
David Steele
13fcbb24e9 Fix container test path being used when --vm=none.
Suggested by Stephen Frost.
2019-10-10 15:09:11 -04:00
David Steele
e732720d85 Disable docker service for --vm=none test.
Starting this service wastes time and this will help catch any regressions where containers are being built when they should not be.
2019-10-10 11:28:37 -04:00
David Steele
9a3ba649e1 Remove code to generate .travis.yml.
Most of the logic has been moved to test/travis.pl so there wasn't much purpose to this code anymore.
2019-10-10 11:25:59 -04:00
David Steele
696e6a7c44 Don't require sudo to run tests with --vm=none.
Run these tests without sudo privileges on Travis to prevent regressions.
2019-10-10 11:21:09 -04:00
David Steele
50d1d0a4e8 Move release note to correct section. 2019-10-10 09:28:50 -04:00
David Steele
47e0ba2502 Fix remote timeout in delta restore.
When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout.

Add keep-alives to prevent remote timeout.

Reported by James Sewell, Jens Wilke.
2019-10-10 09:27:46 -04:00
Cynthia Shang
82baf1e521 Fix typo. 2019-10-10 08:33:40 -04:00
David Steele
7f369006b5 Add gcc 9 support.
A number of tests have been updated and Fedora 30 has been added to the test suite so the unit tests can run on gcc 9.

Stop running unit tests on co6/7 since we appear to have ample unit test coverage.
2019-10-09 15:03:03 -04:00
David Steele
528f4c4347 Remove dependency on aws cli for testing.
This tool was only being used it a few places but was a pretty large dependency.

Rework the forceStorageMove() code using our storage layer and replace one aws cli cp with a storage put.

Also, remove the Dockerfile that was once used to build the Scality S3 test container.
2019-10-09 14:38:24 -04:00
David Steele
ac870b42de Refactor error logic to make sure Db object gets freed immediately.
Because db can be reset to NULL on an error in the try block we need nested try blocks to ensure that db is non-NULL and can be freed on an error after being created.

This is not a production issue because the db will be freed when the temp mem context is freed, but it does affect reproducibility in the tests and is a bit tidier.
2019-10-09 08:43:30 -04:00
David Steele
61c4f64895 Be smarter about which packages are loaded for testing.
Now that our tests are more diversified it makes sense to load only the packages that are needed for each test.

Move the package loads from .travis.yaml to test/travis.pl where we have more control over what is loaded.
2019-10-08 18:56:55 -04:00