1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

2127 Commits

Author SHA1 Message Date
David Steele
3d3003e9ca The check command is implemented partly in C.
Implement switch WAL and archive check in C but leave the rest in Perl for now.

The main idea was to have some real integration tests for the new database code so the rest of the migration can wait.

Reviewed by Cynthia Shang.
2019-08-01 20:35:01 -04:00
David Steele
e4901d50d5 Add Db object to encapsulate PostgreSQL queries and commands.
Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check.

Add the dbGet() helper function to get Db objects easily.

Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together.

Reviewed by Cynthia Shang.
2019-08-01 15:38:27 -04:00
David Steele
f9e1f3a798 Retry S3 RequestTimeTooSkewed errors instead of immediately terminating.
The cause of this error seems to be that a failed request takes so long that a subsequent retry at the http level uses outdated headers.

We're not sure if pgBackRest it to blame here (in one case a kernel downgrade fixed it, in another case an incorrect network driver was the problem) so add retries to hopefully deal with the issue if it is not too persistent.  If SSL_write() has long delays before reporting an error then this will obviously affect backup performance.

Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.
2019-08-01 14:28:30 -04:00
David Steele
2eb3c9f95f Improve error handling for SSL_write().
Error codes were not being caught for SSL_write() so it was hard to see exactly what was happening in error cases.  Report errors to aid in debugging.

Also add a retry for SSL_ERROR_WANT_READ.  Even though we have not been able to reproduce this case it is required by SSL_write() so go ahead and implement it.
2019-08-01 09:58:24 -04:00
David Steele
89c67287bc Improve multi-host handling in protocol helper.
Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
David Steele
893ae24284 Add timeout to walSegmentFind().
Keep trying to locate the WAL segment until timeout.  This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
Cynthia Shang
03b28da1ca Rename control/control module to control/common.
This is more consistent with how other common modules are named.

Contributed by Cynthia Shang.
2019-07-31 11:35:58 -04:00
David Steele
a04baa1bdc Fix incorrect comment (obviously pasted from S3 code). 2019-07-30 11:42:56 -04:00
David Steele
3d892cfb75 Remove extra linefeed. 2019-07-30 11:39:01 -04:00
David Steele
88c1929ec5 Don't pass local config* options to the remote.
The remotes have their own config options (repo-host-config, etc.) so don't pass the local config* options.

This was a regression from the behavior of the Perl code and while there have been no field reports it caused breakage on test systems with multiple configurations.
2019-07-26 08:37:58 -04:00
David Steele
f8b0676fd6 Allow modules to be included for testing without requiring coverage.
Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested.  The include directive allows this.

Update modules that had previously been added to coverage that only need to be included.
2019-07-25 20:15:06 -04:00
David Steele
554d98746a Add repo-s3-port option for setting a non-standard S3 service port.
If this option is set then ports appended to repo-s3-endpoint or repo-s3-host will be ignored.

Setting this option explicitly may be the only way to use a bare ipv6 address with S3 (since multiple colons confuse the parser) but we plan to improve this in the future.
2019-07-25 17:36:51 -04:00
David Steele
d8ca0e5c5b Add Perl interface to C PgQuery object.
This validates that all current queries work with the new interface and removes the dependency on DBD::Pg.
2019-07-25 17:05:39 -04:00
David Steele
415542b4a3 Add PostgreSQL query client.
This direct interface to libpq allows simple queries to be run against PostgreSQL and supports timeouts.

Testing is performed using a shim that can use scripted responses to test all aspects of the client code.  The shim will be very useful for testing backup scenarios on complex topologies.

Reviewed by Cynthia Shang.
2019-07-25 14:50:02 -04:00
David Steele
59f135340d The local command for backup is implemented entirely in C.
The local process is now entirely migrated to C.  Since all major I/O operations are performed in the local process, the vast majority of I/O is now performed in C.

Contributed by David Steele, Cynthia Shang.
2019-07-25 14:34:16 -04:00
David Steele
54ec8f151e Add int rendering to jsonFromVar(). 2019-07-24 19:45:35 -04:00
David Steele
615735e7ee Add new types to JSON render.
Add bool, array, and int64 as valid array subtypes.

Pretty print for the array subtype is not correct but is currently not in use (this can be seen at line 328 in typeJsonTest.c).
2019-07-24 06:52:49 -04:00
David Steele
62f0c7fb37 Add lock bot configuration.
Lock closed issues after ninety days to prevent posting on old issues.
2019-07-22 13:09:51 -04:00
David Steele
38ba458616 Add IoSink filter.
Discard all data passed to the filter.  Useful for calculating size/checksum on a remote system when no data needs to be returned.

Update ioReadDrain() to automatically use the IoSink filter.
2019-07-18 08:42:42 -04:00
Cynthia Shang
d1dd6add48 Remove never-used infoBackupCheckPg() function.
Contributed by Cynthia Shang.
2019-07-17 16:55:21 -04:00
David Steele
3bdba4933d Fix incorrect handling of transfer-encoding response to HEAD request.
The HTTP server can use either content-length or transfer-encoding to indicate that there is content in the response.  HEAD requests do not include content but return all the same headers as GET.  In the HEAD case we were ignoring content-length but not transfer-encoding which led to unexpected eof errors on AWS S3.  Our test server, minio, uses content-length so this was not caught in integration testing.

Ignore all content for HEAD requests (no matter how it is reported) and add a unit test for transfer-encoding to prevent a regression.

Found by Pavel Suderevsky.
2019-07-17 16:49:42 -04:00
Cynthia Shang
6f981c53bb Remove obsolete ignoreMissing parameter.
Contributed by Cynthia Shang.
2019-07-17 15:44:55 -04:00
David Steele
7662d32e60 Fix comment typos and clarify HEAD response behavior. 2019-07-17 15:42:37 -04:00
David Steele
eee67db4d6 Allow pg storage to be remote.
None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
David Steele
30f55a3c2a Add compressed storage feature.
This feature denotes storage that can compress files so that they take up less space than what was written.  Currently this includes the Posix and CIFS drivers.  The stored size of the file will be rechecked after write to determine if the reported size is different.  This check would be wasted on object stores such as S3, and they might not report the file as existing immediately after write.

Also add tests to each storage driver to check features.
2019-07-15 17:36:24 -04:00
David Steele
3e1062825d Allow multiple filters to be pushed to the remote and return results.
Previously only a single filter could be pushed to the remote since order was not being maintained.  Now the filters are strictly ordered.

Results are returned from the remote and set in the local IoFilterGroup so they can be retrieved.

Expand remote filter support to include all filters.
2019-07-15 16:49:46 -04:00
David Steele
d5654375a5 Add ioReadDrain().
Read all data from an IoRead object and discard it.  This is handy for calculating size, hash, etc. when the output is not needed.

Update code where a loop was used before.
2019-07-15 08:44:41 -04:00
Cynthia Shang
cdb75ac8b3 Add constants for path and archive.info/backup.info combinations.
Contributed by Cynthia Shang.
2019-07-15 07:13:36 -04:00
David Steele
ede7df9fb1 Allow NULL in JSON list. 2019-07-14 15:53:31 -04:00
David Steele
c836c483dc Add lstClear() to List object. 2019-07-14 15:42:55 -04:00
David Steele
e10577d0b0 Fix incorrect offline upper bound for ignoring page checksum errors.
For offline backups the upper bound was being set to 0x0000FFFF0000FFFF rather than UINT64_MAX.  This meant that page checksum errors might be ignored for databases with a lot of past WAL in offline mode.

Online mode is not affected since the upper bound is retrieved from pg_start_backup().
2019-07-11 09:13:56 -04:00
David Steele
2fd0ebb78a Fix links broken by non-standard version.
Using version 2.15.1 fixed the duplicate tarball problem but broke the auto-generated links.  Fix them manually since this should not be a common problem.

Reported by Mohamad El-Rifai.
2019-07-10 15:08:35 -04:00
Cynthia Shang
6a89c1526e Revert a2dcdc07.
It is simpler to implement the required logic in stanza-delete rather than add complexity to this function.

Contributed by Cynthia Shang.
2019-07-10 12:04:25 -04:00
David Steele
04646599a7 Remove extraneous test macro. 2019-07-10 06:17:33 -04:00
David Steele
4e7db608dc Clarify that return statements are not removed in production builds. 2019-07-10 06:11:21 -04:00
David Steele
a22a6dc08c Update contributor name. 2019-07-10 06:06:07 -04:00
Cynthia Shang
a2dcdc0711 Update lockStopTest() to optionally return a result rather than error.
Some commands (e.g. stanza-delete) would prefer to throw a customized error.

Contributed by Cynthia Shang.
2019-07-09 16:41:58 -04:00
David Steele
27b3246e85 Exclude more build files from rsync between tests.
Files (especially build.auto.h) were being removed and forcing a full build between separate invocations of test.pl.

This affected ad-hoc testing at the command-line, not a full test run in CI.
2019-07-08 08:29:25 -04:00
David Steele
5e1ed2e8a5 Remove clang static analysis.
This analysis never produced anything but false positives (var might be NULL) but took over a minute per test run and added 600MB to the test container.
2019-07-05 18:34:15 -04:00
David Steele
488fb67294 Force PostgreSQL versions to string for newer versions of JSON:PP.
Since 2.91 JSON::PP has a bias for saving variables that look like numbers as numbers even if they were declared as strings.

Force versions to strings where needed by appending ''.

Update the json-pp-perl package on Ubuntu 18.04 to 2.97 to provide test coverage.
2019-07-05 17:25:01 -04:00
David Steele
9836578520 Remove perl critic and coverage.
No new Perl code is being developed, so these tools are just taking up time and making migrations to newer platforms harder.  There are only a few Perl tests remaining with full coverage so the coverage tool does not warn of loss of coverage in most cases.

Remove both tools and associated libraries.
2019-07-05 16:55:17 -04:00
David Steele
fc21013522 Fix scoping violations exposed by optimizations in gcc 9.
gcc < 9 makes all compound literals function scope, even though the C spec requires them to be invalid outside the current scope.  Since the compiler and valgrind were not enforcing this we had a few violations which caused problems in gcc >= 9.

Even though we are not quite ready to support gcc 9 officially, fix the scoping violations that currently exist in the codebase.

Reported by chrlange, Ned T. Crigler.
2019-07-05 16:25:28 -04:00
David Steele
1708f1d151 Use minio for integration testing.
ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing.  Replace it with minio which starts quickly and ships as a single executable or a tiny container.

Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly.

This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-02 22:20:35 -04:00
David Steele
b9b21315ea Updates for openssl 1.1.1.
Some HTTP error tests were failing after the upgrade to openssl 1.1.1, though the rest of the unit and integration tests worked fine.  This seemed to be related to the very small messages used in the error testing, but it pointed to an issue with the code not being fully compliant, made worse by auto-retry being enabled by default.

Disable auto-retry and implement better error handling to bring the code in line with openssl recommendations.

There's no evidence this is a problem in the field, but having all the tests pass seems like a good idea and the new code is certainly more robust.

Coverage will be complete in the next commit when openssl 1.1.1 is introduced.
2019-07-02 22:09:12 -04:00
David Steele
c55009d0f9 Community yum package can be installed with --var=package=yum.
Like apt, the community yum package can now be installed instead of a user-specified package.
2019-06-27 14:39:11 -04:00
David Steele
b0728c33db Remove Debian package patch merged to upstream. 2019-06-27 14:30:20 -04:00
David Steele
020101b30b Update release notes to explicitly select release commit when tagging. 2019-06-27 09:38:40 -04:00
Cynthia Shang
33e3d316d6 Reverse loop in infoPgSave() to be consistent with infoPgNewLoad().
Contributed by Cynthia Shang.
2019-06-26 19:52:04 -04:00
David Steele
4bffa0c5bb Add test function to create the S3 bucket instead of using aws cli.
Eventually the idea is to remove the dependency on aws cli since Python is a big install.
2019-06-26 15:02:30 -04:00
David Steele
4815752ccc Add Perl interface to C storage layer.
Maintaining the storage layer/drivers in two languages is burdensome.  Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C.  Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents.

The goal is to move the integration tests to C so this interface will eventually be removed.  That being the case, the interface was designed for maximum compatibility to ease the transition.  The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 08:24:58 -04:00