1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1098 Commits

Author SHA1 Message Date
David Steele
ce2bf29998 v2.17: C Migrations and Bug Fixes
Bug Fixes:

* Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)
* Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)

Improvements:

* The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang.)
* Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)

Documentation Bug Fixes:

* Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)

Documentation Improvements:

* Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)
2019-09-03 16:39:32 -04:00
David Steele
3a28b68b8b Disable S3 and encryption on u18 integration tests for mock/all/1.
This test is commonly used for sanity checking but the combination of S3 and encryption makes it hard to use and encourages temporary changes to make it usable.

Acknowledge this and disable S3 and encryption for this test and move them to mock/all/2.
2019-09-02 19:06:12 -04:00
David Steele
05abab5e72 Add OBJECT_DEFINE_MOVE() macro.
The object *Move() functions are very repetitive so replace them with a macro that works similarly to OBJECT_DEFINE_FREE().
2019-08-30 14:36:02 -04:00
David Steele
d1675b7e91 Add ioReadLineParam() to allow return on eof.
ioReadLine() errors on eof because it has previously been used only for protocol reads.

Returning on eof is handy for reading lines from files where eof is not considered an error.
2019-08-28 10:46:54 -04:00
Josh Soref
545ccfa878 Fix typos in the documentation.
Contributed by Josh Soref.
2019-08-26 12:26:00 -04:00
Josh Soref
08bac74d38 Fix typo in stack trace message.
Contributed by Josh Soref.
2019-08-26 12:20:51 -04:00
Josh Soref
c2771e5469 Fix comment typos.
This includes some variable names in tests which don't seem important enough for their own commits.

Contributed by Josh Soref.
2019-08-26 12:05:36 -04:00
David Steele
22aa532be1 Add storage tests for files beginning with dots.
Prevent a regression of the issue fixed in f88012ce by adding some tests.
2019-08-26 11:37:21 -04:00
Josh Soref
8c9df8c290 Fix typo in httpClientReponseHeader() function name.
Contributed by Josh Soref.
2019-08-26 11:35:35 -04:00
David Steele
01c2669b97 Fix exclusions for special files.
Prior to 2.16 the Perl manifest code would skip any file that began with a dot.  This was not intentional but it allowed PostgreSQL socket files to be located in the data directory.  The new C code in 2.16 did not have this unintentional exclusion so socket files in the data directory caused errors.

Worse, the file type error was being thrown before the exclusion check so there was really no way around the issue except to move the socket files out of the data directory.

Special file types (e.g. socket, pipe) will now be automatically skipped and a warning logged to notify the user of the exclusion.  The warning can be suppressed with an explicit --exclude.

Reported by CluelessTechnologist, Janis Puris, Rachid Broum.
2019-08-23 07:47:54 -04:00
David Steele
2862f480cd Add special file type to storageInfo().
There's not much we can do with special files, but higher level logic can at least exclude them gracefully rather than throwing a hard error.
2019-08-23 07:24:25 -04:00
David Steele
f88012cef3 Fix regexp to ignore ./.. directories in the Posix driver.
In versions <= 2.15 the old regexp caused any file or directory beginning with . to be ignored during a backup.  This has caused behavioral differences in 2.16 because the new C code correctly excludes ./.. directories.

This Perl code is only used for testing now, but it should still match the output of the C functions.
2019-08-22 10:18:34 -04:00
David Steele
c002a2ce2f Move info file checksum to the end of the file.
Putting the checksum at the beginning of the file made it impossible to stream the file out when saving.  The entire file had to be held in memory while it was checksummed so the checksum could be written at the beginning.

Instead place the checksum at the end.  This does not break the existing Perl or C code since the read is not order dependent.

There are no plans to improve the Perl code to take advantage of this change, but it will make the C implementation more efficient.

Reviewed by Cynthia Shang.
2019-08-21 19:45:48 -04:00
Cynthia Shang
c733319063 The stanza-create/update/delete commands are implemented entirely in C.
Contributed by Cynthia Shang.
2019-08-21 16:26:28 -04:00
Cynthia Shang
53f27da3a6 Add checkDbConfig() to compare pgBackRest/PostgreSQL configs.
Checking the PostgreSQL-reported path and version against the pgBackRest configuration helps ensure that pgBackRest is operating against the correct cluster.

In Perl this functionality was in the Db object, but check seems like a better place for it in C.

Contributed by Cynthia Shang.
2019-08-21 15:41:52 -04:00
David Steele
8b93fdf349 Add storagePg*() variants to get storage for a specified host id.
Previously the host id to use was pulled from the host-id option or defaulted to 1.

The stanza, check, and backup commands will all need the ability to address a specified pg host, so add functions to make that possible.
2019-08-21 15:26:52 -04:00
Cynthia Shang
fa640f22ad Allow Info* objects to be created from scratch in C.
Previously, info files (e.g. archive.info, backup.info) were created in Perl and only loaded in C.

The upcoming stanza commands in C need to create these files so refactor the Info* objects to allow new, empty objects to be created.  Also, add functions needed to initialize each Info* object to a valid state.

Contributed by Cynthia Shang.
2019-08-21 15:12:00 -04:00
Cynthia Shang
71d96531f8 Fix comment typo.
Contributed by Cynthia Shang.
2019-08-21 12:10:09 -04:00
Cynthia Shang
c4533765fb Add Db object getters missed in e4901d50.
dbOpen() loaded the version and data directory but there was no way to get the data externally.

Contributed by Cynthia Shang.
2019-08-21 11:59:51 -04:00
Cynthia Shang
27e8235812 Add repoIsLocalVerify() to verify repository locality.
Some commands can only be run on a host where the repository is local.  This function centralizes the check and error.

Contributed by Cynthia Shang.
2019-08-21 11:41:36 -04:00
Cynthia Shang
6a09d9294d Require storage when calling pgControlFromFile().
Previously storageLocal() was being used internally but loading pg_control from remote storage is often required.

Also, storagePg() is more appropriate than storageLocal() for all current usage.

Contributed by Cynthia Shang.
2019-08-21 11:29:30 -04:00
David Steele
f6aef6e466 Properly reset conflicting pg-* options for the remote protocol.
The pg1-socket-path and pg1-port options were not being reset when options from a higher index were being pushed down for processing by a remote.  Since remotes only talk to one cluster they always use the options in index 1.  This requires moving options from the original index to 1 before starting the remote.  All options already set on index 1 must be removed if they are not being overwritten.
2019-08-19 21:45:54 -04:00
David Steele
d411321d28 Add reset to temp memory contexts to save memory and processing time.
Processing large datasets in a memory context can lead to high memory usage and long allocation times.  Add a new MEM_CONTEXT_TEMP_RESET_BEGIN() macro that allows temp allocations to be automatically freed after N iterations.
2019-08-19 21:16:10 -04:00
David Steele
7d97d49f41 Add MostCommonValue object.
Calculate the most common value in a list of variants.  If there is a tie then the first value passed to mcvUpdate() wins.

mcvResult() can be called multiple times because it does not end processing, but there is a cost to calculating the result each time
since it is not stored.
2019-08-18 20:46:34 -04:00
David Steele
8aa1e552b0 Add backup type conversion functions.
Convert back and forth between the string and enum representations of backup types.
2019-08-18 20:09:44 -04:00
David Steele
3df075bf40 Fix test writing "null" into manifest files.
"null" is not allowed in the manifest format (null values should be missing instead) but Perl was treating the invalid values written by this test as if they were missing.

Update the test code to remove the values rather than setting them to "null".
2019-08-18 15:29:18 -04:00
David Steele
8edc57b581 Remove RHEL package patch merged to upstream. 2019-08-18 15:19:10 -04:00
David Steele
d0dee2e260 Add bool rendering to jsonFromVar(). 2019-08-17 17:47:13 -04:00
David Steele
8fc1d3883b Fix expire not immediately writing into separate file after backup.
Logging stayed in the backup log until the Perl code started.  Fix this so it logs to the correct file and will still work after the Perl code is removed.
2019-08-17 17:43:56 -04:00
David Steele
bc5385142c Change comment for searchability. 2019-08-09 15:18:42 -04:00
Cynthia Shang
382ed92825 The start/stop commands are implemented entirely in C.
The Perl versions remain because they are still being used by the Perl stanza commands.  Once the stanza commands are migrated they can be removed.

Contributed by Cynthia Shang.
2019-08-09 15:17:18 -04:00
David Steele
fe196cb0df Update RHEL patch for upstream changes.
This isn't correct, but we can't test anything else if the package doesn't build.
2019-08-09 14:52:26 -04:00
David Steele
e9517dcec0 Add hash constants for zero-length data.
No need to calculate a hash when the data length is known to be zero.  Use one of these constants instead.
2019-08-08 18:47:02 -04:00
David Steele
1076b142e3 Remove Debian package patch merged to upstream. 2019-08-08 16:15:42 -04:00
David Steele
969e5817c5 Add cvtZToMode().
The companion function to cvtModeToZ().
2019-08-08 11:59:39 -04:00
David Steele
1f93e147fa Add kvGetDefault() to KeyValue object.
Return the default only if the key is not found, not when the value is NULL.
2019-08-08 11:41:49 -04:00
David Steele
289b47902b Allow NULLs in strEq().
Bring this function more in line with the way varEq() works.  NULL == NULL but NULL != NOT NULL.
2019-08-08 10:50:25 -04:00
Cynthia Shang
feec674b6f Add constant for lock file extension.
Contributed by Cynthia Shang.
2019-08-08 06:48:25 -04:00
David Steele
9e730c1bd6 v2.16: C Migrations and Bug Fixes
Bug Fixes:

* Retry S3 RequestTimeTooSkewed errors instead of immediately terminating. (Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.)
* Fix incorrect handling of transfer-encoding response to HEAD request. (Reported by Pavel Suderevsky.)
* Fix scoping violations exposed by optimizations in gcc 9. (Reported by Christian Lange, Ned T. Crigler.)

Features:

* Add repo-s3-port option for setting a non-standard S3 service port.

Improvements:

* The local command for backup is implemented entirely in C. (Contributed by David Steele, Cynthia Shang.)
* The check command is implemented partly in C. (Reviewed by Cynthia Shang.)
2019-08-05 12:03:04 -04:00
David Steele
3d3003e9ca The check command is implemented partly in C.
Implement switch WAL and archive check in C but leave the rest in Perl for now.

The main idea was to have some real integration tests for the new database code so the rest of the migration can wait.

Reviewed by Cynthia Shang.
2019-08-01 20:35:01 -04:00
David Steele
e4901d50d5 Add Db object to encapsulate PostgreSQL queries and commands.
Migrate functionality from the Perl Db module to C. For now this is just enough to implement the WAL switch check.

Add the dbGet() helper function to get Db objects easily.

Create macros in harnessPq to make writing pq scripts easier by grouping commonly used functions together.

Reviewed by Cynthia Shang.
2019-08-01 15:38:27 -04:00
David Steele
f9e1f3a798 Retry S3 RequestTimeTooSkewed errors instead of immediately terminating.
The cause of this error seems to be that a failed request takes so long that a subsequent retry at the http level uses outdated headers.

We're not sure if pgBackRest it to blame here (in one case a kernel downgrade fixed it, in another case an incorrect network driver was the problem) so add retries to hopefully deal with the issue if it is not too persistent.  If SSL_write() has long delays before reporting an error then this will obviously affect backup performance.

Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.
2019-08-01 14:28:30 -04:00
David Steele
2eb3c9f95f Improve error handling for SSL_write().
Error codes were not being caught for SSL_write() so it was hard to see exactly what was happening in error cases.  Report errors to aid in debugging.

Also add a retry for SSL_ERROR_WANT_READ.  Even though we have not been able to reproduce this case it is required by SSL_write() so go ahead and implement it.
2019-08-01 09:58:24 -04:00
David Steele
89c67287bc Improve multi-host handling in protocol helper.
Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
David Steele
893ae24284 Add timeout to walSegmentFind().
Keep trying to locate the WAL segment until timeout.  This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
Cynthia Shang
03b28da1ca Rename control/control module to control/common.
This is more consistent with how other common modules are named.

Contributed by Cynthia Shang.
2019-07-31 11:35:58 -04:00
David Steele
88c1929ec5 Don't pass local config* options to the remote.
The remotes have their own config options (repo-host-config, etc.) so don't pass the local config* options.

This was a regression from the behavior of the Perl code and while there have been no field reports it caused breakage on test systems with multiple configurations.
2019-07-26 08:37:58 -04:00
David Steele
f8b0676fd6 Allow modules to be included for testing without requiring coverage.
Sometimes it is useful to get at the internals of a module that is not being tested for coverage in order to provide coverage for another module that is being tested.  The include directive allows this.

Update modules that had previously been added to coverage that only need to be included.
2019-07-25 20:15:06 -04:00
David Steele
554d98746a Add repo-s3-port option for setting a non-standard S3 service port.
If this option is set then ports appended to repo-s3-endpoint or repo-s3-host will be ignored.

Setting this option explicitly may be the only way to use a bare ipv6 address with S3 (since multiple colons confuse the parser) but we plan to improve this in the future.
2019-07-25 17:36:51 -04:00
David Steele
d8ca0e5c5b Add Perl interface to C PgQuery object.
This validates that all current queries work with the new interface and removes the dependency on DBD::Pg.
2019-07-25 17:05:39 -04:00