1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

1603 Commits

Author SHA1 Message Date
David Steele
98f6a1cffd Remove unused cfgDefOptionPrefix() function and data.
This function was made obsolete when Perl was removed in f0ef73db.
2020-10-07 12:15:50 -04:00
David Steele
9377d05072
Add repo-azure-endpoint option.
This option allows alternate endpoints (e.g. Azure Government) to be configured.
2020-10-06 17:15:48 -04:00
David Steele
9229d6c866 v2.30: PostgreSQL 13 Support
Bug Fixes:

* Error with hints when backup user cannot read pg_settings. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Mohamed Insaf K.)

Features:

* PostgreSQL 13 support. (Reviewed by Cynthia Shang.)

Improvements:

* Improve PostgreSQL version identification. (Reviewed by Cynthia Shang, Stephen Frost.)
* Improve working directory error message. (Reviewed by Stefan Fercot.)
* Add hint about starting the stanza when WAL segment not found. (Contributed by David Christensen. Reviewed by David Steele.)
* Add hint for protocol version mismatch. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)

Documentation Improvements:

* Add note that pgBackRest versions must match when running remotely. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)
* Move info command text to the reference and link to user guide. (Reviewed by Cynthia Shang. Suggested by Christophe Courtois.)
* Update yum repository path for CentOS/RHEL user guide. (Contributed by Heath Lord. Reviewed by David Steele.)
2020-10-05 11:51:45 -04:00
David Steele
31962377d9
Add hint and documentation for protocol version mismatches.
Update the documentation to explicitly state that versions must match across hosts when running remotely.

Add a hint to the protocol version mismatch error to help the user identify the problem.
2020-09-25 10:30:29 -04:00
David Steele
b096a25b49 Update test containers for PostgreSQL 13.
Add older PostgreSQL versions to the u18 container that were not available before.

This also updates all minor versions for prior versions of PostgreSQL.
2020-09-24 11:19:51 -04:00
Cynthia Shang
ad79932ba5
Add internal verify command.
Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet).

The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
David Steele
927d9adbee
Improve PostgreSQL version identification.
Previously, catalog versions were fixed for all versions which made maintaining the catalog versions during PostgreSQL beta and release candidate cycles very painful. A version of pgBackRest which was functionally compatible was rendered useless by a catalog version bump in PostgreSQL.

Instead use only the control version to identify a PostgreSQL version when possible. Some older versions require a catalog version to positively identify a PostgreSQL version, so include them when required.

Since the catalog number is required to work with tablespaces it will need to be stored. There's already a copy of it in backup.info so use that (even though we have been ignoring it in the C versions).
2020-09-18 16:55:26 -04:00
David Steele
4cd61152f5 Update PostgreSQL 13 test catalog versions missed in 6bb111c1.
These values are not used by the Perl integration tests so maybe it would be better to remove them, but for now just update since they should not be changing again for PG13.
2020-09-17 12:39:30 -04:00
David Steele
9db3143973
Error with hints when backup user cannot read pg_settings.
This condition used to give a not-very-clear error which we have been intending to improve. But in the meantime the changes in fbff299 resulted in a segfault for this condition instead because the data_directory was assumed to be non-NULL.

Fix this by explicitly throwing an error with hints when any row in pg_settings cannot be selected.
2020-09-17 10:35:27 -04:00
David Steele
c71609879b Fix incorrect control/catalog versions in command/stanza test.
These were not caught because backup info load automatically corrects them.
2020-09-16 09:33:37 -04:00
David Steele
8dce7bbb60
Ignore backup_manifest in PG13.
This file is created by pg_basebackup so might be in the data directory if the cluster was restored from a pg_basebackup backup. Also exclude backup_manifest.tmp since it is possible to find that in the backup directory.
2020-09-14 10:15:40 -04:00
David Steele
fc77c51182
Improve working directory error message.
Improve the wording of the error message and add a hint to make it clearer what is wrong and how the user can fix it.

Also change the assert to a regular error since this is not an internal error.
2020-09-11 10:10:25 -04:00
David Steele
dac182c06e Improve alignment in TEST_ERROR output.
Align the error types and messages so they are easier to compare.
2020-09-11 08:23:26 -04:00
Cynthia Shang
ecb112f802 Add infoArchiveMove() and infoBackupMove(). 2020-09-08 13:02:13 -04:00
Cynthia Shang
4239f8a81a Add manifestFree(). 2020-09-08 12:47:35 -04:00
Cynthia Shang
b8efb13bcb Move archiveIdComparator() to archive/common module. 2020-09-08 12:28:56 -04:00
David Christensen
9fd31913a8 Add hint about starting the stanza when WAL segment not found.
If a stop command has been issued the check command fails due to archiving timing out.

Provide a hint to document this situation and point the user in the proper direction.
2020-09-03 07:49:49 -04:00
David Steele
41118e1b68
Allow ProtocolParallel to complete with no jobs.
If the callback never returned any jobs then protocolParallelDone() would never be true. The reason is that the done state was being set in protocolParallelResult(), which never gets called if there are no results.

Calling protocolParallelResult() doesn't make much sense in this case so instead move the done logic to protocolParallelDone().

For current usage of ProtocolParallel we ensure there are jobs before processing so this is not a live issue, but the new behavior is required for future development.
2020-09-02 11:03:49 -04:00
David Steele
1df7d3eee5 v2.29: Auto S3 Credentials on AWS
Bug Fixes:

* Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. (Reviewed by Cynthia Shang. Reported by argdenis.)
* Fix issue with = character in file or database names. (Reviewed by Bastian Wegge, Cynthia Shang. Reported by Brad Nicholson, Bastian Wegge.)

Features:

* Automatically retrieve temporary S3 credentials on AWS instances. (Contributed by David Steele, Stephen Frost. Reviewed by Cynthia Shang, David Youatt, Aleš Zelený, Jeanette Bromage.)
* Add archive-mode option to disable archiving on restore. (Reviewed by Stephen Frost. Suggested by Stephen Frost.)

Improvements:

* PostgreSQL 13 beta3 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
* Asynchronous list/remove for S3/Azure storage. (Reviewed by Cynthia Shang, Stephen Frost.)
* Improve memory usage of unlogged relation detection in manifest build. (Reviewed by Cynthia Shang, Stephen Frost, Brad Nicholson, Oscar. Suggested by Oscar, Brad Nicholson.)
* Proactively close file descriptors after forking async process. (Reviewed by Stephen Frost, Cynthia Shang.)
* Delay backup remote connection close until after archive check. (Contributed by Floris van Nee. Reviewed by David Steele.)
* Improve detailed error output. (Reviewed by Cynthia Shang.)
* Improve TLS error reporting. (Reviewed by Cynthia Shang, Stephen Frost.)

Documentation Bug Fixes:

* Add none to compress-type option reference and fix example. (Reported by Ugo Bellavance, Don Seiler.)
* Add missing azure type in repo-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)
* Fix typo in repo-cipher-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)

Documentation Improvements:

* Clarify that expire must be run regularly when expire-auto is disabled. (Reviewed by Douglas J Hunley. Suggested by Douglas J Hunley.)
2020-08-31 07:28:22 -04:00
David Steele
8c2960fab3
Add archive-mode option to disable archiving on restore.
When restoring a cluster that will be promoted but is not intended to be the new primary, it is important to disable archiving to avoid polluting the repository with useless WAL. This option makes disabling archiving a bit easier.
2020-08-25 15:05:41 -04:00
David Steele
851f2e814e
Automatically retrieve temporary S3 credentials on AWS instances.
Automatically retrieve the role and temporary credentials for S3 when the AWS instance is associated with an IAM role. Credentials are automatically updated when they are <= 5 minutes from expiring.

Basic configuration is to set repo1-s3-key-type=auto. repo1-s3-role can be used to set a specific role, otherwise it will be retrieved automatically.
2020-08-25 10:38:49 -04:00
David Steele
1d2b8ae7bb
Improve detailed error output.
Add more info (command, version, options) to asserts, and errors when debug logging is enabled. This won't cover all cases but might mean we get more info in some circumstances.
2020-08-24 14:51:08 -04:00
David Steele
7fba1f0bbe Consolidate variables into structs in common/stackTrace module.
This makes it easier to determine which variables are used in which parts of the module and reduces the chance of name conflicts while unit testing.
2020-08-21 16:12:44 -04:00
David Steele
0e8e5881d8 Remove invalid call to stackTraceTestFileLineSet().
When testing the common/stack-trace module it is important not to call this test function since the trace stack is empty and it will cause a buffer under run.

Instead use a macro that is only defined under the correct circumstances and add an assert() to catch future regressions.
2020-08-21 11:43:26 -04:00
David Steele
25ee913932 Rearrange members in StackTraceData struct to save memory.
This arrangement is more efficient on 64-bit systems due to alignment. There is no difference on 32-bit systems.
2020-08-20 15:53:01 -04:00
David Steele
859b8a50fd Remove unused parameter from cmdBegin(). 2020-08-20 14:16:36 -04:00
David Steele
fccca0d716 Refactor option logging into a general-purpose function. 2020-08-20 14:11:40 -04:00
David Steele
959f77cd6a
Add general-purpose statistics collector.
Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome.

Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually.

No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible.

The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
David Steele
de0f8c2654
Add user-agent to HTTP requests. 2020-08-18 10:01:24 -04:00
David Steele
d935af5e59 Update *((Type *)lstGet()) pattern to *(Type *)lstGet().
This pattern has extra parens and was only used in a few places so replace it with the general *(Type *)lstGet() pattern.
2020-08-18 07:26:02 -04:00
David Steele
fbee6ec170
Add support for HTTP/1.0.
HTTP/1.0 connections are closed by default after a single response. Other than that, treat 1.0 the same as 1.1.

HTTP/1.0 allows different date formats that we can't parse but for now, at least, we don't need any date headers from 1.0 requests.
2020-08-14 13:11:33 -04:00
David Steele
6bb111c170 PostgreSQL 13 beta3 support. 2020-08-14 10:08:25 -04:00
David Steele
65858d12dd Improve server test harness to support multiple protocols/connections.
The prior implementation only supported a single connection on TLS. This is not flexible enough for complex testing scenarios which might require multiple simultaneous connections on different protocols.

Allow multiple simultaneous connections and add plain sockets as a protocol option. Rename the functions used for server scripting to hrnServerScript*() to make it clear they are related. Improve error messages when less input is received by the server than expected.

Also, do a bit of cleanup and add more comments.
2020-08-10 16:16:11 -04:00
David Steele
7fdbd94e39
Implement IoClient/IoSession interfaces for SocketClient/SocketSession.
Following up on 111d33c, implement the new interfaces for socket client/session. Now HTTP objects can be used over TLS or plain sockets.

This required adding ioSessionFd() and ioSessionRole() to provide the functionality of sckSessionFd() and sckSessionType(). sckClientHost() and sckClientPort don't make sense in a generic interface so they were replaced with ioSessionName().
2020-08-10 16:03:38 -04:00
David Steele
8b34f854f3 Simplify S3 configuration tests and add security token tests.
Rather than calling storageS3New() directly, create the storage by loading a configuration and calling repoStorageGet(). This is a better end-to-end test and cuts down on a lot of redundant tests.

Add tests that include security tokens in error messages to ensure they are redacted.
2020-08-08 15:52:33 -04:00
David Steele
4d22d6eeca
Move file descriptor read/write ready into IoRead/IoWrite.
Move sckSessionReadyRead()/Write() into the IoRead/IoWrite interfaces. This is a more logical place for them and the alternative would be to add them to the IoSession interface, which does not seem like a good idea.

This is mostly a refactor, but a big change is the select() logic in fdRead.c has been replaced by ioReadReady(). This was duplicated code that was being used by our protocol but not TLS. Since we have not had any problems with requiring poll() in the field this seems like a good time to remove our dependence on select().

Also, IoFdWrite now requires a timeout so update where required, mostly in the tests.
2020-08-08 11:23:37 -04:00
David Steele
111d33c123
Add IoClient and IoSession interfaces.
These interfaces allow the HttpClient and HttpSession objects to work with protocols other than TLS, .e.g. plain sockets. This is necessary to allow standard HTTP -- right now only HTTPS is allowed, i.e. HTTP over TLS.

For now only TlsClient and TlsSession have been converted to the new interfaces. SocketClient and SocketSession will also need to be converted but first sckSessionReadyRead() and sckSessionReadyWrite() need to be moved into the IoRead and IoWrite interfaces, since they are not a good fit for IoSession.
2020-08-08 10:39:39 -04:00
David Steele
9b7fd1a894 Improve error message in HARNESS_FORK_END().
Also include expected exit status to make debugging easier.
2020-08-08 10:02:04 -04:00
David Steele
cde2c756ea Rename handle to fd.
Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd.

There were also plenty of places fd was used so now all uses are consistent.

The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
David Steele
a1d7d50223 Fix macro spacing. 2020-08-05 17:30:38 -04:00
David Steele
d5edc3fe37 Remove unused port struct member.
This does not appear to have been used in quite some time and the tests are equally useless because they don't prove the correct port was passed to httpClientNew().
2020-08-05 15:24:13 -04:00
David Steele
d74c8f4b58 Add memContextFreeing().
This can be used to determine if functions are running in a callback from memContextFree(). In this case, data in child mem contexts is not valid.
2020-08-05 14:07:04 -04:00
David Steele
847e61ce21
Improve TLS error reporting.
Before 9f2d647 TLS errors included additional details in at least some cases. After 9f2d647 a connection to an HTTP server threw `TLS error [1]` instead of `unable to negotiate TLS connection: [336031996] unknown protocol`.

Bring back the detailed messages to make debugging TLS errors easier. Since the error routine is now generic the `unable to negotiate TLS connection context` is not available so the error looks like `TLS error [1:336031996] unknown protocol`.
2020-08-04 15:15:24 -04:00
David Steele
e81533bbab
Improve memory usage of unlogged relation detection in manifest build.
This loop was using a lot of memory without freeing it at intervals.

Rewrite to use char arrays when possible to reduce memory that needs to be allocated and freed.
2020-08-04 10:16:51 -04:00
David Steele
5a9856c2f9 Add functions for Zigzag encoding/decoding.
Zigzag encoding places the sign bit in the least significant bit so that -1 is encoded as 1, 1 as 2, etc. This moves as many bits as possible into the low order bits which is good for other types of encoding, e.g. base-128.

See https://en.wikipedia.org/wiki/Variable-length_quantity#Zigzag_encoding.
2020-08-01 09:42:03 -04:00
David Steele
caef2ba482 Accept NULL filenames in hrnStorageInfoListCallback().
This allows it to be used with StorageInfo objects returned from storageInfo(), which do not contain a filename.
2020-07-31 16:35:58 -04:00
David Steele
a8e47c38c6 Add lstGetLast() and lstRemoveLast().
When a list is being treated as a stack it is useful to get/remove the last (top) item.
2020-07-31 16:27:57 -04:00
David Steele
bfb489a82d Add file name to make performance/storage test more realistic.
Also add timing information.
2020-07-31 16:18:56 -04:00
David Steele
9f1e7d88da Remove special protocol encoding for StorageType.
It seems like overkill to encode this when other enums (e.g. StorageInfoLevel) are passed as integers.

Instead note that StorageType values should not be changed and remove the special encoding.
2020-07-31 16:13:05 -04:00
David Steele
ac72e1f193
Preserve Variant parsed from JSON in iniLoad().
The fix for = characters in info files (039d314) added JSON validation but discarded the resulting Variant which means the JSON is being parsed twice. This nearly doubles the time to load a manifest since a lot of complex JSON is involved.

Time to load a million file manifest:
Before 039d314: 7.8s
039d314: 15.5s
This patch: 7.5s

To fix this regression return the Variant in the callback so the caller does not have to parse it again. The new code appears slightly more efficient overall, probably because there are fewer operations against Strings.
2020-07-30 10:59:50 -04:00