1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

3079 Commits

Author SHA1 Message Date
David Steele
53f8e7a9cf
Asynchronous list/remove for S3/Azure storage.
Improve the performance of list/delete operations by using async requests.

It's questionable whether this will have any impact on Azure deletes since they are sent one at a time with little work done in between, but it doesn't hurt to try.
2020-08-20 12:12:21 -04:00
David Steele
9876d7096d Disable s390x builds on Travis-CI (again).
s390x currently is too unreliable to be part of required CI builds. They can sometimes take hours or days to get queued which makes getting commits through our workflow very time-consuming.
2020-08-20 11:24:27 -04:00
David Steele
de0f8c2654
Add user-agent to HTTP requests. 2020-08-18 10:01:24 -04:00
David Steele
4391497a05 Enable s390x builds on Travis-CI.
These were disabled in 6fc25c7a because s390x builds were not being queued. It looks like they are working again.
2020-08-18 08:17:13 -04:00
David Steele
d935af5e59 Update *((Type *)lstGet()) pattern to *(Type *)lstGet().
This pattern has extra parens and was only used in a few places so replace it with the general *(Type *)lstGet() pattern.
2020-08-18 07:26:02 -04:00
David Steele
fbee6ec170
Add support for HTTP/1.0.
HTTP/1.0 connections are closed by default after a single response. Other than that, treat 1.0 the same as 1.1.

HTTP/1.0 allows different date formats that we can't parse but for now, at least, we don't need any date headers from 1.0 requests.
2020-08-14 13:11:33 -04:00
David Steele
6bb111c170 PostgreSQL 13 beta3 support. 2020-08-14 10:08:25 -04:00
Don Seiler
afcc4d193d
Add missing azure type in repo-type option reference. 2020-08-11 14:38:38 -04:00
Don Seiler
f40c7b65fa Fix typo in repo-cipher-type option reference. 2020-08-11 10:41:06 -04:00
David Steele
65858d12dd Improve server test harness to support multiple protocols/connections.
The prior implementation only supported a single connection on TLS. This is not flexible enough for complex testing scenarios which might require multiple simultaneous connections on different protocols.

Allow multiple simultaneous connections and add plain sockets as a protocol option. Rename the functions used for server scripting to hrnServerScript*() to make it clear they are related. Improve error messages when less input is received by the server than expected.

Also, do a bit of cleanup and add more comments.
2020-08-10 16:16:11 -04:00
David Steele
7fdbd94e39
Implement IoClient/IoSession interfaces for SocketClient/SocketSession.
Following up on 111d33c, implement the new interfaces for socket client/session. Now HTTP objects can be used over TLS or plain sockets.

This required adding ioSessionFd() and ioSessionRole() to provide the functionality of sckSessionFd() and sckSessionType(). sckClientHost() and sckClientPort don't make sense in a generic interface so they were replaced with ioSessionName().
2020-08-10 16:03:38 -04:00
Floris van Nee
54c3c39645
Delay backup remote connection close until after archive check.
Only close the remote connection after verifying that the WAL files have been received. This is necessary if the archive_command on the PostgreSQL host is conditional, i.e. archiving only happens while a backup lock is held, to ensure all WAL segments are archived.
2020-08-10 11:35:09 -04:00
David Steele
63ad4786af Remove deleteMax parameter from storageS3New().
This was included primarily for testing but now that storageS3New() is not called directly in the tests the value needs to be modified after configuring storage anyway.

Since there is no need to make this parameter user-configurable just remove it.
2020-08-08 16:06:05 -04:00
David Steele
8b34f854f3 Simplify S3 configuration tests and add security token tests.
Rather than calling storageS3New() directly, create the storage by loading a configuration and calling repoStorageGet(). This is a better end-to-end test and cuts down on a lot of redundant tests.

Add tests that include security tokens in error messages to ensure they are redacted.
2020-08-08 15:52:33 -04:00
David Steele
38f66d4bd3 Add debug log function for HttpClient. 2020-08-08 14:32:21 -04:00
David Steele
426e6933da Redact x-amz-security-token header in errors.
This header should not be displayed to the user in error output, even if it is useless by itself.
2020-08-08 14:18:03 -04:00
David Steele
4d22d6eeca
Move file descriptor read/write ready into IoRead/IoWrite.
Move sckSessionReadyRead()/Write() into the IoRead/IoWrite interfaces. This is a more logical place for them and the alternative would be to add them to the IoSession interface, which does not seem like a good idea.

This is mostly a refactor, but a big change is the select() logic in fdRead.c has been replaced by ioReadReady(). This was duplicated code that was being used by our protocol but not TLS. Since we have not had any problems with requiring poll() in the field this seems like a good time to remove our dependence on select().

Also, IoFdWrite now requires a timeout so update where required, mostly in the tests.
2020-08-08 11:23:37 -04:00
David Steele
111d33c123
Add IoClient and IoSession interfaces.
These interfaces allow the HttpClient and HttpSession objects to work with protocols other than TLS, .e.g. plain sockets. This is necessary to allow standard HTTP -- right now only HTTPS is allowed, i.e. HTTP over TLS.

For now only TlsClient and TlsSession have been converted to the new interfaces. SocketClient and SocketSession will also need to be converted but first sckSessionReadyRead() and sckSessionReadyWrite() need to be moved into the IoRead and IoWrite interfaces, since they are not a good fit for IoSession.
2020-08-08 10:39:39 -04:00
David Steele
9b7fd1a894 Improve error message in HARNESS_FORK_END().
Also include expected exit status to make debugging easier.
2020-08-08 10:02:04 -04:00
David Steele
0b349c0f2f Clear the callback before calling storageReadPosixFreeResource().
If storageReadPosixFreeResource() errors then it will be called again when the object is freed, which is not ideal since it might error in a different way and lose the original error.
2020-08-07 13:44:01 -04:00
David Steele
cde2c756ea Rename handle to fd.
Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd.

There were also plenty of places fd was used so now all uses are consistent.

The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
David Steele
a1d7d50223 Fix macro spacing. 2020-08-05 17:30:38 -04:00
David Steele
d5edc3fe37 Remove unused port struct member.
This does not appear to have been used in quite some time and the tests are equally useless because they don't prove the correct port was passed to httpClientNew().
2020-08-05 15:24:13 -04:00
David Steele
d74c8f4b58 Add memContextFreeing().
This can be used to determine if functions are running in a callback from memContextFree(). In this case, data in child mem contexts is not valid.
2020-08-05 14:07:04 -04:00
David Steele
6fc25c7a69 Disable s390x builds on Travis-CI.
s390x has not been building for about a day which is clogging the integration branch.
2020-08-05 13:38:44 -04:00
David Steele
847e61ce21
Improve TLS error reporting.
Before 9f2d647 TLS errors included additional details in at least some cases. After 9f2d647 a connection to an HTTP server threw `TLS error [1]` instead of `unable to negotiate TLS connection: [336031996] unknown protocol`.

Bring back the detailed messages to make debugging TLS errors easier. Since the error routine is now generic the `unable to negotiate TLS connection context` is not available so the error looks like `TLS error [1:336031996] unknown protocol`.
2020-08-04 15:15:24 -04:00
David Steele
94d3a01f73
Proactively close file descriptors after forking async process.
PostgreSQL may be using most of the available file descriptors when it executes the the archive-get/archive-push commands (especially archive-get). This can lead to problems depending on how many file descriptors are needed for parallelism in the async process.

Proactively free file descriptors between 3 and 1023 to help ensure there are enough available for reasonable values of process-max, i.e. <= 300.
2020-08-04 13:20:01 -04:00
David Steele
e81533bbab
Improve memory usage of unlogged relation detection in manifest build.
This loop was using a lot of memory without freeing it at intervals.

Rewrite to use char arrays when possible to reduce memory that needs to be allocated and freed.
2020-08-04 10:16:51 -04:00
David Steele
e044eaf55f Use forkSafe() in common/exec module.
forkSafe() will throw an exception if the fork fails. This is a pretty unlikely scenario but the function is already there so may as well use it.
2020-08-04 09:23:59 -04:00
David Steele
5a9856c2f9 Add functions for Zigzag encoding/decoding.
Zigzag encoding places the sign bit in the least significant bit so that -1 is encoded as 1, 1 as 2, etc. This moves as many bits as possible into the low order bits which is good for other types of encoding, e.g. base-128.

See https://en.wikipedia.org/wiki/Variable-length_quantity#Zigzag_encoding.
2020-08-01 09:42:03 -04:00
David Steele
f6d22ccb6f Improve comment locations. 2020-07-31 17:33:15 -04:00
David Steele
caef2ba482 Accept NULL filenames in hrnStorageInfoListCallback().
This allows it to be used with StorageInfo objects returned from storageInfo(), which do not contain a filename.
2020-07-31 16:35:58 -04:00
David Steele
a8e47c38c6 Add lstGetLast() and lstRemoveLast().
When a list is being treated as a stack it is useful to get/remove the last (top) item.
2020-07-31 16:27:57 -04:00
David Steele
bfb489a82d Add file name to make performance/storage test more realistic.
Also add timing information.
2020-07-31 16:18:56 -04:00
David Steele
9f1e7d88da Remove special protocol encoding for StorageType.
It seems like overkill to encode this when other enums (e.g. StorageInfoLevel) are passed as integers.

Instead note that StorageType values should not be changed and remove the special encoding.
2020-07-31 16:13:05 -04:00
David Steele
afe1568bb1 Fix missing space. 2020-07-31 15:55:17 -04:00
David Steele
ac72e1f193
Preserve Variant parsed from JSON in iniLoad().
The fix for = characters in info files (039d314) added JSON validation but discarded the resulting Variant which means the JSON is being parsed twice. This nearly doubles the time to load a manifest since a lot of complex JSON is involved.

Time to load a million file manifest:
Before 039d314: 7.8s
039d314: 15.5s
This patch: 7.5s

To fix this regression return the Variant in the callback so the caller does not have to parse it again. The new code appears slightly more efficient overall, probably because there are fewer operations against Strings.
2020-07-30 10:59:50 -04:00
David Steele
3e9dce0d76 Rename strPtr()/strPtrNull() to strZ()/strZNull().
We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern.

As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
David Steele
039d314438
Fix issue with = character in file or database names.
The manifest uses the = character as the key/value separator so = characters in the key cause parsing errors and lead to an error or segfault.

Since the value must be valid JSON we can keep checking the value on the right side of the = and stop building the key when the value is valid. It's a bit hackish but it does seem to do the job without breaking the manifest format.

Unsurprisingly this makes parsing about 50% slower but it's still more than fast enough. Parsing 10 million key/values takes about 6.5s for the old code and 10s for the new code. Since the value is used as JSON downstream we can reclaim most of this time by just passing the JSON value rather than making the callback reparse it. We'll save that for another commit, though.
2020-07-28 14:00:23 -04:00
David Steele
63a93db6fd
Suppress errors when closing local/remote processes.
Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened.

Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
David Steele
d9309f13a4 Add quarterly press release instructions to doc/RELEASE.md. 2020-07-28 11:20:07 -04:00
David Steele
a260d4a53b Add zstd to CentOS/RHEL 6 test container.
Zstd is now required by the upstream yum package.
2020-07-28 08:09:10 -04:00
David Steele
aa37a21084 Add strBaseZ().
Since the base name is at the end of the filename it is possible to return the char pointer without making a copy, which is preferred in some cases.
2020-07-25 13:37:39 -04:00
David Steele
8e140ad7e0 Build new manifest inside a temporary context.
There is a fair amount of temporary data that is created during the build and it should not end up as a permanent part of the manifest.
2020-07-25 11:39:24 -04:00
David Steele
8e776cc537 Add manifestNewBuild() to performance/type test.
Use a test storage driver to allow manifestNewBuild() to be run against a test cluster at any scale without having to write files to disk.

Simplify the test by using the output of manifestNewBuild() to feed manifestSave() and manifestNewLoad().

Also add manifest size to the output.
2020-07-25 11:26:14 -04:00
David Steele
f9d923ca3b Add memContextSize().
Calculates the memory used by the context and all child contexts.

This is primarily useful for debugging but it is not conditional on DEBUG because it is useful for profile/performance tests.
2020-07-25 11:06:25 -04:00
David Steele
216a61d936 Move dummy storage driver to test harness.
The dummy driver is the basis for creating test storage drivers so it makes sense to locate it in the harness where all tests can access it.
2020-07-25 08:44:41 -04:00
David Steele
cbf5dd19c2 Fix non-JSON ini values in tests.
A number of tests used invalid JSON values where an error was expected or the value would be ignored.

Update these tests to use valid JSON values so all values in the file can be validated even if they are not used.
2020-07-24 11:05:09 -04:00
David Steele
9d8f6d9fc2 Error in jsonToVar() when input not entirely consumed.
Something like 3="string" would return an Int64 variant and ignore the invalid portion after the integer. Other JSON interface functions have this check but it was forgotten here.

There are no current issues because of this but we want to be able to validate arbitrary JSON strings and this function was not working correctly for that usage.
2020-07-24 10:47:51 -04:00
Cynthia Shang
78ef442a18
Add storage parameter to pgWalFromFile(). 2020-07-21 16:28:05 -04:00