1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-07 00:35:37 +02:00
Commit Graph

6 Commits

Author SHA1 Message Date
6cc521b6b2 Update storage module to use StringIds.
Use StringIds for the storage types (e.g. STORAGE_S3_TYPE) and configuration settings, e.g. cfgOptS3KeyType.

Also add new config functions and harness config functions to support StringIds.
2021-04-23 13:19:47 -04:00
088662d986 GCS support for repository storage.
GCS and GCS-compatible object stores can now be used for repository storage.
2021-03-05 12:13:51 -05:00
91c7adc834 Allow redactions for HTTP queries.
The Azure storage driver exposes secrets in the query when using SAS authorization. These secrets can show up during logging or when an error occurs.

Allow redaction of queries to prevent secrets from being exposed in logs and errors.
2020-07-14 13:09:48 -04:00
3f4371d7a2 Azure support for repository storage.
Azure and Azure-compatible object stores can now be used for repository storage.

Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
c5892d1291 Asynchronous S3 multipart upload.
When uploading large files the upload is split into multiple parts which are assembled at the end to create the final file. Previously we waited until each part was acknowledged before starting on the processing (i.e. compression, etc.) of the next part.

Now, the request for each part is sent while processing continues and the response is read just before sending the request for the next part. This asynchronous method allows us to continue processing while the S3 server formulates a response.

Testing from outside AWS in a high-bandwidth, low-latency environment showed a 35% improvement in the upload time of 1GB files. The time spent waiting for multipart notifications was reduced by ~300% (this measurement included the final part which is not uploaded asynchronously).

There are still some possible improvements: 1) the creation of the multipart id could be made asynchronous when it looks like the upload will need to be multipart (this may incur cost if the upload turns out not to be multipart). 2) allow more than one async request (this will use more memory).

A fair amount of refactoring was required to make the HTTP responses asynchronous. This may seem like overkill but having well-defined request, response, and session objects will also be advantageous for the upcoming HTTP server functionality.

Another advantage is that the lifecycle of an HttpSession is better defined. We only want to reuse sessions that complete the request/response cycle successfully, otherwise we consider the session to be in a bad state and would prefer to start clean with a new one. Previously, this required complex notifications to mark a session as "successfully done". Now, ownership of the session is passed to the request and then the response and only returned to the client after a successful response. If an error occurs anywhere along the way the session will be automatically closed by the object destructor when the request/response object is freed (depending on which one currently owns the session).
2020-06-24 13:44:00 -04:00
d3c83453de Add repo-create, repo-get, repo-put, and repo-rm commands.
These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage.

All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00