db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Code Issues Releases Activity
Files
aa72c19a831fb0dd15a50a003fbaad427f335353
pgbackrest/src/command/repo/create.c

47 lines
1.8 KiB
C
Raw Normal View History

Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
/***********************************************************************************************************************************
Repository Create Command
***********************************************************************************************************************************/
#include "build.auto.h"
#include "common/debug.h"
#include "common/log.h"
#include "common/memContext.h"
GCS support for repository storage. GCS and GCS-compatible object stores can now be used for repository storage.
2021-03-05 12:13:51 -05:00
#include "common/type/json.h"
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
#include "config/config.h"
#include "storage/helper.h"
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
#include "storage/azure/storage.intern.h"
GCS support for repository storage. GCS and GCS-compatible object stores can now be used for repository storage.
2021-03-05 12:13:51 -05:00
#include "storage/gcs/storage.intern.h"
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
#include "storage/s3/storage.intern.h"
/**********************************************************************************************************************************/
void
cmdRepoCreate(void)
{
FUNCTION_LOG_VOID(logLevelDebug);
MEM_CONTEXT_TEMP_BEGIN()
{
if (strEq(storageType(storageRepo()), STORAGE_S3_TYPE_STR))
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
{
Asynchronous S3 multipart upload. When uploading large files the upload is split into multiple parts which are assembled at the end to create the final file. Previously we waited until each part was acknowledged before starting on the processing (i.e. compression, etc.) of the next part. Now, the request for each part is sent while processing continues and the response is read just before sending the request for the next part. This asynchronous method allows us to continue processing while the S3 server formulates a response. Testing from outside AWS in a high-bandwidth, low-latency environment showed a 35% improvement in the upload time of 1GB files. The time spent waiting for multipart notifications was reduced by ~300% (this measurement included the final part which is not uploaded asynchronously). There are still some possible improvements: 1) the creation of the multipart id could be made asynchronous when it looks like the upload will need to be multipart (this may incur cost if the upload turns out not to be multipart). 2) allow more than one async request (this will use more memory). A fair amount of refactoring was required to make the HTTP responses asynchronous. This may seem like overkill but having well-defined request, response, and session objects will also be advantageous for the upcoming HTTP server functionality. Another advantage is that the lifecycle of an HttpSession is better defined. We only want to reuse sessions that complete the request/response cycle successfully, otherwise we consider the session to be in a bad state and would prefer to start clean with a new one. Previously, this required complex notifications to mark a session as "successfully done". Now, ownership of the session is passed to the request and then the response and only returned to the client after a successful response. If an error occurs anywhere along the way the session will be automatically closed by the object destructor when the request/response object is freed (depending on which one currently owns the session).
2020-06-24 13:44:00 -04:00
storageS3RequestP((StorageS3 *)storageDriver(storageRepoWrite()), HTTP_VERB_PUT_STR, FSLASH_STR);
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
}
else if (strEq(storageType(storageRepo()), STORAGE_AZURE_TYPE_STR))
{
storageAzureRequestP(
(StorageAzure *)storageDriver(storageRepoWrite()), HTTP_VERB_PUT_STR,
Allow redactions for HTTP queries. The Azure storage driver exposes secrets in the query when using SAS authorization. These secrets can show up during logging or when an error occurs. Allow redaction of queries to prevent secrets from being exposed in logs and errors.
2020-07-14 13:09:48 -04:00
.query = httpQueryAdd(httpQueryNewP(), AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_VALUE_CONTAINER_STR));
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
}
GCS support for repository storage. GCS and GCS-compatible object stores can now be used for repository storage.
2021-03-05 12:13:51 -05:00
else if (strEq(storageType(storageRepo()), STORAGE_GCS_TYPE_STR))
{
KeyValue *kvContent = kvPut(kvNew(), GCS_JSON_NAME_VAR, VARSTR(cfgOptionStr(cfgOptRepoGcsBucket)));
storageGcsRequestP(
(StorageGcs *)storageDriver(storageRepoWrite()), HTTP_VERB_POST_STR, .noBucket = true,
.content = BUFSTR(jsonFromKv(kvContent)));
}
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
}
MEM_CONTEXT_TEMP_END();
FUNCTION_LOG_RETURN_VOID();
}
Reference in New Issue Copy Permalink