db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Code Issues Releases Activity
Files
d3dd32a031fa7dab98bbef3eee5c7390a774a9d9
pgbackrest/src/command/repo/create.c

37 lines
1.3 KiB
C
Raw Normal View History

Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
/***********************************************************************************************************************************
Repository Create Command
***********************************************************************************************************************************/
#include "build.auto.h"
#include "common/debug.h"
#include "common/log.h"
#include "common/memContext.h"
#include "config/config.h"
#include "storage/helper.h"
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
#include "storage/azure/storage.intern.h"
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
#include "storage/s3/storage.intern.h"
/**********************************************************************************************************************************/
void
cmdRepoCreate(void)
{
FUNCTION_LOG_VOID(logLevelDebug);
MEM_CONTEXT_TEMP_BEGIN()
{
if (strEq(storageType(storageRepo()), STORAGE_S3_TYPE_STR))
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
{
Asynchronous S3 multipart upload. When uploading large files the upload is split into multiple parts which are assembled at the end to create the final file. Previously we waited until each part was acknowledged before starting on the processing (i.e. compression, etc.) of the next part. Now, the request for each part is sent while processing continues and the response is read just before sending the request for the next part. This asynchronous method allows us to continue processing while the S3 server formulates a response. Testing from outside AWS in a high-bandwidth, low-latency environment showed a 35% improvement in the upload time of 1GB files. The time spent waiting for multipart notifications was reduced by ~300% (this measurement included the final part which is not uploaded asynchronously). There are still some possible improvements: 1) the creation of the multipart id could be made asynchronous when it looks like the upload will need to be multipart (this may incur cost if the upload turns out not to be multipart). 2) allow more than one async request (this will use more memory). A fair amount of refactoring was required to make the HTTP responses asynchronous. This may seem like overkill but having well-defined request, response, and session objects will also be advantageous for the upcoming HTTP server functionality. Another advantage is that the lifecycle of an HttpSession is better defined. We only want to reuse sessions that complete the request/response cycle successfully, otherwise we consider the session to be in a bad state and would prefer to start clean with a new one. Previously, this required complex notifications to mark a session as "successfully done". Now, ownership of the session is passed to the request and then the response and only returned to the client after a successful response. If an error occurs anywhere along the way the session will be automatically closed by the object destructor when the request/response object is freed (depending on which one currently owns the session).
2020-06-24 13:44:00 -04:00
storageS3RequestP((StorageS3 *)storageDriver(storageRepoWrite()), HTTP_VERB_PUT_STR, FSLASH_STR);
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 16:24:34 -04:00
}
else if (strEq(storageType(storageRepo()), STORAGE_AZURE_TYPE_STR))
{
storageAzureRequestP(
(StorageAzure *)storageDriver(storageRepoWrite()), HTTP_VERB_PUT_STR,
.query = httpQueryAdd(httpQueryNew(), AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_VALUE_CONTAINER_STR));
}
Add repo-create, repo-get, repo-put, and repo-rm commands. These commands are generally useful but more importantly they allow removing LibC by providing the Perl integration tests an alternate way to work with repository storage. All the commands are currently internal only and should not be used on production repositories.
2020-03-09 17:15:03 -04:00
}
MEM_CONTEXT_TEMP_END();
FUNCTION_LOG_RETURN_VOID();
}
Reference in New Issue Copy Permalink