db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-04-27 12:12:18 +02:00
Code Issues Releases Activity
pgbackrest/src/protocol/helper.c

187 lines
8.3 KiB
C
Raw Normal View History

Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Protocol Helper
***********************************************************************************************************************************/
#include "common/debug.h"
#include "common/exec.h"
#include "common/memContext.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "crypto/crypto.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "config/config.h"
#include "config/exec.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "config/protocol.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "protocol/helper.h"
/***********************************************************************************************************************************
Constants
***********************************************************************************************************************************/
STRING_EXTERN(PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE);
/***********************************************************************************************************************************
Local variables
***********************************************************************************************************************************/
static struct
{
MemContext *memContext; // Mem context for protocol helper
Exec *remoteExec; // Executed remote
ProtocolClient *remote; // Remote protocol client
} protocolHelper;
/***********************************************************************************************************************************
Is the repository local?
***********************************************************************************************************************************/
bool
repoIsLocal(void)
{
FUNCTION_TEST_VOID();
Remove unused type parameter from FUNCTION_TEST_RETURN(). This parameter was always useless but commit 7333b630 removed all references to it so remove the parameter at all call sites as well. The original intention was probably to allow logging of TEST return values but that never happened.
2019-01-28 15:06:28 +02:00
FUNCTION_TEST_RETURN(!cfgOptionTest(cfgOptRepoHost));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
/***********************************************************************************************************************************
Get the command line required for protocol execution
***********************************************************************************************************************************/
static StringList *
protocolParam(RemoteType remoteType, unsigned int remoteId)
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(ENUM, remoteType);
FUNCTION_LOG_PARAM(UINT, remoteId);
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
ASSERT(remoteType == remoteTypeRepo); // ??? Hard-coded until the function supports db remotes
ASSERT(remoteId == 1); // ??? Hard-coded until the function supports db remotes
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Fixed parameters for ssh command
StringList *result = strLstNew();
strLstAddZ(result, "-o");
strLstAddZ(result, "LogLevel=error");
strLstAddZ(result, "-o");
strLstAddZ(result, "Compression=no");
strLstAddZ(result, "-o");
strLstAddZ(result, "PasswordAuthentication=no");
// Append port if specified
if (cfgOptionTest(cfgOptRepoHostPort))
{
strLstAddZ(result, "-p");
strLstAdd(result, strNewFmt("%d", cfgOptionInt(cfgOptRepoHostPort)));
}
// Append user/host
strLstAdd(result, strNewFmt("%s@%s", strPtr(cfgOptionStr(cfgOptRepoHostUser)), strPtr(cfgOptionStr(cfgOptRepoHost))));
// Append pgbackrest command
KeyValue *optionReplace = kvNew();
// Replace config options with the host versions
if (cfgOptionSource(cfgOptRepoHostConfig) != cfgSourceDefault)
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptConfig))), cfgOption(cfgOptRepoHostConfig));
if (cfgOptionSource(cfgOptRepoHostConfigIncludePath) != cfgSourceDefault)
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptConfigIncludePath))), cfgOption(cfgOptRepoHostConfigIncludePath));
if (cfgOptionSource(cfgOptRepoHostConfigPath) != cfgSourceDefault)
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptConfigPath))), cfgOption(cfgOptRepoHostConfigPath));
// If this is the local or remote command then we need to add the command option
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptCommand))), varNewStr(strNew(cfgCommandName(cfgCommand()))));
// Add the process id -- used when more than one process will be called
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptProcess))), varNewInt(0));
Resolve storage path expressions before passing to remote. Expressions such as <REPO:ARCHIVE> require a stanza name in order to be resolved correctly. However, if the stanza name is passed to the remote then that remote will only work correctly for that one stanza. Instead, resolved the expressions locally but still pass a relative path to the remote. That way, a storage path that is only configured on the remote does not need to be known locally.
2019-02-21 15:40:21 +02:00
// Don't pass the stanza if it is set. It is better if the remote is stanza-agnostic so the client can operate on multiple
// stanzas without starting a new remote. Once the Perl code is removed the stanza option can be removed from the remote
// command.
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptStanza))), NULL);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Add the type
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptType))), varNewStr(strNew("backup")));
StringList *commandExec = cfgExecParam(cfgCmdRemote, optionReplace);
strLstInsert(commandExec, 0, cfgOptionStr(cfgOptRepoHostCmd));
strLstAdd(result, strLstJoin(commandExec, " "));
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN(STRING_LIST, result);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
/***********************************************************************************************************************************
Get the protocol client
***********************************************************************************************************************************/
ProtocolClient *
protocolGet(RemoteType remoteType, unsigned int remoteId)
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(ENUM, remoteType);
FUNCTION_LOG_PARAM(UINT, remoteId);
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Create a mem context to store protocol objects
if (protocolHelper.memContext == NULL)
{
MEM_CONTEXT_BEGIN(memContextTop())
{
protocolHelper.memContext = memContextNew("ProtocolHelper");
}
MEM_CONTEXT_END();
}
// Create protocol object
if (protocolHelper.remote == NULL)
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
// Execute the protocol command
protocolHelper.remoteExec = execNew(
cfgOptionStr(cfgOptCmdSsh), protocolParam(remoteType, remoteId),
strNewFmt("remote-%u process on '%s'", remoteId, strPtr(cfgOptionStr(cfgOptRepoHost))),
(TimeMSec)(cfgOptionDbl(cfgOptProtocolTimeout) * 1000));
execOpen(protocolHelper.remoteExec);
// Create protocol object
protocolHelper.remote = protocolClientNew(
strNewFmt("remote-%u protocol on '%s'", remoteId, strPtr(cfgOptionStr(cfgOptRepoHost))),
PROTOCOL_SERVICE_REMOTE_STR, execIoRead(protocolHelper.remoteExec), execIoWrite(protocolHelper.remoteExec));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Get cipher options from the remote if none are locally configured
if (strEq(cfgOptionStr(cfgOptRepoCipherType), CIPHER_TYPE_NONE_STR))
{
// Options to query
VariantList *param = varLstNew();
varLstAdd(param, varNewStr(strNew(cfgOptionName(cfgOptRepoCipherType))));
varLstAdd(param, varNewStr(strNew(cfgOptionName(cfgOptRepoCipherPass))));
VariantList *optionList = configProtocolOption(protocolHelper.remote, param);
if (!strEq(varStr(varLstGet(optionList, 0)), CIPHER_TYPE_NONE_STR))
{
cfgOptionSet(cfgOptRepoCipherType, cfgSourceConfig, varLstGet(optionList, 0));
cfgOptionSet(cfgOptRepoCipherPass, cfgSourceConfig, varLstGet(optionList, 1));
}
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
protocolClientMove(protocolHelper.remote, execMemContext(protocolHelper.remoteExec));
}
MEM_CONTEXT_END();
}
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelper.remote);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
/***********************************************************************************************************************************
Free the protocol objects and shutdown processes
***********************************************************************************************************************************/
void
protocolFree(void)
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_VOID(logLevelTrace);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
if (protocolHelper.remote != NULL)
{
protocolClientFree(protocolHelper.remote);
execFree(protocolHelper.remoteExec);
protocolHelper.remote = NULL;
}
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN_VOID();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Reference in New Issue Copy Permalink