db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-04-27 12:12:18 +02:00
Code Issues Releases Activity
pgbackrest/src/protocol/helper.c

353 lines
15 KiB
C
Raw Normal View History

Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Protocol Helper
***********************************************************************************************************************************/
#include "common/debug.h"
#include "common/exec.h"
#include "common/memContext.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "crypto/crypto.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "config/config.h"
#include "config/exec.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "config/protocol.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "protocol/helper.h"
/***********************************************************************************************************************************
Constants
***********************************************************************************************************************************/
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
STRING_EXTERN(PROTOCOL_SERVICE_LOCAL_STR, PROTOCOL_SERVICE_LOCAL);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
STRING_EXTERN(PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE);
/***********************************************************************************************************************************
Local variables
***********************************************************************************************************************************/
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
typedef struct ProtocolHelperClient
{
Exec *exec; // Executed client
ProtocolClient *client; // Protocol client
} ProtocolHelperClient;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
static struct
{
MemContext *memContext; // Mem context for protocol helper
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
unsigned int clientRemoteSize; // Remote clients
ProtocolHelperClient *clientRemote;
unsigned int clientLocalSize; // Local clients
ProtocolHelperClient *clientLocal;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
} protocolHelper;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
/***********************************************************************************************************************************
Init local mem context and data structure
***********************************************************************************************************************************/
static void
protocolHelperInit(void)
{
// In the protocol helper has not been initialized
if (protocolHelper.memContext == NULL)
{
// Create a mem context to store protocol objects
MEM_CONTEXT_BEGIN(memContextTop())
{
protocolHelper.memContext = memContextNew("ProtocolHelper");
}
MEM_CONTEXT_END();
}
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Is the repository local?
***********************************************************************************************************************************/
bool
repoIsLocal(void)
{
FUNCTION_TEST_VOID();
Remove unused type parameter from FUNCTION_TEST_RETURN(). This parameter was always useless but commit 7333b630 removed all references to it so remove the parameter at all call sites as well. The original intention was probably to allow logging of TEST return values but that never happened.
2019-01-28 15:06:28 +02:00
FUNCTION_TEST_RETURN(!cfgOptionTest(cfgOptRepoHost));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
/***********************************************************************************************************************************
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Get the command line required for local protocol execution
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
***********************************************************************************************************************************/
static StringList *
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolLocalParam(ProtocolStorageType protocolStorageType, unsigned int protocolId)
{
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
FUNCTION_LOG_PARAM(UINT, protocolId);
FUNCTION_LOG_END();
ASSERT(protocolStorageType == protocolStorageTypeRepo); // ??? Hard-coded until the function supports pg remotes
StringList *result = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
// Option replacements
KeyValue *optionReplace = kvNew();
// Add the command option
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptCommand))), varNewStr(strNew(cfgCommandName(cfgCommand()))));
// Add the process id -- used when more than one process will be called
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptProcess))), varNewInt((int)protocolId));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Add the host id -- for now this is hard-coded to 1
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptHostId))), varNewInt(1));
// Add the type
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptType))), varNewStr(strNew("backup")));
result = strLstMove(cfgExecParam(cfgCmdLocal, optionReplace), MEM_CONTEXT_OLD());
}
MEM_CONTEXT_TEMP_END();
FUNCTION_LOG_RETURN(STRING_LIST, result);
}
/***********************************************************************************************************************************
Get the local protocol client
***********************************************************************************************************************************/
ProtocolClient *
protocolLocalGet(ProtocolStorageType protocolStorageType, unsigned int protocolId)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
FUNCTION_LOG_PARAM(UINT, protocolId);
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperInit();
// Allocate the client cache
if (protocolHelper.clientLocalSize == 0)
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
protocolHelper.clientLocalSize = (unsigned int)cfgOptionInt(cfgOptProcessMax);
protocolHelper.clientLocal = (ProtocolHelperClient *)memNew(
protocolHelper.clientLocalSize * sizeof(ProtocolHelperClient));
}
MEM_CONTEXT_END();
}
ASSERT(protocolId <= protocolHelper.clientLocalSize);
// Create protocol object
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientLocal[protocolId - 1];
if (protocolHelperClient->client == NULL)
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
// Execute the protocol command
protocolHelperClient->exec = execNew(
cfgExe(), protocolLocalParam(protocolStorageType, protocolId),
strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u process", protocolId),
(TimeMSec)(cfgOptionDbl(cfgOptProtocolTimeout) * 1000));
execOpen(protocolHelperClient->exec);
// Create protocol object
protocolHelperClient->client = protocolClientNew(
strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u protocol", protocolId),
PROTOCOL_SERVICE_LOCAL_STR, execIoRead(protocolHelperClient->exec), execIoWrite(protocolHelperClient->exec));
protocolClientMove(protocolHelperClient->client, execMemContext(protocolHelperClient->exec));
}
MEM_CONTEXT_END();
}
FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client);
}
/***********************************************************************************************************************************
Get the command line required for remote protocol execution
***********************************************************************************************************************************/
static StringList *
protocolRemoteParam(ProtocolStorageType protocolStorageType, unsigned int protocolId)
{
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
FUNCTION_LOG_PARAM(UINT, protocolId);
FUNCTION_LOG_END();
ASSERT(protocolStorageType == protocolStorageTypeRepo); // ??? Hard-coded until the function supports pg remotes
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Fixed parameters for ssh command
StringList *result = strLstNew();
strLstAddZ(result, "-o");
strLstAddZ(result, "LogLevel=error");
strLstAddZ(result, "-o");
strLstAddZ(result, "Compression=no");
strLstAddZ(result, "-o");
strLstAddZ(result, "PasswordAuthentication=no");
// Append port if specified
if (cfgOptionTest(cfgOptRepoHostPort))
{
strLstAddZ(result, "-p");
strLstAdd(result, strNewFmt("%d", cfgOptionInt(cfgOptRepoHostPort)));
}
// Append user/host
strLstAdd(result, strNewFmt("%s@%s", strPtr(cfgOptionStr(cfgOptRepoHostUser)), strPtr(cfgOptionStr(cfgOptRepoHost))));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Option replacements
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
KeyValue *optionReplace = kvNew();
// Replace config options with the host versions
if (cfgOptionSource(cfgOptRepoHostConfig) != cfgSourceDefault)
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptConfig))), cfgOption(cfgOptRepoHostConfig));
if (cfgOptionSource(cfgOptRepoHostConfigIncludePath) != cfgSourceDefault)
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptConfigIncludePath))), cfgOption(cfgOptRepoHostConfigIncludePath));
if (cfgOptionSource(cfgOptRepoHostConfigPath) != cfgSourceDefault)
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptConfigPath))), cfgOption(cfgOptRepoHostConfigPath));
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
// Add the command option (or use the current command option if it is valid)
if (!cfgOptionTest(cfgOptCommand))
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptCommand))), varNewStr(strNew(cfgCommandName(cfgCommand()))));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
// Add the process id (or use the current process id if it is valid)
if (!cfgOptionTest(cfgOptProcess))
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptProcess))), varNewInt((int)protocolId));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Resolve storage path expressions before passing to remote. Expressions such as <REPO:ARCHIVE> require a stanza name in order to be resolved correctly. However, if the stanza name is passed to the remote then that remote will only work correctly for that one stanza. Instead, resolved the expressions locally but still pass a relative path to the remote. That way, a storage path that is only configured on the remote does not need to be known locally.
2019-02-21 15:40:21 +02:00
// Don't pass the stanza if it is set. It is better if the remote is stanza-agnostic so the client can operate on multiple
// stanzas without starting a new remote. Once the Perl code is removed the stanza option can be removed from the remote
// command.
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptStanza))), NULL);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Add the type
kvPut(optionReplace, varNewStr(strNew(cfgOptionName(cfgOptType))), varNewStr(strNew("backup")));
StringList *commandExec = cfgExecParam(cfgCmdRemote, optionReplace);
strLstInsert(commandExec, 0, cfgOptionStr(cfgOptRepoHostCmd));
strLstAdd(result, strLstJoin(commandExec, " "));
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN(STRING_LIST, result);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
/***********************************************************************************************************************************
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Get the remote protocol client
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
***********************************************************************************************************************************/
ProtocolClient *
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
protocolRemoteGet(ProtocolStorageType protocolStorageType)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperInit();
// Allocate the client cache
if (protocolHelper.clientRemoteSize == 0)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// The number of remotes allowed is the greater of allowed repo or db configs + 1 (0 is reserved for connections from
// the main process). Since these are static and only one will be true it presents a problem for coverage. We think
// that pg remotes will always be greater but we'll protect that assumption with an assertion.
ASSERT(cfgDefOptionIndexTotal(cfgDefOptPgPath) >= cfgDefOptionIndexTotal(cfgDefOptRepoPath));
protocolHelper.clientRemoteSize = cfgDefOptionIndexTotal(cfgDefOptPgPath) +1;
protocolHelper.clientRemote = (ProtocolHelperClient *)memNew(
protocolHelper.clientRemoteSize * sizeof(ProtocolHelperClient));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
MEM_CONTEXT_END();
}
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
// Determine protocol id for the remote. If the process option is set then use that since we want to remote protocol id to
// match the local protocol id. Otherwise set to 0 since the remote is being started from a main process.
unsigned int protocolId = 0;
if (cfgOptionTest(cfgOptProcess))
protocolId = (unsigned int)cfgOptionInt(cfgOptProcess);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
ASSERT(protocolId < protocolHelper.clientRemoteSize);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Create protocol object
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientRemote[protocolId];
if (protocolHelperClient->client == NULL)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
// Execute the protocol command
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperClient->exec = execNew(
cfgOptionStr(cfgOptCmdSsh), protocolRemoteParam(protocolStorageType, protocolId),
strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u process on '%s'", protocolId, strPtr(cfgOptionStr(cfgOptRepoHost))),
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
(TimeMSec)(cfgOptionDbl(cfgOptProtocolTimeout) * 1000));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
execOpen(protocolHelperClient->exec);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Create protocol object
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperClient->client = protocolClientNew(
strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u protocol on '%s'", protocolId, strPtr(cfgOptionStr(cfgOptRepoHost))),
PROTOCOL_SERVICE_REMOTE_STR, execIoRead(protocolHelperClient->exec), execIoWrite(protocolHelperClient->exec));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Get cipher options from the remote if none are locally configured
if (strEq(cfgOptionStr(cfgOptRepoCipherType), CIPHER_TYPE_NONE_STR))
{
// Options to query
VariantList *param = varLstNew();
varLstAdd(param, varNewStr(strNew(cfgOptionName(cfgOptRepoCipherType))));
varLstAdd(param, varNewStr(strNew(cfgOptionName(cfgOptRepoCipherPass))));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
VariantList *optionList = configProtocolOption(protocolHelperClient->client, param);
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
if (!strEq(varStr(varLstGet(optionList, 0)), CIPHER_TYPE_NONE_STR))
{
cfgOptionSet(cfgOptRepoCipherType, cfgSourceConfig, varLstGet(optionList, 0));
cfgOptionSet(cfgOptRepoCipherPass, cfgSourceConfig, varLstGet(optionList, 1));
}
}
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolClientMove(protocolHelperClient->client, execMemContext(protocolHelperClient->exec));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
MEM_CONTEXT_END();
}
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
/***********************************************************************************************************************************
Free the protocol objects and shutdown processes
***********************************************************************************************************************************/
void
protocolFree(void)
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_VOID(logLevelTrace);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelper.memContext != NULL)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Free remotes
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientRemoteSize; clientIdx++)
{
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientRemote[clientIdx];
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelperClient->client != NULL)
{
protocolClientFree(protocolHelperClient->client);
execFree(protocolHelperClient->exec);
protocolHelperClient->client = NULL;
protocolHelperClient->exec = NULL;
}
}
// Free locals
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientLocalSize; clientIdx++)
{
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientLocal[clientIdx];
if (protocolHelperClient->client != NULL)
{
protocolClientFree(protocolHelperClient->client);
execFree(protocolHelperClient->exec);
protocolHelperClient->client = NULL;
protocolHelperClient->exec = NULL;
}
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN_VOID();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Reference in New Issue Copy Permalink