db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-04-27 12:12:18 +02:00
Code Issues Releases Activity
pgbackrest/src/protocol/helper.c

459 lines
19 KiB
C
Raw Normal View History

Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Protocol Helper
***********************************************************************************************************************************/
Add configure script for improved multi-platform support. Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile. Each C file must include "build.auto.conf" before all other includes and defines. This is enforced by test.pl for includes, but it won't detect incorrect define ordering. Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
#include "build.auto.h"
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
#include <string.h>
Move crypto module to common/crypto. It makes sense for the crypto code to be in common since it is not pgBackRest-specific. Also combine the crypto tests into a single module.
2019-03-10 13:27:30 +02:00
#include "common/crypto/common.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "common/debug.h"
#include "common/exec.h"
#include "common/memContext.h"
#include "config/config.h"
#include "config/exec.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "config/protocol.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "protocol/helper.h"
/***********************************************************************************************************************************
Constants
***********************************************************************************************************************************/
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
STRING_EXTERN(PROTOCOL_SERVICE_LOCAL_STR, PROTOCOL_SERVICE_LOCAL);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
STRING_EXTERN(PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE);
/***********************************************************************************************************************************
Local variables
***********************************************************************************************************************************/
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
typedef struct ProtocolHelperClient
{
Exec *exec; // Executed client
ProtocolClient *client; // Protocol client
} ProtocolHelperClient;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
static struct
{
MemContext *memContext; // Mem context for protocol helper
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
unsigned int clientRemoteSize; // Remote clients
ProtocolHelperClient *clientRemote;
unsigned int clientLocalSize; // Local clients
ProtocolHelperClient *clientLocal;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
} protocolHelper;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
/***********************************************************************************************************************************
Init local mem context and data structure
***********************************************************************************************************************************/
static void
protocolHelperInit(void)
{
// In the protocol helper has not been initialized
if (protocolHelper.memContext == NULL)
{
// Create a mem context to store protocol objects
MEM_CONTEXT_BEGIN(memContextTop())
{
protocolHelper.memContext = memContextNew("ProtocolHelper");
}
MEM_CONTEXT_END();
}
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Is the repository local?
***********************************************************************************************************************************/
bool
repoIsLocal(void)
{
FUNCTION_TEST_VOID();
Remove unused type parameter from FUNCTION_TEST_RETURN(). This parameter was always useless but commit 7333b630 removed all references to it so remove the parameter at all call sites as well. The original intention was probably to allow logging of TEST return values but that never happened.
2019-01-28 15:06:28 +02:00
FUNCTION_TEST_RETURN(!cfgOptionTest(cfgOptRepoHost));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
/***********************************************************************************************************************************
Is pg local?
***********************************************************************************************************************************/
bool
pgIsLocal(void)
{
FUNCTION_TEST_VOID();
FUNCTION_TEST_RETURN(!cfgOptionTest(cfgOptPgHost + (cfgOptionTest(cfgOptHostId) ? cfgOptionUInt(cfgOptHostId) - 1 : 0)));
}
/***********************************************************************************************************************************
Get host id if host-id option is set, otherwise return default id 1
***********************************************************************************************************************************/
unsigned int
protocolHostId(void)
{
FUNCTION_TEST_VOID();
FUNCTION_TEST_RETURN(cfgOptionTest(cfgOptHostId) ? cfgOptionUInt(cfgOptHostId) : 1);
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Get the command line required for local protocol execution
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
***********************************************************************************************************************************/
static StringList *
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolLocalParam(ProtocolStorageType protocolStorageType, unsigned int protocolId)
{
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
FUNCTION_LOG_PARAM(UINT, protocolId);
FUNCTION_LOG_END();
StringList *result = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
// Option replacements
KeyValue *optionReplace = kvNew();
// Add the command option
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_COMMAND_STR), VARSTRZ(cfgCommandName(cfgCommand())));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Add the process id -- used when more than one process will be called
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_PROCESS_STR), VARINT((int)protocolId));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Add the host id -- for now this is hard-coded to 1
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_HOST_ID_STR), VARINT(1));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Add the type
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_TYPE_STR), VARSTRDEF("backup"));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Only enable file logging on the local when requested
kvPut(
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
optionReplace, VARSTR(CFGOPT_LOG_LEVEL_FILE_STR),
cfgOptionBool(cfgOptLogSubprocess) ? cfgOption(cfgOptLogLevelFile) : VARSTRDEF("off"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Always output errors on stderr for debugging purposes
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_LOG_LEVEL_STDERR_STR), VARSTRDEF("error"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
result = strLstMove(cfgExecParam(cfgCmdLocal, optionReplace), MEM_CONTEXT_OLD());
}
MEM_CONTEXT_TEMP_END();
FUNCTION_LOG_RETURN(STRING_LIST, result);
}
/***********************************************************************************************************************************
Get the local protocol client
***********************************************************************************************************************************/
ProtocolClient *
protocolLocalGet(ProtocolStorageType protocolStorageType, unsigned int protocolId)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
FUNCTION_LOG_PARAM(UINT, protocolId);
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperInit();
// Allocate the client cache
if (protocolHelper.clientLocalSize == 0)
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
Update code to use new unsigned int Variant type and config methods.
2019-04-19 11:40:39 -04:00
protocolHelper.clientLocalSize = cfgOptionUInt(cfgOptProcessMax);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelper.clientLocal = (ProtocolHelperClient *)memNew(
protocolHelper.clientLocalSize * sizeof(ProtocolHelperClient));
}
MEM_CONTEXT_END();
}
ASSERT(protocolId <= protocolHelper.clientLocalSize);
// Create protocol object
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientLocal[protocolId - 1];
if (protocolHelperClient->client == NULL)
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
// Execute the protocol command
protocolHelperClient->exec = execNew(
cfgExe(), protocolLocalParam(protocolStorageType, protocolId),
strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u process", protocolId),
(TimeMSec)(cfgOptionDbl(cfgOptProtocolTimeout) * 1000));
execOpen(protocolHelperClient->exec);
// Create protocol object
protocolHelperClient->client = protocolClientNew(
strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u protocol", protocolId),
PROTOCOL_SERVICE_LOCAL_STR, execIoRead(protocolHelperClient->exec), execIoWrite(protocolHelperClient->exec));
protocolClientMove(protocolHelperClient->client, execMemContext(protocolHelperClient->exec));
}
MEM_CONTEXT_END();
}
FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client);
}
/***********************************************************************************************************************************
Get the command line required for remote protocol execution
***********************************************************************************************************************************/
static StringList *
protocolRemoteParam(ProtocolStorageType protocolStorageType, unsigned int protocolId)
{
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
FUNCTION_LOG_PARAM(UINT, protocolId);
FUNCTION_LOG_END();
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// Is this a repo remote?
bool isRepo = protocolStorageType == protocolStorageTypeRepo;
// Get the host index. Default to 0 if host-id is not set.
unsigned int hostIdx = isRepo ? 0 : protocolHostId() - 1;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Fixed parameters for ssh command
StringList *result = strLstNew();
strLstAddZ(result, "-o");
strLstAddZ(result, "LogLevel=error");
strLstAddZ(result, "-o");
strLstAddZ(result, "Compression=no");
strLstAddZ(result, "-o");
strLstAddZ(result, "PasswordAuthentication=no");
// Append port if specified
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
ConfigOption optHostPort = isRepo ? cfgOptRepoHostPort : cfgOptPgHostPort + hostIdx;
if (cfgOptionTest(optHostPort))
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
strLstAddZ(result, "-p");
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strLstAdd(result, strNewFmt("%u", cfgOptionUInt(optHostPort)));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
// Append user/host
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strLstAdd(
result,
strNewFmt(
"%s@%s", strPtr(cfgOptionStr(isRepo ? cfgOptRepoHostUser : cfgOptPgHostUser + hostIdx)),
strPtr(cfgOptionStr(isRepo ? cfgOptRepoHost : cfgOptPgHost + hostIdx))));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Option replacements
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
KeyValue *optionReplace = kvNew();
// Replace config options with the host versions
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
unsigned int optConfig = isRepo ? cfgOptRepoHostConfig : cfgOptPgHostConfig + hostIdx;
if (cfgOptionSource(optConfig) != cfgSourceDefault)
kvPut(optionReplace, VARSTR(CFGOPT_CONFIG_STR), cfgOption(optConfig));
unsigned int optConfigIncludePath = isRepo ? cfgOptRepoHostConfigIncludePath : cfgOptPgHostConfigIncludePath + hostIdx;
if (cfgOptionSource(optConfigIncludePath) != cfgSourceDefault)
kvPut(optionReplace, VARSTR(CFGOPT_CONFIG_INCLUDE_PATH_STR), cfgOption(optConfigIncludePath));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
unsigned int optConfigPath = isRepo ? cfgOptRepoHostConfigPath : cfgOptPgHostConfigPath + hostIdx;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
if (cfgOptionSource(optConfigPath) != cfgSourceDefault)
kvPut(optionReplace, VARSTR(CFGOPT_CONFIG_PATH_STR), cfgOption(optConfigPath));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Add --c option to request a C remote. The rules for when a C remote is required are getting complicated and will get worse when restoreFile() is migrated. Instead, set the --c option when a C remote is required. This option will be removed when the remote is entirely implemented in C.
2019-05-09 08:55:48 -04:00
// Use a C remote
kvPut(optionReplace, VARSTR(CFGOPT_C_STR), VARBOOL(true));
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// Copy pg options to index 0 since that's what the remote will be expecting
if (hostIdx != 0)
{
kvPut(optionReplace, VARSTR(CFGOPT_PG1_PATH_STR), cfgOption(cfgOptPgPath + hostIdx));
if (cfgOptionSource(cfgOptPgSocketPath + hostIdx) != cfgSourceDefault)
kvPut(optionReplace, VARSTR(CFGOPT_PG1_SOCKET_PATH_STR), cfgOption(cfgOptPgSocketPath + hostIdx));
if (cfgOptionSource(cfgOptPgPort + hostIdx) != cfgSourceDefault)
kvPut(optionReplace, VARSTR(CFGOPT_PG1_PORT_STR), cfgOption(cfgOptPgPort + hostIdx));
}
// Remove pg options that are not needed on the remote. This is to reduce clustter and make debugging options easier.
for (unsigned int pgIdx = 1; pgIdx < cfgOptionIndexTotal(cfgOptPgPath); pgIdx++)
{
kvPut(optionReplace, VARSTRZ(cfgOptionName(cfgOptPgPath + pgIdx)), NULL);
kvPut(optionReplace, VARSTRZ(cfgOptionName(cfgOptPgSocketPath + pgIdx)), NULL);
kvPut(optionReplace, VARSTRZ(cfgOptionName(cfgOptPgPort + pgIdx)), NULL);
}
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
// Add the command option (or use the current command option if it is valid)
if (!cfgOptionTest(cfgOptCommand))
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_COMMAND_STR), VARSTRZ(cfgCommandName(cfgCommand())));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
// Add the process id (or use the current process id if it is valid)
if (!cfgOptionTest(cfgOptProcess))
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_PROCESS_STR), VARINT((int)protocolId));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Don't pass log-path or lock-path since these are host specific
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_LOG_PATH_STR), NULL);
kvPut(optionReplace, VARSTR(CFGOPT_LOCK_PATH_STR), NULL);
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Only enable file logging on the remote when requested
kvPut(
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
optionReplace, VARSTR(CFGOPT_LOG_LEVEL_FILE_STR),
cfgOptionBool(cfgOptLogSubprocess) ? cfgOption(cfgOptLogLevelFile) : VARSTRDEF("off"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Always output errors on stderr for debugging purposes
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_LOG_LEVEL_STDERR_STR), VARSTRDEF("error"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Add the type
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
kvPut(optionReplace, VARSTR(CFGOPT_TYPE_STR), isRepo ? VARSTRDEF("backup") : VARSTRDEF("db"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
StringList *commandExec = cfgExecParam(cfgCmdRemote, optionReplace);
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strLstInsert(commandExec, 0, cfgOptionStr(isRepo ? cfgOptRepoHostCmd : cfgOptPgHostCmd + hostIdx));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
strLstAdd(result, strLstJoin(commandExec, " "));
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN(STRING_LIST, result);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
/***********************************************************************************************************************************
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Get the remote protocol client
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
***********************************************************************************************************************************/
ProtocolClient *
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
protocolRemoteGet(ProtocolStorageType protocolStorageType)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_PARAM(ENUM, protocolStorageType);
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// Is this a repo remote?
bool isRepo = protocolStorageType == protocolStorageTypeRepo;
// Get the host index. Default to 0 if host-id is not set.
unsigned int hostIdx = isRepo ? 0 : protocolHostId() - 1;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperInit();
// Allocate the client cache
if (protocolHelper.clientRemoteSize == 0)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// The number of remotes allowed is the greater of allowed repo or pg configs + 1 (0 is reserved for connections from
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// the main process). Since these are static and only one will be true it presents a problem for coverage. We think
// that pg remotes will always be greater but we'll protect that assumption with an assertion.
ASSERT(cfgDefOptionIndexTotal(cfgDefOptPgPath) >= cfgDefOptionIndexTotal(cfgDefOptRepoPath));
Add missing space.
2019-04-29 14:23:37 -04:00
protocolHelper.clientRemoteSize = cfgDefOptionIndexTotal(cfgDefOptPgPath) + 1;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelper.clientRemote = (ProtocolHelperClient *)memNew(
protocolHelper.clientRemoteSize * sizeof(ProtocolHelperClient));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
MEM_CONTEXT_END();
}
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
// Determine protocol id for the remote. If the process option is set then use that since we want to remote protocol id to
Fix segfault when process-max > 8 for archive-push/archive-get. The remote list was at most 9 (based on pg[1-8]-* max index) so anything over 8 wrote into unallocated memory. The remote for the main process is (currently) stored in position zero so do the same for remotes started from locals, since there should only be one. The main process will need to start more remotes in the future which is why there is extra space. Reported by Jens Wilke.
2019-04-29 16:10:27 -04:00
// match the local protocol id (but we'll still save it in position 0 or we'd need to allocated up to process-max slots).
// Otherwise set to 0 since the remote is being started from a main process.
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
unsigned int protocolId = 0;
Fix segfault when process-max > 8 for archive-push/archive-get. The remote list was at most 9 (based on pg[1-8]-* max index) so anything over 8 wrote into unallocated memory. The remote for the main process is (currently) stored in position zero so do the same for remotes started from locals, since there should only be one. The main process will need to start more remotes in the future which is why there is extra space. Reported by Jens Wilke.
2019-04-29 16:10:27 -04:00
unsigned int protocolIdx = 0;
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
if (cfgOptionTest(cfgOptProcess))
Update code to use new unsigned int Variant type and config methods.
2019-04-19 11:40:39 -04:00
protocolId = cfgOptionUInt(cfgOptProcess);
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
Fix segfault when process-max > 8 for archive-push/archive-get. The remote list was at most 9 (based on pg[1-8]-* max index) so anything over 8 wrote into unallocated memory. The remote for the main process is (currently) stored in position zero so do the same for remotes started from locals, since there should only be one. The main process will need to start more remotes in the future which is why there is extra space. Reported by Jens Wilke.
2019-04-29 16:10:27 -04:00
CHECK(protocolIdx < protocolHelper.clientRemoteSize);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Create protocol object
Fix segfault when process-max > 8 for archive-push/archive-get. The remote list was at most 9 (based on pg[1-8]-* max index) so anything over 8 wrote into unallocated memory. The remote for the main process is (currently) stored in position zero so do the same for remotes started from locals, since there should only be one. The main process will need to start more remotes in the future which is why there is extra space. Reported by Jens Wilke.
2019-04-29 16:10:27 -04:00
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientRemote[protocolIdx];
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelperClient->client == NULL)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
unsigned int optHost = isRepo ? cfgOptRepoHost : cfgOptPgHost + hostIdx;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Execute the protocol command
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperClient->exec = execNew(
cfgOptionStr(cfgOptCmdSsh), protocolRemoteParam(protocolStorageType, protocolId),
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u process on '%s'", protocolId, strPtr(cfgOptionStr(optHost))),
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
(TimeMSec)(cfgOptionDbl(cfgOptProtocolTimeout) * 1000));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
execOpen(protocolHelperClient->exec);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Create protocol object
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperClient->client = protocolClientNew(
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u protocol on '%s'", protocolId, strPtr(cfgOptionStr(optHost))),
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
PROTOCOL_SERVICE_REMOTE_STR, execIoRead(protocolHelperClient->exec), execIoWrite(protocolHelperClient->exec));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Get cipher options from the remote if none are locally configured
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
if (isRepo && strEq(cfgOptionStr(cfgOptRepoCipherType), CIPHER_TYPE_NONE_STR))
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
{
// Options to query
VariantList *param = varLstNew();
Automatically generate constants for command and option names. These constants are easier than using cfgOptionName() and cfgCommandName() and lead to cleaner code and simpler to construct messages. String versions are provided. Eventually all the strings will be used in the config structures, but for now they are useful to avoid wrapping with strNew().
2019-04-12 09:03:34 -04:00
varLstAdd(param, varNewStr(CFGOPT_REPO1_CIPHER_TYPE_STR));
varLstAdd(param, varNewStr(CFGOPT_REPO1_CIPHER_PASS_STR));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
VariantList *optionList = configProtocolOption(protocolHelperClient->client, param);
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
if (!strEq(varStr(varLstGet(optionList, 0)), CIPHER_TYPE_NONE_STR))
{
cfgOptionSet(cfgOptRepoCipherType, cfgSourceConfig, varLstGet(optionList, 0));
cfgOptionSet(cfgOptRepoCipherPass, cfgSourceConfig, varLstGet(optionList, 1));
}
}
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolClientMove(protocolHelperClient->client, execMemContext(protocolHelperClient->exec));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
MEM_CONTEXT_END();
}
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Add protocolKeepAlive() to send noops to all remotes. While the local processes are doing their jobs the remote connection from the main process may timeout. Send occasional noops to ensure that doesn't happen.
2019-03-27 20:59:28 +00:00
/***********************************************************************************************************************************
Send keepalives to all remotes
***********************************************************************************************************************************/
void
protocolKeepAlive(void)
{
FUNCTION_LOG_VOID(logLevelTrace);
if (protocolHelper.memContext != NULL)
{
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientRemoteSize; clientIdx++)
{
if (protocolHelper.clientRemote[clientIdx].client != NULL)
protocolClientNoOp(protocolHelper.clientRemote[clientIdx].client);
}
}
FUNCTION_LOG_RETURN_VOID();
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Free the protocol objects and shutdown processes
***********************************************************************************************************************************/
void
protocolFree(void)
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_VOID(logLevelTrace);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelper.memContext != NULL)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Free remotes
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientRemoteSize; clientIdx++)
{
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientRemote[clientIdx];
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelperClient->client != NULL)
{
protocolClientFree(protocolHelperClient->client);
execFree(protocolHelperClient->exec);
protocolHelperClient->client = NULL;
protocolHelperClient->exec = NULL;
}
}
// Free locals
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientLocalSize; clientIdx++)
{
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientLocal[clientIdx];
if (protocolHelperClient->client != NULL)
{
protocolClientFree(protocolHelperClient->client);
execFree(protocolHelperClient->exec);
protocolHelperClient->client = NULL;
protocolHelperClient->exec = NULL;
}
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN_VOID();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Reference in New Issue Copy Permalink