db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-04-25 12:04:48 +02:00
Code Issues Releases Activity
7f6c513be9
pgbackrest/src/protocol/helper.c

844 lines
32 KiB
C
Raw Normal View History

Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Protocol Helper
***********************************************************************************************************************************/
Add configure script for improved multi-platform support. Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile. Each C file must include "build.auto.conf" before all other includes and defines. This is enforced by test.pl for includes, but it won't detect incorrect define ordering. Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
#include "build.auto.h"
Add macros to create constant Variant types. These work almost exactly like the String constant macros. However, a struct per variant type was required which meant custom constructors and destructors for each type. Propagate the variant constants out into the codebase wherever they are useful.
2019-04-17 08:04:22 -04:00
#include <string.h>
Move crypto module to common/crypto. It makes sense for the crypto code to be in common since it is not pgBackRest-specific. Also combine the crypto tests into a single module.
2019-03-10 13:27:30 +02:00
#include "common/crypto/common.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "common/debug.h"
#include "common/exec.h"
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
#include "common/io/client.h"
#include "common/io/socket/client.h"
#include "common/io/socket/server.h"
#include "common/io/tls/client.h"
#include "common/io/tls/server.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "common/memContext.h"
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
#include "config/config.intern.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "config/exec.h"
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
#include "config/load.h"
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
#include "config/parse.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "config/protocol.h"
Add pgIsLocalVerify(). This functionality is required in commands other than restore, so centralize it.
2020-02-12 15:47:07 -07:00
#include "postgres/version.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "protocol/helper.h"
Add pgIsLocalVerify(). This functionality is required in commands other than restore, so centralize it.
2020-02-12 15:47:07 -07:00
#include "version.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Constants
***********************************************************************************************************************************/
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
STRING_EXTERN(PROTOCOL_SERVICE_LOCAL_STR, PROTOCOL_SERVICE_LOCAL);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
STRING_EXTERN(PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE);
/***********************************************************************************************************************************
Local variables
***********************************************************************************************************************************/
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
typedef struct ProtocolHelperClient
{
Exec *exec; // Executed client
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
IoClient *ioClient; // Io client, e.g. TlsClient
IoSession *ioSession; // Io session, e.g. TlsSession
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
ProtocolClient *client; // Protocol client
} ProtocolHelperClient;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
static struct
{
MemContext *memContext; // Mem context for protocol helper
Fix comment indents.
2019-08-21 14:14:30 -04:00
unsigned int clientRemoteSize; // Remote clients
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
ProtocolHelperClient *clientRemote;
Fix comment indents.
2019-08-21 14:14:30 -04:00
unsigned int clientLocalSize; // Local clients
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
ProtocolHelperClient *clientLocal;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
} protocolHelper;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
/***********************************************************************************************************************************
Init local mem context and data structure
***********************************************************************************************************************************/
static void
protocolHelperInit(void)
{
// In the protocol helper has not been initialized
if (protocolHelper.memContext == NULL)
{
// Create a mem context to store protocol objects
MEM_CONTEXT_BEGIN(memContextTop())
{
Use MEM_CONTEXT_NEW_BEGIN() block instead of memContextNew(). A few places were using just memContextNew(), probably because they did not immediately need to create anything in the new context, but it's better if we use the same pattern everywhere, even if it results in a few extra mem context switches.
2020-01-17 11:58:41 -07:00
MEM_CONTEXT_NEW_BEGIN("ProtocolHelper")
{
protocolHelper.memContext = MEM_CONTEXT_NEW();
}
MEM_CONTEXT_NEW_END();
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
}
MEM_CONTEXT_END();
}
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
bool
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
repoIsLocal(unsigned int repoIdx)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(UINT, repoIdx);
FUNCTION_LOG_END();
FUNCTION_LOG_RETURN(BOOL, !cfgOptionIdxTest(cfgOptRepoHost, repoIdx));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add repoIsLocalVerify() to verify repository locality. Some commands can only be run on a host where the repository is local. This function centralizes the check and error. Contributed by Cynthia Shang.
2019-08-21 11:41:36 -04:00
void
repoIsLocalVerify(void)
{
FUNCTION_TEST_VOID();
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
repoIsLocalVerifyIdx(cfgOptionGroupIdxDefault(cfgOptGrpRepo));
FUNCTION_TEST_RETURN_VOID();
}
/**********************************************************************************************************************************/
void
repoIsLocalVerifyIdx(unsigned int repoIdx)
{
FUNCTION_TEST_VOID();
if (!repoIsLocal(repoIdx))
Remove generated config.auto.c file. This file duplicated the command list that already exists in parse.auto.c. Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
2021-07-29 14:40:30 -04:00
THROW_FMT(HostInvalidError, "%s command must be run on the repository host", cfgCommandName());
Add repoIsLocalVerify() to verify repository locality. Some commands can only be run on a host where the repository is local. This function centralizes the check and error. Contributed by Cynthia Shang.
2019-08-21 11:41:36 -04:00
FUNCTION_TEST_RETURN_VOID();
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
bool
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
pgIsLocal(unsigned int pgIdx)
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
{
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
FUNCTION_LOG_BEGIN(logLevelDebug);
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
FUNCTION_LOG_PARAM(UINT, pgIdx);
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
FUNCTION_LOG_END();
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
FUNCTION_LOG_RETURN(BOOL, !cfgOptionIdxTest(cfgOptPgHost, pgIdx));
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
}
Add pgIsLocalVerify(). This functionality is required in commands other than restore, so centralize it.
2020-02-12 15:47:07 -07:00
/**********************************************************************************************************************************/
void
pgIsLocalVerify(void)
{
FUNCTION_TEST_VOID();
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
if (!pgIsLocal(cfgOptionGroupIdxDefault(cfgOptGrpPg)))
Remove generated config.auto.c file. This file duplicated the command list that already exists in parse.auto.c. Combine the data from config.auto.c into parse.auto.c and adjust the interface functions as needed. Quite a few were able to be moved to parse.c as static.
2021-07-29 14:40:30 -04:00
THROW_FMT(HostInvalidError, "%s command must be run on the " PG_NAME " host", cfgCommandName());
Add pgIsLocalVerify(). This functionality is required in commands other than restore, so centralize it.
2020-02-12 15:47:07 -07:00
FUNCTION_TEST_RETURN_VOID();
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Get the command line required for local protocol execution
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
***********************************************************************************************************************************/
static StringList *
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolLocalParam(ProtocolStorageType protocolStorageType, unsigned int hostIdx, unsigned int processId)
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
{
FUNCTION_LOG_BEGIN(logLevelDebug);
Convert ProtocolStorageType enum to StringId. Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType);
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
FUNCTION_LOG_PARAM(UINT, hostIdx);
FUNCTION_LOG_PARAM(UINT, processId);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_END();
StringList *result = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
// Option replacements
KeyValue *optionReplace = kvNew();
// Add the process id -- used when more than one process will be called
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS), VARUINT(processId));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// Add the pg default. Don't do this for repos because the repo default should come from the user or the local should
// handle all the repos equally. Repos don't get special handling like pg primaries or standbys.
Remove repo option. This option was added in advance of the multi-repo functionality but it has no purpose and it is not clear what the validity rules should be. The option will be added back when multi-repo functionality is committed.
2020-12-31 08:12:35 -05:00
if (protocolStorageType == protocolStorageTypePg)
kvPut(optionReplace, VARSTRDEF(CFGOPT_PG), VARUINT(cfgOptionGroupIdxToKey(cfgOptGrpPg, hostIdx)));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Update inaccurate comments.
2020-01-09 11:56:03 -07:00
// Add the remote type
Convert ProtocolStorageType enum to StringId. Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_REMOTE_TYPE), VARSTR(strIdToStr(protocolStorageType)));
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Only enable file logging on the local when requested
kvPut(
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE),
Add StringId as an option type. Rather the converting String to StringIds at runtime, store defaults in StringId format in parse.auto.c and convert user input to StringId during parsing.
2021-11-03 07:27:26 -04:00
cfgOptionBool(cfgOptLogSubprocess) ? VARUINT64(cfgOptionStrId(cfgOptLogLevelFile)) : VARSTRDEF("off"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Always output errors on stderr for debugging purposes
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARSTRDEF("error"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
// Disable output to stdout since it is used by the protocol
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARSTRDEF("off"));
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
Add MEM_CONTEXT_PRIOR() block and update current call sites. This macro block encapsulates the common pattern of switching to the prior (formerly called old) mem context to return results from a function. Also rename MEM_CONTEXT_OLD() to memContextPrior(). This violates our convention of macros being in all caps but memContextPrior() will become a function very soon so this will reduce churn.
2020-01-17 13:29:49 -07:00
result = strLstMove(cfgExecParam(cfgCommand(), cfgCmdRoleLocal, optionReplace, true, false), memContextPrior());
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
}
MEM_CONTEXT_TEMP_END();
FUNCTION_LOG_RETURN(STRING_LIST, result);
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Factor local process exec out of protocolLocalGet(). This allows protocolLocalExec() to be shimmed, which means the local can be run as a child of the test process, simplifying coverage testing.
2021-05-20 18:35:30 -04:00
// Helper to execute the local process. This is a separate function solely so that it can be shimmed during testing.
static void
protocolLocalExec(
ProtocolHelperClient *helper, ProtocolStorageType protocolStorageType, unsigned int hostIdx, unsigned int processId)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM_P(VOID, helper);
FUNCTION_TEST_PARAM(ENUM, protocolStorageType);
FUNCTION_TEST_PARAM(UINT, hostIdx);
FUNCTION_TEST_PARAM(UINT, processId);
FUNCTION_TEST_END();
ASSERT(helper != NULL);
// Execute the protocol command
helper->exec = execNew(
cfgExe(), protocolLocalParam(protocolStorageType, hostIdx, processId),
strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u process", processId), cfgOptionUInt64(cfgOptProtocolTimeout));
execOpen(helper->exec);
// Create protocol object
helper->client = protocolClientNew(
strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u protocol", processId),
PROTOCOL_SERVICE_LOCAL_STR, execIoRead(helper->exec), execIoWrite(helper->exec));
Move noop from protocolClientNew() to local/remote initialization. protocolServerNew() does not automatically process a noop so this made the handshake in the constructors asymmetric. This made testing a bit confusing since an extra noop was needed when cmdLocal()/cmdRemote() were not called (since they processed the noop sent by protocolClientNew()). The extra noops also make complex protocol negotiation (coming in a future commit) more complicated and slower because of the additional round trips.
2021-07-23 08:32:30 -04:00
// Move client to exec context so they are freed together
Factor local process exec out of protocolLocalGet(). This allows protocolLocalExec() to be shimmed, which means the local can be run as a child of the test process, simplifying coverage testing.
2021-05-20 18:35:30 -04:00
protocolClientMove(helper->client, execMemContext(helper->exec));
FUNCTION_TEST_RETURN_VOID();
}
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
ProtocolClient *
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolLocalGet(ProtocolStorageType protocolStorageType, unsigned int hostIdx, unsigned int processId)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
Convert ProtocolStorageType enum to StringId. Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType);
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
FUNCTION_LOG_PARAM(UINT, hostIdx);
FUNCTION_LOG_PARAM(UINT, processId);
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperInit();
// Allocate the client cache
if (protocolHelper.clientLocalSize == 0)
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
Allocate one extra process for backup from standby. The connection to the primary is process-max + 1 so make space for it.
2019-12-05 22:34:38 -05:00
protocolHelper.clientLocalSize = cfgOptionUInt(cfgOptProcessMax) + 1;
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
protocolHelper.clientLocal = memNew(protocolHelper.clientLocalSize * sizeof(ProtocolHelperClient));
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientLocalSize; clientIdx++)
protocolHelper.clientLocal[clientIdx] = (ProtocolHelperClient){.exec = NULL};
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
}
MEM_CONTEXT_END();
}
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
ASSERT(processId <= protocolHelper.clientLocalSize);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Create protocol object
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientLocal[processId - 1];
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelperClient->client == NULL)
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
Factor local process exec out of protocolLocalGet(). This allows protocolLocalExec() to be shimmed, which means the local can be run as a child of the test process, simplifying coverage testing.
2021-05-20 18:35:30 -04:00
protocolLocalExec(protocolHelperClient, protocolStorageType, hostIdx, processId);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
}
MEM_CONTEXT_END();
Move noop from protocolClientNew() to local/remote initialization. protocolServerNew() does not automatically process a noop so this made the handshake in the constructors asymmetric. This made testing a bit confusing since an extra noop was needed when cmdLocal()/cmdRemote() were not called (since they processed the noop sent by protocolClientNew()). The extra noops also make complex protocol negotiation (coming in a future commit) more complicated and slower because of the additional round trips.
2021-07-23 08:32:30 -04:00
// Send noop to catch initialization errors
protocolClientNoOp(protocolHelperClient->client);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
}
FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client);
}
Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
/***********************************************************************************************************************************
Free the protocol client and underlying exec'd process. Log any errors as warnings since it is not worth terminating the process
while closing a local/remote that has already completed its work. The warning will be an indication that something is not right.
***********************************************************************************************************************************/
static void
protocolHelperClientFree(ProtocolHelperClient *protocolHelperClient)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM_P(VOID, protocolHelperClient);
FUNCTION_LOG_END();
if (protocolHelperClient->client != NULL)
{
// Try to shutdown the protocol but only warn on error
TRY_BEGIN()
{
protocolClientFree(protocolHelperClient->client);
}
CATCH_ANY()
{
LOG_WARN(errorMessage());
}
TRY_END();
// Try to end the child process but only warn on error
TRY_BEGIN()
{
execFree(protocolHelperClient->exec);
}
CATCH_ANY()
{
LOG_WARN(errorMessage());
}
TRY_END();
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
// Free the io client/session (there should be no errors)
ioSessionFree(protocolHelperClient->ioSession);
ioClientFree(protocolHelperClient->ioClient);
protocolHelperClient->ioSession = NULL;
protocolHelperClient->ioClient = NULL;
Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
protocolHelperClient->client = NULL;
protocolHelperClient->exec = NULL;
}
FUNCTION_LOG_RETURN_VOID();
}
/**********************************************************************************************************************************/
void
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolLocalFree(unsigned int processId)
Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
{
FUNCTION_LOG_BEGIN(logLevelDebug);
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
FUNCTION_LOG_PARAM(UINT, processId);
Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
FUNCTION_LOG_END();
if (protocolHelper.clientLocal != NULL)
{
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
ASSERT(processId <= protocolHelper.clientLocalSize);
protocolHelperClientFree(&protocolHelper.clientLocal[processId - 1]);
Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
}
FUNCTION_LOG_RETURN_VOID();
}
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
/**********************************************************************************************************************************/
// Helper to check if client is authorized for a stanza
static bool
protocolServerAuthorize(const String *authListStr, const String *const stanza)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM(STRING, authListStr);
FUNCTION_LOG_PARAM(STRING, stanza);
FUNCTION_TEST_END();
ASSERT(authListStr != NULL);
// Empty list is not valid. ??? It would be better if this were done during config parsing.
authListStr = strTrim(strDup(authListStr));
if (strEmpty(authListStr))
THROW(OptionInvalidValueError, "'" CFGOPT_TLS_SERVER_AUTH "' option must have a value");
// If * then all stanzas are authorized
if (strEqZ(authListStr, "*"))
FUNCTION_TEST_RETURN(true);
// Check the list of stanzas for a match with the specified stanza. Each entry will need to be trimmed before comparing.
if (stanza != NULL)
{
StringList *authList = strLstNewSplitZ(authListStr, COMMA_Z);
for (unsigned int authListIdx = 0; authListIdx < strLstSize(authList); authListIdx++)
{
if (strEq(strTrim(strLstGet(authList, authListIdx)), stanza))
FUNCTION_TEST_RETURN(true);
}
}
FUNCTION_TEST_RETURN(false);
}
ProtocolServer *
protocolServer(IoServer *const tlsServer, IoSession *const socketSession)
{
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(IO_SERVER, tlsServer);
FUNCTION_LOG_PARAM(IO_SESSION, socketSession);
FUNCTION_LOG_END();
ProtocolServer *result = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
// Start TLS
IoSession *const tlsSession = ioServerAccept(tlsServer, socketSession);
result = protocolServerNew(
PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE_STR, ioSessionIoRead(tlsSession),
ioSessionIoWrite(tlsSession));
// If session is authenticated
if (ioSessionAuthenticated(tlsSession))
{
TRY_BEGIN()
{
// Get list of authorized stanzas for this client
CHECK(cfgOptionTest(cfgOptTlsServerAuth));
const String *const clientAuthList = strDup(
varStr(kvGet(cfgOptionKv(cfgOptTlsServerAuth), VARSTR(ioSessionPeerName(tlsSession)))));
// Error if the client is not authorized for anything
if (clientAuthList == NULL)
THROW(AccessError, "access denied");
// Get parameter list from the client and load it
const ProtocolServerCommandGetResult command = protocolServerCommandGet(result);
CHECK(command.id == PROTOCOL_COMMAND_CONFIG);
StringList *const paramList = pckReadStrLstP(pckReadNew(command.param));
strLstInsert(paramList, 0, cfgExe());
cfgLoad(strLstSize(paramList), strLstPtr(paramList));
// Error if the client is authorized for the requested stanza
if (!protocolServerAuthorize(clientAuthList, cfgOptionStrNull(cfgOptStanza)))
THROW(AccessError, "access denied");
}
CATCH_ANY()
{
protocolServerError(result, errorCode(), STR(errorMessage()), STR(errorStackTrace()));
RETHROW();
}
TRY_END();
// Ack the config command
protocolServerDataEndPut(result);
ioSessionMove(tlsSession, memContextPrior());
protocolServerMove(result, memContextPrior());
}
// Else the client can only detect that the server is alive
else
{
// Send a data end message and return a NULL server. Do not waste time looking at what the client wrote.
protocolServerDataEndPut(result);
// Set result to NULL so there is no server for the caller to use. The TLS session will be freed when the temp mem
// context ends.
result = NULL;
}
}
MEM_CONTEXT_TEMP_END();
FUNCTION_LOG_RETURN(PROTOCOL_SERVER, result);
}
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
/***********************************************************************************************************************************
Get the command line required for remote protocol execution
***********************************************************************************************************************************/
static StringList *
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolRemoteParam(ProtocolStorageType protocolStorageType, unsigned int hostIdx)
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
{
FUNCTION_LOG_BEGIN(logLevelDebug);
Convert ProtocolStorageType enum to StringId. Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType);
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
FUNCTION_LOG_PARAM(UINT, hostIdx);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_END();
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// Is this a repo remote?
bool isRepo = protocolStorageType == protocolStorageTypeRepo;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Option replacements
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
KeyValue *optionReplace = kvNew();
// Replace config options with the host versions
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
unsigned int optConfig = isRepo ? cfgOptRepoHostConfig : cfgOptPgHostConfig;
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
kvPut(
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
optionReplace, VARSTRDEF(CFGOPT_CONFIG),
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
cfgOptionIdxSource(optConfig, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfig, hostIdx)) : NULL);
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
unsigned int optConfigIncludePath = isRepo ? cfgOptRepoHostConfigIncludePath : cfgOptPgHostConfigIncludePath;
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
Don't pass local config* options to the remote. The remotes have their own config options (repo-host-config, etc.) so don't pass the local config* options. This was a regression from the behavior of the Perl code and while there have been no field reports it caused breakage on test systems with multiple configurations.
2019-07-26 08:37:58 -04:00
kvPut(
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
optionReplace, VARSTRDEF(CFGOPT_CONFIG_INCLUDE_PATH),
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
cfgOptionIdxSource(optConfigIncludePath, hostIdx) != cfgSourceDefault ?
VARSTR(cfgOptionIdxStr(optConfigIncludePath, hostIdx)) : NULL);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
unsigned int optConfigPath = isRepo ? cfgOptRepoHostConfigPath : cfgOptPgHostConfigPath;
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Don't pass local config* options to the remote. The remotes have their own config options (repo-host-config, etc.) so don't pass the local config* options. This was a regression from the behavior of the Perl code and while there have been no field reports it caused breakage on test systems with multiple configurations.
2019-07-26 08:37:58 -04:00
kvPut(
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
optionReplace, VARSTRDEF(CFGOPT_CONFIG_PATH),
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
cfgOptionIdxSource(optConfigPath, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfigPath, hostIdx)) : NULL);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
// Update/remove repo/pg options that are sent to the remote
Improve code that updates/removes pg options passed to a remote. The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated. For example, dc1e7ca2 added the pg-user option but this code was not updated. Instead find the options to update/remove dynamically. The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.
2020-01-09 12:20:13 -07:00
for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++)
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
{
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
// Skip options that are not part of a group
if (!cfgOptionGroup(optionId))
continue;
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
bool remove = false;
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
bool skipHostZero = false;
Improve code that updates/removes pg options passed to a remote. The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated. For example, dc1e7ca2 added the pg-user option but this code was not updated. Instead find the options to update/remove dynamically. The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.
2020-01-09 12:20:13 -07:00
Only pass selected repo options to the remote. Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster. Also fix an outdated comment.
2021-08-05 14:29:26 -04:00
// Remove repo options that are not needed on the remote
Limit option validity by command role. Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here. The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.
2020-12-29 15:49:37 -05:00
if (cfgOptionGroupId(optionId) == cfgOptGrpRepo)
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
{
Only pass selected repo options to the remote. Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster. Also fix an outdated comment.
2021-08-05 14:29:26 -04:00
// If remote type is pg then remove all repo options since they will not be used
if (protocolStorageType == protocolStorageTypePg)
{
remove = true;
}
// Else remove repo options for indexes that are not being passed to the repo. This prevents the remote from getting
// partial info about a repo, which could cause an error during validation.
else
{
for (unsigned int optionIdx = 0; optionIdx < cfgOptionIdxTotal(optionId); optionIdx++)
{
if (cfgOptionIdxTest(optionId, optionIdx) && optionIdx != hostIdx)
kvPut(optionReplace, VARSTRZ(cfgOptionIdxName(optionId, optionIdx)), NULL);
}
}
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
}
Only pass selected repo options to the remote. Options for other repos can cause conflicts and should never be used. Each remote can address exactly one repo or pg cluster. Also fix an outdated comment.
2021-08-05 14:29:26 -04:00
// Remove pg options that are not needed on the remote
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
else
Improve code that updates/removes pg options passed to a remote. The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated. For example, dc1e7ca2 added the pg-user option but this code was not updated. Instead find the options to update/remove dynamically. The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.
2020-01-09 12:20:13 -07:00
{
Limit option validity by command role. Building on 23f5712, limit option validity by role. This is mostly for options that weren't needed for certain roles but were harmless. However, the upcoming multi repository functionality requires the granularity implemented here. The remote role benefits since host options can automatically excluded when building the options. Also, many options that are only required for the default role (e.g. repo-retention-full) no longer need to be passed in tests for other roles.
2020-12-29 15:49:37 -05:00
ASSERT(cfgOptionGroupId(optionId) == cfgOptGrpPg);
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
Fix possibly missing pg1-* options for the remote command. Some pg1-* options are required by the remote so if they are not provided in the remote's configuration file then it may cause a configuration error, depending on the operation. This currently only applies to the pg1-path option. This is still an issue for repo-* options but the same solution cannot be applied because some repo-* options are secure and cannot be passed on the command-line.
2020-05-21 16:09:23 -04:00
// Remove unrequired/defaulted pg options when the remote type is repo since they won't be used
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
if (protocolStorageType == protocolStorageTypeRepo)
{
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
remove = !cfgParseOptionRequired(cfgCommand(), optionId) || cfgParseOptionDefault(cfgCommand(), optionId) != NULL;
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
}
// Move pg options to host index 0 (key 1) so they will be in the default index on the remote host
else
Improve code that updates/removes pg options passed to a remote. The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated. For example, dc1e7ca2 added the pg-user option but this code was not updated. Instead find the options to update/remove dynamically. The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.
2020-01-09 12:20:13 -07:00
{
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
if (hostIdx != 0)
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
{
kvPut(
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
optionReplace, VARSTRZ(cfgOptionIdxName(optionId, 0)),
Store config values as a union instead of a variant. The variants were needed to easily serialize configurations for the Perl code. Unions are more efficient and will allow us to add new types that are not supported by variants, e.g. StringId.
2021-10-22 18:02:20 -04:00
cfgOptionIdxSource(optionId, hostIdx) != cfgSourceDefault ? cfgOptionIdxVar(optionId, hostIdx) : NULL);
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
}
remove = true;
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
skipHostZero = true;
Improve code that updates/removes pg options passed to a remote. The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated. For example, dc1e7ca2 added the pg-user option but this code was not updated. Instead find the options to update/remove dynamically. The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.
2020-01-09 12:20:13 -07:00
}
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
}
Improve code that updates/removes pg options passed to a remote. The prior code was updating/removing hard-coded options but new options are added all the time and there was no indication that this code needed to be updated. For example, dc1e7ca2 added the pg-user option but this code was not updated. Instead find the options to update/remove dynamically. The new code uses prefixes, which is not perfect, but the patterns for pg options are pretty well established and this seems safer than the existing code.
2020-01-09 12:20:13 -07:00
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
// Remove options that have been marked for removal if they are not already null or invalid. This is more efficient because
// cfgExecParam() won't have to search through as large a list looking for overrides.
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
if (remove)
{
// Loop through option indexes
for (unsigned int optionIdx = 0; optionIdx < cfgOptionIdxTotal(optionId); optionIdx++)
{
if (cfgOptionIdxTest(optionId, optionIdx) && !(skipHostZero && optionIdx == 0))
kvPut(optionReplace, VARSTRZ(cfgOptionIdxName(optionId, optionIdx)), NULL);
}
}
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
}
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// Set repo default so the remote only operates on a single repo
if (protocolStorageType == protocolStorageTypeRepo)
kvPut(optionReplace, VARSTRDEF(CFGOPT_REPO), VARUINT(cfgOptionGroupIdxToKey(cfgOptGrpRepo, hostIdx)));
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
// Add the process id if not set. This means that the remote is being started from the main process and should always get a
// process id of 0.
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
if (!cfgOptionTest(cfgOptProcess))
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS), VARINT(0));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Don't pass log-path or lock-path since these are host specific
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_PATH), NULL);
kvPut(optionReplace, VARSTRDEF(CFGOPT_LOCK_PATH), NULL);
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Only enable file logging on the remote when requested
kvPut(
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE),
Add StringId as an option type. Rather the converting String to StringIds at runtime, store defaults in StringId format in parse.auto.c and convert user input to StringId during parsing.
2021-11-03 07:27:26 -04:00
cfgOptionBool(cfgOptLogSubprocess) ? VARUINT64(cfgOptionStrId(cfgOptLogLevelFile)) : VARSTRDEF("off"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// Always output errors on stderr for debugging purposes
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARSTRDEF("error"));
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
// Disable output to stdout since it is used by the protocol
Remove auto-generated option String constants. These are rarely used and end up using more space than they save since most of them are never referenced. Replace VARSTR() with VARSTRDEF() where these constants are being used.
2021-04-22 19:10:13 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARSTRDEF("off"));
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
Update inaccurate comments.
2020-01-09 11:56:03 -07:00
// Add the remote type
Convert ProtocolStorageType enum to StringId. Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
kvPut(optionReplace, VARSTRDEF(CFGOPT_REMOTE_TYPE), VARSTR(strIdToStr(protocolStorageType)));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Factor remote process exec out of protocolRemoteGet(). This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing. The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
FUNCTION_LOG_RETURN(STRING_LIST, cfgExecParam(cfgCommand(), cfgCmdRoleRemote, optionReplace, false, true));
}
// Helper to add SSH parameters when executing the remote via SSH
static StringList *
protocolRemoteParamSsh(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx)
{
FUNCTION_LOG_BEGIN(logLevelDebug);
FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType);
FUNCTION_LOG_PARAM(UINT, hostIdx);
FUNCTION_LOG_END();
StringList *result = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
// Is this a repo remote?
bool isRepo = protocolStorageType == protocolStorageTypeRepo;
// Fixed parameters for ssh command
result = strLstNew();
strLstAddZ(result, "-o");
strLstAddZ(result, "LogLevel=error");
strLstAddZ(result, "-o");
strLstAddZ(result, "Compression=no");
strLstAddZ(result, "-o");
strLstAddZ(result, "PasswordAuthentication=no");
// Append port if specified
ConfigOption optHostPort = isRepo ? cfgOptRepoHostPort : cfgOptPgHostPort;
if (cfgOptionIdxTest(optHostPort, hostIdx))
{
strLstAddZ(result, "-p");
strLstAdd(result, strNewFmt("%u", cfgOptionIdxUInt(optHostPort, hostIdx)));
}
// Append user/host
strLstAdd(
result,
strNewFmt(
"%s@%s", strZ(cfgOptionIdxStr(isRepo ? cfgOptRepoHostUser : cfgOptPgHostUser, hostIdx)),
strZ(cfgOptionIdxStr(isRepo ? cfgOptRepoHost : cfgOptPgHost, hostIdx))));
// Add remote command and parameters
StringList *paramList = protocolRemoteParam(protocolStorageType, hostIdx);
strLstInsert(paramList, 0, cfgOptionIdxStr(isRepo ? cfgOptRepoHostCmd : cfgOptPgHostCmd, hostIdx));
strLstAdd(result, strLstJoin(paramList, " "));
// Move to prior context
strLstMove(result, memContextPrior());
}
MEM_CONTEXT_TEMP_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN(STRING_LIST, result);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Factor remote process exec out of protocolRemoteGet(). This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing. The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
// Helper to execute the local process. This is a separate function solely so that it can be shimmed during testing.
static void
protocolRemoteExec(
ProtocolHelperClient *const helper, const ProtocolStorageType protocolStorageType, const unsigned int hostIdx,
const unsigned int processId)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM_P(VOID, helper);
FUNCTION_TEST_PARAM(ENUM, protocolStorageType);
FUNCTION_TEST_PARAM(UINT, hostIdx);
FUNCTION_TEST_PARAM(UINT, processId);
FUNCTION_TEST_END();
ASSERT(helper != NULL);
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
// Get remote info
const bool isRepo = protocolStorageType == protocolStorageTypeRepo;
const StringId remoteType = cfgOptionIdxStrId(isRepo ? cfgOptRepoHostType : cfgOptPgHostType, hostIdx);
const String *const host = cfgOptionIdxStr(isRepo ? cfgOptRepoHost : cfgOptPgHost, hostIdx);
Factor remote process exec out of protocolRemoteGet(). This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing. The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
// Handle remote types
IoRead *read;
IoWrite *write;
switch (remoteType)
{
// SSH remote
case CFGOPTVAL_REPO_HOST_TYPE_SSH:
{
// Exec SSH
helper->exec = execNew(
cfgOptionStr(cfgOptCmdSsh), protocolRemoteParamSsh(protocolStorageType, hostIdx),
strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u process on '%s'", processId, strZ(host)),
cfgOptionUInt64(cfgOptProtocolTimeout));
execOpen(helper->exec);
read = execIoRead(helper->exec);
write = execIoWrite(helper->exec);
break;
}
// TLS remote
default:
{
ASSERT(remoteType == CFGOPTVAL_REPO_HOST_TYPE_TLS);
// Negotiate TLS
helper->ioClient = tlsClientNew(
sckClientNew(
host, cfgOptionIdxUInt(isRepo ? cfgOptRepoHostPort : cfgOptPgHostPort, hostIdx),
cfgOptionUInt64(cfgOptIoTimeout), cfgOptionUInt64(cfgOptProtocolTimeout)),
host, cfgOptionUInt64(cfgOptIoTimeout), cfgOptionUInt64(cfgOptProtocolTimeout), true,
cfgOptionIdxStrNull(isRepo ? cfgOptRepoHostCaFile : cfgOptPgHostCaFile, hostIdx),
cfgOptionIdxStrNull(isRepo ? cfgOptRepoHostCaPath : cfgOptPgHostCaPath, hostIdx),
cfgOptionIdxStr(isRepo ? cfgOptRepoHostCertFile : cfgOptPgHostCertFile, hostIdx),
cfgOptionIdxStr(isRepo ? cfgOptRepoHostKeyFile : cfgOptPgHostKeyFile, hostIdx));
helper->ioSession = ioClientOpen(helper->ioClient);
read = ioSessionIoRead(helper->ioSession);
write = ioSessionIoWrite(helper->ioSession);
break;
}
}
Factor remote process exec out of protocolRemoteGet(). This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing. The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
// Create protocol object
helper->client = protocolClientNew(
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u %s protocol on '%s'", processId, strZ(strIdToStr(remoteType)), strZ(host)),
PROTOCOL_SERVICE_REMOTE_STR, read, write);
Factor remote process exec out of protocolRemoteGet(). This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing. The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 14:32:41 -04:00
// Remote initialization
switch (remoteType)
{
// SSH remote
case CFGOPTVAL_REPO_HOST_TYPE_SSH:
// Move client to exec context so they are freed together
protocolClientMove(helper->client, execMemContext(helper->exec));
break;
// TLS remote
default:
{
ASSERT(remoteType == CFGOPTVAL_REPO_HOST_TYPE_TLS);
// Pass parameters to server
ProtocolCommand *command = protocolCommandNew(PROTOCOL_COMMAND_CONFIG);
pckWriteStrLstP(protocolCommandParam(command), protocolRemoteParam(protocolStorageType, hostIdx));
protocolClientExecute(helper->client, command, false);
break;
}
}
Factor remote process exec out of protocolRemoteGet(). This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing. The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
FUNCTION_TEST_RETURN_VOID();
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ProtocolClient *
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolRemoteGet(ProtocolStorageType protocolStorageType, unsigned int hostIdx)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelDebug);
Convert ProtocolStorageType enum to StringId. Allows removal of protocolStorageTypeEnum()/protocolStorageTypeStr() and improves debug logging of the enum.
2021-04-28 11:59:04 -04:00
FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType);
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
FUNCTION_LOG_PARAM(UINT, hostIdx);
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// Is this a repo remote?
bool isRepo = protocolStorageType == protocolStorageTypeRepo;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
protocolHelperInit();
// Allocate the client cache
if (protocolHelper.clientRemoteSize == 0)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Remove cfgDefOptionIndexTotal(). This function was only used in one place, which was better served by cfgOptionGroupIdxTotal().
2020-12-14 14:37:23 -05:00
protocolHelper.clientRemoteSize = cfgOptionGroupIdxTotal(isRepo ? cfgOptGrpRepo : cfgOptGrpPg) + 1;
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
protocolHelper.clientRemote = memNew(protocolHelper.clientRemoteSize * sizeof(ProtocolHelperClient));
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientRemoteSize; clientIdx++)
protocolHelper.clientRemote[clientIdx] = (ProtocolHelperClient){.exec = NULL};
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
MEM_CONTEXT_END();
}
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
// Determine protocol id for the remote. If the process option is set then use that since we want the remote protocol id to
// match the local protocol id. Otherwise set to 0 since the remote is being started from a main process and there should only
// be one remote per host.
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
unsigned int processId = 0;
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
if (cfgOptionTest(cfgOptProcess))
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
processId = cfgOptionUInt(cfgOptProcess);
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
CHECK(hostIdx < protocolHelper.clientRemoteSize);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// Create protocol object
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
ProtocolHelperClient *protocolHelperClient = &protocolHelper.clientRemote[hostIdx];
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelperClient->client == NULL)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
MEM_CONTEXT_BEGIN(protocolHelper.memContext)
{
Factor remote process exec out of protocolRemoteGet(). This allows protocolRemoteExec() to be shimmed, which means the remote can be run as a child of the test process, simplifying coverage testing. The shim does not need SSH parameters, so also split those out into a separate function and update the tests to match.
2021-05-25 18:09:29 -04:00
protocolRemoteExec(protocolHelperClient, protocolStorageType, hostIdx, processId);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Move noop from protocolClientNew() to local/remote initialization. protocolServerNew() does not automatically process a noop so this made the handshake in the constructors asymmetric. This made testing a bit confusing since an extra noop was needed when cmdLocal()/cmdRemote() were not called (since they processed the noop sent by protocolClientNew()). The extra noops also make complex protocol negotiation (coming in a future commit) more complicated and slower because of the additional round trips.
2021-07-23 08:32:30 -04:00
// Send noop to catch initialization errors
protocolClientNoOp(protocolHelperClient->client);
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Get cipher options from the remote if none are locally configured
Update CipherType/CipherMode to StringId. As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
if (isRepo && cfgOptionIdxStrId(cfgOptRepoCipherType, hostIdx) == cipherTypeNone)
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
{
// Options to query
VariantList *param = varLstNew();
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
varLstAdd(param, varNewStrZ(cfgOptionIdxName(cfgOptRepoCipherType, hostIdx)));
varLstAdd(param, varNewStrZ(cfgOptionIdxName(cfgOptRepoCipherPass, hostIdx)));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
Improve protocol handlers. Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.
2021-03-16 13:09:34 -04:00
VariantList *optionList = configOptionRemote(protocolHelperClient->client, param);
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
Add StringId as an option type. Rather the converting String to StringIds at runtime, store defaults in StringId format in parse.auto.c and convert user input to StringId during parsing.
2021-11-03 07:27:26 -04:00
if (varUInt64(varLstGet(optionList, 0)) != cipherTypeNone)
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
{
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
cfgOptionIdxSet(cfgOptRepoCipherType, hostIdx, cfgSourceConfig, varLstGet(optionList, 0));
cfgOptionIdxSet(cfgOptRepoCipherPass, hostIdx, cfgSourceConfig, varLstGet(optionList, 1));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
}
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
MEM_CONTEXT_END();
}
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Add protocolRemoteFree() to shutdown a specific remote. Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.
2019-12-07 17:48:53 -05:00
/**********************************************************************************************************************************/
void
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolRemoteFree(unsigned int hostIdx)
Add protocolRemoteFree() to shutdown a specific remote. Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.
2019-12-07 17:48:53 -05:00
{
FUNCTION_LOG_BEGIN(logLevelDebug);
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
FUNCTION_LOG_PARAM(UINT, hostIdx);
Add protocolRemoteFree() to shutdown a specific remote. Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.
2019-12-07 17:48:53 -05:00
FUNCTION_LOG_END();
if (protocolHelper.clientRemote != NULL)
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolHelperClientFree(&protocolHelper.clientRemote[hostIdx]);
Add protocolRemoteFree() to shutdown a specific remote. Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.
2019-12-07 17:48:53 -05:00
FUNCTION_LOG_RETURN_VOID();
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add protocolKeepAlive() to send noops to all remotes. While the local processes are doing their jobs the remote connection from the main process may timeout. Send occasional noops to ensure that doesn't happen.
2019-03-27 20:59:28 +00:00
void
protocolKeepAlive(void)
{
FUNCTION_LOG_VOID(logLevelTrace);
if (protocolHelper.memContext != NULL)
{
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientRemoteSize; clientIdx++)
{
if (protocolHelper.clientRemote[clientIdx].client != NULL)
protocolClientNoOp(protocolHelper.clientRemote[clientIdx].client);
}
}
FUNCTION_LOG_RETURN_VOID();
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
void
protocolFree(void)
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_VOID(logLevelTrace);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
if (protocolHelper.memContext != NULL)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Free remotes
Add protocolRemoteFree() to shutdown a specific remote. Sometimes it is useful to shutdown remotes that are no longer needed instead of waiting for them to be shutdown at program exit.
2019-12-07 17:48:53 -05:00
for (unsigned int clientIdx = 0; clientIdx < protocolHelper.clientRemoteSize; clientIdx++)
Use zero-based indexes when referring to option indexes. There were a number of places in the code where "hostId" was used, but hostId is just the option group index + 1 so this led to a lot of +1 and -1 to convert the id to an index and vice versa. Instead just use the zero based index wherever possible. This is pretty much everywhere except when the host-id option is read or set, or where a message is being formatted for the user. Also fix a bug in protocolRemoteParam() where remotes spawned from the main process could get process ids that were not 0. Only the locals should spawn remotes with process id > 0. This seems to have been harmless since the process id is only a label, but it could be confusing when debugging.
2020-10-26 10:25:16 -04:00
protocolRemoteFree(clientIdx);
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Free locals
Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
for (unsigned int clientIdx = 1; clientIdx <= protocolHelper.clientLocalSize; clientIdx++)
protocolLocalFree(clientIdx);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN_VOID();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Reference in New Issue Copy Permalink