db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-06-18 23:57:33 +02:00
Code Issues Releases Activity
Files
77b0c6c993a0e6ff45a6a99f343c3709a016d152
pgbackrest/test/src/module/protocol/protocolTest.c

901 lines
45 KiB
C
Raw Normal View History

Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Test Protocol
***********************************************************************************************************************************/
#include "common/io/handleRead.h"
#include "common/io/handleWrite.h"
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
#include "common/io/bufferRead.h"
#include "common/io/bufferWrite.h"
Fix reliability of error reporting from local/remote processes. Asserts were only only reported on stderr rather than being returned through the protocol layer. This did not appear to be very reliable. Instead, report the assert through the protocol layer like any other error. Add a stack trace if an assert error or debug logging is enabled.
2019-04-18 10:36:21 -04:00
#include "common/regExp.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "storage/storage.h"
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
#include "storage/posix/storage.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
#include "version.h"
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
#include "common/harnessConfig.h"
#include "common/harnessFork.h"
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
/***********************************************************************************************************************************
Test protocol request handler
***********************************************************************************************************************************/
bool
testServerProtocol(const String *command, const VariantList *paramList, ProtocolServer *server)
{
FUNCTION_HARNESS_BEGIN();
FUNCTION_HARNESS_PARAM(STRING, command);
FUNCTION_HARNESS_PARAM(VARIANT_LIST, paramList);
FUNCTION_HARNESS_PARAM(PROTOCOL_SERVER, server);
FUNCTION_HARNESS_END();
ASSERT(command != NULL);
// Attempt to satisfy the request -- we may get requests that are meant for other handlers
bool found = true;
MEM_CONTEXT_TEMP_BEGIN()
{
if (strEq(command, strNew("assert")))
{
THROW(AssertError, "test assert");
}
else if (strEq(command, strNew("request-simple")))
{
protocolServerResponse(server, varNewBool(true));
}
else if (strEq(command, strNew("request-complex")))
{
protocolServerResponse(server, varNewBool(false));
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(protocolServerIoWrite(server), strNew("LINEOFTEXT"));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
ioWriteFlush(protocolServerIoWrite(server));
}
else
found = false;
}
MEM_CONTEXT_TEMP_END();
FUNCTION_HARNESS_RESULT(BOOL, found);
}
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
/***********************************************************************************************************************************
Test ParallelJobCallback
***********************************************************************************************************************************/
typedef struct TestParallelJobCallback
{
List *jobList; // List of jobs to process
unsigned int jobIdx; // Current index in the list to be processed
bool clientSeen[2]; // Make sure the client idx was seen
} TestParallelJobCallback;
static ProtocolParallelJob *testParallelJobCallback(void *data, unsigned int clientIdx)
{
FUNCTION_TEST_BEGIN();
FUNCTION_TEST_PARAM_P(VOID, data);
FUNCTION_TEST_PARAM(UINT, clientIdx);
FUNCTION_TEST_END();
TestParallelJobCallback *listData = data;
// Mark the client idx as seen
listData->clientSeen[clientIdx] = true;
// Get a new job if there are any left
if (listData->jobIdx < lstSize(listData->jobList))
{
ProtocolParallelJob *job = *(ProtocolParallelJob **)lstGet(listData->jobList, listData->jobIdx);
listData->jobIdx++;
FUNCTION_TEST_RETURN(protocolParallelJobMove(job, memContextCurrent()));
}
FUNCTION_TEST_RETURN(NULL);
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
testRun(void)
{
FUNCTION_HARNESS_VOID();
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
Storage *storageTest = storagePosixNew(
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
strNew(testPath()), STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, true, NULL);
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// *****************************************************************************************************************************
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
if (testBegin("repoIsLocal() and pgIsLocal()"))
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
StringList *argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_BOOL(repoIsLocal(), true, "repo is local");
Add repoIsLocalVerify() to verify repository locality. Some commands can only be run on a host where the repository is local. This function centralizes the check and error. Contributed by Cynthia Shang.
2019-08-21 11:41:36 -04:00
TEST_RESULT_VOID(repoIsLocalVerify(), " local verified");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--repo1-host=remote-host");
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_BOOL(repoIsLocal(), false, "repo is remote");
Add repoIsLocalVerify() to verify repository locality. Some commands can only be run on a host where the repository is local. This function centralizes the check and error. Contributed by Cynthia Shang.
2019-08-21 11:41:36 -04:00
TEST_ERROR_FMT(repoIsLocalVerify(), HostInvalidError, "archive-get command must be run on the repository host");
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--pg1-path=/path/to");
strLstAddZ(argList, "--repo1-retention-full=1");
strLstAddZ(argList, "backup");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
TEST_RESULT_BOOL(pgIsLocal(1), true, "pg is local");
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--pg7-path=/path/to");
strLstAddZ(argList, "--pg7-host=test1");
strLstAddZ(argList, "--host-id=7");
strLstAddZ(argList, "--command=backup");
strLstAddZ(argList, "--type=db");
strLstAddZ(argList, "--process=0");
strLstAddZ(argList, "local");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
TEST_RESULT_BOOL(pgIsLocal(7), false, "pg is remote");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
// *****************************************************************************************************************************
if (testBegin("protocolLocalParam()"))
{
StringList *argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
strPtr(strLstJoin(protocolLocalParam(protocolStorageTypeRepo, 0), "|")),
strPtr(
strNew(
"--command=archive-get|--host-id=1|--log-level-file=off|--log-level-stderr=error|--process=0|--stanza=test1"
"|--type=backup|local")),
"local protocol params");
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--log-subprocess");
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
strPtr(strLstJoin(protocolLocalParam(protocolStorageTypeRepo, 1), "|")),
strPtr(
strNew(
"--command=archive-get|--host-id=1|--log-level-file=info|--log-level-stderr=error|--log-subprocess|--process=1"
"|--stanza=test1|--type=backup|local")),
"local protocol params with replacements");
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// *****************************************************************************************************************************
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
if (testBegin("protocolRemoteParam()"))
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Don't pass local config* options to the remote. The remotes have their own config options (repo-host-config, etc.) so don't pass the local config* options. This was a regression from the behavior of the Perl code and while there have been no field reports it caused breakage on test systems with multiple configurations.
2019-07-26 08:37:58 -04:00
storagePutNP(storageNewWriteNP(storageTest, strNew("pgbackrest.conf")), bufNew(0));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
StringList *argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--repo1-host=repo-host");
strLstAddZ(argList, "--repo1-host-user=repo-host-user");
Don't pass local config* options to the remote. The remotes have their own config options (repo-host-config, etc.) so don't pass the local config* options. This was a regression from the behavior of the Perl code and while there have been no field reports it caused breakage on test systems with multiple configurations.
2019-07-26 08:37:58 -04:00
// Local config settings should never be passed to the remote
strLstAdd(argList, strNewFmt("--config=%s/pgbackrest.conf", testPath()));
strLstAdd(argList, strNewFmt("--config-include-path=%s", testPath()));
strLstAdd(argList, strNewFmt("--config-path=%s", testPath()));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
strPtr(strLstJoin(protocolRemoteParam(protocolStorageTypeRepo, 0, 0), "|")),
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
strPtr(
strNew(
"-o|LogLevel=error|-o|Compression=no|-o|PasswordAuthentication=no|repo-host-user@repo-host"
Add --c option to request a C remote. The rules for when a C remote is required are getting complicated and will get worse when restoreFile() is migrated. Instead, set the --c option when a C remote is required. This option will be removed when the remote is entirely implemented in C.
2019-05-09 08:55:48 -04:00
"|pgbackrest --c --command=archive-get --log-level-file=off --log-level-stderr=error --process=0"
" --stanza=test1 --type=backup remote")),
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
"remote protocol params");
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
strLstAddZ(argList, "--log-subprocess");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
strLstAddZ(argList, "--repo1-host=repo-host");
strLstAddZ(argList, "--repo1-host-port=444");
strLstAddZ(argList, "--repo1-host-config=/path/pgbackrest.conf");
strLstAddZ(argList, "--repo1-host-config-include-path=/path/include");
strLstAddZ(argList, "--repo1-host-config-path=/path/config");
strLstAddZ(argList, "--repo1-host-user=repo-host-user");
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
strPtr(strLstJoin(protocolRemoteParam(protocolStorageTypeRepo, 1, 0), "|")),
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
strPtr(
strNew(
"-o|LogLevel=error|-o|Compression=no|-o|PasswordAuthentication=no|-p|444|repo-host-user@repo-host"
Add --c option to request a C remote. The rules for when a C remote is required are getting complicated and will get worse when restoreFile() is migrated. Instead, set the --c option when a C remote is required. This option will be removed when the remote is entirely implemented in C.
2019-05-09 08:55:48 -04:00
"|pgbackrest --c --command=archive-get --config=/path/pgbackrest.conf --config-include-path=/path/include"
Fix issues with remote/local command logging options. Logging was being enable on local/remote processes even if --log-subprocess was not specified, so fix that. Also, make sure that stderr is enabled at error level as it was on Perl. This helps expose error information for debugging. For remotes, suppress log and lock paths since these are not applicable on remote hosts. These options should be set in the local config if they need to be overridden.
2019-03-16 15:00:02 +04:00
" --config-path=/path/config --log-level-file=info --log-level-stderr=error --log-subprocess --process=1"
Add locking capability to the remote command. When a repository server is configured, commands that modify the repository acquire a remote lock as well as a local lock for extra protection against multiple writers. Instead of the custom logic used in Perl, make remote locking part of the command configuration. This also means that the C remote needs the stanza since it is used to construct the lock name. We may need to revisit this at a later date.
2019-03-27 21:14:06 +00:00
" --stanza=test1 --type=backup remote")),
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
"remote protocol params with replacements");
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--command=archive-get");
strLstAddZ(argList, "--process=3");
strLstAddZ(argList, "--host-id=1");
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strLstAddZ(argList, "--type=backup");
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
strLstAddZ(argList, "--repo1-host=repo-host");
strLstAddZ(argList, "local");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
strPtr(strLstJoin(protocolRemoteParam(protocolStorageTypeRepo, 66, 0), "|")),
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
strPtr(
strNew(
"-o|LogLevel=error|-o|Compression=no|-o|PasswordAuthentication=no|pgbackrest@repo-host"
Add --c option to request a C remote. The rules for when a C remote is required are getting complicated and will get worse when restoreFile() is migrated. Instead, set the --c option when a C remote is required. This option will be removed when the remote is entirely implemented in C.
2019-05-09 08:55:48 -04:00
"|pgbackrest --c --command=archive-get --log-level-file=off --log-level-stderr=error --process=3"
" --stanza=test1 --type=backup remote")),
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
"remote protocol params for backup local");
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--pg1-path=/path/to/1");
strLstAddZ(argList, "--pg1-host=pg1-host");
strLstAddZ(argList, "--repo1-retention-full=1");
strLstAddZ(argList, "backup");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
strPtr(strLstJoin(protocolRemoteParam(protocolStorageTypePg, 1, 0), "|")),
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strPtr(
strNew(
"-o|LogLevel=error|-o|Compression=no|-o|PasswordAuthentication=no|postgres@pg1-host"
"|pgbackrest --c --command=backup --log-level-file=off --log-level-stderr=error --pg1-path=/path/to/1"
" --process=1 --stanza=test1 --type=db remote")),
"remote protocol params for db backup");
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--command=backup");
strLstAddZ(argList, "--process=4");
strLstAddZ(argList, "--host-id=2");
strLstAddZ(argList, "--pg1-path=/path/to/1");
Properly reset conflicting pg-* options for the remote protocol. The pg1-socket-path and pg1-port options were not being reset when options from a higher index were being pushed down for processing by a remote. Since remotes only talk to one cluster they always use the options in index 1. This requires moving options from the original index to 1 before starting the remote. All options already set on index 1 must be removed if they are not being overwritten.
2019-08-19 21:45:54 -04:00
strLstAddZ(argList, "--pg1-socket-path=/socket3");
strLstAddZ(argList, "--pg1-port=1111");
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strLstAddZ(argList, "--pg2-path=/path/to/2");
strLstAddZ(argList, "--pg2-host=pg2-host");
strLstAddZ(argList, "--type=db");
strLstAddZ(argList, "local");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
strPtr(strLstJoin(protocolRemoteParam(protocolStorageTypePg, 1, 1), "|")),
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strPtr(
strNew(
"-o|LogLevel=error|-o|Compression=no|-o|PasswordAuthentication=no|postgres@pg2-host"
"|pgbackrest --c --command=backup --log-level-file=off --log-level-stderr=error --pg1-path=/path/to/2"
" --process=4 --stanza=test1 --type=db remote")),
"remote protocol params for db local");
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAddZ(argList, "--command=backup");
strLstAddZ(argList, "--process=4");
strLstAddZ(argList, "--host-id=3");
strLstAddZ(argList, "--pg1-path=/path/to/1");
strLstAddZ(argList, "--pg3-path=/path/to/3");
strLstAddZ(argList, "--pg3-host=pg3-host");
strLstAddZ(argList, "--pg3-socket-path=/socket3");
strLstAddZ(argList, "--pg3-port=3333");
strLstAddZ(argList, "--type=db");
strLstAddZ(argList, "local");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
strPtr(strLstJoin(protocolRemoteParam(protocolStorageTypePg, 1, 2), "|")),
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
strPtr(
strNew(
"-o|LogLevel=error|-o|Compression=no|-o|PasswordAuthentication=no|postgres@pg3-host"
"|pgbackrest --c --command=backup --log-level-file=off --log-level-stderr=error --pg1-path=/path/to/3"
" --pg1-port=3333 --pg1-socket-path=/socket3 --process=4 --stanza=test1 --type=db remote")),
"remote protocol params for db local");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
Add ProtocolCommand object. This formalizes the creation of protocol commands, which was previously done by creating KeyValue objects manually.
2019-02-27 19:48:30 +02:00
// *****************************************************************************************************************************
if (testBegin("ProtocolCommand"))
{
ProtocolCommand *command = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
TEST_ASSIGN(command, protocolCommandNew(strNew("command1")), "create command");
TEST_RESULT_PTR(protocolCommandParamAdd(command, varNewStr(strNew("param1"))), command, "add param");
TEST_RESULT_PTR(protocolCommandParamAdd(command, varNewStr(strNew("param2"))), command, "add param");
TEST_RESULT_PTR(protocolCommandMove(command, MEM_CONTEXT_OLD()), command, "move protocol command");
TEST_RESULT_PTR(protocolCommandMove(NULL, MEM_CONTEXT_OLD()), NULL, "move null protocol command");
}
MEM_CONTEXT_TEMP_END();
TEST_RESULT_STR(strPtr(protocolCommandToLog(command)), "{command: command1}", "check log");
TEST_RESULT_STR(
strPtr(protocolCommandJson(command)), "{\"cmd\":\"command1\",\"param\":[\"param1\",\"param2\"]}", "check json");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ASSIGN(command, protocolCommandNew(strNew("command2")), "create command");
TEST_RESULT_STR(strPtr(protocolCommandToLog(command)), "{command: command2}", "check log");
TEST_RESULT_STR(strPtr(protocolCommandJson(command)), "{\"cmd\":\"command2\"}", "check json");
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(protocolCommandFree(command), "free command");
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// *****************************************************************************************************************************
if (testBegin("ProtocolClient"))
{
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, true)
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("server read"), HARNESS_FORK_CHILD_READ(), 2000);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("server write"), HARNESS_FORK_CHILD_WRITE());
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteOpen(write);
// Various bogus greetings
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("bogus greeting"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":999}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":null}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":\"bogus\"}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":\"pgBackRest\",\"service\":\"bogus\"}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":\"pgBackRest\",\"service\":\"test\",\"version\":\"bogus\"}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
// Correct greeting with noop
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":\"pgBackRest\",\"service\":\"test\",\"version\":\"" PROJECT_VERSION "\"}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"noop\"}", "noop");
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
// Throw errors
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"noop\"}", "noop with error text");
Improve macros and coverage rules that were hiding missing coverage. The branch coverage exclusion rules were overly broad and included functions that ended in a capital letter, which disabled all coverage for the statement. Improve matching so that all characters in the name must be upper-case for a match. Some macros with internal branches accepted parameters that might contain conditionals. This made it impossible to tell which branches belonged to which, and in any case an overzealous exclusion rule was ignoring all branches in such cases. Add the DEBUG_COVERAGE flag to build a modified version of the macros without any internal branches to be used for coverage testing. In most cases, the branches were optimizations (like checking logWill()) that improve production performance but are not needed for testing. In other cases, a parameter needed to be added to the underlying function to handle the branch during coverage testing. Also tweak the coverage rules so that macros without conditionals are automatically excluded from branch coverage as long as they are not themselves a parameter. Finally, update tests and code where missing coverage was exposed by these changes. Some code was updated to remove existing coverage exclusions when it was a simple change.
2019-05-11 14:51:51 -04:00
ioWriteStrLine(write, strNew("{\"err\":25,\"out\":\"sample error message\",\"errStack\":\"stack data\"}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"noop\"}", "noop with no error text");
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"err\":255}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
// No output expected
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"noop\"}", "noop with parameters returned");
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"out\":[\"bogus\"]}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
// Send output
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"test\"}", "test command");
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"out\":[\"value1\",\"value2\"]}"));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteFlush(write);
// Wait for exit
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"exit\"}", "exit command");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_PARENT_BEGIN()
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("client read"), HARNESS_FORK_PARENT_READ_PROCESS(0), 2000);
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("client write"), HARNESS_FORK_PARENT_WRITE_PROCESS(0));
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
ioWriteOpen(write);
// Various bogus greetings
TEST_ERROR(
protocolClientNew(strNew("test client"), strNew("test"), read, write), JsonFormatError,
Add separate functions to encode/decode each JSON type. In most cases the JSON type is known so this is more efficient than converting to Variant first, both in terms of memory and time. Also rename some of the existing functions for consistency.
2019-04-22 18:41:01 -04:00
"expected '{' at 'bogus greeting'");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
TEST_ERROR(
protocolClientNew(strNew("test client"), strNew("test"), read, write), ProtocolError,
"greeting key 'name' must be string type");
TEST_ERROR(
protocolClientNew(strNew("test client"), strNew("test"), read, write), ProtocolError,
"unable to find greeting key 'name'");
TEST_ERROR(
protocolClientNew(strNew("test client"), strNew("test"), read, write), ProtocolError,
"expected value 'pgBackRest' for greeting key 'name' but got 'bogus'");
TEST_ERROR(
protocolClientNew(strNew("test client"), strNew("test"), read, write), ProtocolError,
"expected value 'test' for greeting key 'service' but got 'bogus'");
TEST_ERROR(
protocolClientNew(strNew("test client"), strNew("test"), read, write), ProtocolError,
"expected value '" PROJECT_VERSION "' for greeting key 'version' but got 'bogus'");
// Correct greeting
ProtocolClient *client = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
TEST_ASSIGN(
client,
protocolClientMove(
protocolClientNew(strNew("test client"), strNew("test"), read, write), MEM_CONTEXT_OLD()),
"create client");
TEST_RESULT_VOID(protocolClientMove(NULL, MEM_CONTEXT_OLD()), "move null client");
}
MEM_CONTEXT_TEMP_END();
TEST_RESULT_PTR(protocolClientIoRead(client), client->read, "get read io");
TEST_RESULT_PTR(protocolClientIoWrite(client), client->write, "get write io");
// Throw errors
Fix reliability of error reporting from local/remote processes. Asserts were only only reported on stderr rather than being returned through the protocol layer. This did not appear to be very reliable. Instead, report the assert through the protocol layer like any other error. Add a stack trace if an assert error or debug logging is enabled.
2019-04-18 10:36:21 -04:00
TEST_ERROR(
protocolClientNoOp(client), AssertError,
Improve macros and coverage rules that were hiding missing coverage. The branch coverage exclusion rules were overly broad and included functions that ended in a capital letter, which disabled all coverage for the statement. Improve matching so that all characters in the name must be upper-case for a match. Some macros with internal branches accepted parameters that might contain conditionals. This made it impossible to tell which branches belonged to which, and in any case an overzealous exclusion rule was ignoring all branches in such cases. Add the DEBUG_COVERAGE flag to build a modified version of the macros without any internal branches to be used for coverage testing. In most cases, the branches were optimizations (like checking logWill()) that improve production performance but are not needed for testing. In other cases, a parameter needed to be added to the underlying function to handle the branch during coverage testing. Also tweak the coverage rules so that macros without conditionals are automatically excluded from branch coverage as long as they are not themselves a parameter. Finally, update tests and code where missing coverage was exposed by these changes. Some code was updated to remove existing coverage exclusions when it was a simple change.
2019-05-11 14:51:51 -04:00
"raised from test client: sample error message\nstack data");
harnessLogLevelSet(logLevelDebug);
TEST_ERROR(
protocolClientNoOp(client), UnknownError,
"raised from test client: no details available\nno stack trace available");
harnessLogLevelReset();
// No output expected
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
TEST_ERROR(protocolClientNoOp(client), AssertError, "no output required by command");
// Get command output
const VariantList *output = NULL;
Add ProtocolCommand object. This formalizes the creation of protocol commands, which was previously done by creating KeyValue objects manually.
2019-02-27 19:48:30 +02:00
TEST_ASSIGN(
output,
varVarLst(protocolClientExecute(client, protocolCommandNew(strNew("test")), true)),
"execute command with output");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
TEST_RESULT_UINT(varLstSize(output), 2, "check output size");
TEST_RESULT_STR(strPtr(varStr(varLstGet(output, 0))), "value1", "check value1");
TEST_RESULT_STR(strPtr(varStr(varLstGet(output, 1))), "value2", "check value2");
// Free client
TEST_RESULT_VOID(protocolClientFree(client), "free client");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_PARENT_END();
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
HARNESS_FORK_END();
}
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// *****************************************************************************************************************************
if (testBegin("ProtocolServer"))
{
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, true)
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("client read"), HARNESS_FORK_CHILD_READ(), 2000);
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("client write"), HARNESS_FORK_CHILD_WRITE());
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
ioWriteOpen(write);
// Check greeting
TEST_RESULT_STR(
strPtr(ioReadLine(read)), "{\"name\":\"pgBackRest\",\"service\":\"test\",\"version\":\"" PROJECT_VERSION "\"}",
"check greeting");
// Noop
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
TEST_RESULT_VOID(ioWriteStrLine(write, strNew("{\"cmd\":\"noop\"}")), "write noop");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_VOID(ioWriteFlush(write), "flush noop");
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{}", "noop result");
// Invalid command
Fix reliability of error reporting from local/remote processes. Asserts were only only reported on stderr rather than being returned through the protocol layer. This did not appear to be very reliable. Instead, report the assert through the protocol layer like any other error. Add a stack trace if an assert error or debug logging is enabled.
2019-04-18 10:36:21 -04:00
KeyValue *result = NULL;
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
TEST_RESULT_VOID(ioWriteStrLine(write, strNew("{\"cmd\":\"bogus\"}")), "write bogus");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_VOID(ioWriteFlush(write), "flush bogus");
Fix reliability of error reporting from local/remote processes. Asserts were only only reported on stderr rather than being returned through the protocol layer. This did not appear to be very reliable. Instead, report the assert through the protocol layer like any other error. Add a stack trace if an assert error or debug logging is enabled.
2019-04-18 10:36:21 -04:00
TEST_ASSIGN(result, varKv(jsonToVar(ioReadLine(read))), "parse error result");
TEST_RESULT_INT(varIntForce(kvGet(result, VARSTRDEF("err"))), 39, " check code");
TEST_RESULT_STR(strPtr(varStr(kvGet(result, VARSTRDEF("out")))), "invalid command 'bogus'", " check message");
TEST_RESULT_BOOL(kvGet(result, VARSTRDEF("errStack")) != NULL, true, " check stack exists");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Simple request
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
TEST_RESULT_VOID(ioWriteStrLine(write, strNew("{\"cmd\":\"request-simple\"}")), "write simple request");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_VOID(ioWriteFlush(write), "flush simple request");
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"out\":true}", "simple request result");
Fix reliability of error reporting from local/remote processes. Asserts were only only reported on stderr rather than being returned through the protocol layer. This did not appear to be very reliable. Instead, report the assert through the protocol layer like any other error. Add a stack trace if an assert error or debug logging is enabled.
2019-04-18 10:36:21 -04:00
// Throw an assert error which will include a stack trace
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
TEST_RESULT_VOID(ioWriteStrLine(write, strNew("{\"cmd\":\"assert\"}")), "write assert");
Fix reliability of error reporting from local/remote processes. Asserts were only only reported on stderr rather than being returned through the protocol layer. This did not appear to be very reliable. Instead, report the assert through the protocol layer like any other error. Add a stack trace if an assert error or debug logging is enabled.
2019-04-18 10:36:21 -04:00
TEST_RESULT_VOID(ioWriteFlush(write), "flush assert error");
TEST_ASSIGN(result, varKv(jsonToVar(ioReadLine(read))), "parse error result");
TEST_RESULT_INT(varIntForce(kvGet(result, VARSTRDEF("err"))), 25, " check code");
TEST_RESULT_STR(strPtr(varStr(kvGet(result, VARSTRDEF("out")))), "test assert", " check message");
TEST_RESULT_BOOL(kvGet(result, VARSTRDEF("errStack")) != NULL, true, " check stack exists");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Complex request -- after process loop has been restarted
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
TEST_RESULT_VOID(ioWriteStrLine(write, strNew("{\"cmd\":\"request-complex\"}")), "write complex request");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_VOID(ioWriteFlush(write), "flush complex request");
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"out\":false}", "complex request result");
TEST_RESULT_STR(strPtr(ioReadLine(read)), "LINEOFTEXT", "complex request result");
// Exit
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
TEST_RESULT_VOID(ioWriteStrLine(write, strNew("{\"cmd\":\"exit\"}")), "write exit");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_VOID(ioWriteFlush(write), "flush exit");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_PARENT_BEGIN()
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("server read"), HARNESS_FORK_PARENT_READ_PROCESS(0), 2000);
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("server write"), HARNESS_FORK_PARENT_WRITE_PROCESS(0));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
ioWriteOpen(write);
// Send greeting
ProtocolServer *server = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
TEST_ASSIGN(
server,
protocolServerMove(
protocolServerNew(strNew("test server"), strNew("test"), read, write), MEM_CONTEXT_OLD()),
"create server");
TEST_RESULT_VOID(protocolServerMove(NULL, MEM_CONTEXT_OLD()), "move null server");
}
MEM_CONTEXT_TEMP_END();
TEST_RESULT_PTR(protocolServerIoRead(server), server->read, "get read io");
TEST_RESULT_PTR(protocolServerIoWrite(server), server->write, "get write io");
TEST_RESULT_VOID(protocolServerHandlerAdd(server, testServerProtocol), "add handler");
Fix reliability of error reporting from local/remote processes. Asserts were only only reported on stderr rather than being returned through the protocol layer. This did not appear to be very reliable. Instead, report the assert through the protocol layer like any other error. Add a stack trace if an assert error or debug logging is enabled.
2019-04-18 10:36:21 -04:00
TEST_RESULT_VOID(protocolServerProcess(server), "run process loop");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_VOID(protocolServerFree(server), "free server");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_PARENT_END();
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
}
HARNESS_FORK_END();
}
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
// *****************************************************************************************************************************
if (testBegin("ProtocolParallel and ProtocolParallelJob"))
{
ProtocolParallelJob *job = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
TEST_ASSIGN(job, protocolParallelJobNew(varNewStr(strNew("test")), protocolCommandNew(strNew("command"))), "new job");
TEST_RESULT_PTR(protocolParallelJobMove(job, MEM_CONTEXT_OLD()), job, "move job");
TEST_RESULT_PTR(protocolParallelJobMove(NULL, MEM_CONTEXT_OLD()), NULL, "move null job");
}
MEM_CONTEXT_TEMP_END();
TEST_ERROR(
protocolParallelJobStateSet(job, protocolParallelJobStateDone), AssertError,
"invalid state transition from 'pending' to 'done'");
TEST_RESULT_VOID(protocolParallelJobStateSet(job, protocolParallelJobStateRunning), "transition to running");
TEST_ERROR(
protocolParallelJobStateSet(job, protocolParallelJobStatePending), AssertError,
"invalid state transition from 'running' to 'pending'");
// Free job
TEST_RESULT_VOID(protocolParallelJobFree(job), "free job");
// -------------------------------------------------------------------------------------------------------------------------
HARNESS_FORK_BEGIN()
{
// Local 1
HARNESS_FORK_CHILD_BEGIN(0, true)
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("server read"), HARNESS_FORK_CHILD_READ(), 10000);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("server write"), HARNESS_FORK_CHILD_WRITE());
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteOpen(write);
// Greeting with noop
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":\"pgBackRest\",\"service\":\"test\",\"version\":\"" PROJECT_VERSION "\"}"));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteFlush(write);
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"noop\"}", "noop");
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{}"));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteFlush(write);
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"command1\",\"param\":[\"param1\",\"param2\"]}", "command1");
sleepMSec(4000);
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"out\":1}"));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteFlush(write);
// Wait for exit
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"exit\"}", "exit command");
}
HARNESS_FORK_CHILD_END();
// Local 2
HARNESS_FORK_CHILD_BEGIN(0, true)
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("server read"), HARNESS_FORK_CHILD_READ(), 10000);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("server write"), HARNESS_FORK_CHILD_WRITE());
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteOpen(write);
// Greeting with noop
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"name\":\"pgBackRest\",\"service\":\"test\",\"version\":\"" PROJECT_VERSION "\"}"));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteFlush(write);
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"noop\"}", "noop");
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{}"));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteFlush(write);
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"command2\",\"param\":[\"param1\"]}", "command2");
sleepMSec(1000);
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"out\":2}"));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteFlush(write);
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"command3\",\"param\":[\"param1\"]}", "command3");
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew("{\"err\":39,\"out\":\"very serious error\"}"));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteFlush(write);
// Wait for exit
TEST_RESULT_STR(strPtr(ioReadLine(read)), "{\"cmd\":\"exit\"}", "exit command");
}
HARNESS_FORK_CHILD_END();
HARNESS_FORK_PARENT_BEGIN()
{
// -----------------------------------------------------------------------------------------------------------------
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
TestParallelJobCallback data = {.jobList = lstNew(sizeof(ProtocolParallelJob *))};
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ProtocolParallel *parallel = NULL;
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
TEST_ASSIGN(parallel, protocolParallelNew(2000, testParallelJobCallback, &data), "create parallel");
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
TEST_RESULT_STR(
strPtr(protocolParallelToLog(parallel)), "{state: pending, clientTotal: 0, jobTotal: 0}", "check log");
// Add client
unsigned int clientTotal = 2;
ProtocolClient *client[HARNESS_FORK_CHILD_MAX];
for (unsigned int clientIdx = 0; clientIdx < clientTotal; clientIdx++)
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(
strNewFmt("client %u read", clientIdx), HARNESS_FORK_PARENT_READ_PROCESS(clientIdx), 2000);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(
strNewFmt("client %u write", clientIdx), HARNESS_FORK_PARENT_WRITE_PROCESS(clientIdx));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteOpen(write);
TEST_ASSIGN(
client[clientIdx],
protocolClientNew(strNewFmt("test client %u", clientIdx), strNew("test"), read, write),
"create client %u", clientIdx);
TEST_RESULT_VOID(protocolParallelClientAdd(parallel, client[clientIdx]), "add client %u", clientIdx);
}
// Attempt to add client without handle io
String *protocolString = strNew(
"{\"name\":\"pgBackRest\",\"service\":\"error\",\"version\":\"" PROJECT_VERSION "\"}\n"
"{}\n");
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioBufferReadNew(BUFSTR(protocolString));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioBufferWriteNew(bufNew(1024));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ioWriteOpen(write);
ProtocolClient *clientError = protocolClientNew(strNew("error"), strNew("error"), read, write);
TEST_ERROR(protocolParallelClientAdd(parallel, clientError), AssertError, "client with read handle is required");
protocolClientFree(clientError);
// Add jobs
ProtocolCommand *command = protocolCommandNew(strNew("command1"));
protocolCommandParamAdd(command, varNewStr(strNew("param1")));
protocolCommandParamAdd(command, varNewStr(strNew("param2")));
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
ProtocolParallelJob *job = protocolParallelJobNew(varNewStr(strNew("job1")), command);
TEST_RESULT_VOID(lstAdd(data.jobList, &job), "add job");
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
command = protocolCommandNew(strNew("command2"));
protocolCommandParamAdd(command, varNewStr(strNew("param1")));
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
job = protocolParallelJobNew(varNewStr(strNew("job2")), command);
TEST_RESULT_VOID(lstAdd(data.jobList, &job), "add job");
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
command = protocolCommandNew(strNew("command3"));
protocolCommandParamAdd(command, varNewStr(strNew("param1")));
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
job = protocolParallelJobNew(varNewStr(strNew("job3")), command);
TEST_RESULT_VOID(lstAdd(data.jobList, &job), "add job");
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
// Process jobs
TEST_RESULT_INT(protocolParallelProcess(parallel), 0, "process jobs");
TEST_RESULT_PTR(protocolParallelResult(parallel), NULL, "check no result");
// Process jobs
TEST_RESULT_INT(protocolParallelProcess(parallel), 1, "process jobs");
TEST_ASSIGN(job, protocolParallelResult(parallel), "get result");
TEST_RESULT_STR(strPtr(varStr(protocolParallelJobKey(job))), "job2", "check key is job2");
Add process id to ProtocolParallelJob. This exposes the process that performed to job to the caller for logging purposes.
2019-04-09 11:01:35 -04:00
TEST_RESULT_BOOL(
protocolParallelJobProcessId(job) >= 1 && protocolParallelJobProcessId(job) <= 2, true,
"check process id is valid");
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
TEST_RESULT_INT(varIntForce(protocolParallelJobResult(job)), 2, "check result is 2");
TEST_RESULT_PTR(protocolParallelResult(parallel), NULL, "check no more results");
// Process jobs
TEST_RESULT_INT(protocolParallelProcess(parallel), 1, "process jobs");
TEST_ASSIGN(job, protocolParallelResult(parallel), "get result");
TEST_RESULT_STR(strPtr(varStr(protocolParallelJobKey(job))), "job3", "check key is job3");
TEST_RESULT_INT(protocolParallelJobErrorCode(job), 39, "check error code");
TEST_RESULT_STR(
strPtr(protocolParallelJobErrorMessage(job)), "raised from test client 1: very serious error",
"check error message");
TEST_RESULT_PTR(protocolParallelJobResult(job), NULL, "check result is null");
TEST_RESULT_PTR(protocolParallelResult(parallel), NULL, "check no more results");
// Process jobs
TEST_RESULT_INT(protocolParallelProcess(parallel), 0, "process jobs");
TEST_RESULT_PTR(protocolParallelResult(parallel), NULL, "check no result");
// Process jobs
TEST_RESULT_INT(protocolParallelProcess(parallel), 1, "process jobs");
TEST_ASSIGN(job, protocolParallelResult(parallel), "get result");
TEST_RESULT_STR(strPtr(varStr(protocolParallelJobKey(job))), "job1", "check key is job1");
TEST_RESULT_INT(varIntForce(protocolParallelJobResult(job)), 1, "check result is 1");
TEST_RESULT_BOOL(protocolParallelDone(parallel), true, "check done");
// Free client
for (unsigned int clientIdx = 0; clientIdx < clientTotal; clientIdx++)
TEST_RESULT_VOID(protocolClientFree(client[clientIdx]), "free client %u", clientIdx);
// Free parallel
TEST_RESULT_VOID(protocolParallelFree(parallel), "free parallel");
}
HARNESS_FORK_PARENT_END();
}
HARNESS_FORK_END();
}
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
// *****************************************************************************************************************************
if (testBegin("protocolGet()"))
{
Add protocolKeepAlive() to send noops to all remotes. While the local processes are doing their jobs the remote connection from the main process may timeout. Send occasional noops to ensure that doesn't happen.
2019-03-27 20:59:28 +00:00
// Call keep alive before any remotes exist
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(protocolKeepAlive(), "keep alive");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Simple protocol start
// -------------------------------------------------------------------------------------------------------------------------
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
StringList *argList = strLstNew();
strLstAddZ(argList, "/usr/bin/pgbackrest");
strLstAddZ(argList, "--stanza=db");
strLstAddZ(argList, "--protocol-timeout=10");
strLstAddZ(argList, "--repo1-host=localhost");
strLstAdd(argList, strNewFmt("--repo1-path=%s", testPath()));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
strLstAddZ(argList, "info");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
ProtocolClient *client = NULL;
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
TEST_RESULT_VOID(protocolFree(), "free protocol objects before anything has been created");
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
TEST_ASSIGN(client, protocolRemoteGet(protocolStorageTypeRepo, 1), "get remote protocol");
TEST_RESULT_PTR(protocolRemoteGet(protocolStorageTypeRepo, 1), client, "get remote cached protocol");
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
TEST_RESULT_PTR(protocolHelper.clientRemote[0].client, client, "check position in cache");
Add protocolKeepAlive() to send noops to all remotes. While the local processes are doing their jobs the remote connection from the main process may timeout. Send occasional noops to ensure that doesn't happen.
2019-03-27 20:59:28 +00:00
TEST_RESULT_VOID(protocolKeepAlive(), "keep alive");
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
TEST_RESULT_VOID(protocolFree(), "free remote protocol objects");
TEST_RESULT_VOID(protocolFree(), "free remote protocol objects again");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Start protocol with local encryption settings
// -------------------------------------------------------------------------------------------------------------------------
storagePut(
storageNewWriteNP(storageTest, strNew("pgbackrest.conf")),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF(
"[global]\n"
"repo1-cipher-type=aes-256-cbc\n"
"repo1-cipher-pass=acbd\n"));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
argList = strLstNew();
strLstAddZ(argList, "/usr/bin/pgbackrest");
strLstAddZ(argList, "--stanza=db");
strLstAddZ(argList, "--protocol-timeout=10");
strLstAdd(argList, strNewFmt("--config=%s/pgbackrest.conf", testPath()));
strLstAddZ(argList, "--repo1-host=localhost");
strLstAdd(argList, strNewFmt("--repo1-path=%s", testPath()));
Fix segfault when process-max > 8 for archive-push/archive-get. The remote list was at most 9 (based on pg[1-8]-* max index) so anything over 8 wrote into unallocated memory. The remote for the main process is (currently) stored in position zero so do the same for remotes started from locals, since there should only be one. The main process will need to start more remotes in the future which is why there is extra space. Reported by Jens Wilke.
2019-04-29 16:10:27 -04:00
strLstAddZ(argList, "--process=999");
Cleanup local/remote protocol interaction from 9367cc46. The command option was not being set correctly when a remote was started from a local. It was being set as 'local' rather than the command that the local was running as. Also automatically select the remote protocol id based on whether it is started from a local (use the local protocol id) or from the main process (use 0). These were not live issues but could cause strange behaviors as new features are added that might be hard to diagnose.
2019-02-28 09:51:19 +02:00
strLstAddZ(argList, "--command=archive-get");
strLstAddZ(argList, "--host-id=1");
strLstAddZ(argList, "--type=db");
strLstAddZ(argList, "local");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_STR(strPtr(cfgOptionStr(cfgOptRepoCipherPass)), "acbd", "check cipher pass before");
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
TEST_ASSIGN(client, protocolRemoteGet(protocolStorageTypeRepo, 1), "get remote protocol");
Fix segfault when process-max > 8 for archive-push/archive-get. The remote list was at most 9 (based on pg[1-8]-* max index) so anything over 8 wrote into unallocated memory. The remote for the main process is (currently) stored in position zero so do the same for remotes started from locals, since there should only be one. The main process will need to start more remotes in the future which is why there is extra space. Reported by Jens Wilke.
2019-04-29 16:10:27 -04:00
TEST_RESULT_PTR(protocolHelper.clientRemote[0].client, client, "check position in cache");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_STR(strPtr(cfgOptionStr(cfgOptRepoCipherPass)), "acbd", "check cipher pass after");
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
TEST_RESULT_VOID(protocolFree(), "free remote protocol objects");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
// Start protocol with remote encryption settings
// -------------------------------------------------------------------------------------------------------------------------
storagePut(
storageNewWriteNP(storageTest, strNew("pgbackrest.conf")),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF(
"[global]\n"
"repo1-cipher-type=aes-256-cbc\n"
"repo1-cipher-pass=dcba\n"));
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
argList = strLstNew();
strLstAddZ(argList, "/usr/bin/pgbackrest");
strLstAddZ(argList, "--stanza=db");
strLstAddZ(argList, "--protocol-timeout=10");
strLstAdd(argList, strNewFmt("--repo1-host-config=%s/pgbackrest.conf", testPath()));
strLstAddZ(argList, "--repo1-host=localhost");
strLstAdd(argList, strNewFmt("--repo1-path=%s", testPath()));
strLstAddZ(argList, "info");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_RESULT_PTR(cfgOptionStr(cfgOptRepoCipherPass), NULL, "check cipher pass before");
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
TEST_ASSIGN(client, protocolRemoteGet(protocolStorageTypeRepo, 1), "get remote protocol");
Migrate remote command to C. Prior to this the Perl remote was used to satisfy C requests. This worked fine but since the remote needed to be migrated to C anyway there was no reason to wait. Add the ProtocolServer object and tweak ProtocolClient to work with it. It was also necessary to add a mechanism to get option values from the remote so that encryption settings could be read and used in the storage object. Update the remote storage objects to comply with the protocol changes and add the storage protocol handler. Ideally this commit would have been broken up into smaller chunks but there are cross-dependencies in the protocol layer and it didn't seem worth the extra effort.
2019-02-19 20:57:38 +02:00
TEST_RESULT_STR(strPtr(cfgOptionStr(cfgOptRepoCipherPass)), "dcba", "check cipher pass after");
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
TEST_RESULT_VOID(protocolFree(), "free remote protocol objects");
// Start db protocol
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "/usr/bin/pgbackrest");
strLstAddZ(argList, "--stanza=db");
strLstAddZ(argList, "--protocol-timeout=10");
strLstAddZ(argList, "--repo1-retention-full=1");
strLstAddZ(argList, "--pg1-host=localhost");
strLstAdd(argList, strNewFmt("--pg1-host-user=%s", testUser()));
strLstAdd(argList, strNewFmt("--pg1-path=%s", testPath()));
strLstAddZ(argList, "backup");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
Improve multi-host handling in protocol helper. Multiple PostgreSQL hosts were supported via the host-id option but there are cases where it is useful to be able to directly specify the host id required, e.g. to iterate through pg* hosts when looking for candidate primaries and standbys during backup.
2019-07-31 20:44:49 -04:00
TEST_ASSIGN(client, protocolRemoteGet(protocolStorageTypePg, 1), "get remote protocol");
Allow pg storage to be remote. None of the currently migrated commands needed remote pg storage but now backup, check, stanza-* will need it.
2019-07-17 14:09:50 -04:00
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
// Start local protocol
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "/usr/bin/pgbackrest");
strLstAddZ(argList, "--stanza=db");
strLstAddZ(argList, "--protocol-timeout=10");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
strLstAddZ(argList, "--process-max=2");
strLstAddZ(argList, "archive-get-async");
Migrate local command to C. The C local is only used for C commands in the main process. Some tweaking of the existing protocolGet() command was required. Originally the idea was to share the function for local and remote requests but the differences (as in Perl) were too great to make that practical.
2019-02-27 22:34:21 +02:00
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
TEST_ASSIGN(client, protocolLocalGet(protocolStorageTypeRepo, 1), "get local protocol");
TEST_RESULT_PTR(protocolLocalGet(protocolStorageTypeRepo, 1), client, "get local cached protocol");
TEST_RESULT_PTR(protocolHelper.clientLocal[0].client, client, "check location in cache");
TEST_RESULT_VOID(protocolFree(), "free local and remote protocol objects");
Add ProtocolClient object and helper functions. This is a complete protocol client implementation in C. Currently there is no C server implementation so the C client is talking to a Perl server. This won't work very long, though, as the protocol format, even though in JSON, has a lot of language-specific structure. While it would be possible to maintain compatibility between C and Perl it's probably not worth the effort in the long run. Just as in Perl there are helper functions to make constructing protocol objects easier. Currently only repository remotes are supported.
2019-01-18 21:32:51 +02:00
}
FUNCTION_HARNESS_RESULT_VOID();
}
Reference in New Issue Copy Permalink