db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-04-21 11:57:01 +02:00
Code Issues Releases Activity
pgbackrest/src/protocol/parallel.c

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

281 lines
11 KiB
C
Raw Normal View History

Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
/***********************************************************************************************************************************
Protocol Parallel Executor
***********************************************************************************************************************************/
Add configure script for improved multi-platform support. Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile. Each C file must include "build.auto.conf" before all other includes and defines. This is enforced by test.pl for includes, but it won't detect incorrect define ordering. Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
#include "build.auto.h"
Add STR() macro to create constant String objects from runtime strings. The STRING_CONST() macro worked fine for constants but was not able to constify strings created at runtime. Add the STR() macro to do this by using strlen() to get the size. Also rename STRING_CONST() to STRDEF() for brevity and to match the other macro name.
2019-04-16 13:39:58 -04:00
#include <string.h>
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
#include <sys/select.h>
#include "common/debug.h"
#include "common/log.h"
Replace OBJECT_DEFINE_MOVE() and OBJECT_DEFINE_FREE() with inlines. Inline functions are more efficient and if they are not used are automatically omitted from the binary. This also makes the implementation of these functions easier to find and removes the need for a declaration. That is, the complete implementation is located in the header rather than being spread between the header and C file.
2021-04-08 10:04:57 -04:00
#include "common/macro.h"
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
#include "common/type/keyValue.h"
#include "common/type/list.h"
#include "protocol/command.h"
Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. Also fix the related issue that the local processes were not being shut down when they completed, which meant that they might timeout before being closed when pgbackrest terminated.
2020-07-28 12:15:33 -04:00
#include "protocol/helper.h"
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
#include "protocol/parallel.h"
/***********************************************************************************************************************************
Object type
***********************************************************************************************************************************/
struct ProtocolParallel
{
TimeMSec timeout; // Max time to wait for jobs before returning
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
ParallelJobCallback *callbackFunction; // Function to get new jobs
void *callbackData; // Data to pass to callback function
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
List *clientList; // List of clients to process jobs
List *jobList; // List of jobs to be processed
ProtocolParallelJob **clientJobList; // Jobs being processing by each client
ProtocolParallelJobState state; // Overall state of job processing
};
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN. This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can). Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
FN_EXTERN ProtocolParallel *
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
protocolParallelNew(TimeMSec timeout, ParallelJobCallback *callbackFunction, void *callbackData)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(UINT64, timeout);
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction);
FUNCTION_LOG_PARAM_P(VOID, callbackData);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_END();
Use a callback to feed jobs to ProtocolParallel. Loading jobs in advance uses a lot of memory in the case that there are millions of jobs to be performed. We haven't seen this yet, but with backup and restore on the horizon it will become the norm. Instead, use a callback so that jobs are only created as they are needed and can be freed as soon as they are completed.
2019-09-18 07:15:16 -04:00
ASSERT(callbackFunction != NULL);
ASSERT(callbackData != NULL);
Improve memory usage of mem contexts. Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible. This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case. Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed. Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree(). Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
OBJ_NEW_BEGIN(ProtocolParallel, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*this = (ProtocolParallel)
{
.timeout = timeout,
.callbackFunction = callbackFunction,
.callbackData = callbackData,
Remove lstNew() constructor in favor of lstNewP(). The old constructor was left around to reduce code churn during the migration but it just makes the code harder to read and search. Remove the old constructor and rename all remaining instances to lstNewP(), which by default has the same semantics.
2020-07-20 15:22:33 -04:00
.clientList = lstNewP(sizeof(ProtocolClient *)),
.jobList = lstNewP(sizeof(ProtocolParallelJob *)),
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
.state = protocolParallelJobStatePending,
};
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
OBJ_NEW_END();
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL, this);
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN. This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can). Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
FN_EXTERN void
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
protocolParallelClientAdd(ProtocolParallel *this, ProtocolClient *client)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this);
FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client);
FUNCTION_LOG_END();
ASSERT(this != NULL);
ASSERT(client != NULL);
ASSERT(this->state == protocolParallelJobStatePending);
Binary protocol. Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol. There are a number advantages: * The pack type is more compact than JSON and are more efficient to render/parse. * Packs are more strictly typed than JSON. * Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state. * There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files. Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C. There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
if (protocolClientIoReadFd(client) == -1)
Rename handle to fd. Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd. There were also plenty of places fd was used so now all uses are consistent. The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
THROW(AssertError, "client with read fd is required");
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
lstAdd(this->clientList, &client);
FUNCTION_LOG_RETURN_VOID();
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN. This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can). Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
FN_EXTERN unsigned int
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
protocolParallelProcess(ProtocolParallel *this)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this);
FUNCTION_LOG_END();
ASSERT(this != NULL);
ASSERT(this->state != protocolParallelJobStateDone);
unsigned int result = 0;
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
MEM_CONTEXT_TEMP_BEGIN()
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// If called for the first time, initialize processing
if (this->state == protocolParallelJobStatePending)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Add MEM_CONTEXT_OBJ_*() macros. These provide a standard way to switch to an object mem context. Update the two different patterns that were used before to the new macros.
2022-04-25 09:12:25 -04:00
MEM_CONTEXT_OBJ_BEGIN(this)
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
{
this->clientJobList = memNewPtrArray(lstSize(this->clientList));
}
Add MEM_CONTEXT_OBJ_*() macros. These provide a standard way to switch to an object mem context. Update the two different patterns that were used before to the new macros.
2022-04-25 09:12:25 -04:00
MEM_CONTEXT_OBJ_END();
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
this->state = protocolParallelJobStateRunning;
}
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Initialize the file descriptor set used for select
fd_set selectSet;
FD_ZERO(&selectSet);
int fdMax = -1;
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Find clients that are running jobs
unsigned int clientRunningTotal = 0;
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
if (this->clientJobList[clientIdx] != NULL)
{
int fd = protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx));
FD_SET(fd, &selectSet);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Find the max file descriptor needed for select()
MAX_ASSIGN(fdMax, fd);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
clientRunningTotal++;
}
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// If clients are running then wait for one to finish
if (clientRunningTotal > 0)
{
// Initialize timeout struct used for select. Recreate this structure each time since Linux (at least) will modify it.
struct timeval timeoutSelect;
timeoutSelect.tv_sec = (time_t)(this->timeout / MSEC_PER_SEC);
timeoutSelect.tv_usec = (suseconds_t)(this->timeout % MSEC_PER_SEC * 1000);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Determine if there is data to be read
int completed = select(fdMax + 1, &selectSet, NULL, NULL, &timeoutSelect);
THROW_ON_SYS_ERROR(completed == -1, AssertError, "unable to select from parallel client(s)");
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// If any jobs have completed then get the results
if (completed > 0)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
ProtocolParallelJob *job = this->clientJobList[clientIdx];
Binary protocol. Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol. There are a number advantages: * The pack type is more compact than JSON and are more efficient to render/parse. * Packs are more strictly typed than JSON. * Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state. * There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files. Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C. There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
if (job != NULL &&
FD_ISSET(
protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx)),
&selectSet))
{
MEM_CONTEXT_TEMP_BEGIN()
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
TRY_BEGIN()
{
ProtocolClient *const client = *(ProtocolClient **)lstGet(this->clientList, clientIdx);
protocolParallelJobResultSet(job, protocolClientDataGet(client));
protocolClientDataEndGet(client);
}
CATCH_ANY()
{
protocolParallelJobErrorSet(job, errorCode(), STR(errorMessage()));
}
TRY_END();
protocolParallelJobStateSet(job, protocolParallelJobStateDone);
this->clientJobList[clientIdx] = NULL;
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
MEM_CONTEXT_TEMP_END();
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
}
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
result = (unsigned int)completed;
}
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Find new jobs to be run
for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// If nothing is running for this client
if (this->clientJobList[clientIdx] == NULL)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Get a new job
ProtocolParallelJob *job = NULL;
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
MEM_CONTEXT_BEGIN(lstMemContext(this->jobList))
{
job = this->callbackFunction(this->callbackData, clientIdx);
}
MEM_CONTEXT_END();
// If a new job was found
if (job != NULL)
{
// Add to the job list
lstAdd(this->jobList, &job);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Put command
protocolClientCommandPut(
Add state to ProtocolClient. This is currently only useful for debugging, but in the future the state may be used for resetting the protocol when something goes wrong.
2021-10-05 14:06:59 -04:00
*(ProtocolClient **)lstGet(this->clientList, clientIdx), protocolParallelJobCommand(job), false);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
// Set client id and running state
protocolParallelJobProcessIdSet(job, clientIdx + 1);
protocolParallelJobStateSet(job, protocolParallelJobStateRunning);
this->clientJobList[clientIdx] = job;
}
// Else no more jobs for this client so free it
else
protocolLocalFree(clientIdx + 1);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
}
}
Fix more memory leaks introduced by the binary protocol in 6a1c0337. Either of these temp mem context blocks fixes the issue of command packs not being freed, but it seems like a good idea to have both in case the code changes.
2021-08-18 08:18:11 -04:00
MEM_CONTEXT_TEMP_END();
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_RETURN(UINT, result);
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN. This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can). Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
FN_EXTERN ProtocolParallelJob *
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
protocolParallelResult(ProtocolParallel *this)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this);
FUNCTION_LOG_END();
ASSERT(this != NULL);
ASSERT(this->state == protocolParallelJobStateRunning);
ProtocolParallelJob *result = NULL;
// Find the next completed job
for (unsigned int jobIdx = 0; jobIdx < lstSize(this->jobList); jobIdx++)
{
ProtocolParallelJob *job = *(ProtocolParallelJob **)lstGet(this->jobList, jobIdx);
if (protocolParallelJobState(job) == protocolParallelJobStateDone)
{
result = protocolParallelJobMove(job, memContextCurrent());
Rename lstRemove() to lstRemoveIdx(). This works better with the names of upcoming functions and seems sensible even in isolation.
2019-09-06 16:50:57 -04:00
lstRemoveIdx(this->jobList, jobIdx);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
break;
}
}
FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL_JOB, result);
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Split VR_EXTERN/FN_EXTERN macros from FV_EXTERN. This should make it a little clearer what the variable (VR) macros are doing since the declaration/definition cannot both be set to extern (but functions can). Splitting the variable macros out also allows them to be changed in the future with little churn, while changing the function macro creates a large amount of churn.
2023-01-02 15:24:51 +07:00
FN_EXTERN bool
Allow ProtocolParallel to complete with no jobs. If the callback never returned any jobs then protocolParallelDone() would never be true. The reason is that the done state was being set in protocolParallelResult(), which never gets called if there are no results. Calling protocolParallelResult() doesn't make much sense in this case so instead move the done logic to protocolParallelDone(). For current usage of ProtocolParallel we ensure there are jobs before processing so this is not a live issue, but the new behavior is required for future development.
2020-09-02 11:03:49 -04:00
protocolParallelDone(ProtocolParallel *this)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this);
FUNCTION_LOG_END();
Allow ProtocolParallel to complete with no jobs. If the callback never returned any jobs then protocolParallelDone() would never be true. The reason is that the done state was being set in protocolParallelResult(), which never gets called if there are no results. Calling protocolParallelResult() doesn't make much sense in this case so instead move the done logic to protocolParallelDone(). For current usage of ProtocolParallel we ensure there are jobs before processing so this is not a live issue, but the new behavior is required for future development.
2020-09-02 11:03:49 -04:00
ASSERT(this != NULL);
ASSERT(this->state != protocolParallelJobStatePending);
// If there are no jobs left then we are done
Add lstEmpty(), strLstEmpty(), and varLstEmpty(). This seems more readable than lst*Size() == 0. Hopefully this will also eliminate usage of lst*Size() > 0/lst*Size() != 0 variants for the inverse.
2021-01-29 14:27:56 -05:00
if (this->state != protocolParallelJobStateDone && lstEmpty(this->jobList))
Allow ProtocolParallel to complete with no jobs. If the callback never returned any jobs then protocolParallelDone() would never be true. The reason is that the done state was being set in protocolParallelResult(), which never gets called if there are no results. Calling protocolParallelResult() doesn't make much sense in this case so instead move the done logic to protocolParallelDone(). For current usage of ProtocolParallel we ensure there are jobs before processing so this is not a live issue, but the new behavior is required for future development.
2020-09-02 11:03:49 -04:00
this->state = protocolParallelJobStateDone;
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_RETURN(BOOL, this->state == protocolParallelJobStateDone);
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Refactor logging functions to never allocate memory. Allocating memory made these functions simpler but it meant that memory was leaking into the calling context when logging was enabled. It is not clear that this was an issue but it seems that trace level logging could result it a lot of memory usage depending on the use case. This also makes it possible to audit allocations returned to the calling context, which will be done in a followup commit. Also rename objToLog() to objNameToLog() since it seemed logical to name the new function objToLog().
2023-01-12 17:14:36 +07:00
FN_EXTERN void
protocolParallelToLog(const ProtocolParallel *const this, StringStatic *const debugLog)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Refactor logging functions to never allocate memory. Allocating memory made these functions simpler but it meant that memory was leaking into the calling context when logging was enabled. It is not clear that this was an issue but it seems that trace level logging could result it a lot of memory usage depending on the use case. This also makes it possible to audit allocations returned to the calling context, which will be done in a followup commit. Also rename objToLog() to objNameToLog() since it seemed logical to name the new function objToLog().
2023-01-12 17:14:36 +07:00
strStcCat(debugLog, "{state: ");
strStcResultSizeInc(debugLog, strIdToLog(this->state, strStcRemains(debugLog), strStcRemainsSize(debugLog)));
strStcFmt(debugLog, ", clientTotal: %u, jobTotal: %u}", lstSize(this->clientList), lstSize(this->jobList));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Reference in New Issue Copy Permalink