db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-07 00:35:37 +02:00
Code Issues Releases Activity
Files
c7a66ac1afb9a55b049dc18493b602b7bf62bda4
pgbackrest/src/protocol/parallelJob.c

143 lines
4.9 KiB
C
Raw Normal View History

Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
/***********************************************************************************************************************************
Protocol Parallel Job
***********************************************************************************************************************************/
Add configure script for improved multi-platform support. Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile. Each C file must include "build.auto.conf" before all other includes and defines. This is enforced by test.pl for includes, but it won't detect incorrect define ordering. Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
#include "build.auto.h"
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
#include "common/debug.h"
#include "common/log.h"
#include "protocol/command.h"
#include "protocol/parallelJob.h"
/***********************************************************************************************************************************
Object type
***********************************************************************************************************************************/
struct ProtocolParallelJob
{
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
ProtocolParallelJobPub pub; // Publicly accessible variables
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
};
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
ProtocolParallelJob *
protocolParallelJobNew(const Variant *key, ProtocolCommand *command)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(VARIANT, key);
FUNCTION_LOG_PARAM(PROTOCOL_COMMAND, command);
FUNCTION_LOG_END();
ProtocolParallelJob *this = NULL;
Improve memory usage of mem contexts. Each mem context can track child contexts, allocations, and a callback. Before this change memory was allocated for tracking all three even if they were not used for a particular context. This made mem contexts unsuitable for String and Variant objects since they are plentiful and need to be as small as possible. This change allows mem contexts to be configured to track any combination of child contexts, allocations, and a callback. In addition, the mem context can be configured to track a single child context and/or allocation, which saves memory and is a common use case. Another benefit is that Variants can own objects (e.g. KeyValue) that they encapsulate. All of this makes memory accounting simpler because mem contexts have names while allocations do not. No more memory is used than before since Variants and Strings still had to store the memory context they were originally allocated in so they could be easily freed. Update the String and Variant objects to use this new functionality. The custom strFree() and varFree() functions are no longer required and can now be a wrapper around objFree(). Lastly, this will allow strMove() and varMove() to be implemented and used in cases where strDup() and varDup() are being used to move a String or Variant to a new context. Since this will be a bit noisy it is saved for a future commit.
2022-05-18 10:52:01 -04:00
OBJ_NEW_BEGIN(ProtocolParallelJob, .childQty = MEM_CONTEXT_QTY_MAX)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
this = OBJ_NEW_ALLOC();
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*this = (ProtocolParallelJob)
{
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
.pub =
{
.state = protocolParallelJobStatePending,
.key = varDup(key),
},
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
};
Designated initializer cleanup. Cleanup designated initializers created in b134175f by moving struct members in or out for clarity.
2020-01-27 17:50:07 -07:00
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
this->pub.command = protocolCommandMove(command, objMemContext(this));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
OBJ_NEW_END();
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL_JOB, this);
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
void
protocolParallelJobErrorSet(ProtocolParallelJob *this, int code, const String *message)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this);
FUNCTION_LOG_PARAM(INT, code);
FUNCTION_LOG_PARAM(STRING, message);
FUNCTION_LOG_END();
ASSERT(this != NULL);
ASSERT(code != 0);
ASSERT(message != NULL);
Add MEM_CONTEXT_OBJ_*() macros. These provide a standard way to switch to an object mem context. Update the two different patterns that were used before to the new macros.
2022-04-25 09:12:25 -04:00
MEM_CONTEXT_OBJ_BEGIN(this)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
this->pub.code = code;
this->pub.message = strDup(message);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Add MEM_CONTEXT_OBJ_*() macros. These provide a standard way to switch to an object mem context. Update the two different patterns that were used before to the new macros.
2022-04-25 09:12:25 -04:00
MEM_CONTEXT_OBJ_END();
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_RETURN_VOID();
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add process id to ProtocolParallelJob. This exposes the process that performed to job to the caller for logging purposes.
2019-04-09 11:01:35 -04:00
void
protocolParallelJobProcessIdSet(ProtocolParallelJob *this, unsigned int processId)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this);
FUNCTION_LOG_PARAM(UINT, processId);
FUNCTION_LOG_END();
ASSERT(this != NULL);
ASSERT(processId > 0);
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
this->pub.processId = processId;
Add process id to ProtocolParallelJob. This exposes the process that performed to job to the caller for logging purposes.
2019-04-09 11:01:35 -04:00
FUNCTION_LOG_RETURN_VOID();
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
void
Binary protocol. Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol. There are a number advantages: * The pack type is more compact than JSON and are more efficient to render/parse. * Packs are more strictly typed than JSON. * Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state. * There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files. Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C. There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
protocolParallelJobResultSet(ProtocolParallelJob *const this, PackRead *const result)
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this);
Binary protocol. Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol. There are a number advantages: * The pack type is more compact than JSON and are more efficient to render/parse. * Packs are more strictly typed than JSON. * Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state. * There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files. Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C. There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
FUNCTION_LOG_PARAM(PACK_READ, result);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_END();
ASSERT(this != NULL);
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
ASSERT(protocolParallelJobErrorCode(this) == 0);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
Allow additional memory to be allocated with a mem context. The primary benefit is that objects can allocate memory for their struct with the context, which saves an additional allocation and makes it easier to read context/allocation dumps. Also, the memory context does not need to be stored with the object since it can be determined using the object pointer. Object pointers cannot be moved, so this means whatever additional memory is allocated cannot be resized. That makes the additional memory ideal for object structs, but not so much for allocating a list that might change size. Mem contexts can no longer be reused since they will probably be the wrong size so their memory is freed on memContextFree(). This still means fewer allocations and frees overall. Interfaces still need to be freed by mem context so the old objMove() and objFree() have been preserved as objMoveContext() and objFreeContext(). This will be addressed in a future commit.
2021-09-01 11:10:35 -04:00
this->pub.result = pckReadMove(result, objMemContext(this));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_RETURN_VOID();
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
void
protocolParallelJobStateSet(ProtocolParallelJob *this, ProtocolParallelJobState state)
{
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this);
Convert ProtocolParallelJobState enum to StringId. Allows removal of protocolParallelJobToConstZ(), which was used only for debugging.
2021-04-28 11:43:08 -04:00
FUNCTION_LOG_PARAM(STRING_ID, state);
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
FUNCTION_LOG_END();
ASSERT(this != NULL);
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
if (this->pub.state == protocolParallelJobStatePending && state == protocolParallelJobStateRunning)
this->pub.state = protocolParallelJobStateRunning;
else if (this->pub.state == protocolParallelJobStateRunning && state == protocolParallelJobStateDone)
this->pub.state = protocolParallelJobStateDone;
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
else
{
THROW_FMT(
Convert ProtocolParallelJobState enum to StringId. Allows removal of protocolParallelJobToConstZ(), which was used only for debugging.
2021-04-28 11:43:08 -04:00
AssertError, "invalid state transition from '%s' to '%s'", strZ(strIdToStr(protocolParallelJobState(this))),
strZ(strIdToStr(state)));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
FUNCTION_LOG_RETURN_VOID();
}
String *
protocolParallelJobToLog(const ProtocolParallelJob *this)
{
return strNewFmt(
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
"{state: %s, key: %s, command: %s, code: %d, message: %s, result: %s}",
Convert ProtocolParallelJobState enum to StringId. Allows removal of protocolParallelJobToConstZ(), which was used only for debugging.
2021-04-28 11:43:08 -04:00
strZ(strIdToStr(protocolParallelJobState(this))), strZ(varToLog(protocolParallelJobKey(this))),
Refactor protocol modules with inline getters/setters. Extend the pattern introduced in 79a2d02c to the protocol modules.
2021-04-08 16:46:42 -04:00
strZ(protocolCommandToLog(protocolParallelJobCommand(this))), protocolParallelJobErrorCode(this),
Binary protocol. Switch from JSON-based to binary protocol for communicating with local and remote process. The pack type is used to implement the binary protocol. There are a number advantages: * The pack type is more compact than JSON and are more efficient to render/parse. * Packs are more strictly typed than JSON. * Each protocol message is written entirely within ProtocolServer/ProtocolClient so is less likely to get interrupted by an error and leave the protocol in a bad state. * There is no limit on message size. Previously this was limited by buffer size without a custom implementation, as was done for read/writing files. Some cruft from the Perl days was removed, specifically allowing NULL messages and stack traces. This is no longer possible in C. There is room for improvement here, in particular locking down the allowed sequence of protocol messages and building a state machine to enforce it. This will be useful for resetting the protocol when it gets in a bad state.
2021-06-24 13:31:16 -04:00
strZ(strToLog(protocolParallelJobErrorMessage(this))),
protocolParallelJobResult(this) == NULL ? NULL_Z : strZ(pckReadToLog(protocolParallelJobResult(this))));
Add ProtocolParallel* objects for parallelizing commands. Allows commands to be easily parallelized if the jobs are broken up into discrete, non-overlapping chunks.
2019-02-27 21:10:52 +02:00
}
Reference in New Issue Copy Permalink