db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Code Issues Releases Activity
Files
95fb002b85897e71820c44519231975d883f87db
pgbackrest/src/command/command.c

244 lines
10 KiB
C
Raw Normal View History

Add command and exit modules.
2018-01-17 09:15:51 -05:00
/***********************************************************************************************************************************
Common Command Routines
***********************************************************************************************************************************/
Add configure script for improved multi-platform support. Use autoconf to provide a basic configure script. WITH_BACKTRACE is yet to be migrated to configure and the unit tests still use a custom Makefile. Each C file must include "build.auto.conf" before all other includes and defines. This is enforced by test.pl for includes, but it won't detect incorrect define ordering. Update packages to call configure and use standard flags to pass options.
2019-04-26 08:08:23 -04:00
#include "build.auto.h"
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
#include <inttypes.h>
Add command and exit modules.
2018-01-17 09:15:51 -05:00
#include <string.h>
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
#include "common/debug.h"
Add command and exit modules.
2018-01-17 09:15:51 -05:00
#include "common/log.h"
#include "common/memContext.h"
Add general-purpose statistics collector. Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome. Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually. No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible. The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
#include "common/stat.h"
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
#include "common/time.h"
Add general-purpose statistics collector. Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome. Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually. No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible. The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
#include "common/type/json.h"
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
#include "config/config.intern.h"
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
#include "config/parse.h"
Add command and exit modules.
2018-01-17 09:15:51 -05:00
#include "version.h"
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
/***********************************************************************************************************************************
Track time command started
***********************************************************************************************************************************/
static TimeMSec timeBegin;
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
static String *cmdOptionStr;
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
cmdInit(void)
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_VOID(logLevelTrace);
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
timeBegin = timeMSec();
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN_VOID();
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
const String *
cmdOption(void)
Add command and exit modules.
2018-01-17 09:15:51 -05:00
{
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
FUNCTION_TEST_VOID();
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
if (cmdOptionStr == NULL)
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
{
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
MEM_CONTEXT_BEGIN(memContextTop())
Add command and exit modules.
2018-01-17 09:15:51 -05:00
{
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
cmdOptionStr = strNew("");
Add command and exit modules.
2018-01-17 09:15:51 -05:00
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
MEM_CONTEXT_TEMP_BEGIN()
Add command and exit modules.
2018-01-17 09:15:51 -05:00
{
Show command parameters as well as command options in initial info message.
2018-04-17 18:47:14 -04:00
// Add command parameters if they exist
const StringList *commandParamList = cfgCommandParam();
Add lstEmpty(), strLstEmpty(), and varLstEmpty(). This seems more readable than lst*Size() == 0. Hopefully this will also eliminate usage of lst*Size() > 0/lst*Size() != 0 variants for the inverse.
2021-01-29 14:27:56 -05:00
if (!strLstEmpty(commandParamList))
Show command parameters as well as command options in initial info message.
2018-04-17 18:47:14 -04:00
{
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
strCatFmt(cmdOptionStr, " [");
Show command parameters as well as command options in initial info message.
2018-04-17 18:47:14 -04:00
for (unsigned int commandParamIdx = 0; commandParamIdx < strLstSize(commandParamList); commandParamIdx++)
{
const String *commandParam = strLstGet(commandParamList, commandParamIdx);
if (commandParamIdx != 0)
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
strCatFmt(cmdOptionStr, ", ");
Show command parameters as well as command options in initial info message.
2018-04-17 18:47:14 -04:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
if (strchr(strZ(commandParam), ' ') != NULL)
commandParam = strNewFmt("\"%s\"", strZ(commandParam));
Show command parameters as well as command options in initial info message.
2018-04-17 18:47:14 -04:00
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
strCat(cmdOptionStr, commandParam);
Show command parameters as well as command options in initial info message.
2018-04-17 18:47:14 -04:00
}
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
strCatFmt(cmdOptionStr, "]");
Show command parameters as well as command options in initial info message.
2018-04-17 18:47:14 -04:00
}
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
// Loop though options and add the ones that are interesting
for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++)
Add command and exit modules.
2018-01-17 09:15:51 -05:00
{
Filter logged command options based on the command definition. Previously, options were being filtered based on what was currently valid. For chained commands (e.g. backup then expire) some options may be valid for the first command but not the second. Filter based on the command definition rather than what is currently valid to avoid logging options that are not valid for subsequent commands. This reduces the number of options logged and will hopefully help avoid confusion and expect log churn.
2019-11-14 16:48:41 -05:00
// Skip the option if not valid for this command. Generally only one command runs at a time, but sometimes
// commands are chained together (e.g. backup and expire) and the second command may not use all the options of
// the first command. Displaying them is harmless but might cause confusion.
Allow option validity to be determined by command role. Validity by command was not granular enough so numerous options needed be marked internal so users would not stumble across them. Options were also needlessly being passed to roles that had no use for them. Introduce per-role validity lists that depend on what roles are valid per command. Also add a check to ensure that only valid roles are used with a command. This commit adds the functionality but does not introduce any new behavior, i.e. all options are valid for all roles that the command is valid for. A subsequent commit will introduce the new role restrictions to make the changes easier to audit.
2020-12-28 09:43:23 -05:00
if (!cfgOptionValid(optionId) || !cfgParseOptionValid(cfgCommand(), cfgCommandRole(), optionId))
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
continue;
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
// Loop through option indexes
unsigned int optionIdxTotal = cfgOptionGroup(optionId) ? cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)) : 1;
for (unsigned int optionIdx = 0; optionIdx < optionIdxTotal; optionIdx++)
Add command and exit modules.
2018-01-17 09:15:51 -05:00
{
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
// If option was negated
if (cfgOptionIdxNegate(optionId, optionIdx))
strCatFmt(cmdOptionStr, " --no-%s", cfgOptionIdxName(optionId, optionIdx));
// If option was reset
else if (cfgOptionIdxReset(optionId, optionIdx))
strCatFmt(cmdOptionStr, " --reset-%s", cfgOptionIdxName(optionId, optionIdx));
// Else not default
else if (cfgOptionIdxSource(optionId, optionIdx) != cfgSourceDefault)
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
{
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
// Don't show redacted options
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
if (cfgParseOptionSecure(optionId))
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
strCatFmt(cmdOptionStr, " --%s=<redacted>", cfgOptionIdxName(optionId, optionIdx));
// Output boolean option
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
else if (cfgParseOptionType(optionId) == cfgOptTypeBoolean)
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
strCatFmt(cmdOptionStr, " --%s", cfgOptionIdxName(optionId, optionIdx));
// Output other options
else
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
{
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
StringList *valueList = NULL;
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
// Generate the values of hash options
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
if (cfgParseOptionType(optionId) == cfgOptTypeHash)
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
{
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
valueList = strLstNew();
const KeyValue *optionKv = cfgOptionIdxKv(optionId, optionIdx);
const VariantList *keyList = kvKeyList(optionKv);
for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++)
{
strLstAdd(
valueList,
strNewFmt(
"%s=%s", strZ(varStr(varLstGet(keyList, keyIdx))),
strZ(varStrForce(kvGet(optionKv, varLstGet(keyList, keyIdx))))));
}
}
// Generate values for list options
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
else if (cfgParseOptionType(optionId) == cfgOptTypeList)
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
{
valueList = strLstNewVarLst(cfgOptionIdxLst(optionId, optionIdx));
}
Replace double type with time in config module. The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand. Also remove Variant doubles since they are no longer used. Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
// Generate time value
Move all parse-related rules to parse module. Data required for parsing was spread between the config and defined modules, mostly for historical reasons because the same data was used by Perl. Requiring all the parse rules to be accessed with function interfaces makes the code more complicated and new rules harder to implement. Instead, move the data to the parse module so in the most complex cases no interface functions are needed. This reduces the total amount of code and paves the way for more complex parse rules.
2020-12-17 09:32:31 -05:00
else if (cfgParseOptionType(optionId) == cfgOptTypeTime)
Replace double type with time in config module. The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand. Also remove Variant doubles since they are no longer used. Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
{
valueList = strLstNew();
Move cvtDoubleToStr() to strNewDbl(). This is a more logical location and it reduces the dependencies required to compile the common/convert module.
2021-01-27 11:50:10 -05:00
strLstAdd(valueList, strNewDbl((double)cfgOptionIdxInt64(optionId, optionIdx) / MSEC_PER_SEC));
Replace double type with time in config module. The C code does not use doubles to represent seconds like the Perl code did so time can be represented as an integer which reduces the number of data types that config has to understand. Also remove Variant doubles since they are no longer used. Note that not all double code was removed since we still need to display times to the user in seconds and it is possible for the times to be fractional. In the future this will likely be simplified by storing the original user input and using that value when the time needs to be displayed.
2020-12-09 08:59:51 -05:00
}
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
// Else only one value
else
{
valueList = strLstNew();
strLstAdd(valueList, varStrForce(cfgOptionIdx(optionId, optionIdx)));
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
}
Add command and exit modules.
2018-01-17 09:15:51 -05:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
// Output options and values
for (unsigned int valueListIdx = 0; valueListIdx < strLstSize(valueList); valueListIdx++)
{
const String *value = strLstGet(valueList, valueListIdx);
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
strCatFmt(cmdOptionStr, " --%s", cfgOptionIdxName(optionId, optionIdx));
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
if (strchr(strZ(value), ' ') != NULL)
value = strNewFmt("\"%s\"", strZ(value));
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
Prepare configuration module for multi-repository support. Refactor the code to allow a dynamic number of indexes for indexed options, e.g. pg-path. Our reliance on getopt_long() still limits the number of indexes we can have per group, but once this limitation is removed the rest of the code should be happy with dynamic numbers of indexes (with a reasonable maximum). Add an option to set a default in each group. This was previously handled by the host-id option but now there is a specific option for each group, pg and repo. These remain internal until they can be fully tested with multi-repo support. They are fully tested for internal usage. Remove the ConfigDefineOption enum and use the ConfigOption enum instead. They are now equal since the indexed options (e.g. cfgOptRepoHost2) have been removed from ConfigOption. Remove the config/config test module and add required tests to the config/parse test module. Parsing is now the only way to load a config so this removes some redundancy. Split new internal config structures and functions into a new header file, config.intern.h. More functions will need to be moved over from config.h but that will need to be done in a future commit to reduce churn. Add repoIdx to repoIsLocal() and storageRepo*(). Multi-repository support requires that repo locality and storage be accessible by index. This allows, for example, multiple repos to be iterated in a loop. This could be done in a separate commit but doesn't seem worth it since the code is related. Remove the type parameter from storageRepoGet(). This parameter existed solely to provide coverage for the case where the storage type was invalid. A better pattern is to check that the type is S3 once all other types have been ruled out.
2020-11-23 15:55:46 -05:00
strCatFmt(cmdOptionStr, "=%s", strZ(value));
}
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
}
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
}
Add command and exit modules.
2018-01-17 09:15:51 -05:00
}
}
}
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
MEM_CONTEXT_TEMP_END();
}
MEM_CONTEXT_END();
}
FUNCTION_TEST_RETURN(cmdOptionStr);
}
/**********************************************************************************************************************************/
void
Remove unused parameter from cmdBegin().
2020-08-20 14:16:36 -04:00
cmdBegin(void)
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
{
Remove unused parameter from cmdBegin().
2020-08-20 14:16:36 -04:00
FUNCTION_LOG_VOID(logLevelTrace);
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
ASSERT(cfgCommand() != cfgCmdNone);
// This is fairly expensive log message to generate so skip it if it won't be output
if (logAny(cfgLogLevelDefault()))
{
MEM_CONTEXT_TEMP_BEGIN()
{
// Basic info on command start
String *info = strNewFmt("%s command begin", strZ(cfgCommandRoleName()));
Remove unused parameter from cmdBegin().
2020-08-20 14:16:36 -04:00
// Free the old option string if it exists. This is needed when more than one command is run in a row so an option
// string gets created for the new command.
strFree(cmdOptionStr);
cmdOptionStr = NULL;
Refactor option logging into a general-purpose function.
2020-08-20 14:11:40 -04:00
Remove unused parameter from cmdBegin().
2020-08-20 14:16:36 -04:00
// Add version and options
strCatFmt(info, " %s:%s", PROJECT_VERSION, strZ(cmdOption()));
Add command and exit modules.
2018-01-17 09:15:51 -05:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
LOG(cfgLogLevelDefault(), 0, strZ(info));
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
}
MEM_CONTEXT_TEMP_END();
Add command and exit modules.
2018-01-17 09:15:51 -05:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN_VOID();
Add command and exit modules.
2018-01-17 09:15:51 -05:00
}
Move extern function comments to headers. This has been the policy for some time but due to migration pressure only new functions and refactors have been following this rule. Now it seems sensible to make a clean sweep and move all the comments that have not been moved already (i.e. most of them). Only obvious typos and gross inaccuracies in the comments have been fixed. For this most part this was a copy and paste operation. Useless comments, e.g. "New object", were not copied. Even so, there are surely many deficient comments left. Some rearranging was done where needed and functions were placed in the proper sections, e.g. "Constructors", "Functions", etc. A few function prototypes were found that not longer had an implementation. These were removed, but there may be more. The coding document has been updated to reflect this policy, which is not new but has never been documented.
2020-04-03 18:01:28 -04:00
/**********************************************************************************************************************************/
Fix non-compliant formatting for function declarations. Per review by Cynthia Shang.
2018-01-31 18:22:25 -05:00
void
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
cmdEnd(int code, const String *errorMessage)
Add command and exit modules.
2018-01-17 09:15:51 -05:00
{
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_BEGIN(logLevelTrace);
FUNCTION_LOG_PARAM(INT, code);
FUNCTION_LOG_PARAM(STRING, errorMessage);
FUNCTION_LOG_END();
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
ASSERT(cfgCommand() != cfgCmdNone);
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
// Skip this log message if it won't be output. It's not too expensive but since we skipped cmdBegin(), may as well.
Improve log performance, simplify macros, rename logWill() to logAny(). Pre-calculate the value used by logAny() to improve performance and make it more likely to be inlined. Move IF_LOG_ANY() into LOG_INTERNAL() to simplify the macros and improve performance of LOG() and LOG_PID(). If the message has no chance of being logged there's no reason to call logInternal(). Rename logWill() to logAny() because it seems more intuitive.
2019-05-11 18:20:57 -04:00
if (logAny(cfgLogLevelDefault()))
Add command and exit modules.
2018-01-17 09:15:51 -05:00
{
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
MEM_CONTEXT_TEMP_BEGIN()
{
Add general-purpose statistics collector. Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome. Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually. No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible. The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
// Output statistics if there are any
const KeyValue *statKv = statToKv();
Add SocketClient object. This functionality was embedded into TlsClient but that was starting to get unwieldy. Add SocketClient to contain all socket-related client functionality.
2020-03-31 12:43:29 -04:00
Add lstEmpty(), strLstEmpty(), and varLstEmpty(). This seems more readable than lst*Size() == 0. Hopefully this will also eliminate usage of lst*Size() > 0/lst*Size() != 0 variants for the inverse.
2021-01-29 14:27:56 -05:00
if (!varLstEmpty(kvKeyList(statKv)))
Add general-purpose statistics collector. Currently each module that needs to collect statistics implements custom code to do so. This is cumbersome. Create a general purpose module for collecting and reporting statistics. Statistics are output in the log at detail level, but there are other uses they could be put to eventually. No new functionality is added. This is just a drop-in replacement for the current statistics, with the advantage of being more flexible. The new stats are slower because they involve a list lookup, but performance testing shows stats can be updated at about 40,000/ms which seems fast enough for our purposes.
2020-08-20 14:04:26 -04:00
LOG_DETAIL_FMT("statistics: %s", strZ(jsonFromKv(statKv)));
Move tls/http statistics output to command/command. This module already has the filtering required to keep these messages from being displayed by default for commands that output to stdout (e.g. info).
2019-05-28 09:50:59 -04:00
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
// Basic info on command end
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
String *info = strNewFmt("%s command end: ", strZ(cfgCommandRoleName()));
Add command and exit modules.
2018-01-17 09:15:51 -05:00
Fix incorrectly reported error return in info logging. A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly. Also improve info messages when an archive segment is or is not found.
2018-09-04 21:46:41 -04:00
if (errorMessage == NULL)
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
{
Add strCatZ(). strCat() did not follow our convention of appending Z to functions that accept zero-terminated strings rather than String objects. Add strCatZ() to accept zero-terminated strings and update strCat() to accept String objects. Use LF_STR where appropriate but don't use other String constants because they do not improve readability.
2020-06-24 12:09:24 -04:00
strCatZ(info, "completed successfully");
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
Use harnessCfgLoadRaw() in command/command and common/exit unit tests. The tests were originally written by loading values directly into the configuration before the parser was available. Update to use harnessCfgLoadRaw() to simplify the tests and make them compatible with upcoming config changes. Note that some unreachable conditions were removed since they could not be reached via a parsed config, only by munging values directly into the config. cfgOptionTest(optionId) was removed because a non-default value must always be set. cfgOptionValid(cfgOptLogTimestamp) was removed because it is true for all commands except for cfgCmdNone, which is checked with an assert.
2020-10-20 14:54:28 -04:00
if (cfgOptionBool(cfgOptLogTimestamp))
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
strCatFmt(info, " (%" PRIu64 "ms)", timeMSec() - timeBegin);
}
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
else
Add strCatZ(). strCat() did not follow our convention of appending Z to functions that accept zero-terminated strings rather than String objects. Add strCatZ() to accept zero-terminated strings and update strCat() to accept String objects. Use LF_STR where appropriate but don't use other String constants because they do not improve readability.
2020-06-24 12:09:24 -04:00
strCat(info, errorMessage);
Add command and exit modules.
2018-01-17 09:15:51 -05:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
LOG(cfgLogLevelDefault(), 0, strZ(info));
Improve logging. Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 10:55:58 -04:00
}
MEM_CONTEXT_TEMP_END();
Add command and exit modules.
2018-01-17 09:15:51 -05:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Print total runtime in milliseconds at command end.
2018-05-18 12:03:39 -04:00
// Reset timeBegin in case there is another command following this one
timeBegin = timeMSec();
Rename FUNCTION_DEBUG_* and consolidate ASSERT_* macros for consistency. Rename FUNCTION_DEBUG_* macros to FUNCTION_LOG_* to more accurately reflect what they do. Further rename FUNCTION_DEBUG_RESULT* macros to FUNCTION_LOG_RETURN* to make it clearer that they return from the function as well as logging. Leave FUNCTION_TEST_* macros as they are. Consolidate the various ASSERT* macros into a single ASSERT macro that is always compiled out of production builds. It was difficult to figure out when an assert would be checked with all the different types in play. When ASSERTs are compiled in they will always be checked regardless of the log level -- tying these two concepts together was not a good idea.
2019-01-21 17:41:59 +02:00
FUNCTION_LOG_RETURN_VOID();
Add command and exit modules.
2018-01-17 09:15:51 -05:00
}
Reference in New Issue Copy Permalink