db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Code Issues Releases Activity
Files
ec173f12fb9c26553b2021473dd62d73a006835a
pgbackrest/test/src/common/harnessInfo.c

123 lines
4.6 KiB
C
Raw Normal View History

Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
/***********************************************************************************************************************************
Harness for Loading Test Configurations
***********************************************************************************************************************************/
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
#include <string.h>
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
#include "common/assert.h"
#include "common/crypto/hash.h"
#include "common/io/bufferRead.h"
#include "common/io/filter/filter.intern.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
#include "common/type/json.h"
#include "info/info.h"
#include "version.h"
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
#include "common/harnessDebug.h"
#include "common/harnessInfo.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
/***********************************************************************************************************************************
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
Add header and checksum to an info file
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
This prevents churn in headers and checksums in the unit tests. We purposefully do not use the checksum macros from the info module
here as a cross-check of that code.
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
***********************************************************************************************************************************/
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
typedef struct HarnessInfoChecksumData
{
MemContext *memContext; // Mem context to use for storing data in this structure
String *sectionLast; // The last section seen during load
IoFilter *checksum; // Checksum calculated from the file
} HarnessInfoChecksumData;
static void
harnessInfoChecksumCallback(void *callbackData, const String *section, const String *key, const String *value)
{
HarnessInfoChecksumData *data = (HarnessInfoChecksumData *)callbackData;
// Calculate checksum
if (data->sectionLast == NULL || !strEq(section, data->sectionLast))
{
if (data->sectionLast != NULL)
ioFilterProcessIn(data->checksum, BUFSTRDEF("},"));
ioFilterProcessIn(data->checksum, BUFSTRDEF("\""));
ioFilterProcessIn(data->checksum, BUFSTR(section));
ioFilterProcessIn(data->checksum, BUFSTRDEF("\":{"));
MEM_CONTEXT_BEGIN(data->memContext)
{
data->sectionLast = strDup(section);
}
MEM_CONTEXT_END();
}
else
ioFilterProcessIn(data->checksum, BUFSTRDEF(","));
Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Use jsonFromStr() to properly quote and escape \. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.
2019-12-13 21:33:13 -05:00
ioFilterProcessIn(data->checksum, BUFSTR(jsonFromStr(key)));
ioFilterProcessIn(data->checksum, BUFSTRDEF(":"));
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
ioFilterProcessIn(data->checksum, BUFSTR(value));
}
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
Buffer *
harnessInfoChecksum(const String *info)
{
FUNCTION_HARNESS_BEGIN();
FUNCTION_HARNESS_PARAM(STRING, info);
FUNCTION_HARNESS_END();
Buffer *result = NULL;
MEM_CONTEXT_TEMP_BEGIN()
{
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
// Initialize callback data
HarnessInfoChecksumData data =
{
.memContext = MEM_CONTEXT_TEMP(),
.checksum = cryptoHashNew(HASH_TYPE_SHA1_STR),
};
// Create buffer with space for data, header, and checksum
result = bufNew(strSize(info) + 256);
bufCat(result, BUFSTRDEF("[backrest]\nbackrest-format="));
bufCat(result, BUFSTR(jsonFromUInt(REPOSITORY_FORMAT)));
bufCat(result, BUFSTRDEF("\nbackrest-version="));
bufCat(result, BUFSTR(jsonFromStr(STRDEF(PROJECT_VERSION))));
bufCat(result, BUFSTRDEF("\n\n"));
bufCat(result, BUFSTR(info));
// Generate checksum by loading ini file
ioFilterProcessIn(data.checksum, BUFSTRDEF("{"));
iniLoad(ioBufferReadNew(result), harnessInfoChecksumCallback, &data);
ioFilterProcessIn(data.checksum, BUFSTRDEF("}}"));
// Append checksum to buffer
bufCat(result, BUFSTRDEF("\n[backrest]\nbackrest-checksum="));
Remove pretty-printing from jsonFromKv() and jsonFromVar(). Now that pretty-printing has been removed from the info command it no longer has a purpose, so remove it.
2019-10-11 13:03:52 -04:00
bufCat(result, BUFSTR(jsonFromVar(ioFilterResult(data.checksum))));
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
bufCat(result, BUFSTRDEF("\n"));
Add MEM_CONTEXT_PRIOR() block and update current call sites. This macro block encapsulates the common pattern of switching to the prior (formerly called old) mem context to return results from a function. Also rename MEM_CONTEXT_OLD() to memContextPrior(). This violates our convention of macros being in all caps but memContextPrior() will become a function very soon so this will reduce churn.
2020-01-17 13:29:49 -07:00
bufMove(result, memContextPrior());
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
}
MEM_CONTEXT_TEMP_END();
FUNCTION_HARNESS_RESULT(BUFFER, result);
}
Buffer *
harnessInfoChecksumZ(const char *info)
{
FUNCTION_HARNESS_BEGIN();
FUNCTION_HARNESS_PARAM(STRINGZ, info);
FUNCTION_HARNESS_END();
Improve performance of info file load/save. Info files required three copies in memory to be loaded (the original string, an ini representation, and the final info object). Not only was this memory inefficient but the Ini object does sequential scans when searching for keys making large files very slow to load. This has not been an issue since archive.info and backup.info are very small, but it becomes a big deal when loading manifests with hundreds of thousands of files. Instead of holding copies of the data in memory, use a callback to deliver the ini data directly to the object when loading. Use a similar method for save to avoid having an intermediate copy. Save is a bit complex because sections/keys must be written in alpha order or older versions of pgBackRest will not calculate the correct checksum. Also move the load retry logic to helper functions rather than embedding it in the Info object. This allows for more flexibility in loading and ensures that stack traces will be available when developing unit tests. Reviewed by Cynthia Shang.
2019-09-06 13:48:28 -04:00
FUNCTION_HARNESS_RESULT(BUFFER, harnessInfoChecksum(STR(info)));
}
/***********************************************************************************************************************************
Test callback that logs the results to a string
***********************************************************************************************************************************/
void
harnessInfoLoadNewCallback(void *callbackData, const String *section, const String *key, const String *value)
{
if (callbackData != NULL)
strCatFmt((String *)callbackData, "[%s] %s=%s\n", strPtr(section), strPtr(key), strPtr(value));
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
}
Reference in New Issue Copy Permalink