db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-03 00:26:59 +02:00
Code Issues Releases Activity
Files
3796b74dcac29d7e7e7f89b69d3fa92b9d105a17
pgbackrest/test/src/module/command/backupCommonTest.c

294 lines
14 KiB
C
Raw Normal View History

Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
/***********************************************************************************************************************************
Test Common Functions and Definitions for Backup and Expire Commands
***********************************************************************************************************************************/
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
#include "common/io/bufferWrite.h"
#include "common/regExp.h"
#include "common/type/json.h"
#include "postgres/interface.h"
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
#include "postgres/interface/static.auto.h"
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
#include "storage/posix/storage.h"
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
testRun(void)
{
FUNCTION_HARNESS_VOID();
// *****************************************************************************************************************************
if (testBegin("backupRegExp()"))
{
String *full = strNew("20181119-152138F");
String *incr = strNew("20181119-152138F_20181119-152152I");
String *diff = strNew("20181119-152138F_20181119-152152D");
// -------------------------------------------------------------------------------------------------------------------------
TEST_ERROR(
backupRegExpP(0),
AssertError, "assertion 'param.full || param.differential || param.incremental' failed");
// -------------------------------------------------------------------------------------------------------------------------
Remove errant tabs and fix spacing.
2019-12-19 16:25:46 -05:00
String *filter = backupRegExpP(.full = true);
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(filter, "^[0-9]{8}\\-[0-9]{6}F$", "full backup regex with anchors");
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
TEST_RESULT_BOOL(regExpMatchOne(filter, incr), false, " does not exactly match incr");
TEST_RESULT_BOOL(regExpMatchOne(filter, diff), false, " does not exactly match diff");
TEST_RESULT_BOOL(regExpMatchOne(filter, full), true, " exactly matches full");
// -------------------------------------------------------------------------------------------------------------------------
filter = backupRegExpP(.full = true, .incremental = true);
Add linefeeds for clarity.
2019-05-16 09:56:48 -04:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
filter, "^[0-9]{8}\\-[0-9]{6}F(\\_[0-9]{8}\\-[0-9]{6}I){0,1}$", "full and optional incr backup regex with anchors");
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
TEST_RESULT_BOOL(regExpMatchOne(filter, incr), true, " match incr");
TEST_RESULT_BOOL(regExpMatchOne(filter, diff), false, " does not match diff");
TEST_RESULT_BOOL(regExpMatchOne(filter, full), true, " match full");
TEST_RESULT_BOOL(
regExpMatchOne(
filter, strNew("12341234-123123F_12341234-123123IG")), false, " does not match with trailing character");
TEST_RESULT_BOOL(
regExpMatchOne(
filter, strNew("A12341234-123123F_12341234-123123I")), false, " does not match with leading character");
// -------------------------------------------------------------------------------------------------------------------------
filter = backupRegExpP(.full = true, .differential = true);
Add linefeeds for clarity.
2019-05-16 09:56:48 -04:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
filter, "^[0-9]{8}\\-[0-9]{6}F(\\_[0-9]{8}\\-[0-9]{6}D){0,1}$", "full and optional diff backup regex with anchors");
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
TEST_RESULT_BOOL(regExpMatchOne(filter, incr), false, " does not match incr");
TEST_RESULT_BOOL(regExpMatchOne(filter, diff), true, " match diff");
TEST_RESULT_BOOL(regExpMatchOne(filter, full), true, " match full");
// -------------------------------------------------------------------------------------------------------------------------
filter = backupRegExpP(.full = true, .incremental = true, .differential = true);
Add linefeeds for clarity.
2019-05-16 09:56:48 -04:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
filter, "^[0-9]{8}\\-[0-9]{6}F(\\_[0-9]{8}\\-[0-9]{6}(D|I)){0,1}$",
"full, optional diff and incr backup regex with anchors");
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
TEST_RESULT_BOOL(regExpMatchOne(filter, incr), true, " match incr");
TEST_RESULT_BOOL(regExpMatchOne(filter, diff), true, " match diff");
TEST_RESULT_BOOL(regExpMatchOne(filter, full), true, " match full");
// -------------------------------------------------------------------------------------------------------------------------
Allow end anchor to be excluded in backupRegExp(). This is useful for matching files in the backup history directory which have characters after the backup label.
2019-12-12 18:52:16 -05:00
filter = backupRegExpP(.incremental = true, .differential = true, .noAnchorEnd = true);
Add linefeeds for clarity.
2019-05-16 09:56:48 -04:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(filter, "^[0-9]{8}\\-[0-9]{6}F\\_[0-9]{8}\\-[0-9]{6}(D|I)", "diff and incr backup regex with anchors");
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
TEST_RESULT_BOOL(regExpMatchOne(filter, incr), true, " match incr");
TEST_RESULT_BOOL(regExpMatchOne(filter, diff), true, " match diff");
TEST_RESULT_BOOL(regExpMatchOne(filter, full), false, " does not match full");
TEST_RESULT_BOOL(
regExpMatchOne(
filter, strNew("A12341234-123123F_12341234-123123I")), false, " does not match with leading character");
// -------------------------------------------------------------------------------------------------------------------------
filter = backupRegExpP(.incremental = true);
Add linefeeds for clarity.
2019-05-16 09:56:48 -04:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(filter, "^[0-9]{8}\\-[0-9]{6}F\\_[0-9]{8}\\-[0-9]{6}I$", "incr backup regex with anchors");
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
TEST_RESULT_BOOL(regExpMatchOne(filter, incr), true, " match incr");
TEST_RESULT_BOOL(regExpMatchOne(filter, diff), false, " does not match diff");
TEST_RESULT_BOOL(regExpMatchOne(filter, full), false, " does not match full");
// -------------------------------------------------------------------------------------------------------------------------
filter = backupRegExpP(.differential = true);
Add linefeeds for clarity.
2019-05-16 09:56:48 -04:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(filter, "^[0-9]{8}\\-[0-9]{6}F\\_[0-9]{8}\\-[0-9]{6}D$", "diff backup regex with anchors");
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
TEST_RESULT_BOOL(regExpMatchOne(filter, incr), false, " does not match incr");
TEST_RESULT_BOOL(regExpMatchOne(filter, diff), true, " match diff");
TEST_RESULT_BOOL(regExpMatchOne(filter, full), false, " does not match full");
}
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
// *****************************************************************************************************************************
if (testBegin("PageChecksum"))
{
TEST_RESULT_UINT(PG_SEGMENT_PAGE_DEFAULT, 131072, "check pages per segment");
// Test pages with all zeros (these are considered valid)
// -------------------------------------------------------------------------------------------------------------------------
Buffer *buffer = bufNew(PG_PAGE_SIZE_DEFAULT * 3);
Buffer *bufferOut = bufNew(0);
bufUsedSet(buffer, bufSize(buffer));
memset(bufPtr(buffer), 0, bufSize(buffer));
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x00)) = (PageHeaderData){.pd_upper = 0};
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x01)) = (PageHeaderData){.pd_upper = 0};
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x02)) = (PageHeaderData){.pd_upper = 0};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
IoWrite *write = ioBufferWriteNew(bufferOut);
Use PG_PAGE_SIZE_DEFAULT constant instead of pageSize variable. Page size is passed around a lot but in fact it can only have one value, PG_PAGE_SIZE_DEFAULT, which is checked when pg_control is loaded. There may be an argument for supporting multiple page sizes in the future but for now just use the constant to simplify the code. There is also a significant performance benefit. Because pageSize was being used in pageChecksumBlock() the main loop was neither unrolled nor vectorized (-funroll-loops -ftree-vectorize) as it is now with a constant loop boundary.
2020-03-05 09:14:27 -05:00
ioFilterGroupAdd(ioWriteFilterGroup(write), pageChecksumNew(0, PG_SEGMENT_PAGE_DEFAULT, 0));
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
ioWriteOpen(write);
ioWrite(write, buffer);
ioWriteClose(write);
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
jsonFromVar(ioFilterGroupResult(ioWriteFilterGroup(write), PAGE_CHECKSUM_FILTER_TYPE_STR)),
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
"{\"align\":true,\"valid\":true}", "all zero pages");
// Single checksum error
// -------------------------------------------------------------------------------------------------------------------------
buffer = bufNew(PG_PAGE_SIZE_DEFAULT * 1);
bufUsedSet(buffer, bufSize(buffer));
memset(bufPtr(buffer), 0, bufSize(buffer));
// Page 0 has bogus checksum
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x00)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.xlogid = 0xF0F0F0F0,
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
.xrecoff = 0xF0F0F0F0,
},
};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
write = ioBufferWriteNew(bufferOut);
Allow multiple filters to be pushed to the remote and return results. Previously only a single filter could be pushed to the remote since order was not being maintained. Now the filters are strictly ordered. Results are returned from the remote and set in the local IoFilterGroup so they can be retrieved. Expand remote filter support to include all filters.
2019-07-15 16:49:46 -04:00
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
ioFilterGroupAdd(
Allow multiple filters to be pushed to the remote and return results. Previously only a single filter could be pushed to the remote since order was not being maintained. Now the filters are strictly ordered. Results are returned from the remote and set in the local IoFilterGroup so they can be retrieved. Expand remote filter support to include all filters.
2019-07-15 16:49:46 -04:00
ioWriteFilterGroup(write),
Use PG_PAGE_SIZE_DEFAULT constant instead of pageSize variable. Page size is passed around a lot but in fact it can only have one value, PG_PAGE_SIZE_DEFAULT, which is checked when pg_control is loaded. There may be an argument for supporting multiple page sizes in the future but for now just use the constant to simplify the code. There is also a significant performance benefit. Because pageSize was being used in pageChecksumBlock() the main loop was neither unrolled nor vectorized (-funroll-loops -ftree-vectorize) as it is now with a constant loop boundary.
2020-03-05 09:14:27 -05:00
pageChecksumNewVar(varVarLst(jsonToVar(strNewFmt("[0,%u,%" PRIu64 "]", PG_SEGMENT_PAGE_DEFAULT, 0xFACEFACE00000000)))));
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
ioWriteOpen(write);
ioWrite(write, buffer);
ioWriteClose(write);
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
jsonFromVar(ioFilterGroupResult(ioWriteFilterGroup(write), PAGE_CHECKSUM_FILTER_TYPE_STR)),
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
"{\"align\":true,\"error\":[0],\"valid\":false}", "single checksum error");
// Various checksum errors some of which will be skipped because of the LSN
// -------------------------------------------------------------------------------------------------------------------------
buffer = bufNew(PG_PAGE_SIZE_DEFAULT * 8 - (PG_PAGE_SIZE_DEFAULT - 512));
bufUsedSet(buffer, bufSize(buffer));
memset(bufPtr(buffer), 0, bufSize(buffer));
// Page 0 has bogus checksum
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x00)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.xlogid = 0xF0F0F0F0,
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
.xrecoff = 0xF0F0F0F0,
},
};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
// Page 1 has bogus checksum but lsn above the limit
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x01)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.xlogid = 0xFACEFACE,
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
.xrecoff = 0x00000000,
},
};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
// Page 2 has bogus checksum
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x02)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
.xrecoff = 0x2,
},
};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
// Page 3 has bogus checksum
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x03)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
.xrecoff = 0x3,
},
};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
// Page 4 has bogus checksum
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x04)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
.xrecoff = 0x4,
},
};
// Page 5 is zero
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x05)) = (PageHeaderData){.pd_upper = 0x00};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
// Page 6 has bogus checksum
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x06)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
.xrecoff = 0x6,
},
};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
// Page 7 has bogus checksum (and is misaligned but large enough to test)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x07)) = (PageHeaderData)
{
.pd_upper = 0x01,
Centralize PostgreSQL page header data structures. These data structures were copied a few places (but only once in the core code) so put them in a place where everyone can use them. To do this create a new file, static.auto.h, to contain data types and macros that have stayed the same through all the versions of PostgreSQL that we support. This allows us to have single, non-versioned set of headers and code for stable data structures like page headers. Migrate a few types from version.auto.h that are required for page header structures and pull the remaining types from PostgreSQL directly. We had previously renamed xlog to wal so update those where required since we won't be modifying the PostgreSQL names anymore.
2020-03-04 13:31:27 -05:00
.pd_lsn = (PageXLogRecPtr)
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
{
.xrecoff = 0x7,
},
};
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
write = ioBufferWriteNew(bufferOut);
Use PG_PAGE_SIZE_DEFAULT constant instead of pageSize variable. Page size is passed around a lot but in fact it can only have one value, PG_PAGE_SIZE_DEFAULT, which is checked when pg_control is loaded. There may be an argument for supporting multiple page sizes in the future but for now just use the constant to simplify the code. There is also a significant performance benefit. Because pageSize was being used in pageChecksumBlock() the main loop was neither unrolled nor vectorized (-funroll-loops -ftree-vectorize) as it is now with a constant loop boundary.
2020-03-05 09:14:27 -05:00
ioFilterGroupAdd(ioWriteFilterGroup(write), pageChecksumNew(0, PG_SEGMENT_PAGE_DEFAULT, 0xFACEFACE00000000));
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
ioWriteOpen(write);
ioWrite(write, buffer);
ioWriteClose(write);
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
jsonFromVar(ioFilterGroupResult(ioWriteFilterGroup(write), PAGE_CHECKSUM_FILTER_TYPE_STR)),
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
"{\"align\":false,\"error\":[0,[2,4],[6,7]],\"valid\":false}", "various checksum errors");
// Impossibly misaligned page
// -------------------------------------------------------------------------------------------------------------------------
buffer = bufNew(256);
bufUsedSet(buffer, bufSize(buffer));
memset(bufPtr(buffer), 0, bufSize(buffer));
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x00)) = (PageHeaderData){.pd_upper = 0};
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
write = ioBufferWriteNew(bufferOut);
Use PG_PAGE_SIZE_DEFAULT constant instead of pageSize variable. Page size is passed around a lot but in fact it can only have one value, PG_PAGE_SIZE_DEFAULT, which is checked when pg_control is loaded. There may be an argument for supporting multiple page sizes in the future but for now just use the constant to simplify the code. There is also a significant performance benefit. Because pageSize was being used in pageChecksumBlock() the main loop was neither unrolled nor vectorized (-funroll-loops -ftree-vectorize) as it is now with a constant loop boundary.
2020-03-05 09:14:27 -05:00
ioFilterGroupAdd(ioWriteFilterGroup(write), pageChecksumNew(0, PG_SEGMENT_PAGE_DEFAULT, 0xFACEFACE00000000));
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
ioWriteOpen(write);
ioWrite(write, buffer);
ioWriteClose(write);
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
jsonFromVar(ioFilterGroupResult(ioWriteFilterGroup(write), PAGE_CHECKSUM_FILTER_TYPE_STR)),
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
"{\"align\":false,\"valid\":false}", "misalignment");
// Two misaligned buffers in a row
// -------------------------------------------------------------------------------------------------------------------------
buffer = bufNew(513);
bufUsedSet(buffer, bufSize(buffer));
memset(bufPtr(buffer), 0, bufSize(buffer));
Use designated initializers to initialize structs. Previously memNew() used memset() to initialize all struct members to 0, NULL, false, etc. While this appears to work in practice, it is a violation of the C specification. For instance, NULL == 0 must be true but neither NULL nor 0 must be represented with all zero bits. Instead use designated initializers to initialize structs. These guarantee that struct members will be properly initialized even if they are not specified in the initializer. Note that due to a quirk in the C99 specification at least one member must be explicitly initialized even if it needs to be the default value. Since pre-zeroed memory is no longer required, adjust memAllocInternal()/memReallocInternal() to return raw memory and update dependent functions accordingly. All instances of memset() have been removed except in debug/test code where needed. Add memMewPtrArray() to allocate an array of pointers and automatically set all pointers to NULL. Rename memGrowRaw() to the more logical memResize().
2020-01-23 14:15:58 -07:00
*(PageHeaderData *)(bufPtr(buffer) + (PG_PAGE_SIZE_DEFAULT * 0x00)) = (PageHeaderData){.pd_upper = 0};
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
write = ioBufferWriteNew(bufferOut);
Use PG_PAGE_SIZE_DEFAULT constant instead of pageSize variable. Page size is passed around a lot but in fact it can only have one value, PG_PAGE_SIZE_DEFAULT, which is checked when pg_control is loaded. There may be an argument for supporting multiple page sizes in the future but for now just use the constant to simplify the code. There is also a significant performance benefit. Because pageSize was being used in pageChecksumBlock() the main loop was neither unrolled nor vectorized (-funroll-loops -ftree-vectorize) as it is now with a constant loop boundary.
2020-03-05 09:14:27 -05:00
ioFilterGroupAdd(ioWriteFilterGroup(write), pageChecksumNew(0, PG_SEGMENT_PAGE_DEFAULT, 0xFACEFACE00000000));
Migrate page checksum filter to C. This filter exactly mimics the behavior of the Perl filter so is a drop-in replacement. The filter is not integrated yet since it requires the Perl-to-C storage layer interface coming in a future commit.
2019-06-17 07:52:03 -04:00
ioWriteOpen(write);
ioWrite(write, buffer);
TEST_ERROR(ioWrite(write, buffer), AssertError, "should not be possible to see two misaligned pages in a row");
}
Add backup type conversion functions. Convert back and forth between the string and enum representations of backup types.
2019-08-18 20:09:44 -04:00
// *****************************************************************************************************************************
if (testBegin("backupType() and backupTypeStr()"))
{
TEST_RESULT_UINT(backupType(strNew("full")), backupTypeFull, "backup type full");
TEST_RESULT_UINT(backupType(strNew("diff")), backupTypeDiff, "backup type diff");
TEST_RESULT_UINT(backupType(strNew("incr")), backupTypeIncr, "backup type incr");
TEST_ERROR(backupType(strNew("bogus")), AssertError, "invalid backup type 'bogus'");
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(backupTypeStr(backupTypeFull), "full", "backup type str full");
TEST_RESULT_STR_Z(backupTypeStr(backupTypeDiff), "diff", "backup type str diff");
TEST_RESULT_STR_Z(backupTypeStr(backupTypeIncr), "incr", "backup type str incr");
Add backup type conversion functions. Convert back and forth between the string and enum representations of backup types.
2019-08-18 20:09:44 -04:00
}
Migrate backupRegExp() to C. Removed the "anchor" parameter because it was never used in any calls in the Perl code so it was just a dead parameter that always defaulted to true. Contributed by Cynthia Shang.
2019-04-15 08:29:25 -04:00
FUNCTION_HARNESS_RESULT_VOID();
}
Reference in New Issue Copy Permalink