db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-11 00:50:20 +02:00
Code Issues Releases Activity
Files
0c05df458287c77df1188a68bbd74e145ed0f059
pgbackrest/test/src/module/command/archiveCommonTest.c

294 lines
16 KiB
C
Raw Normal View History

Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
/***********************************************************************************************************************************
Test Archive Common
***********************************************************************************************************************************/
#include <unistd.h>
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
#include "storage/helper.h"
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
#include "storage/posix/storage.h"
Full abstraction of storage driver interface. The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly. Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC). Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
#include "common/harnessConfig.h"
Add timeout to walSegmentFind(). Keep trying to locate the WAL segment until timeout. This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
#include "common/harnessFork.h"
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
// Create default storage object for testing
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
Storage *storageTest = storagePosixNew(
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
strNew(testPath()), STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, true, NULL);
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
// *****************************************************************************************************************************
if (testBegin("archiveAsyncStatus()"))
{
StringList *argList = strLstNew();
strLstAdd(argList, strNewFmt("--spool-path=%s", testPath()));
strLstAddZ(argList, "--archive-async");
strLstAddZ(argList, "--archive-timeout=1");
strLstAddZ(argList, "--stanza=db");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argList);
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
// -------------------------------------------------------------------------------------------------------------------------
String *segment = strNew("000000010000000100000001");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(archiveAsyncStatus(archiveModePush, segment, false), false, "directory and status file not present");
TEST_RESULT_BOOL(archiveAsyncStatus(archiveModeGet, segment, false), false, "directory and status file not present");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
// -------------------------------------------------------------------------------------------------------------------------
mkdir(strPtr(strNewFmt("%s/archive", testPath())), 0750);
mkdir(strPtr(strNewFmt("%s/archive/db", testPath())), 0750);
mkdir(strPtr(strNewFmt("%s/archive/db/out", testPath())), 0750);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(archiveAsyncStatus(archiveModePush, segment, false), false, "status file not present");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF(BOGUS_STR));
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
TEST_ERROR(
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
archiveAsyncStatus(archiveModePush, segment, false), FormatError,
"000000010000000100000001.ok content must have at least two lines");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF(BOGUS_STR "\n"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_ERROR(
archiveAsyncStatus(archiveModePush, segment, false), FormatError, "000000010000000100000001.ok message must be > 0");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF(BOGUS_STR "\nmessage"));
Add base variants to all integer to string conversion functions. Previously these functions were fixed at base 10 conversion. Add variants that can convert from any base.
2018-11-06 18:32:44 -05:00
TEST_ERROR(
archiveAsyncStatus(archiveModePush, segment, false), FormatError, "unable to convert base 10 string 'BOGUS' to int");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))), NULL);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(archiveAsyncStatus(archiveModePush, segment, false), true, "ok file");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF("0\nwarning"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(archiveAsyncStatus(archiveModePush, segment, false), true, "ok file with warning");
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
harnessLogResult("P00 WARN: warning");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment))),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF("25\nerror"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(archiveAsyncStatus(archiveModePush, segment, false), true, "error status renamed to ok");
Refactor the common/log tests to not depend on common/harnessLog. common/harnessLog was not ideally suited for general testing and made all the tests quite awkward. Instead, move all code used to test the common/log module into the logTest module and repurpose common/harnessLog to do log expect testing for all other tests in a cleaner way. Add a few exceptions for config testing since the log levels are reset by default in config/parse.
2018-07-20 18:51:42 -04:00
harnessLogResult(
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
"P00 WARN: WAL segment '000000010000000100000001' was not pushed due to error [25] and was manually skipped: error");
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
TEST_RESULT_VOID(
storageRemoveP(
storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.ok", strPtr(segment)), .errorOnMissing = true),
"remove ok");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment))), bufNew(0));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_ERROR(
archiveAsyncStatus(archiveModePush, segment, true), AssertError,
"status file '000000010000000100000001.error' has no content");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s.error", strPtr(segment))),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF("25\nmessage"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_ERROR(archiveAsyncStatus(archiveModePush, segment, true), AssertError, "message");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(archiveAsyncStatus(archiveModePush, segment, false), false, "suppress error");
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/global.error")),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF("102\nexecute error"));
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
TEST_ERROR(archiveAsyncStatus(archiveModePush, strNew("anyfile"), true), ExecuteError, "execute error");
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
}
Add walSegmentNext() and walSegmentRange().
2018-04-29 11:47:50 -04:00
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// *****************************************************************************************************************************
if (testBegin("archiveAsyncStatusErrorWrite() and archiveAsyncStatusOkWrite()"))
{
StringList *argList = strLstNew();
strLstAdd(argList, strNewFmt("--spool-path=%s", testPath()));
strLstAddZ(argList, "--stanza=db");
strLstAddZ(argList, "archive-get-async");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchiveGetAsync, argList);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
String *walSegment = strNew("000000010000000100000001");
TEST_RESULT_VOID(
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
archiveAsyncStatusErrorWrite(archiveModeGet, walSegment, 25, strNew("error message")), "write error");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_STR(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
strPtr(strNewBuf(storageGetP(storageNewReadP(storageTest, strNew("archive/db/in/000000010000000100000001.error"))))),
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
"25\nerror message", "check error");
TEST_RESULT_VOID(
storageRemoveP(storageTest, strNew("archive/db/in/000000010000000100000001.error"), .errorOnMissing = true),
"remove error");
TEST_RESULT_VOID(
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
archiveAsyncStatusErrorWrite(archiveModeGet, NULL, 25, strNew("global error message")), "write global error");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_STR(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
strPtr(strNewBuf(storageGetP(storageNewReadP(storageTest, strNew("archive/db/in/global.error"))))),
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
"25\nglobal error message", "check global error");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
storageRemoveP(storageTest, strNew("archive/db/in/global.error"), .errorOnMissing = true),
"remove global error");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(
Allow warnings to be written by archiveAsyncStatusOkWrite(). Migrate this feature to the C code since archive-push will need it.
2019-03-25 09:17:15 +04:00
archiveAsyncStatusOkWrite(archiveModeGet, walSegment, NULL), "write ok file");
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
TEST_RESULT_STR(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
strPtr(strNewBuf(storageGetP(storageNewReadP(storageTest, strNew("archive/db/in/000000010000000100000001.ok"))))),
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
"", "check ok");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
storageRemoveP(storageTest, strNew("archive/db/in/000000010000000100000001.ok"), .errorOnMissing = true),
"remove ok");
Allow warnings to be written by archiveAsyncStatusOkWrite(). Migrate this feature to the C code since archive-push will need it.
2019-03-25 09:17:15 +04:00
TEST_RESULT_VOID(
archiveAsyncStatusOkWrite(archiveModeGet, walSegment, strNew("WARNING")), "write ok file with warning");
TEST_RESULT_STR(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
strPtr(strNewBuf(storageGetP(storageNewReadP(storageTest, strNew("archive/db/in/000000010000000100000001.ok"))))),
Allow warnings to be written by archiveAsyncStatusOkWrite(). Migrate this feature to the C code since archive-push will need it.
2019-03-25 09:17:15 +04:00
"0\nWARNING", "check ok warning");
TEST_RESULT_VOID(
storageRemoveP(storageTest, strNew("archive/db/in/000000010000000100000001.ok"), .errorOnMissing = true),
"remove ok");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
}
Add walSegmentNext() and walSegmentRange().
2018-04-29 11:47:50 -04:00
// *****************************************************************************************************************************
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
if (testBegin("walIsPartial()"))
{
TEST_RESULT_BOOL(walIsPartial(strNew("000000010000000100000001")), false, "not partial");
TEST_RESULT_BOOL(walIsPartial(strNew("FFFFFFFFFFFFFFFFFFFFFFFF.partial")), true, "partial");
}
// *****************************************************************************************************************************
if (testBegin("walIsSegment()"))
{
TEST_RESULT_BOOL(walIsSegment(strNew("000000010000000100000001")), true, "wal segment");
TEST_RESULT_BOOL(walIsSegment(strNew("FFFFFFFFFFFFFFFFFFFFFFFF.partial")), true, "partial wal segment");
TEST_RESULT_BOOL(walIsSegment(strNew("0000001A.history")), false, "history file");
}
// *****************************************************************************************************************************
Move WAL path prefix logic into walPath(). This logic is used by both archive-push and archive-get.
2019-03-16 16:14:10 +04:00
if (testBegin("walPath()"))
{
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
THROW_ON_SYS_ERROR(chdir("/tmp") != 0, PathMissingError, "unable to chdir()");
TEST_RESULT_STR(strPtr(walPath(strNew("/absolute/path"))), "/absolute/path", "absolute path");
TEST_RESULT_STR(strPtr(walPath(strNew("relative/path"))), "/tmp/relative/path", "relative path");
Move WAL path prefix logic into walPath(). This logic is used by both archive-push and archive-get.
2019-03-16 16:14:10 +04:00
}
// *****************************************************************************************************************************
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
if (testBegin("walSegmentFind()"))
{
// Load configuration to set repo-path and stanza
StringList *argList = strLstNew();
strLstAddZ(argList, "--stanza=db");
strLstAdd(argList, strNewFmt("--repo-path=%s", testPath()));
strLstAddZ(argList, "archive-get");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchiveGet, argList);
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
Add timeout to walSegmentFind(). Keep trying to locate the WAL segment until timeout. This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
TEST_RESULT_PTR(walSegmentFind(storageRepo(), strNew("9.6-2"), strNew("123456781234567812345678"), 0), NULL, "no path");
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePathCreateP(storageTest, strNew("archive/db/9.6-2/1234567812345678"));
Add timeout to walSegmentFind(). Keep trying to locate the WAL segment until timeout. This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
TEST_RESULT_PTR(walSegmentFind(storageRepo(), strNew("9.6-2"), strNew("123456781234567812345678"), 0), NULL, "no segment");
TEST_RESULT_PTR(
walSegmentFind(storageRepo(), strNew("9.6-2"), strNew("123456781234567812345678"), 500), NULL,
"no segment after 500ms");
// Check timeout by making the wal segment appear after 250ms
HARNESS_FORK_BEGIN()
{
HARNESS_FORK_CHILD_BEGIN(0, false)
{
sleepMSec(250);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(
Add timeout to walSegmentFind(). Keep trying to locate the WAL segment until timeout. This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
storageTest,
strNew(
"archive/db/9.6-2/1234567812345678/123456781234567812345678-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")),
NULL);
}
HARNESS_FORK_CHILD_END();
HARNESS_FORK_PARENT_BEGIN()
{
TEST_RESULT_STR(
strPtr(walSegmentFind(storageRepo(), strNew("9.6-2"), strNew("123456781234567812345678"), 1000)),
"123456781234567812345678-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "found segment");
}
HARNESS_FORK_PARENT_END();
}
HARNESS_FORK_END();
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
storageTest,
strNew("archive/db/9.6-2/1234567812345678/123456781234567812345678-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.gz")),
NULL);
TEST_ERROR(
Add timeout to walSegmentFind(). Keep trying to locate the WAL segment until timeout. This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
walSegmentFind(storageRepo(), strNew("9.6-2"), strNew("123456781234567812345678"), 0),
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
ArchiveDuplicateError,
"duplicates found in archive for WAL segment 123456781234567812345678:"
" 123456781234567812345678-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
", 123456781234567812345678-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.gz"
"\nHINT: are multiple primaries archiving to this stanza?");
Improve macros and coverage rules that were hiding missing coverage. The branch coverage exclusion rules were overly broad and included functions that ended in a capital letter, which disabled all coverage for the statement. Improve matching so that all characters in the name must be upper-case for a match. Some macros with internal branches accepted parameters that might contain conditionals. This made it impossible to tell which branches belonged to which, and in any case an overzealous exclusion rule was ignoring all branches in such cases. Add the DEBUG_COVERAGE flag to build a modified version of the macros without any internal branches to be used for coverage testing. In most cases, the branches were optimizations (like checking logWill()) that improve production performance but are not needed for testing. In other cases, a parameter needed to be added to the underlying function to handle the branch during coverage testing. Also tweak the coverage rules so that macros without conditionals are automatically excluded from branch coverage as long as they are not themselves a parameter. Finally, update tests and code where missing coverage was exposed by these changes. Some code was updated to remove existing coverage exclusions when it was a simple change.
2019-05-11 14:51:51 -04:00
Add gcc 9 support. A number of tests have been updated and Fedora 30 has been added to the test suite so the unit tests can run on gcc 9. Stop running unit tests on co6/7 since we appear to have ample unit test coverage.
2019-10-09 15:03:03 -04:00
TEST_RESULT_PTR(
Add timeout to walSegmentFind(). Keep trying to locate the WAL segment until timeout. This is useful for the check and backup commands which must wait for segments to arrive in the archive.
2019-07-31 19:58:57 -04:00
walSegmentFind(storageRepo(), strNew("9.6-2"), strNew("123456781234567812345678.partial"), 0), NULL,
Improve macros and coverage rules that were hiding missing coverage. The branch coverage exclusion rules were overly broad and included functions that ended in a capital letter, which disabled all coverage for the statement. Improve matching so that all characters in the name must be upper-case for a match. Some macros with internal branches accepted parameters that might contain conditionals. This made it impossible to tell which branches belonged to which, and in any case an overzealous exclusion rule was ignoring all branches in such cases. Add the DEBUG_COVERAGE flag to build a modified version of the macros without any internal branches to be used for coverage testing. In most cases, the branches were optimizations (like checking logWill()) that improve production performance but are not needed for testing. In other cases, a parameter needed to be added to the underlying function to handle the branch during coverage testing. Also tweak the coverage rules so that macros without conditionals are automatically excluded from branch coverage as long as they are not themselves a parameter. Finally, update tests and code where missing coverage was exposed by these changes. Some code was updated to remove existing coverage exclusions when it was a simple change.
2019-05-11 14:51:51 -04:00
"did not find partial segment");
Migrate walIsPartial(), walIsSegment(), and walSegmentFind() from Perl to C. Also refactor regular expression defines to make them more reusable.
2018-09-07 08:00:18 -07:00
}
// *****************************************************************************************************************************
Add walSegmentNext() and walSegmentRange().
2018-04-29 11:47:50 -04:00
if (testBegin("walSegmentNext()"))
{
TEST_RESULT_STR(
strPtr(walSegmentNext(strNew("000000010000000100000001"), 16 * 1024 * 1024, PG_VERSION_10)),
"000000010000000100000002", "get next");
TEST_RESULT_STR(
strPtr(walSegmentNext(strNew("0000000100000001000000FE"), 16 * 1024 * 1024, PG_VERSION_93)),
"0000000100000001000000FF", "get next");
TEST_RESULT_STR(
strPtr(walSegmentNext(strNew("0000009900000001000000FF"), 16 * 1024 * 1024, PG_VERSION_93)),
"000000990000000200000000", "get next overflow >= 9.3");
TEST_RESULT_STR(
strPtr(walSegmentNext(strNew("0000000100000001000000FE"), 16 * 1024 * 1024, PG_VERSION_92)),
"000000010000000200000000", "get next overflow < 9.3");
TEST_RESULT_STR(
strPtr(walSegmentNext(strNew("000000010000000100000003"), 1024 * 1024 * 1024, PG_VERSION_11)),
"000000010000000200000000", "get next overflow >= 11/1GB");
TEST_RESULT_STR(
strPtr(walSegmentNext(strNew("000000010000006700000FFF"), 1024 * 1024, PG_VERSION_11)),
"000000010000006800000000", "get next overflow >= 11/1MB");
}
// *****************************************************************************************************************************
if (testBegin("walSegmentRange()"))
{
TEST_RESULT_STR(
strPtr(strLstJoin(walSegmentRange(strNew("000000010000000100000000"), 16 * 1024 * 1024, PG_VERSION_92, 1), "|")),
"000000010000000100000000", "get single");
TEST_RESULT_STR(
strPtr(strLstJoin(walSegmentRange(strNew("0000000100000001000000FD"), 16 * 1024 * 1024, PG_VERSION_92, 4), "|")),
"0000000100000001000000FD|0000000100000001000000FE|000000010000000200000000|000000010000000200000001",
"get range < 9.3");
TEST_RESULT_STR(
strPtr(strLstJoin(walSegmentRange(strNew("0000000100000001000000FD"), 16 * 1024 * 1024, PG_VERSION_93, 4), "|")),
"0000000100000001000000FD|0000000100000001000000FE|0000000100000001000000FF|000000010000000200000000",
"get range >= 9.3");
TEST_RESULT_STR(
strPtr(strLstJoin(walSegmentRange(strNew("000000080000000A00000000"), 1024 * 1024 * 1024, PG_VERSION_11, 8), "|")),
"000000080000000A00000000|000000080000000A00000001|000000080000000A00000002|000000080000000A00000003|"
"000000080000000B00000000|000000080000000B00000001|000000080000000B00000002|000000080000000B00000003",
"get range >= 11/1GB");
TEST_RESULT_STR(
strPtr(strLstJoin(walSegmentRange(strNew("000000070000000700000FFE"), 1024 * 1024, PG_VERSION_11, 4), "|")),
"000000070000000700000FFE|000000070000000700000FFF|000000070000000800000000|000000070000000800000001",
"get range >= 11/1MB");
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_RESULT_VOID();
Refactor archive common functions in preparation for parallel async archive-get.
2018-04-29 10:16:59 -04:00
}
Reference in New Issue Copy Permalink