db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-11 00:50:20 +02:00
Code Issues Releases Activity
Files
22fd223fc393908f6f0d5f0a64ee8f93317ce24e
pgbackrest/test/src/module/command/archiveGetTest.c

709 lines
36 KiB
C
Raw Normal View History

Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Archive Get Command
***********************************************************************************************************************************/
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 14:41:03 -05:00
#include "common/compress/helper.h"
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
#include "common/harnessConfig.h"
#include "common/harnessFork.h"
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
#include "common/io/bufferRead.h"
#include "common/io/bufferWrite.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
#include "postgres/interface.h"
#include "postgres/version.h"
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
#include "storage/posix/storage.h"
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
#include "common/harnessInfo.h"
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
#include "common/harnessStorage.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Convert storagePosixNew() to storagePosixNewP(). An upcoming feature requires new parameters for storagePosixNew() and this causes a lot of churn because almost every test creates a Posix storage object. Some refactoring in the tests might reduce this duplication but storagePosixNew() is collecting a lot of parameters so converting to storagePosixNewP() makes sense in any case. There are relatively few call sites in the core code but they still benefit from better readability after this change.
2020-04-30 11:01:38 -04:00
Storage *storageTest = storagePosixNewP(strNew(testPath()), .write = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// *****************************************************************************************************************************
if (testBegin("archiveGetCheck()"))
{
// Load Parameters
StringList *argList = strLstNew();
strLstAddZ(argList, "--stanza=test1");
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
strLstAdd(argList, strNewFmt("--pg1-path=%s/pg", testPath()));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchiveGet, argList);
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Create pg_control file
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
storageNewWriteP(storageTest, strNew("pg/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Control and archive info mismatch
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageTest, strNew("repo/archive/test1/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
Make notion of current PostgreSQL info ID in C align with Perl. The C code was assuming that the current PostgreSQL version in archive.info/backup.info was the most recent item in the history, but this is not always the case with some stanza-upgrade scenarios. If a cluster is restored from before the upgrade and stanza-upgrade is run again, it will revert db-id to the original history item. Instead, load db-id from the db section explicitly as the Perl code does. This did not affect archive-get since it does a reverse scan through the history versions and does not rely on the current version.
2019-03-16 15:27:38 +04:00
"[db]\n"
"db-id=1\n"
"\n"
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
TEST_ERROR(
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL), ArchiveMismatchError,
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"unable to retrieve the archive id for database version '10' and system-id '18072658121562454734'");
// Nothing to find in empty archive dir
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(storageTest, strNew("repo/archive/test1/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
Make notion of current PostgreSQL info ID in C align with Perl. The C code was assuming that the current PostgreSQL version in archive.info/backup.info was the most recent item in the history, but this is not always the case with some stanza-upgrade scenarios. If a cluster is restored from before the upgrade and stanza-upgrade is run again, it will revert db-id to the original history item. Instead, load db-id from the db section explicitly as the Perl code does. This did not affect archive-get since it does a reverse scan through the history versions and does not rely on the current version.
2019-03-16 15:27:38 +04:00
"[db]\n"
"db-id=3\n"
"\n"
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"
"2={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n"
"3={\"db-id\":18072658121562454734,\"db-version\":\"9.6\"}\n"
"4={\"db-id\":18072658121562454734,\"db-version\":\"10\"}"));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Update TEST_RESULT_PTR() to TEST_RESULT_STR() where appropriate. These were missed in d41eea68 when the functionality of TEST_RESULT_STR() was changed. Using TEST_RESULT_STR() instead of TEST_RESULT_PTR() is more type-safe and clearer. Add a comment to make it clear that TEST_RESULT_PTR() should be used only when a better alternative is not available.
2020-06-17 09:46:09 -04:00
TEST_RESULT_STR(
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL).archiveFileActual, NULL, "no segment found");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Write segment into an older archive path
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
storageTest,
strNew(
"repo/archive/test1/10-2/8765432187654321/876543218765432187654321-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")),
NULL);
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL).archiveFileActual,
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"10-2/8765432187654321/876543218765432187654321-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "segment found");
// Write segment into an newer archive path
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
storageNewWriteP(
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
storageTest,
strNew(
"repo/archive/test1/10-4/8765432187654321/876543218765432187654321-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb")),
NULL);
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL).archiveFileActual,
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"10-4/8765432187654321/876543218765432187654321-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "newer segment found");
// Get history file
// -------------------------------------------------------------------------------------------------------------------------
Update TEST_RESULT_PTR() to TEST_RESULT_STR() where appropriate. These were missed in d41eea68 when the functionality of TEST_RESULT_STR() was changed. Using TEST_RESULT_STR() instead of TEST_RESULT_PTR() is more type-safe and clearer. Add a comment to make it clear that TEST_RESULT_PTR() should be used only when a better alternative is not available.
2020-06-17 09:46:09 -04:00
TEST_RESULT_STR(
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
archiveGetCheck(strNew("00000009.history"), cipherTypeNone, NULL).archiveFileActual, NULL, "history file not found");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(storageNewWriteP(storageTest, strNew("repo/archive/test1/10-4/00000009.history")), NULL);
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
archiveGetCheck(strNew("00000009.history"), cipherTypeNone, NULL).archiveFileActual, "10-4/00000009.history",
"history file found");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
}
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// *****************************************************************************************************************************
if (testBegin("queueNeed()"))
{
StringList *argList = strLstNew();
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
strLstAddZ(argList, "--stanza=test1");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, "--archive-async");
Remove indexed option constants. These constants don't scale well as the index total is increased for an option. The core code rarely uses these options and they are easily replaced with cfgOptionName(). The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, "/unused");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAdd(argList, strNewFmt("--spool-path=%s/spool", testPath()));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchiveGet, argList);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
size_t queueSize = 16 * 1024 * 1024;
size_t walSegmentSize = 16 * 1024 * 1024;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_ERROR_FMT(
queueNeed(strNew("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92),
Simplify storage driver info and list functions. The storage driver requires two list functions to be implemented, list and infoList. But the former is a subset of the latter so implementing both in every driver is wasteful. The reason both exist is that in Posix it is cheaper to get a list of names than it is to stat files to get size, time, etc. In S3 these operations are equivalent. Introduce storageInfoLevelType to determine the amount of information required by the caller. That way Posix can work efficiently and all drivers can return only the data required which saves some bandwidth. The storageList() and storageInfoList() functions remain in the storage interface since they are useful -- the only change is simplifying the drivers with no external impact. Note that since list() accepted an expression infoList() must now do so. Checking the expression is optional for the driver but can be used to limit results or save IO costs. Similarly, exists() and pathExists() are just specialized forms of info() so adapt them to call info() instead.
2020-04-06 16:09:18 -04:00
PathMissingError, "unable to list file info for missing path '%s/spool/archive/test1/in'", testPath());
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePathCreateP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
queueNeed(STRDEF("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92),
"000000010000000100000001\n000000010000000100000002\n", "queue size smaller than min");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
queueSize = (16 * 1024 * 1024) * 3;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
queueNeed(strNew("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92),
"000000010000000100000001\n000000010000000100000002\n000000010000000100000003\n", "empty queue");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Buffer *walSegmentBuffer = bufNew(walSegmentSize);
memset(bufPtr(walSegmentBuffer), 0, walSegmentSize);
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE", walSegmentBuffer);
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF", walSegmentBuffer);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
queueNeed(strNew("0000000100000001000000FE"), false, queueSize, walSegmentSize, PG_VERSION_92),
"000000010000000200000000\n000000010000000200000001\n", "queue has wal < 9.3");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
storageListP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN)), "0000000100000001000000FE\n", "check queue");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
walSegmentSize = 1024 * 1024;
queueSize = walSegmentSize * 5;
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/junk", "JUNK");
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFE", walSegmentBuffer);
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFF", walSegmentBuffer);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
queueNeed(strNew("000000010000000A00000FFD"), true, queueSize, walSegmentSize, PG_VERSION_11),
"000000010000000B00000000\n000000010000000B00000001\n000000010000000B00000002\n", "queue has wal >= 9.3");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
TEST_STORAGE_LIST(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000A00000FFE\n000000010000000A00000FFF\n");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// *****************************************************************************************************************************
if (testBegin("cmdArchiveGetAsync()"))
{
harnessLogLevelSet(logLevelDetail);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// Arguments that must be included
StringList *argBaseList = strLstNew();
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH_PG);
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH_REPO);
hrnCfgArgRawZ(argBaseList, cfgOptSpoolPath, TEST_PATH_SPOOL);
hrnCfgArgRawBool(argBaseList, cfgOptArchiveAsync, true);
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test2");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("command must be run on the pg host");
StringList *argList = strLstDup(argBaseList);
hrnCfgArgRawZ(argList, cfgOptPgHost, BOGUS_STR);
harnessCfgLoadRole(cfgCmdArchiveGet, cfgCmdRoleAsync, argList);
TEST_ERROR(cmdArchiveGetAsync(), HostInvalidError, "archive-get command must be run on the PostgreSQL host");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Make archive-get locality error generate a global.error file. Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "global.error\n", .remove = true);
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("error on no segments");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
harnessCfgLoadRole(cfgCmdArchiveGet, cfgCmdRoleAsync, argList);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_ERROR(cmdArchiveGetAsync(), ParamInvalidError, "at least one wal segment is required");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Make archive-get locality error generate a global.error file. Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "global.error\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("no segments to find");
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
strLstAddZ(argList, "000000010000000100000001");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchiveGet, cfgCmdRoleAsync, argList);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
harnessLogResult(
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
"P01 DETAIL: unable to find 000000010000000100000001 in the archive");
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000001.ok\n", .remove = true);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on invalid compressed segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
harnessLogResult(
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
"P01 WARN: could not get 000000010000000100000001 from the repo1 archive (will be retried):"
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
" [29] raised from local-1 protocol: unexpected eof in compressed data");
TEST_STORAGE_LIST(
storageSpool(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000001.error\n000000010000000100000001.pgbackrest.tmp\n");
TEST_STORAGE_REMOVE(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.error");
TEST_STORAGE_REMOVE(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("single segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
// There should be a temp file left over. Make sure it still exists to test that temp files are removed on retry.
TEST_STORAGE_EXISTS(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.pgbackrest.tmp");
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
harnessLogResult(
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
"P01 DETAIL: found 000000010000000100000001 in the repo1 archive");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000001\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("multiple segments where some are missing or errored");
argList = strLstDup(argBaseList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "0000000100000001000000FE");
strLstAddZ(argList, "0000000100000001000000FF");
strLstAddZ(argList, "000000010000000200000000");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchiveGet, cfgCmdRoleAsync, argList);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/0000000100000001000000FE-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// Create segment duplicates
HRN_STORAGE_PUT_EMPTY(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_EMPTY(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
harnessLogResult(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000\n"
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
"P01 DETAIL: found 0000000100000001000000FE in the repo1 archive\n"
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"P01 DETAIL: unable to find 0000000100000001000000FF in the archive\n"
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
"P01 WARN: could not get 000000010000000200000000 from the repo1 archive (will be retried): "
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"[45] raised from local-1 protocol: duplicates found in archive for WAL segment 000000010000000200000000: "
"000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, "
"000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n"
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
" HINT: are multiple primaries archiving to this stanza?");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN,
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"0000000100000001000000FE\n0000000100000001000000FF.ok\n000000010000000200000000.error\n",
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
.remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("global error on invalid executable");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
argList = strLstNew();
strLstAddZ(argList, "pgbackrest-bogus");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH_PG);
hrnCfgArgRawZ(argList, cfgOptRepoPath, TEST_PATH_REPO);
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH_SPOOL);
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
hrnCfgArgRawZ(argList, cfgOptStanza, "test2");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
strLstAddZ(argList, CFGCMD_ARCHIVE_GET ":" CONFIG_COMMAND_ROLE_ASYNC);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "0000000100000001000000FE");
strLstAddZ(argList, "0000000100000001000000FF");
strLstAddZ(argList, "000000010000000200000000");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_ERROR(
cmdArchiveGetAsync(), ExecuteError,
"local-1 process terminated unexpectedly [102]: unable to execute 'pgbackrest-bogus': [2] No such file or directory");
harnessLogResult(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
strNewBuf(storageGetP(storageNewReadP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_IN "/global.error")))),
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
"102\nlocal-1 process terminated unexpectedly [102]: unable to execute 'pgbackrest-bogus': "
"[2] No such file or directory",
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
"check global error");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "global.error\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
}
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// *****************************************************************************************************************************
if (testBegin("cmdArchiveGet()"))
{
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
harnessLogLevelSet(logLevelDetail);
// Arguments that must be included. Use raw config here because we need to keep the
StringList *argBaseList = strLstNew();
strLstAddZ(argBaseList, "pgbackrest-bogus");
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH_PG);
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH_REPO);
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test1");
hrnCfgArgRawZ(argBaseList, cfgOptArchiveTimeout, "1");
strLstAddZ(argBaseList, CFGCMD_ARCHIVE_GET);
// -------------------------------------------------------------------------------------------------------------------------
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_TITLE("command must be run on the pg host");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
StringList *argList = strLstDup(argBaseList);
hrnCfgArgRawZ(argList, cfgOptPgHost, BOGUS_STR);
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_ERROR(cmdArchiveGet(), HostInvalidError, "archive-get command must be run on the PostgreSQL host");
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "WAL segment to get required");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000100000001");
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "path to copy WAL segment required");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
storagePathCreateP(storageTest, strNewFmt("%s/pg/pg_wal", testPath()));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
strLstAddZ(argList, TEST_PATH_PG "/pg_wal/RECOVERYXLOG");
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR_FMT(
cmdArchiveGet(), FileMissingError,
"unable to load info file '%s/archive/test1/archive.info' or '%s/archive/test1/archive.info.copy':\n"
"FileMissingError: " STORAGE_ERROR_READ_MISSING "\n"
"FileMissingError: " STORAGE_ERROR_READ_MISSING "\n"
"HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
"HINT: is archive_command configured correctly in postgresql.conf?\n"
"HINT: has a stanza-create been performed?\n"
"HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
" scheme.",
strZ(cfgOptionStr(cfgOptRepoPath)), strZ(cfgOptionStr(cfgOptRepoPath)),
strZ(strNewFmt("%s/archive/test1/archive.info", strZ(cfgOptionStr(cfgOptRepoPath)))),
strZ(strNewFmt("%s/archive/test1/archive.info.copy", strZ(cfgOptionStr(cfgOptRepoPath)))));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// !!! IS THIS TEST NEEDED
argList = strLstDup(argBaseList);
strLstAddZ(argList, "00000001.history");
strLstAddZ(argList, TEST_PATH_PG "/pg_wal/RECOVERYHISTORY");
strLstAddZ(argList, "--archive-async");
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR_FMT(
cmdArchiveGet(), FileMissingError,
"unable to load info file '%s/archive/test1/archive.info' or '%s/archive/test1/archive.info.copy':\n"
"FileMissingError: " STORAGE_ERROR_READ_MISSING "\n"
"FileMissingError: " STORAGE_ERROR_READ_MISSING "\n"
"HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
"HINT: is archive_command configured correctly in postgresql.conf?\n"
"HINT: has a stanza-create been performed?\n"
"HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
" scheme.",
strZ(cfgOptionStr(cfgOptRepoPath)), strZ(cfgOptionStr(cfgOptRepoPath)),
strZ(strNewFmt("%s/archive/test1/archive.info", strZ(cfgOptionStr(cfgOptRepoPath)))),
strZ(strNewFmt("%s/archive/test1/archive.info.copy", strZ(cfgOptionStr(cfgOptRepoPath)))));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// Make sure the process times out when there is nothing to get
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH_SPOOL);
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
strLstAddZ(argList, "000000010000000100000001");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, "pg_wal/RECOVERYXLOG");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
THROW_ON_SYS_ERROR(chdir(strZ(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
Fix archive-push/archive-get when PGDATA is symlinked. Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "timeout getting WAL segment");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: unable to find 000000010000000100000001 in the archive asynchronously");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// Check for missing WAL
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "successful get of missing WAL");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: unable to find 000000010000000100000001 in the archive asynchronously");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
storageExistsP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok")), false,
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
"check OK file was removed");
// Write out a WAL segment for success
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", "SHOULD-BE-A-REAL-WAL-FILE");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
TEST_RESULT_VOID(
harnessLogResult("P00 INFO: found 000000010000000100000001 in the archive asynchronously"), "check log");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", "RECOVERYXLOG\n", .remove = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// Write more WAL segments (in this case queue should be full)
// -------------------------------------------------------------------------------------------------------------------------
strLstAddZ(argList, "--archive-get-queue-max=48");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", "SHOULD-BE-A-REAL-WAL-FILE");
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000002", "SHOULD-BE-A-REAL-WAL-FILE");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
TEST_RESULT_VOID(harnessLogResult("P00 INFO: found 000000010000000100000001 in the archive asynchronously"), "check log");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", "RECOVERYXLOG\n", .remove = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// Make sure the process times out when it can't get a lock
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(
Allow multiple remote locks from the same main process. Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed. The exec-id is also useful for associating remote logs with main logs for debugging purposes.
2020-11-23 12:41:54 -05:00
lockAcquire(
cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptStanza), STRDEF("999-dededede"), cfgLockType(), 30000, true),
"acquire lock");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_VOID(lockClear(true), "clear lock");
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "timeout waiting for lock");
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: unable to find 000000010000000100000001 in the archive asynchronously");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
strLstAddZ(argList, BOGUS_STR);
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
TEST_ERROR(cmdArchiveGet(), ParamInvalidError, "extra parameters found");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_TITLE("pg version does not match archive.info");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
pgControlTestToBuffer((PgControl){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}));
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
argBaseList = strLstNew();
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH_PG);
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH_REPO);
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test1");
argList = strLstDup(argBaseList);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
strLstAddZ(argList, "01ABCDEF01ABCDEF01ABCDEF");
strLstAddZ(argList, TEST_PATH_PG "/pg_wal/RECOVERYXLOG");
harnessCfgLoad(cfgCmdArchiveGet, argList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_ERROR(
cmdArchiveGet(), ArchiveMismatchError,
"unable to retrieve the archive id for database version '11' and system-id '18072658121562454734'");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("pg version does not match archive.info");
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0x8888888888888888}));
TEST_ERROR(
cmdArchiveGet(), ArchiveMismatchError,
"unable to retrieve the archive id for database version '10' and system-id '9838263505978427528'");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("file is missing");
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "get");
harnessLogResult("P00 INFO: unable to find 01ABCDEF01ABCDEF01ABCDEF in the archive");
TEST_STORAGE_LIST_EMPTY(storageTest, TEST_PATH_PG "/pg_wal");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get WAL segment");
Buffer *buffer = bufNew(16 * 1024 * 1024);
memset(bufPtr(buffer), 0, bufSize(buffer));
bufUsedSet(buffer, bufSize(buffer));
HRN_STORAGE_PUT(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
buffer);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_UINT(
storageInfoP(storageTest, STRDEF(TEST_PATH_PG "/pg_wal/RECOVERYXLOG")).size, 16 * 1024 * 1024, "check size");
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", "RECOVERYXLOG\n", .remove = true);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get from prior db-id");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n"
"2={\"db-id\":10000000000000000000,\"db-version\":\"11\"}\n"
"3={\"db-id\":18072658121562454734,\"db-version\":\"10\"}");
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", "RECOVERYXLOG\n", .remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_REMOVE(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get partial");
buffer = bufNew(16 * 1024 * 1024);
memset(bufPtr(buffer), 0xFF, bufSize(buffer));
bufUsedSet(buffer, bufSize(buffer));
HRN_STORAGE_PUT(
storageRepoWrite(),
STORAGE_REPO_ARCHIVE "/10-3/000000010000000100000001.partial-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
buffer);
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000100000001.partial");
strLstAddZ(argList, TEST_PATH_PG "/pg_wal/RECOVERYXLOG");
harnessCfgLoad(cfgCmdArchiveGet, argList);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: found 000000010000000100000001.partial in the repo1 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", "RECOVERYXLOG\n", .remove = true);
TEST_STORAGE_REMOVE(
storageRepoWrite(),
STORAGE_REPO_ARCHIVE "/10-3/000000010000000100000001.partial-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get missing history");
argList = strLstDup(argBaseList);
strLstAddZ(argList, "00000001.history");
strLstAddZ(argList, TEST_PATH_PG "/pg_wal/RECOVERYHISTORY");
harnessCfgLoad(cfgCmdArchiveGet, argList);
TEST_RESULT_INT(cmdArchiveGet(), 1, "get");
harnessLogResult("P00 INFO: unable to find 00000001.history in the archive");
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", NULL);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get history");
HRN_STORAGE_PUT(storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/00000001.history", BUFSTRDEF("HISTORY"));
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: found 00000001.history in the repo1 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_RESULT_UINT(storageInfoP(storageTest, STRDEF(TEST_PATH_PG "/pg_wal/RECOVERYHISTORY")).size, 7, "check size");
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", "RECOVERYHISTORY\n", .remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get compressed and encrypted WAL segment");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[cipher]\n"
"cipher-pass=\"" TEST_CIPHER_PASS_ARCHIVE "\"\n"
"\n"
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}",
.cipherType = cipherTypeAes256Cbc);
HRN_STORAGE_PUT(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
buffer, .compressType = compressTypeGz, .cipherType = cipherTypeAes256Cbc, .cipherPass = TEST_CIPHER_PASS_ARCHIVE);
// Add encryption options
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
argList = strLstDup(argBaseList);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawZ(argList, cfgOptRepoCipherType, CIPHER_TYPE_AES_256_CBC);
hrnCfgEnvRawZ(cfgOptRepoCipherPass, TEST_CIPHER_PASS);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "01ABCDEF01ABCDEF01ABCDEF");
strLstAddZ(argList, TEST_PATH_PG "/pg_wal/RECOVERYXLOG");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
harnessCfgLoad(cfgCmdArchiveGet, argList);
hrnCfgEnvRemoveRaw(cfgOptRepoCipherPass);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Improve logging in archive-get command. Append "asynchronously" to messages when the async process fetched the file (not in the actual async process log, though). Add "repo1" to make it clear what archive we are talking about. This is not very useful by itself but soon we'll be able to add the archive id, which is very useful. Add constants for messages that are used multiple times to ensure they stay consistent.
2021-01-13 10:24:47 -05:00
harnessLogResult("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(storageTest, TEST_PATH_PG "/pg_wal", "RECOVERYXLOG\n");
TEST_RESULT_UINT(
storageInfoP(storageTest, STRDEF(TEST_PATH_PG "/pg_wal/RECOVERYXLOG")).size, 16 * 1024 * 1024, "check size");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("call protocol function directly");
// Start a protocol server
Buffer *serverWrite = bufNew(8192);
IoWrite *serverWriteIo = ioBufferWriteNew(serverWrite);
ioWriteOpen(serverWriteIo);
ProtocolServer *server = protocolServerNew(
strNew("test"), strNew("test"), ioBufferReadNew(bufNew(0)), serverWriteIo);
bufUsedSet(serverWrite, 0);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// Add archive-async and spool path
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH_SPOOL);
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
hrnCfgEnvRawZ(cfgOptRepoCipherPass, TEST_CIPHER_PASS);
harnessCfgLoadRole(cfgCmdArchiveGet, cfgCmdRoleLocal, argList);
hrnCfgEnvRemoveRaw(cfgOptRepoCipherPass);
// Setup protocol command
VariantList *paramList = varLstNew();
varLstAdd(paramList, varNewStrZ("01ABCDEF01ABCDEF01ABCDEF"));
TEST_RESULT_BOOL(
archiveGetProtocol(PROTOCOL_COMMAND_ARCHIVE_GET_STR, paramList, server), true, "protocol archive get");
TEST_RESULT_STR_Z(strNewBuf(serverWrite), "{\"out\":0}\n", "check result");
TEST_STORAGE_LIST(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000002\n01ABCDEF01ABCDEF01ABCDEF\n");
bufUsedSet(serverWrite, 0);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("invalid protocol command");
TEST_RESULT_BOOL(archiveGetProtocol(strNew(BOGUS_STR), paramList, server), false, "invalid function");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_RESULT_VOID();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Reference in New Issue Copy Permalink