db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-06-06 23:16:26 +02:00
Code Issues Releases Activity
pgbackrest/test/src/module/command/archiveGetTest.c

519 lines
26 KiB
C
Raw Normal View History

Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Archive Get Command
***********************************************************************************************************************************/
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
#include "postgres/interface.h"
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
#include "postgres/version.h"
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
#include "common/harnessConfig.h"
#include "common/harnessFork.h"
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
#include "compress/gzipCompress.h"
Full abstraction of storage driver interface. The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly. Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC). Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
#include "storage/driver/posix/storage.h"
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Full abstraction of storage driver interface. The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly. Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC). Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
Storage *storageTest = storageDriverPosixInterface(
storageDriverPosixNew(strNew(testPath()), STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, true, NULL));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// *****************************************************************************************************************************
if (testBegin("archiveGetCheck()"))
{
// Load Parameters
StringList *argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
strLstAdd(argList, strNewFmt("--pg1-path=%s/db", testPath()));
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
// Create pg_control file
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
storagePutNP(
storageNewWriteNP(storageTest, strNew("db/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Control and archive info mismatch
// -------------------------------------------------------------------------------------------------------------------------
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test1/archive.info")),
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
bufNewZ(
"[backrest]\n"
"backrest-checksum=\"0a415a03fa3faccb4ac171759895478469e9e19e\"\n"
"backrest-format=5\n"
"backrest-version=\"2.06\"\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
TEST_ERROR(
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL), ArchiveMismatchError,
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"unable to retrieve the archive id for database version '10' and system-id '18072658121562454734'");
// Nothing to find in empty archive dir
// -------------------------------------------------------------------------------------------------------------------------
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test1/archive.info")),
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
bufNewZ(
"[backrest]\n"
"backrest-checksum=\"f7617b5c4c9f212f40b9bc3d8ec7f97edbbf96af\"\n"
"backrest-format=5\n"
"backrest-version=\"2.06\"\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"
"2={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n"
"3={\"db-id\":18072658121562454734,\"db-version\":\"9.6\"}\n"
"4={\"db-id\":18072658121562454734,\"db-version\":\"10\"}"));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
TEST_RESULT_PTR(
archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL).archiveFileActual, NULL, "no segment found");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Write segment into an older archive path
// -------------------------------------------------------------------------------------------------------------------------
storagePutNP(
storageNewWriteNP(
storageTest,
strNew(
"repo/archive/test1/10-2/8765432187654321/876543218765432187654321-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")),
NULL);
TEST_RESULT_STR(
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
strPtr(archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL).archiveFileActual),
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"10-2/8765432187654321/876543218765432187654321-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "segment found");
// Write segment into an newer archive path
// -------------------------------------------------------------------------------------------------------------------------
storagePutNP(
storageNewWriteNP(
storageTest,
strNew(
"repo/archive/test1/10-4/8765432187654321/876543218765432187654321-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb")),
NULL);
TEST_RESULT_STR(
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
strPtr(archiveGetCheck(strNew("876543218765432187654321"), cipherTypeNone, NULL).archiveFileActual),
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"10-4/8765432187654321/876543218765432187654321-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "newer segment found");
// Get history file
// -------------------------------------------------------------------------------------------------------------------------
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
TEST_RESULT_PTR(
archiveGetCheck(strNew("00000009.history"), cipherTypeNone, NULL).archiveFileActual, NULL, "history file not found");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
storagePutNP(storageNewWriteNP(storageTest, strNew("repo/archive/test1/10-4/00000009.history")), NULL);
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
TEST_RESULT_STR(
strPtr(
archiveGetCheck(
strNew("00000009.history"), cipherTypeNone, NULL).archiveFileActual), "10-4/00000009.history",
"history file found");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
}
// *****************************************************************************************************************************
if (testBegin("archiveGetFile()"))
{
// Load Parameters
StringList *argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
strLstAddZ(argList, "--stanza=test1");
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
strLstAdd(argList, strNewFmt("--pg1-path=%s/db", testPath()));
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
// Create pg_control file
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
storagePutNP(
storageNewWriteNP(storageTest, strNew("db/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Create archive.info
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test1/archive.info")),
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
bufNewZ(
"[backrest]\n"
"backrest-checksum=\"8a041a4128eaa2c08a23dd1f04934627795946ff\"\n"
"backrest-format=5\n"
"backrest-version=\"2.06\"\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}"));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Nothing to copy
// -------------------------------------------------------------------------------------------------------------------------
String *archiveFile = strNew("01ABCDEF01ABCDEF01ABCDEF");
String *walDestination = strNewFmt("%s/db/pg_wal/RECOVERYXLOG", testPath());
storagePathCreateNP(storageTest, strPath(walDestination));
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
TEST_RESULT_INT(archiveGetFile(archiveFile, walDestination, cipherTypeNone, NULL), 1, "WAL segment missing");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
// Create a WAL segment to copy
// -------------------------------------------------------------------------------------------------------------------------
Buffer *buffer = bufNew(16 * 1024 * 1024);
memset(bufPtr(buffer), 0, bufSize(buffer));
bufUsedSet(buffer, bufSize(buffer));
storagePutNP(
storageNewWriteNP(
storageTest,
strNew(
"repo/archive/test1/10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")),
buffer);
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
TEST_RESULT_INT(archiveGetFile(archiveFile, walDestination, cipherTypeNone, NULL), 0, "WAL segment copied");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
TEST_RESULT_BOOL(storageExistsNP(storageTest, walDestination), true, " check exists");
TEST_RESULT_INT(storageInfoNP(storageTest, walDestination).size, 16 * 1024 * 1024, " check size");
storageRemoveP(
storageTest,
strNew("repo/archive/test1/10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"),
.errorOnMissing = true);
storageRemoveP(storageTest, walDestination, .errorOnMissing = true);
// Create a compressed WAL segment to copy
// -------------------------------------------------------------------------------------------------------------------------
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
StorageFileWrite *infoWrite = storageNewWriteNP(storageTest, strNew("repo/archive/test1/archive.info"));
ioWriteFilterGroupSet(
storageFileWriteIo(infoWrite),
ioFilterGroupAdd(
ioFilterGroupNew(),
cipherBlockFilter(cipherBlockNew(cipherModeEncrypt, cipherTypeAes256Cbc, bufNewStr(strNew("12345678")), NULL))));
storagePutNP(
infoWrite,
bufNewZ(
"[backrest]\n"
"backrest-checksum=\"60bfcb0a5a2c91d203c11d7f1924e99dcdfa0b80\"\n"
"backrest-format=5\n"
"backrest-version=\"2.06\"\n"
"\n"
"[cipher]\n"
"cipher-pass=\"worstpassphraseever\"\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}"));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
StorageFileWrite *destination = storageNewWriteNP(
storageTest,
strNew(
"repo/archive/test1/10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"));
IoFilterGroup *filterGroup = ioFilterGroupNew();
ioFilterGroupAdd(filterGroup, gzipCompressFilter(gzipCompressNew(3, false)));
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
ioFilterGroupAdd(
filterGroup,
cipherBlockFilter(
cipherBlockNew(cipherModeEncrypt, cipherTypeAes256Cbc, bufNewStr(strNew("worstpassphraseever")), NULL)));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
ioWriteFilterGroupSet(storageFileWriteIo(destination), filterGroup);
storagePutNP(destination, buffer);
Enable encryption for archive-get command in C. The decryption filter was added in archiveGetFile() and archiveGetCheck() was modified to return the WAL decryption key stored in archive.info. The rest was plumbing. The mock/archive/1 integration test added encryption to provide coverage for the new code paths while mock/archive/2 dropped encryption to provide coverage for the existing code paths. This caused some churn in the expect logs but there was no change in behavior.
2018-11-28 14:56:26 -05:00
TEST_RESULT_INT(
archiveGetFile(
archiveFile, walDestination, cipherTypeAes256Cbc, strNew("12345678")), 0, "WAL segment copied");
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
TEST_RESULT_BOOL(storageExistsNP(storageTest, walDestination), true, " check exists");
TEST_RESULT_INT(storageInfoNP(storageTest, walDestination).size, 16 * 1024 * 1024, " check size");
}
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// *****************************************************************************************************************************
if (testBegin("queueNeed()"))
{
StringList *argList = strLstNew();
strLstAddZ(argList, "pgbackrest");
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
strLstAddZ(argList, "--stanza=test1");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, "--archive-async");
strLstAdd(argList, strNewFmt("--spool-path=%s/spool", testPath()));
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
size_t queueSize = 16 * 1024 * 1024;
size_t walSegmentSize = 16 * 1024 * 1024;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_ERROR_FMT(
queueNeed(strNew("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92),
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
PathOpenError, "unable to open path '%s/spool/archive/test1/in' for read: [2] No such file or directory", testPath());
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storagePathCreateNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_STR(
strPtr(strLstJoin(queueNeed(strNew("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92), "|")),
"000000010000000100000001|000000010000000100000002", "queue size smaller than min");
// -------------------------------------------------------------------------------------------------------------------------
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
queueSize = (16 * 1024 * 1024) * 3;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_STR(
strPtr(strLstJoin(queueNeed(strNew("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92), "|")),
"000000010000000100000001|000000010000000100000002|000000010000000100000003", "empty queue");
// -------------------------------------------------------------------------------------------------------------------------
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Buffer *walSegmentBuffer = bufNew(walSegmentSize);
memset(bufPtr(walSegmentBuffer), 0, walSegmentSize);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
storagePutNP(
storageNewWriteNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE")), walSegmentBuffer);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
storagePutNP(
storageNewWriteNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF")), walSegmentBuffer);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_STR(
strPtr(strLstJoin(queueNeed(strNew("0000000100000001000000FE"), false, queueSize, walSegmentSize, PG_VERSION_92), "|")),
"000000010000000200000000|000000010000000200000001", "queue has wal < 9.3");
TEST_RESULT_STR(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
strPtr(strLstJoin(storageListNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN)), "|")),
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
"0000000100000001000000FE", "check queue");
// -------------------------------------------------------------------------------------------------------------------------
walSegmentSize = 1024 * 1024;
queueSize = walSegmentSize * 5;
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
storagePutNP(storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN "/junk")), bufNewZ("JUNK"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
storagePutNP(
storageNewWriteNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFE")), walSegmentBuffer);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
storagePutNP(
storageNewWriteNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFF")), walSegmentBuffer);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_STR(
strPtr(strLstJoin(queueNeed(strNew("000000010000000A00000FFD"), true, queueSize, walSegmentSize, PG_VERSION_11), "|")),
"000000010000000B00000000|000000010000000B00000001|000000010000000B00000002", "queue has wal >= 9.3");
TEST_RESULT_STR(
strPtr(strLstJoin(strLstSort(storageListNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_IN)), sortOrderAsc), "|")),
"000000010000000A00000FFE|000000010000000A00000FFF", "check queue");
}
// *****************************************************************************************************************************
if (testBegin("cmdArchiveGet()"))
{
StringList *argList = strLstNew();
Add separate archive-get-async command. This command was previously forked off from the archive-get command which required a bit of artificial option and log manipulation. A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-02-20 15:52:07 +02:00
strLstAddZ(argList, "pgbackrest-bogus"); // Break this until async tests are setup correctly
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, "--archive-timeout=1");
strLstAdd(argList, strNewFmt("--log-path=%s", testPath()));
strLstAdd(argList, strNewFmt("--log-level-file=debug"));
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
strLstAddZ(argList, "--stanza=test1");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, "archive-get");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
{
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "WAL segment to get required");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
}
HARNESS_FORK_END();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
StringList *argListTemp = strLstDup(argList);
String *walSegment = strNew("000000010000000100000001");
strLstAdd(argListTemp, walSegment);
harnessCfgLoad(strLstSize(argListTemp), strLstPtr(argListTemp));
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
{
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "path to copy WAL segment required");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
}
HARNESS_FORK_END();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
storagePutNP(
storageNewWriteNP(storageTest, strNew("db/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
pgControlTestToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
storagePathCreateNP(storageTest, strNewFmt("%s/db/pg_wal", testPath()));
String *walFile = strNewFmt("%s/db/pg_wal/RECOVERYXLOG", testPath());
strLstAdd(argListTemp, walFile);
strLstAdd(argListTemp, strNewFmt("--pg1-path=%s/db", testPath()));
harnessCfgLoad(strLstSize(argListTemp), strLstPtr(argListTemp));
// Test this in a fork so we can use different Perl options in later tests
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
HARNESS_FORK_BEGIN()
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
{
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
TEST_ERROR_FMT(
Improve info error messages introduced in 74b72df9. - Add detail to errors when info files are loaded with incorrect encryption settings. - Throw FileMissingError rather than FileOpenError when both copies of the info file are missing. - If one file is present (but errors) and the other is missing, then return the error for the file that was present. Contributed by Cynthia Shang.
2018-12-10 16:32:41 -05:00
cmdArchiveGet(), FileMissingError,
Improve error message when info files are missing/corrupt. The previous error message only showed the last error. In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked. Show both errors below a generic "unable to load" error. Details are now given explaining exactly why the primary and copy failed. Previously if one file could not be loaded a warning would be output. This has been removed because it is not clear what the user should do in this case. Should they do a stanza-create --force? Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
"unable to load info file '%s/archive/test1/archive.info' or '%s/archive/test1/archive.info.copy':\n"
"FileMissingError: unable to open '%s/archive/test1/archive.info' for read: [2] No such file or directory\n"
"FileMissingError: unable to open '%s/archive/test1/archive.info.copy' for read: [2] No such file or"
" directory\n"
"HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
"HINT: is archive_command configured correctly in postgresql.conf?\n"
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"HINT: has a stanza-create been performed?\n"
"HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
" scheme.",
Improve error message when info files are missing/corrupt. The previous error message only showed the last error. In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked. Show both errors below a generic "unable to load" error. Details are now given explaining exactly why the primary and copy failed. Previously if one file could not be loaded a warning would be output. This has been removed because it is not clear what the user should do in this case. Should they do a stanza-create --force? Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
strPtr(cfgOptionStr(cfgOptRepoPath)), strPtr(cfgOptionStr(cfgOptRepoPath)),
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
strPtr(cfgOptionStr(cfgOptRepoPath)), strPtr(cfgOptionStr(cfgOptRepoPath)));
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
HARNESS_FORK_END();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
argListTemp = strLstDup(argList);
strLstAdd(argListTemp, strNewFmt("--pg1-path=%s/db", testPath()));
strLstAddZ(argListTemp, "00000001.history");
strLstAdd(argListTemp, walFile);
strLstAddZ(argListTemp, "--archive-async");
harnessCfgLoad(strLstSize(argListTemp), strLstPtr(argListTemp));
// Test this in a fork so we can use different Perl options in later tests
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
HARNESS_FORK_BEGIN()
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
{
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
TEST_ERROR_FMT(
Improve info error messages introduced in 74b72df9. - Add detail to errors when info files are loaded with incorrect encryption settings. - Throw FileMissingError rather than FileOpenError when both copies of the info file are missing. - If one file is present (but errors) and the other is missing, then return the error for the file that was present. Contributed by Cynthia Shang.
2018-12-10 16:32:41 -05:00
cmdArchiveGet(), FileMissingError,
Improve error message when info files are missing/corrupt. The previous error message only showed the last error. In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked. Show both errors below a generic "unable to load" error. Details are now given explaining exactly why the primary and copy failed. Previously if one file could not be loaded a warning would be output. This has been removed because it is not clear what the user should do in this case. Should they do a stanza-create --force? Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
"unable to load info file '%s/archive/test1/archive.info' or '%s/archive/test1/archive.info.copy':\n"
"FileMissingError: unable to open '%s/archive/test1/archive.info' for read: [2] No such file or directory\n"
"FileMissingError: unable to open '%s/archive/test1/archive.info.copy' for read: [2] No such file or"
" directory\n"
"HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
"HINT: is archive_command configured correctly in postgresql.conf?\n"
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
"HINT: has a stanza-create been performed?\n"
"HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
" scheme.",
Improve error message when info files are missing/corrupt. The previous error message only showed the last error. In addition, some errors were missed (such as directory permission errors) that could prevent the copy from being checked. Show both errors below a generic "unable to load" error. Details are now given explaining exactly why the primary and copy failed. Previously if one file could not be loaded a warning would be output. This has been removed because it is not clear what the user should do in this case. Should they do a stanza-create --force? Maybe the best idea is to automatically repair the corrupt file, but on the other hand that might just spread corruption if pgBackRest makes the wrong choice.
2018-11-28 18:41:21 -05:00
strPtr(cfgOptionStr(cfgOptRepoPath)), strPtr(cfgOptionStr(cfgOptRepoPath)),
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
strPtr(cfgOptionStr(cfgOptRepoPath)), strPtr(cfgOptionStr(cfgOptRepoPath)));
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Add HARNESS_FORK for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.
2018-05-06 08:56:42 -04:00
HARNESS_FORK_END();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// Make sure the process times out when there is nothing to get
// -------------------------------------------------------------------------------------------------------------------------
strLstAdd(argList, strNewFmt("--spool-path=%s/spool", testPath()));
strLstAddZ(argList, "--archive-async");
strLstAdd(argList, walSegment);
strLstAddZ(argList, "pg_wal/RECOVERYXLOG");
Add separate archive-get-async command. This command was previously forked off from the archive-get command which required a bit of artificial option and log manipulation. A separate command is easier to test and will work on platforms that don't have fork(), e.g. Windows.
2019-02-20 15:52:07 +02:00
strLstAdd(argList, strNewFmt("--pg1-path=%s/db", testPath()));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
{
TEST_RESULT_INT(cmdArchiveGet(), 1, "timeout getting WAL segment");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
}
HARNESS_FORK_END();
Make archive-get info messages consistent between C and Perl implementations. The info messages were spread around and logged differently based on the execution path and in some cases logged nothing at all. Temporarily track the async server status with a flag so that info messages are not output in the async process. The async process will be refactored as a separate command to be exec'd in a future commit.
2018-09-11 12:30:48 -04:00
harnessLogResult("P00 INFO: unable to find 000000010000000100000001 in the archive");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// Check for missing WAL
// -------------------------------------------------------------------------------------------------------------------------
storagePutNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageNewWriteNP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s.ok", strPtr(walSegment))), NULL);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
{
TEST_RESULT_INT(cmdArchiveGet(), 1, "successful get of missing WAL");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
}
HARNESS_FORK_END();
Make archive-get info messages consistent between C and Perl implementations. The info messages were spread around and logged differently based on the execution path and in some cases logged nothing at all. Temporarily track the async server status with a flag so that info messages are not output in the async process. The async process will be refactored as a separate command to be exec'd in a future commit.
2018-09-11 12:30:48 -04:00
harnessLogResult("P00 INFO: unable to find 000000010000000100000001 in the archive");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(
storageExistsNP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s.ok", strPtr(walSegment))), false,
"check OK file was removed");
// Write out a WAL segment for success
// -------------------------------------------------------------------------------------------------------------------------
storagePutNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageNewWriteNP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strPtr(walSegment))),
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
bufNewZ("SHOULD-BE-A-REAL-WAL-FILE"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
{
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
}
HARNESS_FORK_END();
TEST_RESULT_VOID(harnessLogResult("P00 INFO: found 000000010000000100000001 in the archive"), "check log");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
TEST_RESULT_BOOL(
storageExistsNP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strPtr(walSegment))), false,
"check WAL segment was removed from source");
TEST_RESULT_BOOL(storageExistsNP(storageTest, walFile), true, "check WAL segment was moved to destination");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
storageRemoveP(storageTest, walFile, .errorOnMissing = true);
// Write more WAL segments (in this case queue should be full)
// -------------------------------------------------------------------------------------------------------------------------
strLstAddZ(argList, "--archive-get-queue-max=48");
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
String *walSegment2 = strNew("000000010000000100000002");
storagePutNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageNewWriteNP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strPtr(walSegment))),
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
bufNewZ("SHOULD-BE-A-REAL-WAL-FILE"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
storagePutNP(
Merge protocol storage helper into storage helper. These are separated the same way in the Perl code where the remote storage driver is located in the Protocol module. However, in the C code the intention is to implement the remote storage driver as a regular driver in the storage layer rather than making a special case out of it. So, merge the storage helpers. This also has the benefit of making the code a bit simpler. Also separate storageSpool() and storageSpoolWrite() to make it clearer which operations require write access and to maintain consistency with the other storage helper functions.
2018-09-16 14:12:53 -04:00
storageNewWriteNP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strPtr(walSegment2))),
Add bufNewZ() to Buffer object. This constructor creates a Buffer object directly from a zero-terminated string. The old way was to create a String object first, then convert that to a Buffer using bufNewStr(). Updated in all places that used the old pattern.
2018-09-26 18:46:52 +01:00
bufNewZ("SHOULD-BE-A-REAL-WAL-FILE"));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
{
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
}
HARNESS_FORK_END();
TEST_RESULT_VOID(harnessLogResult("P00 INFO: found 000000010000000100000001 in the archive"), "check log");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(storageExistsNP(storageTest, walFile), true, "check WAL segment was moved");
// Make sure the process times out when it can't get a lock
// -------------------------------------------------------------------------------------------------------------------------
TEST_RESULT_VOID(
Construct Wait object in milliseconds instead of fractional seconds. The Wait object accepted a double in the constructor for wait time but used TimeMSec internally. This was done for compatibility with the Perl code. Instead, use TimeMSec in the Wait constructor and make changes as needed to calling code. Note that Perl still uses a double for its Wait object so translation is needed in some places. There are no plans to update the Perl code as it will become obsolete.
2018-11-08 08:37:57 -05:00
lockAcquire(cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptStanza), cfgLockType(), 30000, true), "acquire lock");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_VOID(lockClear(true), "clear lock");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
HARNESS_FORK_BEGIN()
{
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_BEGIN(0, false)
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
{
TEST_RESULT_INT(cmdArchiveGet(), 1, "timeout waiting for lock");
}
Improve fork harness to allow multiple children and setup pipes automatically. There was a lot of extra boilerplate involved in setting up pipes so that is now automated. In some cases testing with multiple children is useful so allow that as well.
2019-02-27 18:07:16 +02:00
HARNESS_FORK_CHILD_END();
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
}
HARNESS_FORK_END();
Make archive-get info messages consistent between C and Perl implementations. The info messages were spread around and logged differently based on the execution path and in some cases logged nothing at all. Temporarily track the async server status with a flag so that info messages are not output in the async process. The async process will be refactored as a separate command to be exec'd in a future commit.
2018-09-11 12:30:48 -04:00
harnessLogResult("P00 INFO: unable to find 000000010000000100000001 in the archive");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
strLstAddZ(argList, BOGUS_STR);
harnessCfgLoad(strLstSize(argList), strLstPtr(argList));
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
TEST_ERROR(cmdArchiveGet(), ParamInvalidError, "extra parameters found");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_RESULT_VOID();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Reference in New Issue Copy Permalink