db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-13 01:00:23 +02:00
Code Issues Releases Activity
Files
5c65d88f6254176c257f0bf58ca5c9f59bb99177
pgbackrest/test/src/module/command/archivePushTest.c

675 lines
35 KiB
C
Raw Normal View History

The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
/***********************************************************************************************************************************
Test Archive Push Command
***********************************************************************************************************************************/
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
#include "common/io/bufferRead.h"
#include "common/io/bufferWrite.h"
#include "common/io/handleRead.h"
#include "common/io/handleWrite.h"
#include "common/time.h"
#include "postgres/version.h"
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
#include "storage/posix/storage.h"
Full abstraction of storage driver interface. The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly. Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC). Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
Split cfgLoad() into multiple functions to make testing easier. Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
#include "common/harnessConfig.h"
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
#include "common/harnessFork.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
#include "common/harnessInfo.h"
Split cfgLoad() into multiple functions to make testing easier. Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
Fix non-compliant formatting for function declarations. Per review by Cynthia Shang.
2018-01-31 18:22:25 -05:00
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// Create default storage object for testing
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
Storage *storageTest = storagePosixNew(
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
strNew(testPath()), STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, true, NULL);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Start a protocol server to test the protocol directly
Buffer *serverWrite = bufNew(8192);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *serverWriteIo = ioBufferWriteNew(serverWrite);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteOpen(serverWriteIo);
ProtocolServer *server = protocolServerNew(
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
strNew("test"), strNew("test"), ioBufferReadNew(bufNew(0)), serverWriteIo);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
bufUsedSet(serverWrite, 0);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// *****************************************************************************************************************************
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
if (testBegin("archivePushReadyList(), archivePushProcessList(), and archivePushDrop()"))
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
{
StringList *argList = strLstNew();
strLstAddZ(argList, "--stanza=db");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAdd(argList, strNewFmt("--pg1-path=%s/db", testPath()));
strLstAdd(argList, strNewFmt("--spool-path=%s/spool", testPath()));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePushAsync, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storagePathCreateNP(storagePgWrite(), strNew("pg_wal/archive_status"));
storagePathCreateNP(storageTest, strNew("spool/archive/db/out"));
// Create ok files to indicate WAL that has already been archived
storagePutNP(
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok")), NULL);
storagePutNP(
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000003.ok")), NULL);
storagePutNP(
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000004.ok")), NULL);
storagePutNP(
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000005.error")), NULL);
storagePutNP(
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000006.error")), NULL);
storagePutNP(
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/global.error")), NULL);
// Create ready files for wal that still needs to be archived
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/archive_status/000000010000000100000002.ready")), NULL);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/archive_status/000000010000000100000003.ready")), NULL);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/archive_status/000000010000000100000005.ready")), NULL);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/archive_status/000000010000000100000006.ready")), NULL);
TEST_RESULT_STR(
strPtr(strLstJoin(archivePushProcessList(strNewFmt("%s/db/pg_wal", testPath())), "|")),
"000000010000000100000002|000000010000000100000005|000000010000000100000006", "ready list");
TEST_RESULT_STR(
strPtr(strLstJoin(strLstSort(storageListNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc), "|")),
"000000010000000100000003.ok", "remaining status list");
// Test drop
// -------------------------------------------------------------------------------------------------------------------------
StringList *argListDrop = strLstDup(argList);
strLstAdd(argListDrop, strNewFmt("--archive-push-queue-max=%zu", (size_t)1024 * 1024 * 1024));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePushAsync, argListDrop);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Write the files that we claim are in pg_wal
Buffer *walBuffer = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
pgWalTestToBuffer((PgWal){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}, walBuffer);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000002")), walBuffer);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000003")), walBuffer);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000005")), walBuffer);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000006")), walBuffer);
// Queue max is high enough that no WAL will be dropped
TEST_RESULT_BOOL(
archivePushDrop(strNew("pg_wal"), archivePushProcessList(strNewFmt("%s/db/pg_wal", testPath()))), false,
"wal is not dropped");
// Now set queue max low enough that WAL will be dropped
argListDrop = strLstDup(argList);
strLstAdd(argListDrop, strNewFmt("--archive-push-queue-max=%zu", (size_t)16 * 1024 * 1024 * 2));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePushAsync, argListDrop);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
archivePushDrop(strNew("pg_wal"), archivePushProcessList(strNewFmt("%s/db/pg_wal", testPath()))), true,
"wal is dropped");
}
// *****************************************************************************************************************************
if (testBegin("archivePushCheck()"))
{
StringList *argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
strLstAdd(argList, strNewFmt("--pg1-path=%s/pg", testPath()));
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Check mismatched pg_control and archive.info
// -------------------------------------------------------------------------------------------------------------------------
storagePutNP(
storageNewWriteNP(storageTest, strNew("pg/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
pgControlTestToBuffer((PgControl){.version = PG_VERSION_96, .systemId = 0xFACEFACEFACEFACE}));
// Create incorrect archive info
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"));
TEST_ERROR(
archivePushCheck(cipherTypeNone, NULL), ArchiveMismatchError,
"PostgreSQL version 9.6, system-id 18072658121562454734 do not match stanza version 9.4, system-id 5555555555555555555"
"\nHINT: are you archiving to the correct stanza?");
// Fix the version
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.6\"}\n"));
TEST_ERROR(
archivePushCheck(cipherTypeNone, NULL), ArchiveMismatchError,
"PostgreSQL version 9.6, system-id 18072658121562454734 do not match stanza version 9.6, system-id 5555555555555555555"
"\nHINT: are you archiving to the correct stanza?");
// Fix archive info
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"9.6\"}\n"));
ArchivePushCheckResult result = {0};
TEST_ASSIGN(result, archivePushCheck(cipherTypeNone, NULL), "get archive check result");
TEST_RESULT_UINT(result.pgVersion, PG_VERSION_96, "check pg version");
TEST_RESULT_UINT(result.pgSystemId, 0xFACEFACEFACEFACE, "check pg system id");
TEST_RESULT_UINT(result.pgWalSegmentSize, 16 * 1024 * 1024, "check wal segment size");
TEST_RESULT_STR(strPtr(result.archiveId), "9.6-1", "check archive id");
TEST_RESULT_STR(strPtr(result.archiveCipherPass), NULL, "check archive cipher pass (not set in this test)");
}
// *****************************************************************************************************************************
if (testBegin("Synchronous cmdArchivePush(), archivePushFile() and archivePushProtocol()"))
{
StringList *argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argList);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
TEST_ERROR(cmdArchivePush(), ParamRequiredError, "WAL segment to push required");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Create pg_control and archive.info
// -------------------------------------------------------------------------------------------------------------------------
strLstAdd(argList, strNewFmt("--pg1-path=%s/pg", testPath()));
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
StringList *argListTemp = strLstDup(argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argListTemp, "pg_wal/000000010000000100000001");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storagePutNP(
storageNewWriteNP(storageTest, strNew("pg/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
pgControlTestToBuffer((PgControl){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}));
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"11\"}\n"));
// Generate WAL with incorrect headers and try to push them
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Buffer *walBuffer1 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer1, bufSize(walBuffer1));
memset(bufPtr(walBuffer1), 0, bufSize(walBuffer1));
pgWalTestToBuffer((PgWal){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}, walBuffer1);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000001")), walBuffer1);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
THROW_ON_SYS_ERROR(chdir(strPtr(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
TEST_ERROR(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
cmdArchivePush(), ArchiveMismatchError,
strPtr(
strNewFmt(
"WAL file '%s/pg/pg_wal/000000010000000100000001' version 10, system-id 18072658121562454734 do not match"
" stanza version 11, system-id 18072658121562454734",
testPath())));
memset(bufPtr(walBuffer1), 0, bufSize(walBuffer1));
pgWalTestToBuffer((PgWal){.version = PG_VERSION_11, .systemId = 0xECAFECAFECAFECAF}, walBuffer1);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000001")), walBuffer1);
TEST_ERROR(
cmdArchivePush(), ArchiveMismatchError,
strPtr(
strNewFmt(
"WAL file '%s/pg/pg_wal/000000010000000100000001' version 11, system-id 17055110554209741999 do not match"
" stanza version 11, system-id 18072658121562454734",
testPath())));
// Generate valid WAL and push them
// -------------------------------------------------------------------------------------------------------------------------
memset(bufPtr(walBuffer1), 0, bufSize(walBuffer1));
pgWalTestToBuffer((PgWal){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}, walBuffer1);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000001")), walBuffer1);
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
harnessLogResult("P00 INFO: pushed WAL file '000000010000000100000001' to the archive");
TEST_RESULT_BOOL(
storageExistsNP(
storageTest,
strNewFmt(
"repo/archive/test/11-1/0000000100000001/000000010000000100000001-%s.gz",
TEST_64BIT() ? "3e5ecd22712f319b2420d5b901fd29f4f6be2336" : "6903dce7e3cd64ba9a6134056405eaeb8dedcd37")),
true, "check repo for WAL file");
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment again");
harnessLogResult(
"P00 WARN: WAL file '000000010000000100000001' already exists in the archive with the same checksum\n"
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P00 INFO: pushed WAL file '000000010000000100000001' to the archive");
// Now create a new WAL buffer with a different checksum to test checksum errors
Buffer *walBuffer2 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer2, bufSize(walBuffer2));
memset(bufPtr(walBuffer2), 0xFF, bufSize(walBuffer2));
pgWalTestToBuffer((PgWal){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}, walBuffer2);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000001")), walBuffer2);
TEST_ERROR(cmdArchivePush(), ArchiveDuplicateError, "WAL file '000000010000000100000001' already exists in the archive");
// Save it to a new file instead
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "pg_wal/000000010000000100000002");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/000000010000000100000002")), walBuffer2);
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
harnessLogResult("P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
TEST_RESULT_BOOL(
storageExistsNP(
storageTest,
strNewFmt(
"repo/archive/test/11-1/0000000100000001/000000010000000100000002-%s.gz",
TEST_64BIT() ? "edad2f5a9d8a03ee3c09e8ce92c771e0d20232f5" : "e7c81f5513e0c6e3f19b9dbfc450019165994dda")),
true, "check repo for WAL file");
// Push a history file
// -------------------------------------------------------------------------------------------------------------------------
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "pg_wal/00000001.history");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/00000001.history")), BUFSTRDEF("FAKEHISTORY"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push a history file");
harnessLogResult("P00 INFO: pushed WAL file '00000001.history' to the archive");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
storageExistsNP(storageTest, strNew("repo/archive/test/11-1/00000001.history")), true, "check repo for history file");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Check drop functionality
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/archive_status/000000010000000100000001.ready")), NULL);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_wal/archive_status/000000010000000100000002.ready")), NULL);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=16m");
strLstAddZ(argListTemp, "pg_wal/000000010000000100000002");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "drop WAL file");
harnessLogResult("P00 WARN: dropped WAL file '000000010000000100000002' because archive queue exceeded 16MB");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=1GB");
strLstAddZ(argListTemp, "pg_wal/000000010000000100000002");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push WAL file again");
harnessLogResult(
"P00 WARN: WAL file '000000010000000100000002' already exists in the archive with the same checksum\n"
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
// Check protocol function directly
// -------------------------------------------------------------------------------------------------------------------------
VariantList *paramList = varLstNew();
varLstAdd(paramList, varNewStr(strNewFmt("%s/pg/pg_wal/000000010000000100000002", testPath())));
varLstAdd(paramList, varNewStrZ("11-1"));
varLstAdd(paramList, varNewUInt64(PG_VERSION_11));
varLstAdd(paramList, varNewUInt64(0xFACEFACEFACEFACE));
varLstAdd(paramList, varNewStrZ("000000010000000100000002"));
varLstAdd(paramList, varNewUInt64(cipherTypeNone));
varLstAdd(paramList, NULL);
varLstAdd(paramList, varNewBool(false));
varLstAdd(paramList, varNewInt(6));
TEST_RESULT_BOOL(
archivePushProtocol(PROTOCOL_COMMAND_ARCHIVE_PUSH_STR, paramList, server), true, "protocol archive put");
TEST_RESULT_STR(
strPtr(strNewBuf(serverWrite)),
"{\"out\":\"WAL file '000000010000000100000002' already exists in the archive with the same checksum"
"\\nHINT: this is valid in some recovery scenarios but may also indicate a problem.\"}\n",
"check result");
bufUsedSet(serverWrite, 0);
// Check invalid protocol function
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(archivePushProtocol(strNew(BOGUS_STR), paramList, server), false, "invalid function");
// Create a new encrypted repo to test encryption
// -------------------------------------------------------------------------------------------------------------------------
storagePathRemoveP(storageTest, strNew("repo"), .errorOnMissing = true, .recurse = true);
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
StorageWrite *infoWrite = storageNewWriteNP(storageTest, strNew("repo/archive/test/archive.info"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
ioFilterGroupAdd(
ioWriteFilterGroup(storageWriteIo(infoWrite)), cipherBlockNew(cipherModeEncrypt, cipherTypeAes256Cbc,
BUFSTRDEF("badpassphrase"), NULL));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Storage object improvements. * Convert all functions to variadic functions. * Enforce read-only storage. * Add storageLocalWrite() helper function. Add storageExists(), storagePathCreate(), storageRemove(), and storageStat(). * Add StorageFile object and storageOpenRead()/storageOpenWrite().
2018-04-03 12:25:21 -04:00
storagePutNP(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
infoWrite,
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[cipher]\n"
"cipher-pass=\"badsubpassphrase\"\n"
"\n"
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"11\"}"));
// Push encrypted WAL segment
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "pg_wal/000000010000000100000002");
strLstAddZ(argListTemp, "--repo1-cipher-type=aes-256-cbc");
strLstAddZ(argListTemp, "--no-compress");
setenv("PGBACKREST_REPO1_CIPHER_PASS", "badpassphrase", true);
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
unsetenv("PGBACKREST_REPO1_CIPHER_PASS");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
harnessLogResult("P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
storageExistsNP(
storageTest,
strNewFmt(
"repo/archive/test/11-1/0000000100000001/000000010000000100000002-%s",
TEST_64BIT() ? "edad2f5a9d8a03ee3c09e8ce92c771e0d20232f5" : "e7c81f5513e0c6e3f19b9dbfc450019165994dda")),
true, "check repo for WAL file");
}
// *****************************************************************************************************************************
if (testBegin("Asynchronous cmdArchivePush() and cmdArchivePushAsync()"))
{
harnessLogLevelSet(logLevelDetail);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Call with a bogus exe name so the async process will error out and we can make sure timeouts work
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// -------------------------------------------------------------------------------------------------------------------------
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
StringList *argList = strLstNew();
strLstAddZ(argList, "pgbackrest-bogus");
strLstAddZ(argList, "--stanza=test");
strLstAddZ(argList, "--archive-async");
strLstAddZ(argList, "--archive-timeout=1");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
strLstAdd(argList, strNewFmt("--lock-path=%s/lock", testPath()));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAdd(argList, strNewFmt("--spool-path=%s/spool", testPath()));
strLstAdd(argList, strNewFmt("--pg1-path=%s/pg", testPath()));
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
strLstAddZ(argList, "archive-push");
strLstAddZ(argList, "pg_wal/bogus");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
THROW_ON_SYS_ERROR(chdir(testPath()) != 0, PathMissingError, "unable to chdir()");
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
TEST_ERROR(
cmdArchivePush(), ArchiveTimeoutError,
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"unable to push WAL file 'bogus' to the archive asynchronously after 1 second(s)");
// Create pg_control and archive.info
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
strLstAddZ(argList, "--archive-async");
strLstAddZ(argList, "--no-compress");
strLstAdd(argList, strNewFmt("--spool-path=%s/spool", testPath()));
strLstAdd(argList, strNewFmt("--pg1-path=%s/pg", testPath()));
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
strLstAddZ(argList, "--log-subprocess");
storagePutNP(
storageNewWriteNP(storageTest, strNew("pg/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
pgControlTestToBuffer((PgControl){.version = PG_VERSION_94, .systemId = 0xAAAABBBBCCCCDDDD}));
storagePutNP(
storageNewWriteNP(storageTest, strNew("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":12297848147757817309,\"db-version\":\"9.4\"}\n"));
// Write out an error file that will be ignored on the first pass, then the async process will write a new one
// -------------------------------------------------------------------------------------------------------------------------
StringList *argListTemp = strLstDup(argList);
strLstAdd(argListTemp, strNewFmt("%s/pg/pg_xlog/000000010000000100000001", testPath()));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storagePathCreateNP(storagePgWrite(), strNew("pg_xlog/archive_status"));
storagePutNP(
storageNewWriteNP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.error")),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF("25\nbogus error"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_ERROR(cmdArchivePush(), AssertError, "no WAL files to process");
storageRemoveP(storageSpoolWrite(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/global.error"), .errorOnMissing = true);
// Acquire a lock so the async process will not be able to run -- this will result in a timeout
// -------------------------------------------------------------------------------------------------------------------------
argListTemp = strLstDup(argList);
strLstAdd(argListTemp, strNewFmt("%s/pg/pg_xlog/000000010000000100000001", testPath()));
strLstAddZ(argListTemp, "--archive-timeout=1");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
THROW_ON_SYS_ERROR(chdir(strPtr(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
HARNESS_FORK_BEGIN()
{
HARNESS_FORK_CHILD_BEGIN(0, true)
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("child read"), HARNESS_FORK_CHILD_READ(), 2000);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("child write"), HARNESS_FORK_CHILD_WRITE());
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteOpen(write);
lockAcquire(cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptStanza), cfgLockType(), 30000, true);
// Let the parent know the lock has been acquired and wait for the parent to allow lock release
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteStrLine(write, strNew(""));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteFlush(write);
ioReadLine(read);
lockRelease(true);
}
HARNESS_FORK_CHILD_END();
HARNESS_FORK_PARENT_BEGIN()
{
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoRead *read = ioHandleReadNew(strNew("parent read"), HARNESS_FORK_PARENT_READ_PROCESS(0), 2000);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioReadOpen(read);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *write = ioHandleWriteNew(strNew("parent write"), HARNESS_FORK_PARENT_WRITE_PROCESS(0));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteOpen(write);
// Wait for the child to acquire the lock
ioReadLine(read);
TEST_ERROR(
cmdArchivePush(), ArchiveTimeoutError,
"unable to push WAL file '000000010000000100000001' to the archive asynchronously after 1 second(s)");
// Notify the child to release the lock
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteLine(write, bufNew(0));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteFlush(write);
}
HARNESS_FORK_PARENT_END();
}
HARNESS_FORK_END();
// Actually push a WAL file
// -------------------------------------------------------------------------------------------------------------------------
argListTemp = strLstDup(argList);
strLstAdd(argListTemp, strNewFmt("%s/pg/pg_xlog/000000010000000100000001", testPath()));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_xlog/archive_status/000000010000000100000001.ready")), NULL);
Buffer *walBuffer1 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer1, bufSize(walBuffer1));
memset(bufPtr(walBuffer1), 0xFF, bufSize(walBuffer1));
pgWalTestToBuffer((PgWal){.version = PG_VERSION_94, .systemId = 0xAAAABBBBCCCCDDDD}, walBuffer1);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_xlog/000000010000000100000001")), walBuffer1);
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
harnessLogResult("P00 INFO: pushed WAL file '000000010000000100000001' to the archive asynchronously");
TEST_RESULT_BOOL(
storageExistsNP(
storageTest,
strNewFmt(
"repo/archive/test/9.4-1/0000000100000001/000000010000000100000001-%s",
TEST_64BIT() ? "f81d63dd5e258cd607534f3531bbd71442797e37" : "02d228126281e8e102b35a2737e45a0527946296")),
true, "check repo for WAL file");
// Direct tests of the async function
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
strLstAddZ(argList, "--no-compress");
strLstAdd(argList, strNewFmt("--spool-path=%s/spool", testPath()));
strLstAdd(argList, strNewFmt("--pg1-path=%s/pg", testPath()));
strLstAdd(argList, strNewFmt("--repo1-path=%s/repo", testPath()));
strLstAddZ(argList, "--log-subprocess");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePushAsync, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_ERROR(cmdArchivePushAsync(), ParamRequiredError, "WAL path to push required");
// Check that global.error is created
// -------------------------------------------------------------------------------------------------------------------------
// Remove data from prior tests
storagePathRemoveP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR, .recurse = true);
storagePathCreateNP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR);
storagePathRemoveP(storageRepoWrite(), strNew(STORAGE_REPO_ARCHIVE "/9.4-1"), .recurse = true);
storagePathCreateNP(storageRepoWrite(), strNew(STORAGE_REPO_ARCHIVE "/9.4-1"));
storagePathRemoveP(storagePgWrite(), strNew("pg_xlog/archive_status"), .recurse = true);
storagePathCreateNP(storagePgWrite(), strNew("pg_xlog/archive_status"));
strLstAdd(argList, strNewFmt("%s/pg/pg_xlog", testPath()));
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePushAsync, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_ERROR(cmdArchivePushAsync(), AssertError, "no WAL files to process");
TEST_RESULT_STR(
strPtr(strNewBuf(storageGetNP(storageNewReadNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/global.error"))))),
"25\nno WAL files to process", "check global.error");
TEST_RESULT_STR(
strPtr(strLstJoin(strLstSort(storageListNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc), "|")),
"global.error", "check status files");
// Push WAL
// -------------------------------------------------------------------------------------------------------------------------
// Recreate ready file for WAL 1
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_xlog/archive_status/000000010000000100000001.ready")), NULL);
// Create a ready file for WAL 2 but don't create the segment yet -- this will test the file error
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_xlog/archive_status/000000010000000100000002.ready")), NULL);
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segments");
harnessLogResult(
strPtr(
strNewFmt(
"P00 INFO: push 2 WAL file(s) to archive: 000000010000000100000001...000000010000000100000002\n"
Add process id to C archive-get and archive-push logging. This was missed in the original migration. There was no functional issue, but logging the process ids is useful for debugging.
2019-04-09 11:08:27 -04:00
"P01 DETAIL: pushed WAL file '000000010000000100000001' to the archive\n"
"P01 WARN: could not push WAL file '000000010000000100000002' to the archive (will be retried): "
Refactoring path support in the storage module. Not all storage types support paths as a physical thing that must be created/destroyed. Add a feature to determine which drivers use paths and simplify the driver API as much as possible given that knowledge and by implementing as much path logic as possible in the Storage object. Remove the ignoreMissing parameter from pathSync() since it is not used and makes little sense. Create a standard list of error messages for the drivers to use and apply them where the code was modified -- there is plenty of work still to be done here.
2019-05-26 12:41:15 -04:00
"[55] raised from local-1 protocol: " STORAGE_ERROR_READ_MISSING,
strPtr(strNewFmt("%s/pg/pg_xlog/000000010000000100000002", testPath())))));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
storageExistsNP(
storageTest,
strNewFmt(
"repo/archive/test/9.4-1/0000000100000001/000000010000000100000001-%s",
TEST_64BIT() ? "f81d63dd5e258cd607534f3531bbd71442797e37" : "02d228126281e8e102b35a2737e45a0527946296")),
true, "check repo for WAL 1 file");
TEST_RESULT_STR(
strPtr(strLstJoin(strLstSort(storageListNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc), "|")),
"000000010000000100000001.ok|000000010000000100000002.error", "check status files");
// Create WAL 2 segment
Buffer *walBuffer2 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer2, bufSize(walBuffer2));
memset(bufPtr(walBuffer2), 0x0C, bufSize(walBuffer2));
pgWalTestToBuffer((PgWal){.version = PG_VERSION_94, .systemId = 0xAAAABBBBCCCCDDDD}, walBuffer2);
storagePutNP(storageNewWriteNP(storagePgWrite(), strNew("pg_xlog/000000010000000100000002")), walBuffer2);
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=1gb");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePushAsync, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segments");
harnessLogResult(
"P00 INFO: push 1 WAL file(s) to archive: 000000010000000100000002\n"
Add process id to C archive-get and archive-push logging. This was missed in the original migration. There was no functional issue, but logging the process ids is useful for debugging.
2019-04-09 11:08:27 -04:00
"P01 DETAIL: pushed WAL file '000000010000000100000002' to the archive");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
storageExistsNP(
storageTest,
strNewFmt(
"repo/archive/test/9.4-1/0000000100000001/000000010000000100000002-%s",
TEST_64BIT() ? "0aea6fa5d53500ce548b84a86bc3a29ae77fa048" : "408822a89ef44ef6740e785743bf1b870d8024a2")),
true, "check repo for WAL 2 file");
TEST_RESULT_STR(
strPtr(strLstJoin(strLstSort(storageListNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc), "|")),
"000000010000000100000001.ok|000000010000000100000002.ok", "check status files");
// Check that drop functionality works
// -------------------------------------------------------------------------------------------------------------------------
// Remove status files
storagePathRemoveP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR, .recurse = true);
storagePathCreateNP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR);
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=16m");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePushAsync, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segments");
harnessLogResult(
"P00 INFO: push 2 WAL file(s) to archive: 000000010000000100000001...000000010000000100000002\n"
"P00 WARN: dropped WAL file '000000010000000100000001' because archive queue exceeded 16MB\n"
"P00 WARN: dropped WAL file '000000010000000100000002' because archive queue exceeded 16MB");
TEST_RESULT_STR(
strPtr(
strNewBuf(
storageGetNP(
storageNewReadNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok"))))),
"0\ndropped WAL file '000000010000000100000001' because archive queue exceeded 16MB", "check WAL 1 warning");
TEST_RESULT_STR(
strPtr(
strNewBuf(
storageGetNP(
storageNewReadNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000002.ok"))))),
"0\ndropped WAL file '000000010000000100000002' because archive queue exceeded 16MB", "check WAL 2 warning");
TEST_RESULT_STR(
strPtr(strLstJoin(strLstSort(storageListNP(storageSpool(), strNew(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc), "|")),
"000000010000000100000001.ok|000000010000000100000002.ok", "check status files");
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_RESULT_VOID();
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
}
Reference in New Issue Copy Permalink