db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-13 01:00:23 +02:00
Code Issues Releases Activity
Files
d55b9471a89163ed362c64960316c292e51cdd8e
pgbackrest/test/src/module/command/archivePushTest.c

1033 lines
54 KiB
C
Raw Normal View History

The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
/***********************************************************************************************************************************
Test Archive Push Command
***********************************************************************************************************************************/
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
#include "common/io/bufferRead.h"
#include "common/io/bufferWrite.h"
Rename handle to fd. Pretty much everywhere handle is used what is really meant is file descriptor (fd). This terminology got migrated over from Perl and is just not quite correct, or at least not as correct as fd. There were also plenty of places fd was used so now all uses are consistent. The Perl code was not updated but might be in a future commit.
2020-08-05 18:25:07 -04:00
#include "common/io/fdRead.h"
#include "common/io/fdWrite.h"
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
#include "common/time.h"
#include "postgres/version.h"
Simplify storage object names. Remove "File" and "Driver" from object names so they are shorter and easier to keep consistent. Also remove the "driver" directory so storage implementations are visible directly under "storage".
2019-05-03 15:46:15 -04:00
#include "storage/posix/storage.h"
Full abstraction of storage driver interface. The external storage interfaces (Storage, StorageFileRead, etc.) have been stable for a while, but internally they were calling the posix driver functions directly. Create driver interfaces for storage, fileRead, and fileWrite and remove all references to the posix driver outside storage/driver/posix (with the exception of a direct call to pathRemove() in Perl LibC). Posix is still the only available driver so more adjustment may be needed, but this should represent the bulk of the changes.
2018-09-14 16:08:33 -04:00
Split cfgLoad() into multiple functions to make testing easier. Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
#include "common/harnessConfig.h"
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
#include "common/harnessFork.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
#include "common/harnessInfo.h"
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
#include "common/harnessPostgres.h"
Split cfgLoad() into multiple functions to make testing easier. Mainly this helps with unit tests that need to do log expect testing. Add harnessCfgLoad() test function, which allows a new config to be loaded for unit testing without resetting log functions, opening a log file, or taking locks.
2018-04-13 16:05:52 -04:00
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
Fix non-compliant formatting for function declarations. Per review by Cynthia Shang.
2018-01-31 18:22:25 -05:00
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// Create default storage object for testing
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
Storage *storageTest = storagePosixNewP(TEST_PATH_STR, .write = true);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Start a protocol server to test the protocol directly
Buffer *serverWrite = bufNew(8192);
Improve type safety of interfaces and drivers. The function pointer casting used when creating drivers made changing interfaces difficult and led to slightly divergent driver implementations. Unit testing caught production-level errors but there were a lot of small issues and the process was harder than it should have been. Use void pointers instead so that no casts are required. Introduce the THIS_VOID and THIS() macros to make dealing with void pointers a little safer. Since we don't want to expose void pointers in header files, driver functions have been removed from the headers and the various driver objects return their interface type. This cuts down on accessor methods and the vast majority of those functions were not being used. Move functions that are still required to .intern.h. Remove the special "C" crypto functions that were used in libc and instead use the standard interface.
2019-05-02 17:52:24 -04:00
IoWrite *serverWriteIo = ioBufferWriteNew(serverWrite);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteOpen(serverWriteIo);
ProtocolServer *server = protocolServerNew(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
STRDEF("test"), STRDEF("test"), ioBufferReadNew(bufNew(0)), serverWriteIo);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
bufUsedSet(serverWrite, 0);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// *****************************************************************************************************************************
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
if (testBegin("archivePushReadyList(), archivePushProcessList(), and archivePushDrop()"))
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
{
StringList *argList = strLstNew();
strLstAddZ(argList, "--stanza=db");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--pg1-path=" TEST_PATH "/db");
strLstAddZ(argList, "--spool-path=" TEST_PATH "/spool");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
strLstAddZ(argList, "--" CFGOPT_ARCHIVE_ASYNC);
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePathCreateP(storagePgWrite(), STRDEF("pg_wal/archive_status"));
storagePathCreateP(storageTest, STRDEF("spool/archive/db/out"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Create ok files to indicate WAL that has already been archived
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok")), NULL);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000003.ok")), NULL);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000004.ok")), NULL);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000005.error")), NULL);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000006.error")), NULL);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/global.error")), NULL);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Create ready files for wal that still needs to be archived
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/archive_status/000000010000000100000002.ready")), NULL);
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/archive_status/000000010000000100000003.ready")), NULL);
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/archive_status/000000010000000100000005.ready")), NULL);
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/archive_status/000000010000000100000006.ready")), NULL);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
archivePushProcessList(STRDEF(TEST_PATH "/db/pg_wal")),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000002\n000000010000000100000005\n000000010000000100000006\n", "ready list");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strLstSort(storageListP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000003.ok\n", "remaining status list");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Test drop
// -------------------------------------------------------------------------------------------------------------------------
StringList *argListDrop = strLstDup(argList);
strLstAdd(argListDrop, strNewFmt("--archive-push-queue-max=%zu", (size_t)1024 * 1024 * 1024));
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argListDrop);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Write the files that we claim are in pg_wal
Buffer *walBuffer = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}, walBuffer);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000002")), walBuffer);
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000003")), walBuffer);
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000005")), walBuffer);
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000006")), walBuffer);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Queue max is high enough that no WAL will be dropped
TEST_RESULT_BOOL(
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
archivePushDrop(STRDEF("pg_wal"), archivePushProcessList(STRDEF(TEST_PATH "/db/pg_wal"))), false,
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"wal is not dropped");
// Now set queue max low enough that WAL will be dropped
argListDrop = strLstDup(argList);
strLstAdd(argListDrop, strNewFmt("--archive-push-queue-max=%zu", (size_t)16 * 1024 * 1024 * 2));
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argListDrop);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
archivePushDrop(STRDEF("pg_wal"), archivePushProcessList(STRDEF(TEST_PATH "/db/pg_wal"))), true,
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"wal is dropped");
}
// *****************************************************************************************************************************
if (testBegin("archivePushCheck()"))
{
StringList *argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--pg1-path=" TEST_PATH "/pg");
strLstAddZ(argList, "--repo1-path=" TEST_PATH "/repo");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Check mismatched pg_control and archive.info
// -------------------------------------------------------------------------------------------------------------------------
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("pg/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_96, .systemId = 0xFACEFACEFACEFACE}));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Create incorrect archive info
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"));
TEST_ERROR(
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
archivePushCheck(true), RepoInvalidError,
"unable to find a valid repository:\n"
"repo1: [ArchiveMismatchError] PostgreSQL version 9.6, system-id 18072658121562454734 do not match repo1 stanza version"
" 9.4, system-id 5555555555555555555"
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"\nHINT: are you archiving to the correct stanza?");
// Fix the version
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.6\"}\n"));
TEST_ERROR(
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
archivePushCheck(true), RepoInvalidError,
"unable to find a valid repository:\n"
"repo1: [ArchiveMismatchError] PostgreSQL version 9.6, system-id 18072658121562454734 do not match repo1 stanza version"
" 9.6, system-id 5555555555555555555"
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"\nHINT: are you archiving to the correct stanza?");
// Fix archive info
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"9.6\"}\n"));
ArchivePushCheckResult result = {0};
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
TEST_ASSIGN(result, archivePushCheck(true), "get archive check result");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_UINT(result.pgVersion, PG_VERSION_96, "check pg version");
TEST_RESULT_UINT(result.pgSystemId, 0xFACEFACEFACEFACE, "check pg system id");
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
ArchivePushFileRepoData *repoData = lstGet(result.repoList, 0);
TEST_RESULT_UINT(repoData->repoIdx, 0, "check repo idx");
TEST_RESULT_STR_Z(repoData->archiveId, "9.6-1", "check archive id");
TEST_RESULT_UINT(repoData->cipherType, cipherTypeNone, "check cipher type");
TEST_RESULT_STR_Z(repoData->cipherPass, NULL, "check cipher pass (not set in this test)");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("mismatched repos when pg-path not present");
argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--repo2-path=" TEST_PATH "/repo2");
strLstAddZ(argList, "--repo4-path=" TEST_PATH "/repo4");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
harnessCfgLoad(cfgCmdArchivePush, argList);
// repo2 has correct info
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo2/archive/test/archive.info")),
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
harnessInfoChecksumZ(
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"9.6\"}\n"));
// repo4 has incorrect info
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo4/archive/test/archive.info")),
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
harnessInfoChecksumZ(
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"));
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
TEST_ASSIGN(result, archivePushCheck(false), "get archive check result");
TEST_RESULT_UINT(result.pgVersion, PG_VERSION_96, "check pg version");
TEST_RESULT_UINT(result.pgSystemId, 0xFACEFACEFACEFACE, "check pg system id");
TEST_RESULT_STRLST_Z(
result.errorList,
"repo4: [ArchiveMismatchError] repo2 stanza version 9.6, system-id 18072658121562454734 do not match repo4 stanza"
" version 9.4, system-id 5555555555555555555\n"
"HINT: are you archiving to the correct stanza?\n",
"check error list");
repoData = lstGet(result.repoList, 0);
TEST_RESULT_UINT(repoData->repoIdx, 0, "check repo idx");
TEST_RESULT_STR_Z(repoData->archiveId, "9.6-1", "check archive id");
TEST_RESULT_UINT(repoData->cipherType, cipherTypeNone, "check cipher type");
TEST_RESULT_STR_Z(repoData->cipherPass, NULL, "check cipher pass (not set in this test)");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("matched repos when pg-path not present");
// repo4 has correct info
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo4/archive/test/archive.info")),
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
"\n"
"[db:history]\n"
"1={\"db-id\":5555555555555555555,\"db-version\":\"9.4\"}\n"
"2={\"db-id\":18072658121562454734,\"db-version\":\"9.6\"}\n"));
TEST_ASSIGN(result, archivePushCheck(false), "get archive check result");
TEST_RESULT_UINT(result.pgVersion, PG_VERSION_96, "check pg version");
TEST_RESULT_UINT(result.pgSystemId, 0xFACEFACEFACEFACE, "check pg system id");
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
repoData = lstGet(result.repoList, 0);
TEST_RESULT_UINT(repoData->repoIdx, 0, "check repo idx");
TEST_RESULT_STR_Z(repoData->archiveId, "9.6-1", "check repo2 archive id");
TEST_RESULT_UINT(repoData->cipherType, cipherTypeNone, "check repo2 cipher pass");
TEST_RESULT_STR_Z(repoData->cipherPass, NULL, "check repo2 cipher pass (not set in this test)");
repoData = lstGet(result.repoList, 1);
TEST_RESULT_UINT(repoData->repoIdx, 1, "check repo idx");
TEST_RESULT_STR_Z(repoData->archiveId, "9.6-2", "check repo4 archive id");
TEST_RESULT_UINT(repoData->cipherType, cipherTypeNone, "check repo4 cipher type");
TEST_RESULT_STR_Z(repoData->cipherPass, NULL, "check repo4 cipher pass (not set in this test)");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
}
// *****************************************************************************************************************************
Improve protocol handlers. Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.
2021-03-16 13:09:34 -04:00
if (testBegin("Synchronous cmdArchivePush(), archivePushFile() and archivePushFileProtocol()"))
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
{
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_TITLE("command must be run on the pg host");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
StringList *argList = strLstNew();
Remove indexed option constants. These constants don't scale well as the index total is increased for an option. The core code rarely uses these options and they are easily replaced with cfgOptionName(). The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
hrnCfgArgRawZ(argList, cfgOptPgHost, "host");
hrnCfgArgRawZ(argList, cfgOptPgPath, "/pg");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
strLstAddZ(argList, "--" CFGOPT_STANZA "=test2");
Rename default command role to main. Main makes more sense because we refer to the main process in the code, not the default process. The word default is pretty overloaded anyway.
2021-05-20 14:39:47 -04:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleMain, argList);
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_ERROR(cmdArchivePush(), HostInvalidError, "archive-push command must be run on the PostgreSQL host");
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argList, "--stanza=test");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argList);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
TEST_ERROR(cmdArchivePush(), ParamRequiredError, "WAL segment to push required");
Fix archive-push/archive-get when PGDATA is symlinked. Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
// -------------------------------------------------------------------------------------------------------------------------
StringList *argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "pg_wal/000000010000000100000001");
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
TEST_ERROR(
cmdArchivePush(), OptionRequiredError,
"option 'pg1-path' must be specified when relative wal paths are used"
"\nHINT: is %f passed to archive-push instead of %p?"
"\nHINT: PostgreSQL may pass relative paths even with %p depending on the environment.");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Create pg_control and archive.info
// -------------------------------------------------------------------------------------------------------------------------
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--pg1-path=" TEST_PATH "/pg");
strLstAddZ(argList, "--repo1-path=" TEST_PATH "/repo");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Fix archive-push/archive-get when PGDATA is symlinked. Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
argListTemp = strLstDup(argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argListTemp, "pg_wal/000000010000000100000001");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("pg/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"11\"}\n"));
// Generate WAL with incorrect headers and try to push them
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Buffer *walBuffer1 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer1, bufSize(walBuffer1));
memset(bufPtr(walBuffer1), 0, bufSize(walBuffer1));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}, walBuffer1);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000001")), walBuffer1);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
THROW_ON_SYS_ERROR(chdir(strZ(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
Move async forking and more error handling to C. The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
TEST_ERROR(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
cmdArchivePush(), ArchiveMismatchError,
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
"WAL file '" TEST_PATH "/pg/pg_wal/000000010000000100000001' version 10, system-id 18072658121562454734 do not match"
Allow key replacements in TEST_ERROR*() macros. Allow standard replacements (e.g. {[path]}) in TEST_ERROR*() macros. Update command/archive-push unit test as an example of usage.
2021-03-23 18:20:26 -04:00
" stanza version 11, system-id 18072658121562454734");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
memset(bufPtr(walBuffer1), 0, bufSize(walBuffer1));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .systemId = 0xECAFECAFECAFECAF}, walBuffer1);
Add archive-header-check option. Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located. Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
const char *walBuffer1Sha1 = strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, walBuffer1)));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000001")), walBuffer1);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_ERROR(
cmdArchivePush(), ArchiveMismatchError,
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
"WAL file '" TEST_PATH "/pg/pg_wal/000000010000000100000001' version 11, system-id 17055110554209741999 do not match"
Allow key replacements in TEST_ERROR*() macros. Allow standard replacements (e.g. {[path]}) in TEST_ERROR*() macros. Update command/archive-push unit test as an example of usage.
2021-03-23 18:20:26 -04:00
" stanza version 11, system-id 18072658121562454734");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Add archive-header-check option. Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located. Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("push by ignoring the invalid header");
argListTemp = strLstDup(argList);
hrnCfgArgRawNegate(argListTemp, cfgOptArchiveHeaderCheck);
strLstAddZ(argListTemp, "pg_wal/000000010000000100000001");
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: pushed WAL file '000000010000000100000001' to the archive");
Add archive-header-check option. Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located. Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
TEST_RESULT_BOOL(
storageExistsP(
storageRepoIdx(0),
strNewFmt(STORAGE_REPO_ARCHIVE "/11-1/000000010000000100000001-%s.gz", walBuffer1Sha1)),
true, "check repo for WAL file");
TEST_STORAGE_REMOVE(
storageRepoIdxWrite(0), strZ(strNewFmt(STORAGE_REPO_ARCHIVE "/11-1/000000010000000100000001-%s.gz", walBuffer1Sha1)));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Generate valid WAL and push them
// -------------------------------------------------------------------------------------------------------------------------
Add archive-header-check option. Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located. Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "pg_wal/000000010000000100000001");
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
memset(bufPtr(walBuffer1), 0, bufSize(walBuffer1));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}, walBuffer1);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
// Check sha1 checksum against fixed values once to make sure they are not getting munged. After this we'll calculate them
// directly from the buffers to reduce the cost of maintaining checksums.
Add archive-header-check option. Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located. Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
walBuffer1Sha1 = TEST_64BIT() ?
Add support for testing on 64-bit big-endian architectures. In particular add support for s390x but we hope this will work for other 64-bit big-endian architectures. Run basic unit tests on Travis CI for 390x.
2020-07-20 09:59:16 -04:00
(TEST_BIG_ENDIAN() ? "1c5f963d720bb199d7935dbd315447ea2ec3feb2" : "aae7591a1dbc58f21d0d004886075094f622e6dd") :
"28a13fd8cf6fcd9f9a8108aed4c8bcc58040863a";
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000001")), walBuffer1);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: pushed WAL file '000000010000000100000001' to the archive");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storageExistsP(
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
storageTest, strNewFmt("repo/archive/test/11-1/0000000100000001/000000010000000100000001-%s.gz", walBuffer1Sha1)),
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
true, "check repo for WAL file");
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment again");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
"P00 WARN: WAL file '000000010000000100000001' already exists in the repo1 archive with the same checksum\n"
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P00 INFO: pushed WAL file '000000010000000100000001' to the archive");
// Now create a new WAL buffer with a different checksum to test checksum errors
Buffer *walBuffer2 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer2, bufSize(walBuffer2));
memset(bufPtr(walBuffer2), 0xFF, bufSize(walBuffer2));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}, walBuffer2);
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
const char *walBuffer2Sha1 = strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, walBuffer2)));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/000000010000000100000001")), walBuffer2);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
TEST_ERROR(
cmdArchivePush(), ArchiveDuplicateError,
"WAL file '000000010000000100000001' already exists in the repo1 archive with a different checksum");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Allow pg-path1 to be optional for synchronous archive-push. If the WAL path is absolute then pg1-path should be optional but in fact it was required to load pg_control. Skip the pg_control check when pg1-path is not specified. The check against the stanza version/system-id remains to protect the repo from corruption.
2020-05-01 10:30:35 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("WAL with absolute path and no pg1-path");
argListTemp = strLstNew();
strLstAddZ(argListTemp, "--" CFGOPT_STANZA "=test");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
hrnCfgArgRawZ(argListTemp, cfgOptRepoPath, TEST_PATH "/repo");
strLstAddZ(argListTemp, TEST_PATH "/pg/pg_wal/000000010000000100000002");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
TEST_RESULT_VOID(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storageTest, STRDEF("pg/pg_wal/000000010000000100000002")), walBuffer2), "write WAL");
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
// Create tmp file to make it look like a prior push failed partway through to ensure that retries work
TEST_RESULT_VOID(
storagePutP(
storageNewWriteP(
storageTest,
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
strNewFmt("repo/archive/test/11-1/0000000100000001/000000010000000100000002-%s.gz.pgbackrest.tmp",
walBuffer2Sha1)),
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
BUFSTRDEF("PARTIAL")),
"write WAL tmp file");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
TEST_RESULT_BOOL(
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
storageExistsP(
storageTest, strNewFmt("repo/archive/test/11-1/0000000100000001/000000010000000100000002-%s.gz", walBuffer2Sha1)),
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
true, "check repo for WAL file");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storageExistsP(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
storageTest,
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
strNewFmt("repo/archive/test/11-1/0000000100000001/000000010000000100000002-%s.gz.pgbackrest.tmp", walBuffer2Sha1)),
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
false, "check WAL tmp file is gone");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Push a history file
// -------------------------------------------------------------------------------------------------------------------------
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "pg_wal/00000001.history");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/00000001.history")), BUFSTRDEF("FAKEHISTORY"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push a history file");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: pushed WAL file '00000001.history' to the archive");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageExistsP(storageTest, STRDEF("repo/archive/test/11-1/00000001.history")), true, "check repo for history file");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Check drop functionality
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/archive_status/000000010000000100000001.ready")), NULL);
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_wal/archive_status/000000010000000100000002.ready")), NULL);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=16m");
strLstAddZ(argListTemp, "pg_wal/000000010000000100000002");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "drop WAL file");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 WARN: dropped WAL file '000000010000000100000002' because archive queue exceeded 16MB");
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=1GB");
strLstAddZ(argListTemp, "pg_wal/000000010000000100000002");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push WAL file again");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
"P00 WARN: WAL file '000000010000000100000002' already exists in the repo1 archive with the same checksum\n"
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
// Check protocol function directly
// -------------------------------------------------------------------------------------------------------------------------
VariantList *paramList = varLstNew();
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
varLstAdd(paramList, varNewStrZ(TEST_PATH "/pg/pg_wal/000000010000000100000002"));
Add archive-header-check option. Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located. Therefore, disabling this check is fairly safe but should only be done when required, e.g. if the WAL is encrypted.
2021-03-25 15:33:50 -04:00
varLstAdd(paramList, varNewBool(true));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
varLstAdd(paramList, varNewUInt64(PG_VERSION_11));
varLstAdd(paramList, varNewUInt64(0xFACEFACEFACEFACE));
varLstAdd(paramList, varNewStrZ("000000010000000100000002"));
varLstAdd(paramList, varNewBool(false));
varLstAdd(paramList, varNewInt(6));
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
varLstAdd(paramList, varNewVarLst(varLstNewStrLst(strLstNew())));
varLstAdd(paramList, varNewUInt(1));
varLstAdd(paramList, varNewUInt(0));
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
varLstAdd(paramList, varNewStrZ("11-1"));
varLstAdd(paramList, varNewUInt64(cipherTypeNone));
varLstAdd(paramList, NULL);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Improve protocol handlers. Make protocol handlers have one function per command. This allows the logic of finding the handler to be in ProtocolServer, isolates each command to a function, and removes the need to test the "not found" condition for each handler.
2021-03-16 13:09:34 -04:00
TEST_RESULT_VOID(archivePushFileProtocol(paramList, server), "protocol archive put");
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
strNewBuf(serverWrite),
Refactor archive-push command warnings to work like archive-get. Warnings are logged individually in the async log rather than all together.
2021-02-26 15:58:11 -05:00
"{\"out\":[[\"WAL file '000000010000000100000002' already exists in the repo1 archive with the same checksum"
"\\nHINT: this is valid in some recovery scenarios but may also indicate a problem.\"]]}\n",
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"check result");
bufUsedSet(serverWrite, 0);
// -------------------------------------------------------------------------------------------------------------------------
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
TEST_TITLE("multiple repos, one encrypted");
// Remove old repo
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePathRemoveP(storageTest, STRDEF("repo"), .errorOnMissing = true, .recurse = true);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// repo2 is encrypted
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
StorageWrite *infoWrite = storageNewWriteP(storageTest, STRDEF("repo2/archive/test/archive.info"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Allow protocol compression when read/writing remote files. If the file is compressible (i.e. not encrypted or already compressed) it can be marked as such in storageNewRead()/storageNewWrite(). If the file is being read from/written to a remote it will be compressed in transit using gzip. Simplify filter group handling by having the IoRead/IoWrite objects create the filter group automatically. This removes the need for a lot of NULL checking and has a negligible effect on performance since a filter group needs to be created eventually unless the source file is missing. Allow filters to be created using a VariantList so filter parameters can be passed to the remote.
2019-06-24 10:20:47 -04:00
ioFilterGroupAdd(
ioWriteFilterGroup(storageWriteIo(infoWrite)), cipherBlockNew(cipherModeEncrypt, cipherTypeAes256Cbc,
BUFSTRDEF("badpassphrase"), NULL));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
infoWrite,
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[cipher]\n"
"cipher-pass=\"badsubpassphrase\"\n"
"\n"
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"11\"}"));
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// repo3 is not encrypted
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo3/archive/test/archive.info")),
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
harnessInfoChecksumZ(
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"11\"}"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Push encrypted WAL segment
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
argListTemp = strLstNew();
hrnCfgArgRawZ(argListTemp, cfgOptStanza, "test");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
hrnCfgArgKeyRawZ(argListTemp, cfgOptPgPath, 1, TEST_PATH "/pg");
hrnCfgArgKeyRawZ(argListTemp, cfgOptRepoPath, 2, TEST_PATH "/repo2");
Update CipherType/CipherMode to StringId. As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
hrnCfgArgKeyRawStrId(argListTemp, cfgOptRepoCipherType, 2, cipherTypeAes256Cbc);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgEnvKeyRawZ(cfgOptRepoCipherPass, 2, "badpassphrase");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
hrnCfgArgKeyRawZ(argListTemp, cfgOptRepoPath, 3, TEST_PATH "/repo3");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgArgRawNegate(argListTemp, cfgOptCompress);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argListTemp, "pg_wal/000000010000000100000002");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgEnvKeyRemoveRaw(cfgOptRepoCipherPass, 2);
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storageExistsP(
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
storageTest, strNewFmt("repo2/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)),
true, "check repo2 for WAL file");
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo3/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)),
true, "check repo3 for WAL file");
// -------------------------------------------------------------------------------------------------------------------------
Add write fault-tolerance to archive-push command. The archive-push command will continue to push even after it gets a write error on one or more repos. The idea is to archive to as many repos as possible even we still need to throw an error to PostgreSQL to prevent it from removing the WAL file.
2021-02-26 16:52:59 -05:00
TEST_TITLE("write error on one repo but other repo succeeds");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
storageRemoveP(
storageTest, strNewFmt("repo2/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1),
.errorOnMissing = true);
Add write fault-tolerance to archive-push command. The archive-push command will continue to push even after it gets a write error on one or more repos. The idea is to archive to as many repos as possible even we still need to throw an error to PostgreSQL to prevent it from removing the WAL file.
2021-02-26 16:52:59 -05:00
storageRemoveP(
storageTest, strNewFmt("repo3/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1),
.errorOnMissing = true);
HRN_STORAGE_MODE(storageTest, "repo2/archive/test/11-1/0000000100000001", .mode = 0500);
TEST_ERROR(
cmdArchivePush(), CommandError,
strZ(
strNewFmt(
"archive-push command encountered error(s):\n"
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
"repo2: [FileOpenError] unable to open file '" TEST_PATH "/repo2/archive/test/11-1/0000000100000001"
Add write fault-tolerance to archive-push command. The archive-push command will continue to push even after it gets a write error on one or more repos. The idea is to archive to as many repos as possible even we still need to throw an error to PostgreSQL to prevent it from removing the WAL file.
2021-02-26 16:52:59 -05:00
"/000000010000000100000002-%s' for write: [13] Permission denied", walBuffer2Sha1)));
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo2/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)),
false, "check repo2 for no WAL file");
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo3/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)),
true, "check repo3 for WAL file");
HRN_STORAGE_MODE(storageTest, "repo2/archive/test/11-1/0000000100000001");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("push WAL to one repo");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
"P00 WARN: WAL file '000000010000000100000002' already exists in the repo3 archive with the same checksum\n"
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo2/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)),
true, "check repo2 for WAL file");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("WAL already exists in both repos");
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
"P00 WARN: WAL file '000000010000000100000002' already exists in the repo2 archive with the same checksum\n"
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
Refactor archive-push command warnings to work like archive-get. Warnings are logged individually in the async log rather than all together.
2021-02-26 15:58:11 -05:00
"P00 WARN: WAL file '000000010000000100000002' already exists in the repo3 archive with the same checksum\n"
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P00 INFO: pushed WAL file '000000010000000100000002' to the archive");
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("push succeeds on one repo when other repo fails to load archive.info");
TEST_STORAGE_REMOVE(
storageTest, strZ(strNewFmt("repo2/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)));
TEST_STORAGE_REMOVE(
storageTest, strZ(strNewFmt("repo3/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)));
HRN_STORAGE_MODE(storageTest, "repo2", .mode = 0200);
TEST_ERROR(
cmdArchivePush(), CommandError,
"archive-push command encountered error(s):\n"
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
"repo2: [FileOpenError] unable to load info file '" TEST_PATH "/repo2/archive/test/archive.info' or"
" '" TEST_PATH "/repo2/archive/test/archive.info.copy':\n"
"FileOpenError: unable to open file '" TEST_PATH "/repo2/archive/test/archive.info' for read: [13] Permission denied\n"
"FileOpenError: unable to open file '" TEST_PATH "/repo2/archive/test/archive.info.copy' for read:"
" [13] Permission denied\n"
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
"HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
"HINT: is archive_command configured correctly in postgresql.conf?\n"
"HINT: has a stanza-create been performed?\n"
"HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.");
// Make sure WAL got pushed to repo3
TEST_STORAGE_REMOVE(
storageTest, strZ(strNewFmt("repo3/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)));
HRN_STORAGE_MODE(storageTest, "repo2");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("push succeeds on one repo when other repo fails to read path");
HRN_STORAGE_MODE(storageTest, "repo2/archive/test/11-1", .mode = 0200);
TEST_ERROR(
cmdArchivePush(), CommandError,
"archive-push command encountered error(s):\n"
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
"repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test/11-1/0000000100000001':"
" [13] Permission denied");
Improve archive-push command fault tolerance. 3b8f0ef missed some cases that could cause archive-push to fail: * Checking archive info. * Checking to see if a WAL segment already exists. These cases are now handled so archive-push can succeed on any valid repos.
2021-03-25 12:54:49 -04:00
// Make sure WAL got pushed to repo3
TEST_STORAGE_REMOVE(
storageTest, strZ(strNewFmt("repo3/archive/test/11-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)));
HRN_STORAGE_MODE(storageTest, "repo2/archive/test/11-1");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
}
// *****************************************************************************************************************************
if (testBegin("Asynchronous cmdArchivePush() and cmdArchivePushAsync()"))
{
harnessLogLevelSet(logLevelDetail);
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_TITLE("command must be run on the pg host");
StringList *argList = strLstNew();
Remove indexed option constants. These constants don't scale well as the index total is increased for an option. The core code rarely uses these options and they are easily replaced with cfgOptionName(). The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
hrnCfgArgRawZ(argList, cfgOptPgHost, "host");
hrnCfgArgRawZ(argList, cfgOptPgPath, "/pg");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
strLstAddZ(argList, "--" CFGOPT_SPOOL_PATH "=/spool");
strLstAddZ(argList, "--" CFGOPT_STANZA "=test2");
strLstAddZ(argList, "--" CFGOPT_ARCHIVE_ASYNC);
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argList);
TEST_ERROR(cmdArchivePush(), HostInvalidError, "archive-push command must be run on the PostgreSQL host");
Allow pg-path1 to be optional for synchronous archive-push. If the WAL path is absolute then pg1-path should be optional but in fact it was required to load pg_control. Skip the pg_control check when pg1-path is not specified. The check against the stanza version/system-id remains to protect the repo from corruption.
2020-05-01 10:30:35 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("pg1-path must be set when async");
argList = strLstNew();
strLstAddZ(argList, "--" CFGOPT_SPOOL_PATH "=/spool");
strLstAddZ(argList, "--" CFGOPT_STANZA "=test2");
strLstAddZ(argList, "--" CFGOPT_ARCHIVE_ASYNC);
strLstAddZ(argList, "/000000010000000100000001");
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argList);
TEST_ERROR(cmdArchivePush(), OptionRequiredError, "'archive-push' command in async mode requires option 'pg1-path'");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Call with a bogus exe name so the async process will error out and we can make sure timeouts work
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
argList = strLstNew();
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argList, "pgbackrest-bogus");
strLstAddZ(argList, "--stanza=test");
strLstAddZ(argList, "--archive-async");
strLstAddZ(argList, "--archive-timeout=1");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--lock-path=" TEST_PATH "/lock");
strLstAddZ(argList, "--spool-path=" TEST_PATH " /spool");
strLstAddZ(argList, "--pg1-path=" TEST_PATH "/pg");
strLstAddZ(argList, "--repo1-path=" TEST_PATH "/repo");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argList, "archive-push");
strLstAddZ(argList, "pg_wal/bogus");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoadRaw(strLstSize(argList), strLstPtr(argList));
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
Fix archive-push/archive-get when PGDATA is symlinked. Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
storagePathCreateP(storageTest, cfgOptionStr(cfgOptPgPath));
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
THROW_ON_SYS_ERROR(chdir(strZ(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
Full branch coverage for command/help/help, common/error, common/ini, and common/log modules.
2018-05-05 09:38:09 -04:00
TEST_ERROR(
cmdArchivePush(), ArchiveTimeoutError,
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"unable to push WAL file 'bogus' to the archive asynchronously after 1 second(s)");
// Create pg_control and archive.info
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
strLstAddZ(argList, "--archive-async");
strLstAddZ(argList, "--no-compress");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--spool-path=" TEST_PATH "/spool");
strLstAddZ(argList, "--pg1-path=" TEST_PATH "/pg");
strLstAddZ(argList, "--repo1-path=" TEST_PATH "/repo");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argList, "--log-subprocess");
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("pg/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)),
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_94, .systemId = 0xAAAABBBBCCCCDDDD}));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo/archive/test/archive.info")),
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
harnessInfoChecksumZ(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":12297848147757817309,\"db-version\":\"9.4\"}\n"));
// Write out an error file that will be ignored on the first pass, then the async process will write a new one
// -------------------------------------------------------------------------------------------------------------------------
StringList *argListTemp = strLstDup(argList);
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argListTemp, TEST_PATH "/pg/pg_xlog/000000010000000100000001");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePathCreateP(storagePgWrite(), STRDEF("pg_xlog/archive_status"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.error")),
Add macros to create constant Buffer objects. These are more efficient than creating buffers in place when needed. After replacement discovered that bufNewStr() and BufNewZ() were not being used in the core code so removed them. This required using the macros in tests which is not the usual pattern.
2019-04-20 08:16:17 -04:00
BUFSTRDEF("25\nbogus error"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_ERROR(cmdArchivePush(), AssertError, "no WAL files to process");
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageRemoveP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/global.error"), .errorOnMissing = true);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Acquire a lock so the async process will not be able to run -- this will result in a timeout
// -------------------------------------------------------------------------------------------------------------------------
argListTemp = strLstDup(argList);
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argListTemp, TEST_PATH "/pg/pg_xlog/000000010000000100000001");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argListTemp, "--archive-timeout=1");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
THROW_ON_SYS_ERROR(chdir(strZ(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
Use getcwd() to construct path when WAL path is relative. Using pg1-path, as we were doing previously, could lead to WAL being copied to/from unexpected places. PostgreSQL sets the current working directory to PGDATA so we can use that to resolve relative paths.
2019-10-30 14:55:25 +01:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
HARNESS_FORK_BEGIN()
{
HARNESS_FORK_CHILD_BEGIN(0, true)
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
IoRead *read = ioFdReadNew(STRDEF("child read"), HARNESS_FORK_CHILD_READ(), 2000);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioReadOpen(read);
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
IoWrite *write = ioFdWriteNew(STRDEF("child write"), HARNESS_FORK_CHILD_WRITE(), 2000);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteOpen(write);
Allow multiple remote locks from the same main process. Improve locking on remote processes by introducing an exec-id that is unique to the main process and passed to all remote processes. This allows the remote processes to determine if a lock is held by a remote from the same main process. If so, the lock is allowed. The exec-id is also useful for associating remote logs with main logs for debugging purposes.
2020-11-23 12:41:54 -05:00
lockAcquire(
cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptStanza), STRDEF("555-fefefefe"), cfgLockType(), 30000, true);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Let the parent know the lock has been acquired and wait for the parent to allow lock release
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
ioWriteStrLine(write, strNew());
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteFlush(write);
ioReadLine(read);
lockRelease(true);
}
HARNESS_FORK_CHILD_END();
HARNESS_FORK_PARENT_BEGIN()
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
IoRead *read = ioFdReadNew(STRDEF("parent read"), HARNESS_FORK_PARENT_READ_PROCESS(0), 2000);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioReadOpen(read);
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
IoWrite *write = ioFdWriteNew(STRDEF("parent write"), HARNESS_FORK_PARENT_WRITE_PROCESS(0), 2000);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteOpen(write);
// Wait for the child to acquire the lock
ioReadLine(read);
TEST_ERROR(
cmdArchivePush(), ArchiveTimeoutError,
"unable to push WAL file '000000010000000100000001' to the archive asynchronously after 1 second(s)");
// Notify the child to release the lock
Add ioWriteStr() and ioWriteStrLine(). These function names make it clearer what is being written. The old ioWriteLine() has been repurposed to write buffers.
2019-04-22 18:46:29 -04:00
ioWriteLine(write, bufNew(0));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
ioWriteFlush(write);
}
HARNESS_FORK_PARENT_END();
}
HARNESS_FORK_END();
// Actually push a WAL file
// -------------------------------------------------------------------------------------------------------------------------
argListTemp = strLstDup(argList);
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argListTemp, TEST_PATH "/pg/pg_xlog/000000010000000100000001");
Allow most unit tests to run outside of a container. Three major changes were required to get this working: 1) Provide the path to pgbackrest in the build directory when running outside a container. Tests in a container will continue to install and run against /usr/bin/pgbackrest. 1) Set a per-test lock path so tests don't conflict on the default /tmp/pgbackrest path. Also set a per-test log-path while we are at it. 2) Use localhost instead of a custom host for TLS test connections. Tests in containers will continue to update /etc/hosts and use the custom host. Add infrastructure and update harnessCfgLoad*() to get the correct exe and paths loaded for testing. Since new tests are required to verify that running outside a container works, also rework the tests in Travis CI to provide coverage within a reasonable amount of time. Mainly, break up to doc tests by VM and run an abbreviated unit test suite on co6 and co7.
2019-10-08 12:06:30 -04:00
harnessCfgLoad(cfgCmdArchivePush, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_xlog/archive_status/000000010000000100000001.ready")), NULL);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Buffer *walBuffer1 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer1, bufSize(walBuffer1));
memset(bufPtr(walBuffer1), 0xFF, bufSize(walBuffer1));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_94, .systemId = 0xAAAABBBBCCCCDDDD}, walBuffer1);
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
const char *walBuffer1Sha1 = strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, walBuffer1)));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_xlog/000000010000000100000001")), walBuffer1);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePush(), "push the WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: pushed WAL file '000000010000000100000001' to the archive asynchronously");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storageExistsP(
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
storageTest, strNewFmt("repo/archive/test/9.4-1/0000000100000001/000000010000000100000001-%s", walBuffer1Sha1)),
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
true, "check repo for WAL file");
// Direct tests of the async function
// -------------------------------------------------------------------------------------------------------------------------
argList = strLstNew();
strLstAddZ(argList, "--stanza=test");
Remove deprecated option from command/archive-push unit test.
2020-12-10 17:58:59 -05:00
hrnCfgArgRawZ(argList, cfgOptCompressType, "none");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--spool-path=" TEST_PATH "/spool");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
strLstAddZ(argList, "--" CFGOPT_ARCHIVE_ASYNC);
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, "--pg1-path=" TEST_PATH "/pg");
strLstAddZ(argList, "--repo1-path=" TEST_PATH "/repo");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
strLstAddZ(argList, "--log-subprocess");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_ERROR(cmdArchivePushAsync(), ParamRequiredError, "WAL path to push required");
// Check that global.error is created
// -------------------------------------------------------------------------------------------------------------------------
// Remove data from prior tests
storagePathRemoveP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR, .recurse = true);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePathCreateP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePathRemoveP(storagePgWrite(), STRDEF("pg_xlog/archive_status"), .recurse = true);
storagePathCreateP(storagePgWrite(), STRDEF("pg_xlog/archive_status"));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_xlog");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argList);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_ERROR(cmdArchivePushAsync(), AssertError, "no WAL files to process");
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strNewBuf(storageGetP(storageNewReadP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/global.error")))),
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"25\nno WAL files to process", "check global.error");
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strLstSort(storageListP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"global.error\n", "check status files");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// -------------------------------------------------------------------------------------------------------------------------
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
TEST_TITLE("add repo, push already pushed WAL and new WAL");
// Add repo3
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 3, TEST_PATH "/repo3");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argList);
storagePutP(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageNewWriteP(storageTest, STRDEF("repo3/archive/test/archive.info")),
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
harnessInfoChecksumZ(
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":12297848147757817309,\"db-version\":\"9.4\"}\n"));
Log warnings in archive-push async log. These warnings were only being reported to PostgreSQL on the console. Now they are also recorded in the async log increasing the chance that they will be seen. This also improves coverage by requiring a warning during async processing to have a test case, which has been added.
2020-11-09 16:10:59 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Recreate ready file for WAL 1
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_xlog/archive_status/000000010000000100000001.ready")), NULL);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo/archive/test/9.4-1/0000000100000001/000000010000000100000001-%s", walBuffer1Sha1)),
true, "check repo1 for WAL 1 file");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Create a ready file for WAL 2 but don't create the segment yet -- this will test the file error
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_xlog/archive_status/000000010000000100000002.ready")), NULL);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segments");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
TEST_RESULT_LOG_FMT(
"P00 INFO: push 2 WAL file(s) to archive: 000000010000000100000001...000000010000000100000002\n"
"P01 WARN: WAL file '000000010000000100000001' already exists in the repo1 archive with the same checksum\n"
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P01 DETAIL: pushed WAL file '000000010000000100000001' to the archive\n"
"P01 WARN: could not push WAL file '000000010000000100000002' to the archive (will be retried): "
"[55] raised from local-1 protocol: " STORAGE_ERROR_READ_MISSING,
TEST_PATH "/pg/pg_xlog/000000010000000100000002");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storageExistsP(
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
storageTest, strNewFmt("repo/archive/test/9.4-1/0000000100000001/000000010000000100000001-%s", walBuffer1Sha1)),
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
true, "check repo1 for WAL 1 file");
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo/archive/test/9.4-1/0000000100000001/000000010000000100000001-%s", walBuffer1Sha1)),
true, "check repo3 for WAL 1 file");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strLstSort(storageListP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000001.ok\n000000010000000100000002.error\n", "check status files");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Log warnings in archive-push async log. These warnings were only being reported to PostgreSQL on the console. Now they are also recorded in the async log increasing the chance that they will be seen. This also improves coverage by requiring a warning during async processing to have a test case, which has been added.
2020-11-09 16:10:59 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("create and push previously missing WAL");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Create WAL 2 segment
Buffer *walBuffer2 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer2, bufSize(walBuffer2));
memset(bufPtr(walBuffer2), 0x0C, bufSize(walBuffer2));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_94, .systemId = 0xAAAABBBBCCCCDDDD}, walBuffer2);
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
const char *walBuffer2Sha1 = strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, walBuffer2)));
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_xlog/000000010000000100000002")), walBuffer2);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=1gb");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segments");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"P00 INFO: push 1 WAL file(s) to archive: 000000010000000100000002\n"
Add process id to C archive-get and archive-push logging. This was missed in the original migration. There was no functional issue, but logging the process ids is useful for debugging.
2019-04-09 11:08:27 -04:00
"P01 DETAIL: pushed WAL file '000000010000000100000002' to the archive");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_BOOL(
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storageExistsP(
Reduce reliance on static checksums in unit tests. Testing against static checksums is valuable but it can be become burdensome when supporting multiple architectures. Reduce the number of tests we are doing against static checksums when the architecture can cause the checksum to vary.
2020-07-20 09:47:43 -04:00
storageTest, strNewFmt("repo/archive/test/9.4-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)),
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
true, "check repo1 for WAL 2 file");
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo3/archive/test/9.4-1/0000000100000001/000000010000000100000002-%s", walBuffer2Sha1)),
true, "check repo3 for WAL 2 file");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strLstSort(storageListP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000001.ok\n000000010000000100000002.ok\n", "check status files");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("push wal 2 again to get warnings from both repos");
// Remove the OK file so the WAL gets pushed again
storageRemoveP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000002.ok"));
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segments");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
"P00 INFO: push 1 WAL file(s) to archive: 000000010000000100000002\n"
"P01 WARN: WAL file '000000010000000100000002' already exists in the repo1 archive with the same checksum\n"
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
Refactor archive-push command warnings to work like archive-get. Warnings are logged individually in the async log rather than all together.
2021-02-26 15:58:11 -05:00
"P01 WARN: WAL file '000000010000000100000002' already exists in the repo3 archive with the same checksum\n"
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
" HINT: this is valid in some recovery scenarios but may also indicate a problem.\n"
"P01 DETAIL: pushed WAL file '000000010000000100000002' to the archive");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("create and push WAL 3 to both repos");
// Create WAL 3 segment
Buffer *walBuffer3 = bufNew((size_t)16 * 1024 * 1024);
bufUsedSet(walBuffer3, bufSize(walBuffer3));
memset(bufPtr(walBuffer3), 0x44, bufSize(walBuffer3));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_94, .systemId = 0xAAAABBBBCCCCDDDD}, walBuffer3);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
const char *walBuffer3Sha1 = strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, walBuffer3)));
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_xlog/000000010000000100000003")), walBuffer3);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// Create ready file
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storagePutP(storageNewWriteP(storagePgWrite(), STRDEF("pg_xlog/archive_status/000000010000000100000003.ready")), NULL);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segment");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
"P00 INFO: push 1 WAL file(s) to archive: 000000010000000100000003\n"
"P01 DETAIL: pushed WAL file '000000010000000100000003' to the archive");
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo/archive/test/9.4-1/0000000100000001/000000010000000100000003-%s", walBuffer3Sha1)),
true, "check repo1 for WAL 3 file");
TEST_RESULT_BOOL(
storageExistsP(
storageTest, strNewFmt("repo3/archive/test/9.4-1/0000000100000001/000000010000000100000003-%s", walBuffer3Sha1)),
true, "check repo3 for WAL 3 file");
// Remove the ready file to prevent WAL 3 from being considered for the next test
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageRemoveP(storagePgWrite(), STRDEF("pg_xlog/archive_status/000000010000000100000003.ready"), .errorOnMissing = true);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
// Check that drop functionality works
// -------------------------------------------------------------------------------------------------------------------------
// Remove status files
storagePathRemoveP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR, .recurse = true);
Remove *MP() macros variants. Adding a dummy column which is always set by the P() macro allows a single macro to be used for parameters or no parameters without violating C's prohibition on the {} initializer. -Wmissing-field-initializers remains disabled because it still gives wildly different results between versions of gcc.
2019-11-17 15:10:40 -05:00
storagePathCreateP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
argListTemp = strLstDup(argList);
strLstAddZ(argListTemp, "--archive-push-queue-max=16m");
Fix options being ignored by asynchronous commands. The local, remote, archive-get-async, and archive-push-async commands were used to run functionality that was not directly available to the user. Unfortunately that meant they would not pick up options from the command that the user expected, e.g. backup, archive-get, etc. Remove the internal commands and add roles which allow pgBackRest to determine what functionality is required without implementing special commands. This way the options are loaded from the expected command section. Since remote is no longer a specific command with its own options, more manipulation is required when calling remote. This might be something we can improve in the config system but it may be worth leaving as is because it is a one-off, for now at least.
2020-01-15 12:24:58 -07:00
harnessCfgLoadRole(cfgCmdArchivePush, cfgCmdRoleAsync, argListTemp);
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
TEST_RESULT_VOID(cmdArchivePushAsync(), "push WAL segments");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"P00 INFO: push 2 WAL file(s) to archive: 000000010000000100000001...000000010000000100000002\n"
"P00 WARN: dropped WAL file '000000010000000100000001' because archive queue exceeded 16MB\n"
"P00 WARN: dropped WAL file '000000010000000100000002' because archive queue exceeded 16MB");
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
strNewBuf(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageGetP(storageNewReadP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000001.ok")))),
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"0\ndropped WAL file '000000010000000100000001' because archive queue exceeded 16MB", "check WAL 1 warning");
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
strNewBuf(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
storageGetP(storageNewReadP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/000000010000000100000002.ok")))),
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
"0\ndropped WAL file '000000010000000100000002' because archive queue exceeded 16MB", "check WAL 2 warning");
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strLstSort(storageListP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT)), sortOrderAsc),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000001.ok\n000000010000000100000002.ok\n", "check status files");
The archive-push command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-03-29 13:26:33 +00:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*(). When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed. Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
FUNCTION_HARNESS_RETURN_VOID();
The archive-push command is now partially coded in C. This allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process.
2018-01-17 15:52:00 -05:00
}
Reference in New Issue Copy Permalink