db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-11 00:50:20 +02:00
Code Issues Releases Activity
Files
e634fd85cedc1ae3aef7c4fa2556d7d73d05cb8a
pgbackrest/test/src/module/command/archiveGetTest.c

1190 lines
61 KiB
C
Raw Normal View History

Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Archive Get Command
***********************************************************************************************************************************/
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
#include "common/io/fdRead.h"
#include "common/io/fdWrite.h"
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Minor fixes for command/archive-get unit test. Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test. Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet(). Move some harness headers down to the correct section.
2021-05-24 12:40:27 -04:00
#include "common/harnessConfig.h"
#include "common/harnessFork.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
#include "common/harnessInfo.h"
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
#include "common/harnessPostgres.h"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
#include "common/harnessProtocol.h"
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
#include "common/harnessStorage.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
Add missing static keywords in test modules.
2021-10-18 12:22:48 -04:00
static void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// *****************************************************************************************************************************
if (testBegin("queueNeed()"))
{
StringList *argList = strLstNew();
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
Remove indexed option constants. These constants don't scale well as the index total is increased for an option. The core code rarely uses these options and they are easily replaced with cfgOptionName(). The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, "/unused");
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
size_t queueSize = 16 * 1024 * 1024;
size_t walSegmentSize = 16 * 1024 * 1024;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("path missing");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
TEST_ERROR(
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
queueNeed(STRDEF("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_95),
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
PathMissingError, "unable to list file info for missing path '" TEST_PATH "/spool/archive/test1/in'");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("queue size too small");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
queueNeed(STRDEF("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_95),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000001\n000000010000000100000002\n", "queue size smaller than min");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("queue empty");
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
queueSize = (16 * 1024 * 1024) * 3;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
queueNeed(STRDEF("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_95),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000001\n000000010000000100000002\n000000010000000100000003\n", "empty queue");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
TEST_TITLE("ok/junk status files");
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Buffer *walSegmentBuffer = bufNew(walSegmentSize);
memset(bufPtr(walSegmentBuffer), 0, walSegmentSize);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
walSegmentSize = 1024 * 1024;
queueSize = walSegmentSize * 5;
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/junk", "JUNK");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// Bad OK file with wrong length (just to make sure this does not cause strSubN() issues)
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/AAA.ok", "0\nWARNING");
// OK file with warnings somehow left over from a prior run
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFD.ok", "0\nWARNING");
// Valid queued WAL segments (one with an OK file containing warnings)
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFE", walSegmentBuffer);
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFF", walSegmentBuffer);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFF.ok", "0\nWARNING2");
// Empty OK file indicating a WAL segment not found at the end of the queue
HRN_STORAGE_PUT_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000B00000000.ok");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
queueNeed(STRDEF("000000010000000A00000FFD"), true, queueSize, walSegmentSize, PG_VERSION_11),
Remove support for PostgreSQL 9.3. Per our policy to support five EOL versions of PostgreSQL, 9.3 is no longer supported by pgBackRest. Remove all logic associated with 9.3 and update the tests.
2023-11-09 12:59:12 -03:00
"000000010000000B00000000\n000000010000000B00000001\n000000010000000B00000002\n", "queue has wal");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_LIST(
storageSpool(), STORAGE_SPOOL_ARCHIVE_IN,
"000000010000000A00000FFE\n000000010000000A00000FFF\n000000010000000A00000FFF.ok\n");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// *****************************************************************************************************************************
if (testBegin("cmdArchiveGetAsync()"))
{
harnessLogLevelSet(logLevelDetail);
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
// Install local command handler shim
static const ProtocolServerHandler testLocalHandlerList[] = {PROTOCOL_SERVER_HANDLER_ARCHIVE_GET_LIST};
Add LENGTH_OF() macro. Determining the length of arrays that could be calculated at compile time was a bit piecemeal, with special macros used sometimes and with the math done directly other times. This macro makes the task easier, uses less space, and automatically adjusts when the type changes.
2022-04-07 19:00:15 -04:00
hrnProtocolLocalShimInstall(testLocalHandlerList, LENGTH_OF(testLocalHandlerList));
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// Arguments that must be included
StringList *argBaseList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH "/repo");
hrnCfgArgRawZ(argBaseList, cfgOptSpoolPath, TEST_PATH "/spool");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawBool(argBaseList, cfgOptArchiveAsync, true);
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test2");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("command must be run on the pg host");
StringList *argList = strLstDup(argBaseList);
hrnCfgArgRawZ(argList, cfgOptPgHost, BOGUS_STR);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_ERROR(cmdArchiveGetAsync(), HostInvalidError, "archive-get command must be run on the PostgreSQL host");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/global.error",
"72\narchive-get command must be run on the PostgreSQL host", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Make archive-get locality error generate a global.error file. Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("error on no segments");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_ERROR(cmdArchiveGetAsync(), ParamInvalidError, "at least one wal segment is required");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/global.error", "96\nat least one wal segment is required",
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Make archive-get locality error generate a global.error file. Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
Prevent invalid recovery when backup_label removed. If backup_label is removed from a restored backup then PostgreSQL will instead use checkpoint information from pg_control to attempt (what is thinks is) crash recovery. This will nearly always result in a corrupt cluster because the checkpoint will not be from the beginning of the backup, and even if it is, the end point will not be specified, which could lead to recovery stopping too early. To prevent this, invalidate the checkpoint LSN in pg_control on restore. If backup_label is removed then recovery will still fail because PostgreSQL will not be able to find the invalid checkpoint. The LSN of the checkpoint is not logged but it will be visible in pg_controldata output as 0/DEAD. This value is invalid because PostgreSQL always skips the first WAL segment when initializing a cluster.
2024-03-10 17:08:42 +13:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("pg_control from backup is not valid without backup_label");
HRN_PG_CONTROL_PUT(storagePgWrite(), PG_VERSION_10, .checkpoint = PG_CONTROL_CHECKPOINT_INVALID);
strLstAddZ(argList, "000000010000000100000001");
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
TEST_ERROR(
cmdArchiveGetAsync(), FormatError,
"pg_control from backup is not valid without backup_label\n"
"HINT: was the backup_label file removed?");
TEST_RESULT_LOG(
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001");
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "global.error\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("no segments to find");
Prevent invalid recovery when backup_label removed. If backup_label is removed from a restored backup then PostgreSQL will instead use checkpoint information from pg_control to attempt (what is thinks is) crash recovery. This will nearly always result in a corrupt cluster because the checkpoint will not be from the beginning of the backup, and even if it is, the end point will not be specified, which could lead to recovery stopping too early. To prevent this, invalidate the checkpoint LSN in pg_control on restore. If backup_label is removed then recovery will still fail because PostgreSQL will not be able to find the invalid checkpoint. The LSN of the checkpoint is not logged but it will be visible in pg_controldata output as 0/DEAD. This value is invalid because PostgreSQL always skips the first WAL segment when initializing a cluster.
2024-03-10 17:08:42 +13:00
// Success not that backup_label exists
HRN_STORAGE_PUT_EMPTY(storagePgWrite(), PG_FILE_BACKUPLABEL);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
"P00 DETAIL: unable to find 000000010000000100000001 in the archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on path permission");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-1", .mode = 0400);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo/archive/test2/10-1"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/0000000100000001': [13] Permission denied\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 WARN: [RepoInvalidError] unable to find a valid repository");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.error",
"103\n"
"unable to find a valid repository\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo/archive/test2/10-1/0000000100000001':"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [13] Permission denied",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
HRN_STORAGE_MODE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-1");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on invalid compressed segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"P01 WARN: [FileReadError] raised from local-1 shim protocol: unable to get 000000010000000100000001:\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" repo1: 10-1/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.error",
"42\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"raised from local-1 shim protocol: unable to get 000000010000000100000001:\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"repo1: 10-1/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz [FormatError]"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" unexpected eof in compressed data",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_LIST(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000001.pgbackrest.tmp\n");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz",
.remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("single segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
// There should be a temp file left over. Make sure it still exists to test that temp files are removed on retry.
TEST_STORAGE_EXISTS(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.pgbackrest.tmp");
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 000000010000000100000001 in the repo1: 10-1 archive");
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("single segment with one invalid file");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n"
"2={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
"P01 WARN: repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 000000010000000100000001 in the repo1: 10-1 archive");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok",
"0\n"
"repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz [FormatError]"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" unexpected eof in compressed data",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("single segment with one invalid file");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n"
"2={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
"P01 WARN: repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 000000010000000100000001 in the repo1: 10-1 archive");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok",
"0\n"
"repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz [FormatError]"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" unexpected eof in compressed data",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz",
.remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_TITLE("multiple segments where some are missing or errored and mismatched repo");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgKeyRawZ(argBaseList, cfgOptRepoPath, 2, TEST_PATH "/repo2");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "0000000100000001000000FE");
strLstAddZ(argList, "0000000100000001000000FF");
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_INFO_PUT(
storageRepoIdxWrite(1), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_11_Z ",\"db-version\":\"11\"}\n");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/0000000100000001000000FE-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// Create segment duplicates
HRN_STORAGE_PUT_EMPTY(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_EMPTY(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo2: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" '" HRN_PG_SYSTEMID_10_Z "'\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 0000000100000001000000FE in the repo1: 10-1 archive\n"
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
"P00 DETAIL: unable to find 0000000100000001000000FF in the archive");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE.ok",
"0\n"
"repo2: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
" '" HRN_PG_SYSTEMID_10_Z "'",
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF.ok",
"0\n"
"repo2: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
" '" HRN_PG_SYSTEMID_10_Z "'",
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_TITLE("error on duplicates now that no segments are missing, repo with bad perms");
// Fix repo 2 archive info but break archive path
HRN_INFO_PUT(
storageRepoIdxWrite(1), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1", .mode = 0400);
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/0000000100000001000000FF-efefefefefefefefefefefefefefefefefefefef");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/0000000100000001': [13] Permission denied\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/0000000100000002': [13] Permission denied\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 0000000100000001000000FE in the repo1: 10-1 archive\n"
"P01 DETAIL: found 0000000100000001000000FF in the repo1: 10-1 archive\n"
"P00 WARN: [ArchiveDuplicateError] duplicates found for WAL segment 000000010000000200000000:\n"
Sort WAL segment names when reporting duplicates. Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output. Found on arm64.
2021-10-02 16:29:31 -04:00
" repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
", 10-1/0000000100000002/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n"
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
" HINT: are multiple primaries archiving to this stanza?");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE", .remove = true);
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE.ok",
"0\n"
"repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1/0000000100000001':"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [13] Permission denied",
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF", .remove = true);
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF.ok",
"0\n"
"repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1/0000000100000001':"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [13] Permission denied",
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.error",
"45\n"
"duplicates found for WAL segment 000000010000000200000000:\n"
Sort WAL segment names when reporting duplicates. Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output. Found on arm64.
2021-10-02 16:29:31 -04:00
"repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, 10-1/0000000100000002"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"HINT: are multiple primaries archiving to this stanza?\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1" \
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/0000000100000002': [13] Permission denied",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
HRN_STORAGE_MODE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on duplicates");
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000200000000\n"
"P00 WARN: [ArchiveDuplicateError] duplicates found for WAL segment 000000010000000200000000:\n"
Sort WAL segment names when reporting duplicates. Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output. Found on arm64.
2021-10-02 16:29:31 -04:00
" repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
", 10-1/0000000100000002/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" HINT: are multiple primaries archiving to this stanza?");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.error",
"45\n"
"duplicates found for WAL segment 000000010000000200000000:\n"
Sort WAL segment names when reporting duplicates. Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output. Found on arm64.
2021-10-02 16:29:31 -04:00
"repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa, 10-1/0000000100000002"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"HINT: are multiple primaries archiving to this stanza?",
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
.remove = true);
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("warn on invalid file");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgKeyRawZ(argBaseList, cfgOptRepoPath, 3, TEST_PATH "/repo3");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_INFO_PUT(
storageRepoIdxWrite(2), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"11\"}\n");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(2), "10-1", .mode = 0400);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" '" HRN_PG_SYSTEMID_10_Z "'\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P01 WARN: repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 000000010000000200000000 in the repo2: 10-1 archive");
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.ok",
"0\n"
"repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" '" HRN_PG_SYSTEMID_10_Z "'\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error with warnings");
Add jobRetry parameter to HRN_CFG_LOAD(). Allow the default of 0 to be overridden to test retry behavior for commands.
2021-10-01 17:15:36 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync, .jobRetry = 1);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" '" HRN_PG_SYSTEMID_10_Z "'\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"P01 WARN: [FileReadError] raised from local-1 shim protocol: unable to get 000000010000000200000000:\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
" repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
" repo2: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Improve retry error messages. Centralize the code to allow it to be used in more places and update the protocol/server module to use the new code. Since the time measurements make testing difficult, also add time and errorRetry harnesses to allow specific data to be used for testing. In the case of errorRetry, the production behavior is turned off by default during testing and only enabled for the errorRetry test module.
2023-04-06 11:03:46 +04:00
" [RETRY DETAIL OMITTED]");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.error",
"42\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"raised from local-1 shim protocol: unable to get 000000010000000200000000:\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"repo2: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Improve retry error messages. Centralize the code to allow it to be used in more places and update the protocol/server module to use the new code. Since the time measurements make testing difficult, also add time and errorRetry harnesses to allow specific data to be used for testing. In the case of errorRetry, the production behavior is turned off by default during testing and only enabled for the errorRetry test module.
2023-04-06 11:03:46 +04:00
"[RETRY DETAIL OMITTED]\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" '" HRN_PG_SYSTEMID_10_Z "'",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000200000000.pgbackrest.tmp\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("global error on invalid executable");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
// Uninstall local command handler shim
hrnProtocolLocalShimUninstall();
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argList, cfgOptRepoPath, TEST_PATH "/repo");
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
hrnCfgArgRawZ(argList, cfgOptStanza, "test2");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "0000000100000001000000FE");
strLstAddZ(argList, "0000000100000001000000FF");
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync, .exeBogus = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_ERROR(
cmdArchiveGetAsync(), ExecuteError,
"local-1 process terminated unexpectedly [102]: unable to execute 'pgbackrest-bogus': [2] No such file or directory");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strNewBuf(storageGetP(storageNewReadP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_IN "/global.error")))),
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"102\nlocal-1 process terminated unexpectedly [102]: unable to execute 'pgbackrest-bogus':"
" [2] No such file or directory",
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
"check global error");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "global.error\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
}
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// *****************************************************************************************************************************
if (testBegin("cmdArchiveGet()"))
{
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
harnessLogLevelSet(logLevelDetail);
StringList *argBaseList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH "/repo");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test1");
hrnCfgArgRawZ(argBaseList, cfgOptArchiveTimeout, "1");
// -------------------------------------------------------------------------------------------------------------------------
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_TITLE("command must be run on the pg host");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
StringList *argList = strLstDup(argBaseList);
hrnCfgArgRawZ(argList, cfgOptPgHost, BOGUS_STR);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_ERROR(cmdArchiveGet(), HostInvalidError, "archive-get command must be run on the PostgreSQL host");
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("segment parameter not specified");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "WAL segment to get required");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("path parameter not specified");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000100000001");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "path to copy WAL segment required");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("no valid repo");
Add HRN_PG_CONTROL_PUT() and HRN_PG_CONTROL_TIME(). These macros simplify management of pg_control test files. Centralize time updates for pg_control in the command/backup module. This caused some time updates in the logs. Finally, move the postgres module after the storage module so it can use storage macros.
2021-11-30 13:23:11 -05:00
HRN_PG_CONTROL_PUT(storagePgWrite(), PG_VERSION_10);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Replace TEST_ERROR_FMT() with TEST_ERROR() where possible. Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.
2021-05-22 11:59:43 -04:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"P00 WARN: repo1: [FileMissingError] unable to load info file '" TEST_PATH "/repo/archive/test1/archive.info' or"
" '" TEST_PATH "/repo/archive/test1/archive.info.copy':\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info' for read\n"
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info.copy' for"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" read\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
" HINT: is archive_command configured correctly in postgresql.conf?\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" scheme.");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("no valid repo - async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
strLstAddZ(argList, "00000001.history");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYHISTORY");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Replace TEST_ERROR_FMT() with TEST_ERROR() where possible. Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.
2021-05-22 11:59:43 -04:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"P00 WARN: repo1: [FileMissingError] unable to load info file '" TEST_PATH "/repo/archive/test1/archive.info' or"
" '" TEST_PATH "/repo/archive/test1/archive.info.copy':\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info' for read\n"
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info.copy' for"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" read\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
" HINT: is archive_command configured correctly in postgresql.conf?\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" scheme.");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("WAL not found - timeout");
// Make sure the process times out when there is nothing to get
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
strLstAddZ(argList, "000000010000000100000001");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, "pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
THROW_ON_SYS_ERROR(chdir(strZ(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
Fix archive-push/archive-get when PGDATA is symlinked. Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
TEST_ERROR(
cmdArchiveGet(), ArchiveTimeoutError,
Add hint to check the log on archive-get/archive-push async error. If this error is thrown rather than a specific error returned from the async process, it means the async process is unable to write the status files for some reason and the only way to get the error is out of the async log. This hint includes the exact async log path and name to make finding errors easier.
2022-05-02 08:49:13 -04:00
"unable to get WAL file '000000010000000100000001' from the archive asynchronously after 1 second(s)\n"
"HINT: check '" HRN_PATH "/test1-archive-get-async.log' for errors.");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("check for missing WAL");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
TEST_ERROR(
cmdArchiveGet(), ArchiveTimeoutError,
Add hint to check the log on archive-get/archive-push async error. If this error is thrown rather than a specific error returned from the async process, it means the async process is unable to write the status files for some reason and the only way to get the error is out of the async log. This hint includes the exact async log path and name to make finding errors easier.
2022-05-02 08:49:13 -04:00
"unable to get WAL file '000000010000000100000001' from the archive asynchronously after 1 second(s)\n"
"HINT: check '" HRN_PATH "/test1-archive-get-async.log' for errors.");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
storageExistsP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok")), false,
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
"check OK file was removed");
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("write WAL segment for success");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storagePgWrite(), "pg_wal");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", "SHOULD-BE-A-REAL-WAL-FILE");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 000000010000000100000001 in the archive asynchronously");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("write WAL segments for success - queue full");
hrnCfgArgRawZ(argList, cfgOptArchiveGetQueueMax, "48");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
// Write more WAL segments (in this case queue should be full)
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", "SHOULD-BE-A-REAL-WAL-FILE");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok", "0\nwarning about x");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000002", "SHOULD-BE-A-REAL-WAL-FILE");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
"P00 WARN: warning about x\n"
"P00 INFO: found 000000010000000100000001 in the archive asynchronously");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Minor fixes for command/archive-get unit test. Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test. Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet(). Move some harness headers down to the correct section.
2021-05-24 12:40:27 -04:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000002\n", .remove = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("unable to get lock");
// Make sure the process times out when it can't get a lock
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_BEGIN()
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
{
Improve HRN_FORK*() macros. Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults. Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
2021-07-13 14:22:53 -04:00
HRN_FORK_CHILD_BEGIN()
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
{
Add lock module initialization. Each call to lockAcquireP() passed enough information to initialize the lock system. This was somewhat inefficient and as locks become more complicated it will lead to more code duplication. Since a process can only take one type of lock it makes sense to do most of the initialization up front. Also reduce the log level of lockRelease() since it is only called at exit and the lock will be released in any case.
2023-03-25 14:07:31 +07:00
lockInit(cfgOptionStr(cfgOptLockPath), STRDEF("999-dededede"), cfgOptionStr(cfgOptStanza), cfgLockType());
TEST_RESULT_VOID(lockAcquireP(.timeout = 30000, .returnOnNoLock = true), "acquire lock");
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros. This removes a lot of boiler plate where every instance needs to create these interfaces. Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes. In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
// Notify parent that lock has been acquired
HRN_FORK_CHILD_NOTIFY_PUT();
// Wait for parent to allow release lock
HRN_FORK_CHILD_NOTIFY_GET();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
lockRelease(true);
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_CHILD_END();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_PARENT_BEGIN()
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
{
Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros. This removes a lot of boiler plate where every instance needs to create these interfaces. Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes. In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
// Wait for child to acquire lock
HRN_FORK_PARENT_NOTIFY_GET(0);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
TEST_ERROR(
cmdArchiveGet(), ArchiveTimeoutError,
Add hint to check the log on archive-get/archive-push async error. If this error is thrown rather than a specific error returned from the async process, it means the async process is unable to write the status files for some reason and the only way to get the error is out of the async log. This hint includes the exact async log path and name to make finding errors easier.
2022-05-02 08:49:13 -04:00
"unable to get WAL file '000000010000000100000001' from the archive asynchronously after 1 second(s)\n"
"HINT: check '" HRN_PATH "/test1-archive-get-async.log' for errors.");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros. This removes a lot of boiler plate where every instance needs to create these interfaces. Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes. In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
// Notify child to release lock
HRN_FORK_PARENT_NOTIFY_PUT(0);
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_PARENT_END();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_END();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("too many parameters specified");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, BOGUS_STR);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
TEST_ERROR(cmdArchiveGet(), ParamInvalidError, "extra parameters found");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_TITLE("pg version does not match archive.info");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add HRN_PG_CONTROL_PUT() and HRN_PG_CONTROL_TIME(). These macros simplify management of pg_control test files. Centralize time updates for pg_control in the command/backup module. This caused some time updates in the logs. Finally, move the postgres module after the storage module so it can use storage macros.
2021-11-30 13:23:11 -05:00
HRN_PG_CONTROL_PUT(storagePgWrite(), PG_VERSION_11);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
argBaseList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH "/repo");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test1");
argList = strLstDup(argBaseList);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
strLstAddZ(argList, "01ABCDEF01ABCDEF01ABCDEF");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 WARN: repo1: [ArchiveMismatchError] unable to retrieve the archive id for database version '11' and system-id"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" '" HRN_PG_SYSTEMID_11_Z "'");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Fix incorrect test title.
2021-04-21 18:27:18 -04:00
TEST_TITLE("pg system id does not match archive.info");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Add HRN_PG_CONTROL_PUT() and HRN_PG_CONTROL_TIME(). These macros simplify management of pg_control test files. Centralize time updates for pg_control in the command/backup module. This caused some time updates in the logs. Finally, move the postgres module after the storage module so it can use storage macros.
2021-11-30 13:23:11 -05:00
HRN_PG_CONTROL_PUT(storagePgWrite(), PG_VERSION_10, .systemId = 1);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 WARN: repo1: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" '" HRN_PG_SYSTEMID_10_1_Z "'");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("file is missing");
Add HRN_PG_CONTROL_PUT() and HRN_PG_CONTROL_TIME(). These macros simplify management of pg_control test files. Centralize time updates for pg_control in the command/backup module. This caused some time updates in the logs. Finally, move the postgres module after the storage module so it can use storage macros.
2021-11-30 13:23:11 -05:00
HRN_PG_CONTROL_PUT(storagePgWrite(), PG_VERSION_10);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: unable to find 01ABCDEF01ABCDEF01ABCDEF in the archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST_EMPTY(storagePg(), "pg_wal");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get WAL segment");
Buffer *buffer = bufNew(16 * 1024 * 1024);
memset(bufPtr(buffer), 0, bufSize(buffer));
bufUsedSet(buffer, bufSize(buffer));
HRN_STORAGE_PUT(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
buffer);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1: 10-1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add pg-version-force option for fork integration. Forks may update pg_control version or WAL magic without affecting the structures that pgBackRest depends on. This option forces pgBackRest to treat a cluster as the specified version when it cannot be automatically identified.
2023-03-09 02:23:15 +01:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get WAL segment with a modified control/catalog version");
// Modify control/catalog version and use the --pg-version option
Rename HRN_PG_CONTROL_OVERRIDE_PUT(). Rename to HRN_PG_CONTROL_OVERRIDE_VERSION_PUT() since other types of overrides are possible (and coming).
2023-07-26 10:45:12 +02:00
HRN_PG_CONTROL_OVERRIDE_VERSION_PUT(storagePgWrite(), PG_VERSION_10, 1501, .catalogVersion = 202211111);
Add pg-version-force option for fork integration. Forks may update pg_control version or WAL magic without affecting the structures that pgBackRest depends on. This option forces pgBackRest to treat a cluster as the specified version when it cannot be automatically identified.
2023-03-09 02:23:15 +01:00
TEST_ERROR(
cmdArchiveGet(), VersionNotSupportedError,
"unexpected control version = 1501 and catalog version = 202211111\n"
"HINT: is this version of PostgreSQL supported?");
StringList *argListTemp = strLstDup(argList);
hrnCfgArgRawZ(argListTemp, cfgOptPgVersionForce, "10");
HRN_CFG_LOAD(cfgCmdArchivePush, argListTemp);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1: 10-1 archive");
// Reset control file and command options
HRN_PG_CONTROL_PUT(storagePgWrite(), PG_VERSION_10);
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
// Clean-up pg_wal directory
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_RESULT_UINT(storageInfoP(storagePg(), STRDEF("pg_wal/RECOVERYXLOG")).size, 16 * 1024 * 1024, "check size");
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add duplicate WAL test to synchronous archive-get.
2021-01-13 13:01:40 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on duplicate WAL segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb");
TEST_ERROR(
cmdArchiveGet(), ArchiveDuplicateError,
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"duplicates found for WAL segment 01ABCDEF01ABCDEF01ABCDEF:\n"
Sort WAL segment names when reporting duplicates. Make the output consistent even when files are listed in a different order. This is purely for testing purposes, but there is no harm in consistent output. Found on arm64.
2021-10-02 16:29:31 -04:00
"repo1: 10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
", 10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n"
Add duplicate WAL test to synchronous archive-get.
2021-01-13 13:01:40 -05:00
"HINT: are multiple primaries archiving to this stanza?");
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePg(), "pg_wal", NULL);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
.remove = true);
Add duplicate WAL test to synchronous archive-get.
2021-01-13 13:01:40 -05:00
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get from prior db-id");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n"
"2={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}\n"
"3={\"db-id\":" HRN_PG_SYSTEMID_11_Z ",\"db-version\":\"11\"}\n"
"4={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1: 10-1 archive");
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get from current db-id");
HRN_STORAGE_PUT_EMPTY(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-4/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1: 10-4 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-4/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get partial");
buffer = bufNew(16 * 1024 * 1024);
memset(bufPtr(buffer), 0xFF, bufSize(buffer));
bufUsedSet(buffer, bufSize(buffer));
HRN_STORAGE_PUT(
storageRepoWrite(),
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
STORAGE_REPO_ARCHIVE "/10-4/000000010000000100000001.partial-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
buffer);
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000100000001.partial");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 000000010000000100000001.partial in the repo1: 10-4 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(),
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
STORAGE_REPO_ARCHIVE "/10-4/000000010000000100000001.partial-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", .remove = true);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get missing history");
argList = strLstDup(argBaseList);
strLstAddZ(argList, "00000001.history");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYHISTORY");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: unable to find 00000001.history in the archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePg(), "pg_wal", NULL);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get history");
HRN_STORAGE_PUT(storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/00000001.history", BUFSTRDEF("HISTORY"));
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 00000001.history in the repo1: 10-1 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_RESULT_UINT(storageInfoP(storagePg(), STRDEF("pg_wal/RECOVERYHISTORY")).size, 7, "check size");
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYHISTORY\n", .remove = true);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_TITLE("get compressed and encrypted WAL segment with invalid repo");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[cipher]\n"
"cipher-pass=\"" TEST_CIPHER_PASS_ARCHIVE "\"\n"
"\n"
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}",
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
.cipherType = cipherTypeAes256Cbc);
HRN_STORAGE_PUT(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
buffer, .compressType = compressTypeGz, .cipherType = cipherTypeAes256Cbc, .cipherPass = TEST_CIPHER_PASS_ARCHIVE);
// Add encryption options
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 1, TEST_PATH "/repo-bogus");
hrnCfgArgKeyRawFmt(argList, cfgOptRepoPath, 2, TEST_PATH "/repo");
Update CipherType/CipherMode to StringId. As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
hrnCfgArgKeyRawStrId(argList, cfgOptRepoCipherType, 2, cipherTypeAes256Cbc);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgEnvKeyRawZ(cfgOptRepoCipherPass, 2, TEST_CIPHER_PASS);
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "01ABCDEF01ABCDEF01ABCDEF");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgEnvKeyRemoveRaw(cfgOptRepoCipherPass, 2);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [FileMissingError] unable to load info file '" TEST_PATH "/repo-bogus/archive/test1/archive.info'"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" or '" TEST_PATH "/repo-bogus/archive/test1/archive.info.copy':\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" FileMissingError: unable to open missing file '" TEST_PATH "/repo-bogus/archive/test1/archive.info'"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" for read\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" FileMissingError: unable to open missing file '" TEST_PATH "/repo-bogus/archive/test1/archive.info.copy'"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" for read\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
" HINT: is archive_command configured correctly in postgresql.conf?\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use --no-archive-check to disable archive checks during backup if you have an alternate"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" archiving scheme.\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_RESULT_UINT(storageInfoP(storagePg(), STRDEF("pg_wal/RECOVERYXLOG")).size, 16 * 1024 * 1024, "check size");
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("repo1 has info but bad permissions");
HRN_INFO_PUT(
storageRepoIdxWrite(0), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=2\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"2={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-2", .mode = 0400);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo-bogus/archive/test1/10-2"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/01ABCDEF01ABCDEF': [13] Permission denied\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("all repos have info but bad permissions");
HRN_STORAGE_MODE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1", .mode = 0400);
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo-bogus/archive/test1/10-2"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/01ABCDEF01ABCDEF': [13] Permission denied\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo/archive/test1/10-1"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
"/01ABCDEF01ABCDEF': [13] Permission denied");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_MODE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-2");
HRN_STORAGE_MODE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("unable to get from one repo");
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
HRN_STORAGE_PUT_EMPTY(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
storageRepoIdxWrite(0),
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
STORAGE_REPO_ARCHIVE "/10-2/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 WARN: repo1: 10-2/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" [FormatError] unexpected eof in compressed data\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("unable to get from all repos");
HRN_STORAGE_MODE(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz", .mode = 0200);
TEST_ERROR(
cmdArchiveGet(), FileReadError,
"unable to get 01ABCDEF01ABCDEF01ABCDEF:\n"
"repo1: 10-2/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz [FormatError]"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" unexpected eof in compressed data\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"repo2: 10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz [FileOpenError]"
Reformat code with uncrustify. uncrustify has been configured to be as close to the current format as possible but the following changes were required: * Break long struct initializiers out of function calls. * Bit fields get extra spacing. * Strings that continue from the previous line no longer indented. * Ternary operators that do not fit on a single line moved to the next line first. * Align under parens for multi-line if statements. * Macros in header #if blocks are no longer indented. * Purposeful lack of function indentation in tests has been removed. Currently uncrustify does not completely reflow the code so there are some edge cases that might not be caught. However, this still represents a huge improvement and the formatting can be refined going forward. Support code for uncrustify will be in a followup commit.
2023-01-30 11:55:54 +07:00
" unable to open file '" TEST_PATH "/repo/archive/test1/10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF"
"-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz' for read: [13] Permission denied");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_MODE(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("repo is specified so invalid repo is skipped");
hrnCfgArgRawZ(argList, cfgOptRepo, "2");
hrnCfgEnvKeyRawZ(cfgOptRepoCipherPass, 2, TEST_CIPHER_PASS);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
hrnCfgEnvKeyRemoveRaw(cfgOptRepoCipherPass, 2);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("no segments to find with existing ok file");
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argList, cfgOptRepoPath, TEST_PATH "/repo");
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgRawZ(argList, cfgOptArchiveTimeout, "10");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
strLstAddZ(argList, "000000010000000100000001");
strLstAddZ(argList, "pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
HRN_INFO_PUT(
storageRepoIdxWrite(0), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=2\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"2={\"db-id\":" HRN_PG_SYSTEMID_10_Z ",\"db-version\":\"10\"}");
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
// Put a warning in the file to show that it was read and later overwritten
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok", "0\nshould not be output");
TEST_RESULT_VOID(cmdArchiveGet(), "get async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: unable to find 000000010000000100000001 in the archive asynchronously");
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
// Check that the ok file is missing since it should have been removed on the first loop and removed again on a subsequent
// loop once the async process discovered that the file was missing and wrote the ok file again.
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add archive-missing-retry option. Retry a WAL segment that was previously reported as missing by the archive-get command. This prevents notifications in the spool path from a prior restore from being used and possibly causing a recovery failure if consistency has not been reached. Disabling this option allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary. With retries enabled, a steady stream of WAL being archived will cause PostgreSQL to continue getting WAL from the archive rather than switch to streaming. When disabling this option it is important to ensure that the spool path for the stanza is empty. The restore command does this automatically if the spool path is configured at restore time. Otherwise, it is up to the user to ensure the spool path is empty.
2022-02-23 09:14:27 -06:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("do not retry missing segment");
argList = strLstNew();
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argList, cfgOptRepoPath, TEST_PATH "/repo");
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgRawZ(argList, cfgOptArchiveTimeout, "10");
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
hrnCfgArgRawBool(argList, cfgOptArchiveMissingRetry, false);
strLstAddZ(argList, "000000010000000100000001");
strLstAddZ(argList, "pg_wal/RECOVERYXLOG");
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
// Make sure that a WAL segment is found when the ok file is missing
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
TEST_RESULT_VOID(cmdArchiveGet(), "get async");
TEST_RESULT_LOG("P00 INFO: found 000000010000000100000001 in the archive asynchronously");
// Remove the ok file created by the async process
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000002.ok\n", .remove = true);
// Write an ok file
HRN_STORAGE_PUT_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok");
// Missing should be returned since archive-missing-retry=n
TEST_RESULT_VOID(cmdArchiveGet(), "get async");
TEST_RESULT_LOG("P00 INFO: unable to find 000000010000000100000001 in the archive asynchronously");
// Check that the ok file was removed
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*(). When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed. Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
FUNCTION_HARNESS_RETURN_VOID();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Reference in New Issue Copy Permalink