db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-11 00:50:20 +02:00
Code Issues Releases Activity
Files
71047a9d6d1eea71b3fbd430983541a54049cc69
pgbackrest/test/src/module/command/archiveGetTest.c

1134 lines
59 KiB
C
Raw Normal View History

Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Archive Get Command
***********************************************************************************************************************************/
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
#include "common/io/fdRead.h"
#include "common/io/fdWrite.h"
Migrate local, unencrypted, non-S3 archive-get command to C. The archive-get command will only be executed in C if the repository is local, unencrypted, and type posix or cifs. Admittedly a limited use case, but this is just the first step in migrating the archive-get command entirely into C. This is a direct migration from the Perl code (including messages) to integrate as seamlessly with the remaining Perl code as possible. It should not be possible to determine if the C version is running unless debug-level logging is enabled.
2018-09-11 15:42:31 -04:00
Minor fixes for command/archive-get unit test. Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test. Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet(). Move some harness headers down to the correct section.
2021-05-24 12:40:27 -04:00
#include "common/harnessConfig.h"
#include "common/harnessFork.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
#include "common/harnessInfo.h"
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
#include "common/harnessPostgres.h"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
#include "common/harnessProtocol.h"
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
#include "common/harnessStorage.h"
Add harnessInfoChecksum/Z() to ease creation of test info files.
2019-04-23 14:02:30 -04:00
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
Enable -Wstrict-prototypes and update all void functions to conform.
2018-08-03 19:19:14 -04:00
testRun(void)
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
{
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
FUNCTION_HARNESS_VOID();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// *****************************************************************************************************************************
if (testBegin("queueNeed()"))
{
StringList *argList = strLstNew();
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
Remove indexed option constants. These constants don't scale well as the index total is increased for an option. The core code rarely uses these options and they are easily replaced with cfgOptionName(). The tests had started to make use of the constants, so provide functions that build the option name from the optionId and, optionally, the optionKey.
2020-10-19 14:03:48 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, "/unused");
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
size_t queueSize = 16 * 1024 * 1024;
size_t walSegmentSize = 16 * 1024 * 1024;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("path missing");
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
TEST_ERROR(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
queueNeed(STRDEF("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92),
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
PathMissingError, "unable to list file info for missing path '" TEST_PATH "/spool/archive/test1/in'");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("queue size too small");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
queueNeed(STRDEF("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92),
"000000010000000100000001\n000000010000000100000002\n", "queue size smaller than min");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("queue empty");
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 10:24:42 +01:00
queueSize = (16 * 1024 * 1024) * 3;
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
queueNeed(STRDEF("000000010000000100000001"), false, queueSize, walSegmentSize, PG_VERSION_92),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000100000001\n000000010000000100000002\n000000010000000100000003\n", "empty queue");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("pg version earlier than 9.3");
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Buffer *walSegmentBuffer = bufNew(walSegmentSize);
memset(bufPtr(walSegmentBuffer), 0, walSegmentSize);
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE", walSegmentBuffer);
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF", walSegmentBuffer);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
queueNeed(STRDEF("0000000100000001000000FE"), false, queueSize, walSegmentSize, PG_VERSION_92),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000200000000\n000000010000000200000001\n", "queue has wal < 9.3");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "0000000100000001000000FE\n", .comment = "check queue");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_TITLE("pg >= 9.3 and ok/junk status files");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
walSegmentSize = 1024 * 1024;
queueSize = walSegmentSize * 5;
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/junk", "JUNK");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// Bad OK file with wrong length (just to make sure this does not cause strSubN() issues)
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/AAA.ok", "0\nWARNING");
// OK file with warnings somehow left over from a prior run
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFD.ok", "0\nWARNING");
// Valid queued WAL segments (one with an OK file containing warnings)
Add storage test macros for common operations. The macros should make it much easier to write complex tests, especially when compression and encryption are involved. Update the command/archiveGet test to show how the new macros are used.
2021-01-08 16:14:26 -05:00
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFE", walSegmentBuffer);
HRN_STORAGE_PUT(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFF", walSegmentBuffer);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000A00000FFF.ok", "0\nWARNING2");
// Empty OK file indicating a WAL segment not found at the end of the queue
HRN_STORAGE_PUT_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000B00000000.ok");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
queueNeed(STRDEF("000000010000000A00000FFD"), true, queueSize, walSegmentSize, PG_VERSION_11),
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
"000000010000000B00000000\n000000010000000B00000001\n000000010000000B00000002\n", "queue has wal >= 9.3");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_LIST(
storageSpool(), STORAGE_SPOOL_ARCHIVE_IN,
"000000010000000A00000FFE\n000000010000000A00000FFF\n000000010000000A00000FFF.ok\n");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// *****************************************************************************************************************************
if (testBegin("cmdArchiveGetAsync()"))
{
harnessLogLevelSet(logLevelDetail);
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
// Install local command handler shim
static const ProtocolServerHandler testLocalHandlerList[] = {PROTOCOL_SERVER_HANDLER_ARCHIVE_GET_LIST};
hrnProtocolLocalShimInstall(testLocalHandlerList, PROTOCOL_SERVER_HANDLER_LIST_SIZE(testLocalHandlerList));
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// Arguments that must be included
StringList *argBaseList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH "/repo");
hrnCfgArgRawZ(argBaseList, cfgOptSpoolPath, TEST_PATH "/spool");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawBool(argBaseList, cfgOptArchiveAsync, true);
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test2");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("command must be run on the pg host");
StringList *argList = strLstDup(argBaseList);
hrnCfgArgRawZ(argList, cfgOptPgHost, BOGUS_STR);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_ERROR(cmdArchiveGetAsync(), HostInvalidError, "archive-get command must be run on the PostgreSQL host");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/global.error",
"72\narchive-get command must be run on the PostgreSQL host", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Make archive-get locality error generate a global.error file. Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("error on no segments");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_ERROR(cmdArchiveGetAsync(), ParamInvalidError, "at least one wal segment is required");
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/global.error", "96\nat least one wal segment is required",
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Make archive-get locality error generate a global.error file. Moving this error into the try block ensures that a global.error file is generated, which will be seen by archive-get.
2021-01-08 16:29:56 -05:00
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("no segments to find");
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
strLstAddZ(argList, "000000010000000100000001");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
"P00 DETAIL: unable to find 000000010000000100000001 in the archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on path permission");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-1", .mode = 0400);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo/archive/test2/10-1"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"/0000000100000001': [13] Permission denied\n"
"P00 WARN: [RepoInvalidError] unable to find a valid repository");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.error",
"103\n"
"unable to find a valid repository\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo/archive/test2/10-1/0000000100000001':"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" [13] Permission denied",
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
HRN_STORAGE_MODE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-1");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on invalid compressed segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "get async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"P01 WARN: [FileReadError] raised from local-1 shim protocol: unable to get 000000010000000100000001:\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" repo1: 10-1/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz"
" [FormatError] unexpected eof in compressed data");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.error",
"42\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"raised from local-1 shim protocol: unable to get 000000010000000100000001:\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"repo1: 10-1/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz [FormatError]"
" unexpected eof in compressed data",
.remove = true);
TEST_STORAGE_LIST(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000001.pgbackrest.tmp\n");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz",
.remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("single segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
// There should be a temp file left over. Make sure it still exists to test that temp files are removed on retry.
TEST_STORAGE_EXISTS(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.pgbackrest.tmp");
Automatic retry for backup, restore, archive-get, and archive-push. If a local command, e.g. backupFile(), fails it will stop the entire process. Instead, retry local commands to deal with transient errors. Remove special logic in the S3 storage driver to retry RequestTimeTooSkewed errors since this is now handled by the general retry mechanism in the places where it is most likely to happen, i.e. file read/write. Also, this error should have been entirely eliminated by the asynchronous TLS implementation.
2020-07-14 15:05:31 -04:00
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 000000010000000100000001 in the repo1: 10-1 archive");
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("single segment with one invalid file");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n"
"2={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
"P01 WARN: repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz"
" [FormatError] unexpected eof in compressed data\n"
"P01 DETAIL: found 000000010000000100000001 in the repo1: 10-1 archive");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok",
"0\n"
"repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz [FormatError]"
" unexpected eof in compressed data",
.remove = true);
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("single segment with one invalid file");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n"
"2={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000100000001\n"
"P01 WARN: repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz"
" [FormatError] unexpected eof in compressed data\n"
"P01 DETAIL: found 000000010000000100000001 in the repo1: 10-1 archive");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok",
"0\n"
"repo1: 10-2/0000000100000001/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz [FormatError]"
" unexpected eof in compressed data",
.remove = true);
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-2/000000010000000100000001-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd.gz",
.remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_TITLE("multiple segments where some are missing or errored and mismatched repo");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgKeyRawZ(argBaseList, cfgOptRepoPath, 2, TEST_PATH "/repo2");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "0000000100000001000000FE");
strLstAddZ(argList, "0000000100000001000000FF");
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_INFO_PUT(
storageRepoIdxWrite(1), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"11\"}\n");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/0000000100000001000000FE-abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// Create segment duplicates
HRN_STORAGE_PUT_EMPTY(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_EMPTY(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo2: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '18072658121562454734'\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 0000000100000001000000FE in the repo1: 10-1 archive\n"
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
"P00 DETAIL: unable to find 0000000100000001000000FF in the archive");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE.ok",
"0\n"
"repo2: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '18072658121562454734'",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF.ok",
"0\n"
"repo2: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '18072658121562454734'",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_TITLE("error on duplicates now that no segments are missing, repo with bad perms");
// Fix repo 2 archive info but break archive path
HRN_INFO_PUT(
storageRepoIdxWrite(1), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1", .mode = 0400);
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/0000000100000001000000FF-efefefefefefefefefefefefefefefefefefefef");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1"
"/0000000100000001': [13] Permission denied\n"
"P00 WARN: repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1"
"/0000000100000002': [13] Permission denied\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 0000000100000001000000FE in the repo1: 10-1 archive\n"
"P01 DETAIL: found 0000000100000001000000FF in the repo1: 10-1 archive\n"
"P00 WARN: [ArchiveDuplicateError] duplicates found for WAL segment 000000010000000200000000:\n"
" repo1: 10-1/0000000100000002/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
", 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
" HINT: are multiple primaries archiving to this stanza?");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE", .remove = true);
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FE.ok",
"0\n"
"repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1/0000000100000001':"
" [13] Permission denied",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF", .remove = true);
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/0000000100000001000000FF.ok",
"0\n"
"repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1/0000000100000001':"
" [13] Permission denied",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.error",
"45\n"
"duplicates found for WAL segment 000000010000000200000000:\n"
"repo1: 10-1/0000000100000002/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb, 10-1/0000000100000002"
"/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
"HINT: are multiple primaries archiving to this stanza?\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo2/archive/test2/10-1" \
"/0000000100000002': [13] Permission denied",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
HRN_STORAGE_MODE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on duplicates");
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000200000000\n"
"P00 WARN: [ArchiveDuplicateError] duplicates found for WAL segment 000000010000000200000000:\n"
" repo1: 10-1/0000000100000002/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
", 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
" HINT: are multiple primaries archiving to this stanza?");
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.error",
"45\n"
"duplicates found for WAL segment 000000010000000200000000:\n"
"repo1: 10-1/0000000100000002/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb, 10-1/0000000100000002"
"/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
"HINT: are multiple primaries archiving to this stanza?",
.remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
.remove = true);
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("warn on invalid file");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgKeyRawZ(argBaseList, cfgOptRepoPath, 3, TEST_PATH "/repo3");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_INFO_PUT(
storageRepoIdxWrite(2), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"11\"}\n");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(2), "10-1", .mode = 0400);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '18072658121562454734'\n"
"P01 WARN: repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
" [FormatError] unexpected eof in compressed data\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P01 DETAIL: found 000000010000000200000000 in the repo2: 10-1 archive");
TEST_STORAGE_GET(
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.ok",
"0\n"
"repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '18072658121562454734'\n"
"repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
" [FormatError] unexpected eof in compressed data",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
TEST_STORAGE_GET_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000", .remove = true);
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error with warnings");
Add jobRetry parameter to HRN_CFG_LOAD(). Allow the default of 0 to be overridden to test retry behavior for commands.
2021-10-01 17:15:36 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync, .jobRetry = 1);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
TEST_RESULT_VOID(cmdArchiveGetAsync(), "archive async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 INFO: get 1 WAL file(s) from archive: 000000010000000200000000\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"P00 WARN: repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '18072658121562454734'\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"P01 WARN: [FileReadError] raised from local-1 shim protocol: unable to get 000000010000000200000000:\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
" repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
" [FormatError] unexpected eof in compressed data\n"
" repo2: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
" [FormatError] unexpected eof in compressed data");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_STORAGE_GET(
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000200000000.error",
"42\n"
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
"raised from local-1 shim protocol: unable to get 000000010000000200000000:\n"
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
"repo1: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
" [FormatError] unexpected eof in compressed data\n"
"repo2: 10-1/0000000100000002/000000010000000200000000-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
" [FormatError] unexpected eof in compressed data\n"
"repo3: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '18072658121562454734'",
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
.remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000200000000.pgbackrest.tmp\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
// -------------------------------------------------------------------------------------------------------------------------
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_TITLE("global error on invalid executable");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Add local process shim to archive-get/archive-push unit tests. Use the local process shim introduced in ef63750e in the archive-get/archive-push unit tests.
2021-05-25 11:06:05 -04:00
// Uninstall local command handler shim
hrnProtocolLocalShimUninstall();
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argList, cfgOptRepoPath, TEST_PATH "/repo");
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
hrnCfgArgRawZ(argList, cfgOptStanza, "test2");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "0000000100000001000000FE");
strLstAddZ(argList, "0000000100000001000000FF");
strLstAddZ(argList, "000000010000000200000000");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .role = cfgCmdRoleAsync, .exeBogus = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
TEST_ERROR(
cmdArchiveGetAsync(), ExecuteError,
"local-1 process terminated unexpectedly [102]: unable to execute 'pgbackrest-bogus': [2] No such file or directory");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
"P00 INFO: get 3 WAL file(s) from archive: 0000000100000001000000FE...000000010000000200000000");
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
Change meaning of TEST_RESULT_STR() macro. This macro was created before the String object existed so subsequent usage with String always included a lot of strPtr() wrapping. TEST_RESULT_STR_Z() had already been introduced but a wholesale replacement of TEST_RESULT_STR() was not done since the priority was on the C migration. Update all calls to (old) TEST_RESULT_STR() with one of the following variants: (new) TEST_RESULT_STR(), TEST_RESULT_STR_Z(), TEST_RESULT_Z(), TEST_RESULT_Z_STR().
2019-12-26 18:08:27 -07:00
TEST_RESULT_STR_Z(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
strNewBuf(storageGetP(storageNewReadP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_IN "/global.error")))),
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
"102\nlocal-1 process terminated unexpectedly [102]: unable to execute 'pgbackrest-bogus': "
"[2] No such file or directory",
Use a single file to handle global errors in async archiving. The prior behavior on a global error (i.e. not file specific) was to write an individual error file for each WAL file being processed. On retry each of these error files would be removed, and if the error was persistent, they would then be recreated. In a busy environment this could mean tens or hundreds of thousands of files. Another issue was that the error files could not be written until a list of WAL files to process had been generated. This was easy enough for archive-get but archive-push requires more processing and any errors that happened when generating the list would only be reported in the pgBackRest log rather than the PostgreSQL log. Instead write a global.error file that applies to any WAL file that does not have an explicit ok or error file. This reduces churn and allows more errors to be reported directly to PostgreSQL.
2019-03-25 08:12:38 +04:00
"check global error");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "global.error\n", .remove = true);
The archive-get command is implemented entirely in C. This new implementation should behave exactly like the old Perl code with the exception of a few updated log messages. Remove as much of the Perl code as possible without breaking other commands.
2019-02-27 23:03:02 +02:00
}
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// *****************************************************************************************************************************
if (testBegin("cmdArchiveGet()"))
{
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
harnessLogLevelSet(logLevelDetail);
StringList *argBaseList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH "/repo");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test1");
hrnCfgArgRawZ(argBaseList, cfgOptArchiveTimeout, "1");
// -------------------------------------------------------------------------------------------------------------------------
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_TITLE("command must be run on the pg host");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
StringList *argList = strLstDup(argBaseList);
hrnCfgArgRawZ(argList, cfgOptPgHost, BOGUS_STR);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Error when archive-get/archive-push/restore are not run on a PostgreSQL host. This error was lost during the migration to C. The error that occurred instead (generally an SSH auth error) was hard to debug. Restore the original behavior by throwing an error immediately if pg1-host is configured for any of these commands. reset-pg1-host can be used to suppress the error when required.
2020-02-12 17:18:48 -07:00
TEST_ERROR(cmdArchiveGet(), HostInvalidError, "archive-get command must be run on the PostgreSQL host");
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("segment parameter not specified");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "WAL segment to get required");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("path parameter not specified");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000100000001");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_ERROR(cmdArchiveGet(), ParamRequiredError, "path to copy WAL segment required");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("no valid repo");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Replace TEST_ERROR_FMT() with TEST_ERROR() where possible. Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.
2021-05-22 11:59:43 -04:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [FileMissingError] unable to load info file '" TEST_PATH "/repo/archive/test1/archive.info' or '"
TEST_PATH "/repo/archive/test1/archive.info.copy':\n"
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info' for read\n"
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info.copy' for"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" read\n"
" HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
" HINT: is archive_command configured correctly in postgresql.conf?\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
" scheme.");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("no valid repo - async");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
strLstAddZ(argList, "00000001.history");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYHISTORY");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Replace TEST_ERROR_FMT() with TEST_ERROR() where possible. Some calls did not need TEST_ERROR_FMT() at all and others could be converted by replacing parameters with available defines, e.g. TEST_PATH.
2021-05-22 11:59:43 -04:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [FileMissingError] unable to load info file '" TEST_PATH "/repo/archive/test1/archive.info' or '"
TEST_PATH "/repo/archive/test1/archive.info.copy':\n"
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info' for read\n"
" FileMissingError: unable to open missing file '" TEST_PATH "/repo/archive/test1/archive.info.copy' for"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" read\n"
" HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
" HINT: is archive_command configured correctly in postgresql.conf?\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving"
" scheme.");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("WAL not found - timeout");
// Make sure the process times out when there is nothing to get
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
argList = strLstDup(argBaseList);
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
strLstAddZ(argList, "000000010000000100000001");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, "pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Rename strPtr()/strPtrNull() to strZ()/strZNull(). We use the Z suffix in many functions to indicate that we are expecting a zero-terminated string so make this function conform to the pattern. As a bonus the new name is a bit shorter, which is a good quality in a commonly-used function.
2020-07-30 07:49:06 -04:00
THROW_ON_SYS_ERROR(chdir(strZ(cfgOptionStr(cfgOptPgPath))) != 0, PathMissingError, "unable to chdir()");
Fix archive-push/archive-get when PGDATA is symlinked. Commit 7168e074 tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call.
2019-12-11 14:36:39 -05:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
TEST_ERROR(
cmdArchiveGet(), ArchiveTimeoutError,
"unable to get WAL file '000000010000000100000001' from the archive asynchronously after 1 second(s)");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("check for missing WAL");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_EMPTY(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
TEST_ERROR(
cmdArchiveGet(), ArchiveTimeoutError,
"unable to get WAL file '000000010000000100000001' from the archive asynchronously after 1 second(s)");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
TEST_RESULT_BOOL(
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
storageExistsP(storageSpool(), STRDEF(STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok")), false,
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
"check OK file was removed");
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("write WAL segment for success");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storagePgWrite(), "pg_wal");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", "SHOULD-BE-A-REAL-WAL-FILE");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 000000010000000100000001 in the archive asynchronously");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("write WAL segments for success - queue full");
hrnCfgArgRawZ(argList, cfgOptArchiveGetQueueMax, "48");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
// Write more WAL segments (in this case queue should be full)
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001", "SHOULD-BE-A-REAL-WAL-FILE");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok", "0\nwarning about x");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000002", "SHOULD-BE-A-REAL-WAL-FILE");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Remove forks from command/archive-get test. These were required to deal with the legacy Perl code being unable to load new options between tests. The C code does not have this issue so remove the forks and update process ids in the log tests.
2021-01-06 11:36:42 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "successful get");
Disable flapping archive/get unit on CentOS 6. This test has been flapping since 9b9396c7. It seems to be some kind of timing issue since all integration tests pass and this unit passes on all other VMs. It only happens on Travis and is not reproducible in any development environment that we have tried. For now, disable the test since the constant flapping is causing major delays in testing and quite a bit of time has been spent trying to identify the root cause. We are actively developing these tests and hope the issue will be identified during the course of normal development. A number of improvements were made to the tests while searching for this issue. While none of them helped, it makes sense to keep the improvements.
2018-10-02 17:54:43 +01:00
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
"P00 WARN: warning about x\n"
"P00 INFO: found 000000010000000100000001 in the archive asynchronously");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Minor fixes for command/archive-get unit test. Test and remove WAL segment 000000010000000100000002 in the test where it is created rather than as a byproduct of a much later test. Remove incorrect local role from test. It worked, but was not the correct command role to be using when calling cmdArchiveGet(). Move some harness headers down to the correct section.
2021-05-24 12:40:27 -04:00
TEST_STORAGE_LIST(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN, "000000010000000100000002\n", .remove = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("unable to get lock");
// Make sure the process times out when it can't get a lock
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_BEGIN()
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
{
Improve HRN_FORK*() macros. Simplify HRN_FORK_CHILD_BEGIN() by adding optional parameters with the common defaults. Add _FD() to macros that retrieve file descriptors to make their purpose clearer.
2021-07-13 14:22:53 -04:00
HRN_FORK_CHILD_BEGIN()
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
{
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
TEST_RESULT_VOID(
lockAcquire(
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptStanza), STRDEF("999-dededede"), cfgLockType(), 30000,
true),
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
"acquire lock");
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros. This removes a lot of boiler plate where every instance needs to create these interfaces. Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes. In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
// Notify parent that lock has been acquired
HRN_FORK_CHILD_NOTIFY_PUT();
// Wait for parent to allow release lock
HRN_FORK_CHILD_NOTIFY_GET();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
lockRelease(true);
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_CHILD_END();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_PARENT_BEGIN()
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
{
Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros. This removes a lot of boiler plate where every instance needs to create these interfaces. Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes. In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
// Wait for child to acquire lock
HRN_FORK_PARENT_NOTIFY_GET(0);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
TEST_ERROR(
cmdArchiveGet(), ArchiveTimeoutError,
"unable to get WAL file '000000010000000100000001' from the archive asynchronously after 1 second(s)");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Automatically create IoRead/IoWrite interfaces in HRN_FORK*() macros. This removes a lot of boiler plate where every instance needs to create these interfaces. Also add HRN_FORK_*_NOTIFY*() macros to standardize synchronizing between the parent and child processes. In both cases update the tests with the new macros.
2021-07-14 14:31:57 -04:00
// Notify child to release lock
HRN_FORK_PARENT_NOTIFY_PUT(0);
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_PARENT_END();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
}
Rename HARNESS_FORK*() macros to HRN_FORK*(). This matches the new pattern for harness macro naming and is shorter.
2021-07-13 11:58:23 -04:00
HRN_FORK_END();
Remove lockClear(). This function has not been used since the switch to the fork/exec model. lockClear() was still used in one test (other than the lock test) so update the test and remove the function.
2021-04-14 15:41:55 -04:00
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_TITLE("too many parameters specified");
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
strLstAddZ(argList, BOGUS_STR);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList, .exeBogus = true);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
TEST_ERROR(cmdArchiveGet(), ParamInvalidError, "extra parameters found");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_TITLE("pg version does not match archive.info");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_11, .systemId = 0xFACEFACEFACEFACE}));
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
argBaseList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argBaseList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argBaseList, cfgOptRepoPath, TEST_PATH "/repo");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
hrnCfgArgRawZ(argBaseList, cfgOptStanza, "test1");
argList = strLstDup(argBaseList);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
strLstAddZ(argList, "01ABCDEF01ABCDEF01ABCDEF");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 WARN: repo1: [ArchiveMismatchError] unable to retrieve the archive id for database version '11' and system-id"
" '18072658121562454734'");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Fix incorrect test title.
2021-04-21 18:27:18 -04:00
TEST_TITLE("pg system id does not match archive.info");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0x8888888888888888}));
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 WARN: repo1: [ArchiveMismatchError] unable to retrieve the archive id for database version '10' and system-id"
" '9838263505978427528'");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("file is missing");
HRN_STORAGE_PUT(
storagePgWrite(), PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL,
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgControlToBuffer((PgControl){.version = PG_VERSION_10, .systemId = 0xFACEFACEFACEFACE}));
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: unable to find 01ABCDEF01ABCDEF01ABCDEF in the archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST_EMPTY(storagePg(), "pg_wal");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get WAL segment");
Buffer *buffer = bufNew(16 * 1024 * 1024);
memset(bufPtr(buffer), 0, bufSize(buffer));
bufUsedSet(buffer, bufSize(buffer));
HRN_STORAGE_PUT(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
buffer);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1: 10-1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_RESULT_UINT(storageInfoP(storagePg(), STRDEF("pg_wal/RECOVERYXLOG")).size, 16 * 1024 * 1024, "check size");
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add duplicate WAL test to synchronous archive-get.
2021-01-13 13:01:40 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("error on duplicate WAL segment");
HRN_STORAGE_PUT_EMPTY(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb");
TEST_ERROR(
cmdArchiveGet(), ArchiveDuplicateError,
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"duplicates found for WAL segment 01ABCDEF01ABCDEF01ABCDEF:\n"
"repo1: 10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
", 10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
Add duplicate WAL test to synchronous archive-get.
2021-01-13 13:01:40 -05:00
"HINT: are multiple primaries archiving to this stanza?");
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePg(), "pg_wal", NULL);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
.remove = true);
Add duplicate WAL test to synchronous archive-get.
2021-01-13 13:01:40 -05:00
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get from prior db-id");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"2={\"db-id\":18072658121562454734,\"db-version\":\"10\"}\n"
"3={\"db-id\":10000000000000000000,\"db-version\":\"11\"}\n"
"4={\"db-id\":18072658121562454734,\"db-version\":\"10\"}");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1: 10-1 archive");
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get from current db-id");
HRN_STORAGE_PUT_EMPTY(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-4/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
Improve archive-get performance. Check that archive files exist in the main process instead of the local process. This means that the archive.info file only needs to be loaded once per execution rather than once per file to get. Stop looking when a file is missing or in error. PostgreSQL will never request anything past the missing file so there is no point in getting them. This also reduces "unable to find" logging in the async process. Cache results of storageList() when looking for multiple files to reduce storage I/O. Look for all requested archive files in the archive-id where the first file is found. They may not all be there, but this reduces the number of list calls. If subsequent files are in another archive id they will be found on the next archive-get call.
2021-01-15 10:15:52 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo1: 10-4 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
TEST_STORAGE_EXISTS(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-4/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
.remove = true);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get partial");
buffer = bufNew(16 * 1024 * 1024);
memset(bufPtr(buffer), 0xFF, bufSize(buffer));
bufUsedSet(buffer, bufSize(buffer));
HRN_STORAGE_PUT(
storageRepoWrite(),
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
STORAGE_REPO_ARCHIVE "/10-4/000000010000000100000001.partial-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
buffer);
argList = strLstDup(argBaseList);
strLstAddZ(argList, "000000010000000100000001.partial");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 000000010000000100000001.partial in the repo1: 10-4 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
TEST_STORAGE_EXISTS(
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
storageRepoWrite(),
Add optional remove to TEST_STORAGE_EXISTS(). This allows TEST_STORAGE_EXISTS() to be used in most cases where TEST_STORAGE_REMOVE() was used before. Rename TEST_STORAGE_REMOVE() to HRN_STORAGE_REMOVE() now that is is no longer used as a test. Still allow an error when the file is missing just to help keep tests tidy.
2021-06-08 14:51:23 -04:00
STORAGE_REPO_ARCHIVE "/10-4/000000010000000100000001.partial-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", .remove = true);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get missing history");
argList = strLstDup(argBaseList);
strLstAddZ(argList, "00000001.history");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYHISTORY");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 1, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: unable to find 00000001.history in the archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePg(), "pg_wal", NULL);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("get history");
HRN_STORAGE_PUT(storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/00000001.history", BUFSTRDEF("HISTORY"));
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: found 00000001.history in the repo1: 10-1 archive");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_RESULT_UINT(storageInfoP(storagePg(), STRDEF("pg_wal/RECOVERYHISTORY")).size, 7, "check size");
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYHISTORY\n", .remove = true);
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
// -------------------------------------------------------------------------------------------------------------------------
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_TITLE("get compressed and encrypted WAL segment with invalid repo");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE,
"[cipher]\n"
"cipher-pass=\"" TEST_CIPHER_PASS_ARCHIVE "\"\n"
"\n"
"[db]\n"
"db-id=1\n"
"\n"
"[db:history]\n"
"1={\"db-id\":18072658121562454734,\"db-version\":\"10\"}",
.cipherType = cipherTypeAes256Cbc);
HRN_STORAGE_PUT(
storageRepoWrite(), STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
buffer, .compressType = compressTypeGz, .cipherType = cipherTypeAes256Cbc, .cipherPass = TEST_CIPHER_PASS_ARCHIVE);
// Add encryption options
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 1, TEST_PATH "/repo-bogus");
hrnCfgArgKeyRawFmt(argList, cfgOptRepoPath, 2, TEST_PATH "/repo");
Update CipherType/CipherMode to StringId. As in 6cc521b, this allows option values and enums to be easily mapped together.
2021-04-28 11:36:20 -04:00
hrnCfgArgKeyRawStrId(argList, cfgOptRepoCipherType, 2, cipherTypeAes256Cbc);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgEnvKeyRawZ(cfgOptRepoCipherPass, 2, TEST_CIPHER_PASS);
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
Add tests for the archive-get command. Provide additional coverage for less common use cases.
2021-01-12 18:20:28 -05:00
strLstAddZ(argList, "01ABCDEF01ABCDEF01ABCDEF");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
strLstAddZ(argList, TEST_PATH "/pg/pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgEnvKeyRemoveRaw(cfgOptRepoCipherPass, 2);
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [FileMissingError] unable to load info file '" TEST_PATH "/repo-bogus/archive/test1/archive.info'"
" or '" TEST_PATH "/repo-bogus/archive/test1/archive.info.copy':\n"
" FileMissingError: unable to open missing file '" TEST_PATH "/repo-bogus/archive/test1/archive.info'"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" for read\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" FileMissingError: unable to open missing file '" TEST_PATH "/repo-bogus/archive/test1/archive.info.copy'"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
" for read\n"
" HINT: archive.info cannot be opened but is required to push/get WAL segments.\n"
" HINT: is archive_command configured correctly in postgresql.conf?\n"
" HINT: has a stanza-create been performed?\n"
" HINT: use --no-archive-check to disable archive checks during backup if you have an alternate"
" archiving scheme.\n"
"P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_RESULT_UINT(storageInfoP(storagePg(), STRDEF("pg_wal/RECOVERYXLOG")).size, 16 * 1024 * 1024, "check size");
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("repo1 has info but bad permissions");
HRN_INFO_PUT(
storageRepoIdxWrite(0), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=2\n"
"\n"
"[db:history]\n"
"2={\"db-id\":18072658121562454734,\"db-version\":\"10\"}");
Add HRN_STORAGE_PATH_CREATE() macro. Update command/archive-get test to show how it is used. Also move one path create that was much earlier than it needed to be.
2021-05-28 15:23:18 -04:00
HRN_STORAGE_PATH_CREATE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-2", .mode = 0400);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo-bogus/archive/test1/10-2"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"/01ABCDEF01ABCDEF': [13] Permission denied\n"
"P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
Replace storageTest with storagePg*() in command/archive-get test. All instances of storageTest are better represented with storagePg*(), which allows TEST_PATH and TEST_PATH_PG to be omitted. Also remove some headers which are no longer needed.
2021-05-27 16:50:18 -04:00
TEST_STORAGE_LIST(storagePgWrite(), "pg_wal", "RECOVERYXLOG\n", .remove = true);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("all repos have info but bad permissions");
HRN_STORAGE_MODE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1", .mode = 0400);
TEST_ERROR(cmdArchiveGet(), RepoInvalidError, "unable to find a valid repository");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo1: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo-bogus/archive/test1/10-2"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"/01ABCDEF01ABCDEF': [13] Permission denied\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: repo2: [PathOpenError] unable to list file info for path '" TEST_PATH "/repo/archive/test1/10-1"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"/01ABCDEF01ABCDEF': [13] Permission denied");
HRN_STORAGE_MODE(storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/10-2");
HRN_STORAGE_MODE(storageRepoIdxWrite(1), STORAGE_REPO_ARCHIVE "/10-1");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("unable to get from one repo");
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
HRN_STORAGE_PUT_EMPTY(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
storageRepoIdxWrite(0),
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
STORAGE_REPO_ARCHIVE "/10-2/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"P00 WARN: repo1: 10-2/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz"
" [FormatError] unexpected eof in compressed data\n"
"P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("unable to get from all repos");
HRN_STORAGE_MODE(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz", .mode = 0200);
TEST_ERROR(
cmdArchiveGet(), FileReadError,
"unable to get 01ABCDEF01ABCDEF01ABCDEF:\n"
"repo1: 10-2/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz [FormatError]"
" unexpected eof in compressed data\n"
"repo2: 10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz [FileOpenError]"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" unable to open file '" TEST_PATH "/repo/archive/test1/10-1/01ABCDEF01ABCDEF/01ABCDEF01ABCDEF01ABCDEF"
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
"-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz' for read: [13] Permission denied");
HRN_STORAGE_MODE(
storageRepoIdxWrite(1),
STORAGE_REPO_ARCHIVE "/10-1/01ABCDEF01ABCDEF01ABCDEF-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.gz");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("repo is specified so invalid repo is skipped");
hrnCfgArgRawZ(argList, cfgOptRepo, "2");
hrnCfgEnvKeyRawZ(cfgOptRepoCipherPass, 2, TEST_CIPHER_PASS);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Add archive-get command multi-repo support. Repositories will be searched in order for the requested archive file. Errors will be reported as warnings as long as a valid copy of the archive file is found.
2021-02-23 15:34:28 -05:00
hrnCfgEnvKeyRemoveRaw(cfgOptRepoCipherPass, 2);
TEST_RESULT_INT(cmdArchiveGet(), 0, "get");
Update command/archive tests to use standard patterns. Includes archiveCommon, archiveGet and archivePush. Also fixed a test that was looking in repo instead of repo3 in the original archivePush to use the repo3 path as stated by the comment (line 879 in original tests and line 855 in new tests).
2021-07-02 12:22:11 -04:00
TEST_RESULT_LOG("P00 INFO: found 01ABCDEF01ABCDEF01ABCDEF in the repo2: 10-1 archive");
Refactor common/archiveGet unit test. The test was pretty old and written in stages during the migration, so storage use was a bit archaic and the organization was poor. Update using the new storage macros and reorganize the tests to provide better coverage.
2021-01-08 16:48:32 -05:00
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("no segments to find with existing ok file");
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptPgPath, TEST_PATH "/pg");
hrnCfgArgRawZ(argList, cfgOptRepoPath, TEST_PATH "/repo");
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgRawZ(argList, cfgOptArchiveTimeout, "10");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argList, cfgOptSpoolPath, TEST_PATH "/spool");
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
hrnCfgArgRawBool(argList, cfgOptArchiveAsync, true);
strLstAddZ(argList, "000000010000000100000001");
strLstAddZ(argList, "pg_wal/RECOVERYXLOG");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdArchiveGet, argList);
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
HRN_INFO_PUT(
storageRepoIdxWrite(0), INFO_ARCHIVE_PATH_FILE,
"[db]\n"
"db-id=2\n"
"\n"
"[db:history]\n"
"2={\"db-id\":18072658121562454734,\"db-version\":\"10\"}");
// Put a warning in the file to show that it was read and later overwritten
HRN_STORAGE_PUT_Z(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN "/000000010000000100000001.ok", "0\nshould not be output");
TEST_RESULT_VOID(cmdArchiveGet(), "get async");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 INFO: unable to find 000000010000000100000001 in the archive asynchronously");
Fix issues with leftover spool files from a prior restore. If an ok file (which indicates the WAL segment was not found) is present on the first iteration of the loop then remove it and spawn the async process to retry. This action also resets the queue. Also error if no response is received from the async process rather than returning not found. PostgreSQL will respond the same either way, but this allows us to determine when something is going wrong with the async process. Update archiveAsyncStatus() to allow warnings to be suppressed. It is better to retry if no WAL segment was found before warning because the warning might be stale.
2021-05-13 17:51:39 -04:00
// Check that the ok file is missing since it should have been removed on the first loop and removed again on a subsequent
// loop once the async process discovered that the file was missing and wrote the ok file again.
TEST_STORAGE_LIST_EMPTY(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN);
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Add stack trace macros to all functions. Low-level functions only include stack trace in test builds while higher-level functions ship with stack trace built-in. Stack traces include all parameters passed to the function but production builds only create the parameter list when the log level is set high enough, i.e. debug or trace depending on the function.
2018-05-18 11:57:32 -04:00
Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*(). When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed. Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
FUNCTION_HARNESS_RETURN_VOID();
Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
2018-04-30 17:27:39 -04:00
}
Reference in New Issue Copy Permalink