db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Code Issues Releases Activity
Files
136d309dd4bc1ada9f3d775f036b62292fda390b
pgbackrest/test/src/module/command/verifyTest.c

1448 lines
92 KiB
C
Raw Normal View History

Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
/***********************************************************************************************************************************
Test Stanza Commands
***********************************************************************************************************************************/
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
#include "common/io/bufferRead.h"
#include "postgres/interface.h"
#include "postgres/version.h"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
#include "storage/posix/storage.h"
#include "common/harnessConfig.h"
#include "common/harnessInfo.h"
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
#include "common/harnessPostgres.h"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
#include "common/harnessPq.h"
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
#include "common/harnessProtocol.h"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
void
testRun(void)
{
FUNCTION_HARNESS_VOID();
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
// Install local command handler shim
static const ProtocolServerHandler testLocalHandlerList[] = {PROTOCOL_SERVER_HANDLER_VERIFY_LIST};
hrnProtocolLocalShimInstall(testLocalHandlerList, PROTOCOL_SERVER_HANDLER_LIST_SIZE(testLocalHandlerList));
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
StringList *argListBase = strLstNew();
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
hrnCfgArgRawZ(argListBase, cfgOptStanza, "db");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argListBase, cfgOptRepoPath, TEST_PATH "/repo");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
const char *fileContents = "acefile";
uint64_t fileSize = 7;
const String *fileChecksum = STRDEF("d1cd8a7d11daa26814b93eb604e1d49ab4b43770");
#define TEST_BACKUP_DB1_94 \
"db-catalog-version=201409291\n" \
"db-control-version=942\n" \
"db-id=1\n" \
"db-system-id=6625592122879095702\n" \
"db-version=\"9.4\"\n"
#define TEST_BACKUP_DB2_11 \
"db-catalog-version=201707211\n" \
"db-control-version=1100\n" \
"db-id=2\n" \
"db-system-id=6626363367545678089\n" \
"db-version=\"11\"\n"
#define TEST_BACKUP_DB1_CURRENT_FULL1 \
"20181119-152138F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.28dev\"," \
"\"backup-archive-start\":\"000000010000000000000002\",\"backup-archive-stop\":\"000000010000000000000002\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1482182846,\"backup-timestamp-stop\":1482182861,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB1_CURRENT_FULL2 \
"20181119-152800F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB1_CURRENT_FULL3 \
"20181119-152900F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000010000000000000004\",\"backup-archive-stop\":\"000000010000000000000004\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB1_HISTORY \
"1={\"db-catalog-version\":201409291,\"db-control-version\":942,\"db-system-id\":6625592122879095702," \
"\"db-version\":\"9.4\"}"
#define TEST_BACKUP_DB2_HISTORY \
"2={\"db-catalog-version\":201707211,\"db-control-version\":1100,\"db-system-id\":6626363367545678089," \
"\"db-version\":\"11\"}"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_BACKUP_INFO_MULTI_HISTORY_BASE \
"[backup:current]\n" \
TEST_BACKUP_DB1_CURRENT_FULL1 \
TEST_BACKUP_DB1_CURRENT_FULL2 \
TEST_BACKUP_DB1_CURRENT_FULL3 \
"\n" \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_BACKUP_DB2_HISTORY
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_ARCHIVE_INFO_BASE \
"[db]\n" \
"db-id=1\n" \
"db-system-id=6625592122879095702\n" \
"db-version=\"9.4\"\n" \
"\n" \
"[db:history]\n" \
"1={\"db-id\":6625592122879095702,\"db-version\":\"9.4\"}"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE \
"[db]\n" \
"db-id=2\n" \
"db-system-id=6626363367545678089\n" \
"db-version=\"11\"\n" \
"\n" \
"[db:history]\n" \
"1={\"db-id\":6625592122879095702,\"db-version\":\"9.4\"}\n" \
"2={\"db-id\":6626363367545678089,\"db-version\":\"11\"}"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
#define TEST_MANIFEST_HEADER \
"[backup]\n" \
"backup-label=null\n" \
"backup-timestamp-copy-start=0\n" \
"backup-timestamp-start=0\n" \
"backup-timestamp-stop=0\n" \
"backup-type=\"full\"\n"
#define TEST_MANIFEST_DB_92 \
"\n" \
"[backup:db]\n" \
"db-catalog-version=201204301\n" \
"db-control-version=922\n" \
"db-id=1\n" \
"db-system-id=6625592122879095702\n" \
"db-version=\"9.2\"\n"
#define TEST_MANIFEST_DB_94 \
"\n" \
"[backup:db]\n" \
"db-catalog-version=201409291\n" \
"db-control-version=942\n" \
"db-id=1\n" \
"db-system-id=6625592122879095702\n" \
"db-version=\"9.4\"\n"
#define TEST_MANIFEST_OPTION_ALL \
"\n" \
"[backup:option]\n" \
"option-archive-check=false\n" \
"option-archive-copy=false\n" \
"option-checksum-page=false\n" \
"option-compress=false\n" \
"option-compress-type=\"none\"\n" \
"option-hardlink=false\n" \
"option-online=false\n"
#define TEST_MANIFEST_OPTION_ARCHIVE_TRUE \
"\n" \
"[backup:option]\n" \
"option-archive-check=true\n" \
"option-archive-copy=true\n"
#define TEST_MANIFEST_TARGET \
"\n" \
"[backup:target]\n" \
"pg_data={\"path\":\"/pg/base\",\"type\":\"path\"}\n"
#define TEST_MANIFEST_DB \
"\n" \
"[db]\n" \
"postgres={\"db-id\":12173,\"db-last-system-id\":12168}\n"
#define TEST_MANIFEST_FILE \
"\n" \
"[target:file]\n" \
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"master\":true" \
",\"size\":4,\"timestamp\":1565282114}\n"
#define TEST_MANIFEST_FILE_DEFAULT \
"\n" \
"[target:file:default]\n" \
"group=\"group1\"\n" \
"master=false\n" \
"mode=\"0600\"\n" \
"user=\"user1\"\n"
#define TEST_MANIFEST_LINK \
"\n" \
"[target:link]\n" \
"pg_data/pg_stat={\"destination\":\"../pg_stat\"}\n"
#define TEST_MANIFEST_LINK_DEFAULT \
"\n" \
"[target:link:default]\n" \
"group=\"group1\"\n" \
"user=false\n"
#define TEST_MANIFEST_PATH \
"\n" \
"[target:path]\n" \
"pg_data={\"user\":\"user1\"}\n" \
#define TEST_MANIFEST_PATH_DEFAULT \
"\n" \
"[target:path:default]\n" \
"group=false\n" \
"mode=\"0700\"\n" \
"user=\"user1\"\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_INVALID_BACKREST_INFO \
"[backrest]\n" \
"backrest-checksum=\"BOGUS\"\n" \
"backrest-format=5\n" \
"backrest-version=\"2.28\"\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// *****************************************************************************************************************************
if (testBegin("verifyManifestFile()"))
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_BACKUP_LABEL_FULL "20181119-152138F"
Manifest *manifest = NULL;
unsigned int jobErrorTotal = 0;
VerifyBackupResult backupResult = {.backupLabel = strNewZ(TEST_BACKUP_LABEL_FULL)};
InfoPg *infoPg = NULL;
TEST_ASSIGN(
infoPg, infoArchivePg(infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_ARCHIVE_INFO_BASE)))),
"infoPg from archive.info");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("manifest.copy exists, no manifest main, manifest db version not in history, not current db");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_92
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "manifest db section mismatch");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set - pg version mismatch");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest.copy'"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" for read\n"
"P00 ERROR: [028]: '20181119-152138F' may not be recoverable - PG data (id 1, version 9.2, system-id "
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"6625592122879095702) is not in the backup.info history, skipping");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("rerun test with db-system-id invalid and no main");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE);
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
"\n"
"[backup:db]\n"
"db-catalog-version=201409291\n"
"db-control-version=942\n"
"db-id=1\n"
"db-system-id=0\n"
"db-version=\"9.4\"\n"
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "manifest copy - invalid system-id");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set - pg system-id mismatch");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest' for read\n"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"P00 WARN: 20181119-152138F/backup.manifest is missing or unusable, using copy\n"
"P00 ERROR: [028]: '20181119-152138F' may not be recoverable - PG data (id 1, version 9.4, system-id 0) is "
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"not in the backup.info history, skipping");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("rerun copy test with db-id invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
"\n"
"[backup:db]\n"
"db-catalog-version=201409291\n"
"db-control-version=942\n"
"db-id=0\n"
"db-system-id=6625592122879095702\n"
"db-version=\"9.4\"\n"
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "manifest copy - invalid db-id");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set - pg db-id mismatch");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest' for read\n"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"P00 WARN: 20181119-152138F/backup.manifest is missing or unusable, using copy\n"
"P00 ERROR: [028]: '20181119-152138F' may not be recoverable - PG data (id 0, version 9.4, system-id "
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"6625592122879095702) is not in the backup.info history, skipping");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("missing main manifest, errored copy");
backupResult.status = backupValid;
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
storageRepoWrite(), TEST_PATH "/repo/" STORAGE_PATH_BACKUP "/db/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
INFO_COPY_EXT, TEST_INVALID_BACKREST_INFO, .comment = "invalid manifest copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest' for read\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P00 WARN: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"<REPO:BACKUP>/20181119-152138F/backup.manifest.copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("current backup true");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
storageRepoWrite(), TEST_PATH "/repo/" STORAGE_PATH_BACKUP "/db/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE,
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_INVALID_BACKREST_INFO, .comment = "invalid manifest");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, true, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P00 WARN: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"<REPO:BACKUP>/20181119-152138F/backup.manifest\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P00 WARN: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"<REPO:BACKUP>/20181119-152138F/backup.manifest.copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write a valid manifest with a manifest copy that is invalid
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "valid manifest");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, true, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR_NE(manifest, NULL, "manifest set");
TEST_RESULT_UINT(backupResult.status, backupValid, "manifest usable");
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
TEST_RESULT_LOG("P00 WARN: backup '20181119-152138F' manifest.copy does not match manifest");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
}
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// *****************************************************************************************************************************
if (testBegin("verifyCreateArchiveIdRange()"))
{
VerifyWalRange *walRangeResult = NULL;
unsigned int errTotal = 0;
StringList *walFileList = strLstNew();
VerifyArchiveResult archiveResult =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.archiveId = strNewZ("9.4-1"),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
.walRangeList = lstNewP(sizeof(VerifyWalRange), .comparator = lstComparatorStr),
};
List *archiveIdResultList = lstNewP(sizeof(VerifyArchiveResult), .comparator = archiveIdComparator);
lstAdd(archiveIdResultList, &archiveResult);
VerifyArchiveResult *archiveIdResult = lstGetLast(archiveIdResultList);
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("Single WAL");
archiveIdResult->pgWalInfo.size = PG_WAL_SEGMENT_SIZE_DEFAULT;
archiveIdResult->pgWalInfo.version = PG_VERSION_94;
strLstAddZ(walFileList, "000000020000000200000000-daa497dba64008db824607940609ba1cd7c6c501.gz");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 0, "no errors");
TEST_RESULT_UINT(lstSize(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList), 1, "single range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 0),
"get range");
TEST_RESULT_STR_Z(walRangeResult->start, "000000020000000200000000", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000000", "stop range");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("Duplicate WAL only - no range, all removed from list");
lstClear(archiveIdResult->walRangeList);
// Add a duplicate
strLstAddZ(walFileList, "000000020000000200000000");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 1, "duplicate WAL error");
TEST_RESULT_UINT(strLstSize(walFileList), 0, "all WAL removed from WAL file list");
TEST_RESULT_UINT(lstSize(archiveIdResult->walRangeList), 0, "no range");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 ERROR: [028]: duplicate WAL '000000020000000200000000' for '9.4-1' exists, skipping");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("FF Wal not skipped > 9.2, duplicates at beginning and end of list are removed");
errTotal = 0;
strLstAddZ(walFileList, "000000020000000100000000-daa497dba64008db824607940609ba1cd7c6c501.gz");
strLstAddZ(walFileList, "000000020000000100000000");
strLstAddZ(walFileList, "000000020000000100000000-aaaaaadba64008db824607940609ba1cd7c6c501");
strLstAddZ(walFileList, "0000000200000001000000FD-daa497dba64008db824607940609ba1cd7c6c501.gz");
strLstAddZ(walFileList, "0000000200000001000000FE-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz");
strLstAddZ(walFileList, "0000000200000001000000FF-daa497dba64008db824607940609ba1cd7c6c501");
strLstAddZ(walFileList, "000000020000000200000000");
strLstAddZ(walFileList, "000000020000000200000001");
strLstAddZ(walFileList, "000000020000000200000001");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 2, "triplicate WAL error at beginning, duplicate WAL at end");
TEST_RESULT_UINT(strLstSize(walFileList), 4, "only duplicate WAL removed from WAL list");
TEST_RESULT_UINT(lstSize(archiveIdResultList), 1, "single archiveId result");
TEST_RESULT_UINT(lstSize(archiveIdResult->walRangeList), 1, "single range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 0),
"get range");
TEST_RESULT_STR_Z(walRangeResult->start, "0000000200000001000000FD", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000000", "stop range");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 ERROR: [028]: duplicate WAL '000000020000000100000000' for '9.4-1' exists, skipping\n"
"P00 ERROR: [028]: duplicate WAL '000000020000000200000001' for '9.4-1' exists, skipping");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("FF Wal skipped <= 9.2, duplicates in middle of list removed");
// Clear the range lists and rerun the test with PG_VERSION_92 to ensure FF is reported as an error
lstClear(archiveIdResult->walRangeList);
errTotal = 0;
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
archiveIdResult->archiveId = strNewZ("9.2-1");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
archiveIdResult->pgWalInfo.version = PG_VERSION_92;
strLstAddZ(walFileList, "000000020000000200000001");
strLstAddZ(walFileList, "000000020000000200000001");
strLstAddZ(walFileList, "000000020000000200000002");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 2, "error reported");
TEST_RESULT_UINT(lstSize(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList), 2, "multiple ranges");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 0),
"get range");
TEST_RESULT_STR_Z(walRangeResult->start, "0000000200000001000000FD", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000000", "stop range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 1),
"get second range");
TEST_RESULT_STR_Z(walRangeResult->start, "000000020000000200000002", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000002", "stop range");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 ERROR: [028]: invalid WAL '0000000200000001000000FF' for '9.2-1' exists, skipping\n"
"P00 ERROR: [028]: duplicate WAL '000000020000000200000001' for '9.2-1' exists, skipping");
Add TEST_RESULT_STRLST_*() unit test macros. This avoids the need for strLstJoin() when testing lists. Lists are \n delimited (rather than command or pipe) so that non-trivial lists can be more easily diff'd.
2021-01-08 12:49:33 -05:00
TEST_RESULT_STRLST_Z(
walFileList,
"0000000200000001000000FD-daa497dba64008db824607940609ba1cd7c6c501.gz\n"
"0000000200000001000000FE-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz\n"
"000000020000000200000000\n000000020000000200000002\n",
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"skipped files removed");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("Rerun <= 9.2, missing FF not a gap");
// Clear the range lists, rerun the PG_VERSION_92 test to ensure the missing FF is not considered a gap
lstClear(archiveIdResult->walRangeList);
errTotal = 0;
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 0, "error reported");
TEST_RESULT_UINT(lstSize(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList), 2, "multiple ranges");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 0),
"get range");
TEST_RESULT_STR_Z(walRangeResult->start, "0000000200000001000000FD", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000000", "stop range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 1),
"get second range");
TEST_RESULT_STR_Z(walRangeResult->start, "000000020000000200000002", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000002", "stop range");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("version > 9.2, missing FF is a gap");
// Clear the range lists and update the version > 9.2 so missing FF is considered a gap in the WAL ranges
lstClear(archiveIdResult->walRangeList);
errTotal = 0;
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
archiveIdResult->archiveId = strNewZ("9.6-1");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
archiveIdResult->pgWalInfo.version = PG_VERSION_96;
strLstAddZ(walFileList, "000000020000000200000003-123456");
strLstAddZ(walFileList, "000000020000000200000004-123456");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 0, "no errors");
TEST_RESULT_UINT(lstSize(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList), 3, "multiple ranges");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 0),
"get first range");
TEST_RESULT_STR_Z(walRangeResult->start, "0000000200000001000000FD", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "0000000200000001000000FE", "stop range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 1),
"get second range");
TEST_RESULT_STR_Z(walRangeResult->start, "000000020000000200000000", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000000", "stop range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 2),
"get third range");
TEST_RESULT_STR_Z(walRangeResult->start, "000000020000000200000002", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000004", "stop range");
}
// *****************************************************************************************************************************
if (testBegin("verifyPgHistory()"))
{
// Create backup.info
InfoBackup *backupInfo = NULL;
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_ASSIGN(
backupInfo, infoBackupNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_BACKUP_INFO_MULTI_HISTORY_BASE))),
"backup.info multi-history");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - missing history");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Create archive.info - history mismatch
InfoArchive *archiveInfo = NULL;
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
"db-system-id=6626363367545678089\n"
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
"2={\"db-id\":6626363367545678089,\"db-version\":\"11\"}"))), "archive.info missing history");
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - system id");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
"db-system-id=6626363367545678089\n"
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
"1={\"db-id\":6625592122879095777,\"db-version\":\"9.4\"}\n"
"2={\"db-id\":6626363367545678089,\"db-version\":\"11\"}"))), "archive.info history system id mismatch");
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - version");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
"db-system-id=6626363367545678089\n"
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
"1={\"db-id\":6625592122879095702,\"db-version\":\"9.5\"}\n"
"2={\"db-id\":6626363367545678089,\"db-version\":\"11\"}"))), "archive.info history version mismatch");
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - id");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
"db-system-id=6626363367545678089\n"
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
"3={\"db-id\":6625592122879095702,\"db-version\":\"9.4\"}\n"
"2={\"db-id\":6626363367545678089,\"db-version\":\"11\"}"))), "archive.info history id mismatch");
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
}
// *****************************************************************************************************************************
if (testBegin("verifySetBackupCheckArchive(), verifyLogInvalidResult(), verifyRender()"))
{
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifySetBackupCheckArchive()");
InfoBackup *backupInfo = NULL;
InfoArchive *archiveInfo = NULL;
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_ASSIGN(
backupInfo, infoBackupNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_BACKUP_INFO_MULTI_HISTORY_BASE))),
"backup.info multi-history");
TEST_ASSIGN(archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE))),
"archive.info multi-history");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
InfoPg *pgHistory = infoArchivePg(archiveInfo);
StringList *backupList= strLstNew();
strLstAddZ(backupList, "20181119-152138F");
strLstAddZ(backupList, "20181119-152900F");
StringList *archiveIdList = strLstComparatorSet(strLstNew(), archiveIdComparator);
strLstAddZ(archiveIdList, "9.4-1");
strLstAddZ(archiveIdList, "11-2");
unsigned int errTotal = 0;
// Add backup to end of list
strLstAddZ(backupList, "20181119-153000F");
strLstAddZ(archiveIdList, "12-3");
TEST_RESULT_STR_Z(
verifySetBackupCheckArchive(backupList, backupInfo, archiveIdList, pgHistory, &errTotal),
"20181119-153000F", "current backup, missing archive");
TEST_RESULT_UINT(errTotal, 1, "error logged");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 ERROR: [044]: archiveIds '12-3' are not in the archive.info history list");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
errTotal = 0;
strLstAddZ(archiveIdList, "13-4");
TEST_RESULT_STR_Z(
verifySetBackupCheckArchive(backupList, backupInfo, archiveIdList, pgHistory, &errTotal),
"20181119-153000F", "test multiple archiveIds on disk not in archive.info");
TEST_RESULT_UINT(errTotal, 1, "error logged");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 ERROR: [044]: archiveIds '12-3, 13-4' are not in the archive.info history list");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifyLogInvalidResult() - missing file");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_RESULT_UINT(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
verifyLogInvalidResult(STORAGE_REPO_ARCHIVE_STR, verifyFileMissing, 0, STRDEF("missingfilename")),
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
0, "file missing message");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 WARN: file missing 'missingfilename'");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifyRender() - missing file, empty invalidList");
List *archiveIdResultList = lstNewP(sizeof(VerifyArchiveResult), .comparator = archiveIdComparator);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
List *backupResultList = lstNewP(sizeof(VerifyBackupResult), .comparator = lstComparatorStr);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
VerifyArchiveResult archiveIdResult =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.archiveId = strNewZ("9.6-1"),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
.totalWalFile = 1,
.walRangeList = lstNewP(sizeof(VerifyWalRange), .comparator = lstComparatorStr),
};
VerifyWalRange walRange =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.start = strNewZ("0"),
.stop = strNewZ("2"),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
.invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr),
};
lstAdd(archiveIdResult.walRangeList, &walRange);
lstAdd(archiveIdResultList, &archiveIdResult);
TEST_RESULT_STR_Z(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
verifyRender(archiveIdResultList, backupResultList),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"Results:\n"
" archiveId: 9.6-1, total WAL checked: 1, total valid WAL: 0\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
" missing: 0, checksum invalid: 0, size invalid: 0, other: 0\n"
" backup: none found", "archive: no invalid file list");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
VerifyInvalidFile invalidFile =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.fileName = strNewZ("file"),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
.reason = verifyFileMissing,
};
lstAdd(walRange.invalidFileList, &invalidFile);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
VerifyBackupResult backupResult =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.backupLabel = strNewZ("test-backup-label"),
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
.status = backupInvalid,
.totalFileVerify = 1,
.invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr),
};
lstAdd(backupResult.invalidFileList, &invalidFile);
lstAdd(backupResultList, &backupResult);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_STR_Z(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
verifyRender(archiveIdResultList, backupResultList),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"Results:\n"
" archiveId: 9.6-1, total WAL checked: 1, total valid WAL: 0\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
" missing: 1, checksum invalid: 0, size invalid: 0, other: 0\n"
" backup: test-backup-label, status: invalid, total files checked: 1, total valid files: 0\n"
" missing: 1, checksum invalid: 0, size invalid: 0, other: 0", "archive file missing, backup file missing");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifyAddInvalidWalFile() - file missing (coverage test)");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_VOID(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
verifyAddInvalidWalFile(archiveIdResult.walRangeList, verifyFileMissing, STRDEF("test"), STRDEF("3")), "coverage test");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
// *****************************************************************************************************************************
if (testBegin("cmdVerify() - info files"))
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info invalid checksum, neither backup copy nor archive infos exist");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_INVALID_BACKREST_INFO, .comment = "invalid backup.info");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(cmdVerify(), RuntimeError, "2 fatal errors encountered, see log for details");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 WARN: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
"<REPO:BACKUP>/backup.info\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info.copy' for read\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 ERROR: [029]: No usable backup.info file\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info' for read\n"
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info.copy' for read\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"P00 ERROR: [029]: No usable archive.info file");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info invalid checksum, backup.info.copy valid, archive.info not exist, archive copy checksum invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_INVALID_BACKREST_INFO,
.comment = "invalid archive.info.copy");
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT,
"[backup:current]\n"
TEST_BACKUP_DB1_CURRENT_FULL1
"\n"
"[db]\n"
TEST_BACKUP_DB1_94
"\n"
"[db:history]\n"
TEST_BACKUP_DB1_HISTORY,
.comment = "valid backup.info.copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(cmdVerify(), RuntimeError, "1 fatal errors encountered, see log for details");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 WARN: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS'"
" <REPO:BACKUP>/backup.info\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info' for read\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 WARN: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS'"
" <REPO:ARCHIVE>/archive.info.copy\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"P00 ERROR: [029]: No usable archive.info file");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info and copy valid but checksum mismatch, archive.info checksum invalid, archive.info copy valid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_BACKUP_INFO_MULTI_HISTORY_BASE, .comment = "valid backup.info");
HRN_STORAGE_PUT_Z(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_INVALID_BACKREST_INFO, .comment = "invalid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_BASE, .comment = "valid archive.info.copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(cmdVerify(), RuntimeError, "1 fatal errors encountered, see log for details");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 WARN: backup.info.copy does not match backup.info\n"
"P00 WARN: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS'"
" <REPO:ARCHIVE>/archive.info\n"
"P00 ERROR: [029]: backup info file and archive info file do not match\n"
" archive: id = 1, version = 9.4, system-id = 6625592122879095702\n"
" backup : id = 2, version = 11, system-id = 6626363367545678089\n"
" HINT: this may be a symptom of repository corruption!");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info and copy valid and checksums match, archive.info and copy valid, but checksum mismatch");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_BACKUP_INFO_MULTI_HISTORY_BASE,
.comment = "valid backup.info.copy");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_VOID(cmdVerify(), "usable backup and archive info files");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 WARN: archive.info.copy does not match archive.info\n"
"P00 WARN: no archives or backups exist in the repo");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info valid, copy invalid, archive.info valid, copy invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, .comment = "remove backup.info.copy");
HRN_STORAGE_REMOVE(storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, .comment = "remove archive.info.copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_VOID(cmdVerify(), "usable backup and archive info files");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info.copy' for read\n"
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info.copy' for read\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"P00 WARN: no archives or backups exist in the repo");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info and copy missing, archive.info and copy valid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(storageRepoWrite(), INFO_BACKUP_PATH_FILE);
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid and matching archive.info.copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(cmdVerify(), RuntimeError, "1 fatal errors encountered, see log for details");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info' for read\n"
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info.copy' for read\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"P00 ERROR: [029]: No usable backup.info file");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
// *****************************************************************************************************************************
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
if (testBegin("verifyFile()"))
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_TITLE("zero-sized file in archive");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
String *filePathName = strNewZ(STORAGE_REPO_ARCHIVE "/testfile");
HRN_STORAGE_PUT_EMPTY(storageRepoWrite(), strZ(filePathName));
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_UINT(verifyFile(filePathName, STRDEF(HASH_TYPE_SHA1_ZERO), 0, NULL), verifyOk, "file ok");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("file size invalid in archive");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(storageRepoWrite(), strZ(filePathName), fileContents);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_RESULT_UINT(verifyFile(filePathName, fileChecksum, 0, NULL), verifySizeInvalid, "file size invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("file missing in archive");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_UINT(
verifyFile(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
strNewFmt(STORAGE_REPO_ARCHIVE "/missingFile"), fileChecksum, 0, NULL), verifyFileMissing, "file missing");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("encrypted/compressed file in backup");
// Create a compressed encrypted repo file in backup
filePathName = strNewZ(STORAGE_REPO_BACKUP "/testfile");
HRN_STORAGE_PUT_Z(
storageRepoWrite(), strZ(filePathName), fileContents, .compressType = compressTypeGz, .cipherType = cipherTypeAes256Cbc,
.cipherPass = "pass");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
strCatZ(filePathName, ".gz");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_UINT(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
verifyFile(filePathName, fileChecksum, fileSize, STRDEF("pass")), verifyOk, "file encrypted compressed ok");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_UINT(
verifyFile(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
filePathName, STRDEF("badchecksum"), fileSize, STRDEF("pass")), verifyChecksumMismatch,
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"file encrypted compressed checksum mismatch");
}
// *****************************************************************************************************************************
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
if (testBegin("cmdVerify(), verifyProcess() - errors"))
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
{
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
//--------------------------------------------------------------------------------------------------------------------------
Make --repo optional for remaining commands except stanza-delete. Some commands (repo-*, verify) still required the --repo option but it makes sense to give them the same treatment as backup and simply use the first repo when one is not specified. This leaves stanza-delete as the only remaining command that requires --repo. This is by design to enhance safe usage.
2021-03-03 09:21:06 -05:00
// Load Parameters with multi-repo
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
StringList *argList = strLstDup(argListBase);
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 4, TEST_PATH "/repo4");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Store valid archive/backup info files
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("valid info files, WAL files present, no backups");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_NO_CURRENT_BACKUP \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_NO_CURRENT_BACKUP, .comment = "no current backups");
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_NO_CURRENT_BACKUP, .comment = "no current backups copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Create WAL file with just header info and small WAL size
Buffer *walBuffer = bufNew((size_t)(1024 * 1024));
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
(PgWal){.version = PG_VERSION_11, .systemId = 6626363367545678089, .size = 1024 * 1024}, walBuffer);
const char *walBufferSha1 = strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, walBuffer)));
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
strZ(strNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-%s", walBufferSha1)), walBuffer,
.comment = "valid WAL");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-bad817043007aa2100c44c712bcb456db705dab9",
walBuffer, .comment = "duplicate WAL");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log detail level to capture ranges (there should be none)
harnessLogLevelSet(logLevelDetail);
TEST_ERROR(cmdVerify(), RuntimeError, "1 fatal errors encountered, see log for details");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
"P00 WARN: no backups exist in the repo\n"
"P00 ERROR: [028]: duplicate WAL '000000020000000700000FFE' for '11-2' exists, skipping\n"
"P00 WARN: path '11-2/0000000200000007' does not contain any valid WAL to be processed\n"
"P00 INFO: Results:\n"
" archiveId: 11-2, total WAL checked: 2, total valid WAL: 0\n"
" missing: 0, checksum invalid: 0, size invalid: 0, other: 0\n"
" backup: none found");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
harnessLogLevelReset();
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-bad817043007aa2100c44c712bcb456db705dab9",
.comment = "remove duplicate WAL");
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/9.4-1", .comment = "empty path for old archiveId");
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/11-2/0000000100000000", .comment = "empty timeline path");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4",
walBuffer, .compressType = compressTypeGz, .comment = "first WAL compressed - but checksum failure");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
strZ(
strNewFmt(
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFF-%s",
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, BUFSTRDEF("invalidsize")))))),
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
BUFSTRDEF("invalidsize"), .comment = "WAL - invalid size");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
strZ(strNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000008/000000020000000800000000-%s", walBufferSha1)),
walBuffer, .comment = "WAL - continue range");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log detail level to capture ranges
harnessLogLevelSet(logLevelDetail);
// Test verifyProcess directly
unsigned int errorTotal = 0;
TEST_RESULT_STR_Z(
verifyProcess(&errorTotal),
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
"Results:\n"
" archiveId: 9.4-1, total WAL checked: 0, total valid WAL: 0\n"
" archiveId: 11-2, total WAL checked: 4, total valid WAL: 2\n"
" missing: 0, checksum invalid: 1, size invalid: 1, other: 0\n"
" backup: none found",
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"verifyProcess() results");
TEST_RESULT_UINT(errorTotal, 2, "errors");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
"P00 WARN: no backups exist in the repo\n"
"P00 WARN: archive path '9.4-1' is empty\n"
"P00 WARN: path '11-2/0000000100000000' does not contain any valid WAL to be processed\n"
"P01 ERROR: [028]: invalid checksum "
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
"P01 ERROR: [028]: invalid size "
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFD, wal stop: 000000020000000800000000");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("valid info files, start next timeline");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// Load Parameters - single default repo
argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
strZ(strNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000008/000000020000000800000002-%s", walBufferSha1)),
walBuffer, .comment = "WAL - starts next range");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
strZ(strNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000300000000/000000030000000000000000-%s", walBufferSha1)),
walBuffer, .comment = "WAL - starts next timeline");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
strZ(strNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000300000000/000000030000000000000001-%s", walBufferSha1)),
walBuffer, .comment = "WAL - end next timeline");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log level to errors only
harnessLogLevelSet(logLevelError);
TEST_ERROR(cmdVerify(), RuntimeError, "2 fatal errors encountered, see log for details");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
"P01 ERROR: [028]: invalid checksum "
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
"P01 ERROR: [028]: invalid size "
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_TITLE("valid info files - various archive/backup errors");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// Load Parameters - single non-default repo
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 2, TEST_PATH "/repo");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
hrnCfgArgRawZ(argList, cfgOptStanza, "db");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgArgRawZ(argList, cfgOptRepo, "2");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9",
walBuffer, .modeFile = 0200, .comment = "WAL - file not readable");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152800F", .comment = "prior backup path missing manifests");
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152810F/" BACKUP_MANIFEST_FILE,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_DB,
.comment = "manifest without target files");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create full backup with files
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/PG_VERSION", "BOGUS",
.comment = "put checksum-error backup file");
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/testzero",
.comment = "put zero-size backup file");
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/testvalid", fileContents,
.comment = "put valid file");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write manifests for full backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
String *manifestContent = strNewFmt(
"[backup]\n"
"backup-label=\"20181119-152900F\"\n"
"backup-timestamp-copy-start=0\n"
"backup-timestamp-start=0\n"
"backup-timestamp-stop=0\n"
"backup-type=\"full\"\n"
"\n"
"[backup:db]\n"
TEST_BACKUP_DB2_11
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
"pg_data/testvalid={\"checksum\":\"%s\",\"master\":true,\"size\":7,\"timestamp\":1565282114}\n"
"pg_data/testzero={\"repo-size\":20,\"size\":0,\"timestamp\":1601405663}\n"
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
strZ(fileChecksum));
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE, strZ(manifestContent),
.comment = "valid manifest");
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
strZ(manifestContent), .comment = "valid manifest copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
// Create a manifest for the dependent that has references
manifestContent = strNewFmt(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"[backup]\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"backup-label=\"20181119-152900F_20181119-152909D\"\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"backup-timestamp-copy-start=0\n"
"backup-timestamp-start=0\n"
"backup-timestamp-stop=0\n"
"backup-type=\"diff\"\n"
"\n"
"[backup:db]\n"
TEST_BACKUP_DB2_11
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
"\n"
"[target:file]\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"master\":true,"
"\"reference\":\"20181119-152900F\",\"size\":4,\"timestamp\":1565282114}\n"
"pg_data/testfile={\"checksum\":\"%s\",\"master\":true,\"reference\":\"20181119-152900F\",\"size\":7,"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"\"timestamp\":1565282114}\n"
"pg_data/testfile2={\"checksum\":\"%s\",\"master\":true,\"size\":7,\"timestamp\":1565282114}\n"
"pg_data/testmissing="
"{\"checksum\":\"123473f470864e067ee3a22e64b47b0a1c356abc\",\"size\":7,\"timestamp\":1565282114}\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"pg_data/testother={\"checksum\":\"%s\",\"master\":true,\"reference\":\"UNPROCESSEDBACKUP\",\"size\":7,"
"\"timestamp\":1565282114}\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
strZ(fileChecksum), strZ(fileChecksum), strZ(fileChecksum));
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write manifests for dependent backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE,
strZ(manifestContent), .comment = "manifest to dependent");
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
strZ(manifestContent), .comment = "manifest copy to dependent");
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0),
STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/pg_data/testfile2", fileContents,
.comment = "put valid file to dependent");
// Create an unprocessed backup label with a file that will be referenced in the dependent manifest
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0),
STORAGE_REPO_BACKUP "/UNPROCESSEDBACKUP/pg_data/testother", fileContents, .modeFile = 0200,
.comment = "put unreadable file to unprocessed backup");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create in-progress backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-153000F",
.comment = "create empty backup path for newest backup so in-progress");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log level to capture ranges
harnessLogLevelSet(logLevelDetail);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_ERROR(cmdVerify(), RuntimeError, "7 fatal errors encountered, see log for details");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 WARN: archive path '9.4-1' is empty\n"
"P00 WARN: path '11-2/0000000100000000' does not contain any valid WAL to be processed\n"
"P01 ERROR: [028]: invalid checksum "
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
"P01 ERROR: [028]: invalid size "
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P01 ERROR: [039]: invalid result "
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9: [41] raised from "
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"local-1 shim protocol: unable to open file '" TEST_PATH "/repo/archive/db/"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9' for read:"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
" [13] Permission denied\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152800F/backup.manifest' for read\n"
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152800F/backup.manifest.copy'"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" for read\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P00 WARN: manifest missing for '20181119-152800F' - backup may have expired\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152810F/backup.manifest.copy'"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" for read\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P00 ERROR: [028]: backup '20181119-152810F' manifest does not contain any target files to verify\n"
"P01 ERROR: [028]: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 ERROR: [028]: file missing '20181119-152900F_20181119-152909D/pg_data/testmissing'\n"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"P00 WARN: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-153000F/backup.manifest' for read\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P00 INFO: backup '20181119-153000F' appears to be in progress, skipping\n"
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
"P01 ERROR: [039]: invalid result UNPROCESSEDBACKUP/pg_data/testother: [41] raised from local-1 shim protocol:"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" unable to open file '" TEST_PATH "/repo/backup/db/UNPROCESSEDBACKUP/pg_data/testother' for read: [13]"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" Permission denied\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFD, wal stop: 000000020000000800000000\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000800000002, wal stop: 000000020000000800000003\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000030000000000000000, wal stop: 000000030000000000000001\n"
"P00 INFO: Results:\n"
" archiveId: 9.4-1, total WAL checked: 0, total valid WAL: 0\n"
" archiveId: 11-2, total WAL checked: 8, total valid WAL: 5\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
" missing: 0, checksum invalid: 1, size invalid: 1, other: 1\n"
" backup: 20181119-152800F, status: manifest missing, total files checked: 0, total valid files: 0\n"
" backup: 20181119-152810F, status: invalid, total files checked: 0, total valid files: 0\n"
" backup: 20181119-152900F, status: invalid, total files checked: 3, total valid files: 2\n"
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0\n"
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 5, "
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
"total valid files: 2\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
" missing: 1, checksum invalid: 1, size invalid: 0, other: 1\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20181119-153000F, status: in-progress, total files checked: 0, total valid files: 0");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
harnessLogLevelReset();
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
}
// *****************************************************************************************************************************
if (testBegin("cmdVerify()"))
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
#define TEST_BACKUP_DB1_CURRENT_FULL3_DIFF1 \
"20181119-152900F_20181119-152909D={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000010000000000000006\",\"backup-archive-stop\":\"000000010000000000000007\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB2_CURRENT_FULL1 \
"20201119-163000F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000020000000000000001\",\"backup-archive-stop\":\"000000020000000000000001\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":2,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("prior backup verification incomplete - referenced file checked");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
#define TEST_BACKUP_INFO \
"[backup:current]\n" \
TEST_BACKUP_DB1_CURRENT_FULL3 \
TEST_BACKUP_DB1_CURRENT_FULL3_DIFF1 \
TEST_BACKUP_DB2_CURRENT_FULL1 \
"\n" \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_BACKUP_INFO);
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_BACKUP_INFO);
// Create valid full backup for DB1
#define TEST_MANIFEST_FULL_DB1 \
TEST_MANIFEST_HEADER \
TEST_MANIFEST_DB_94 \
TEST_MANIFEST_OPTION_ALL \
TEST_MANIFEST_TARGET \
TEST_MANIFEST_DB \
TEST_MANIFEST_FILE \
TEST_MANIFEST_FILE_DEFAULT \
TEST_MANIFEST_LINK \
TEST_MANIFEST_LINK_DEFAULT \
TEST_MANIFEST_PATH \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_PATH_DEFAULT
// Write manifests for full backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE, TEST_MANIFEST_FULL_DB1,
.comment = "valid manifest - full");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, TEST_MANIFEST_FULL_DB1,
.comment = "valid manifest copy - full");
// Create valid diff backup for DB1
#define TEST_MANIFEST_DIFF_DB1 \
TEST_MANIFEST_HEADER \
TEST_MANIFEST_DB_94 \
TEST_MANIFEST_OPTION_ALL \
TEST_MANIFEST_TARGET \
TEST_MANIFEST_DB \
"\n" \
"[target:file]\n" \
"pg_data/PG_VERSION=" \
"{\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"master\":true,\"reference\":\"20181119-152900F\"," \
"\"size\":4,\"timestamp\":1565282114}\n" \
TEST_MANIFEST_FILE_DEFAULT \
TEST_MANIFEST_LINK \
TEST_MANIFEST_LINK_DEFAULT \
TEST_MANIFEST_PATH \
TEST_MANIFEST_PATH_DEFAULT
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write manifests for diff backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE,
TEST_MANIFEST_DIFF_DB1, .comment = "valid manifest - diff");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, TEST_MANIFEST_DIFF_DB1, .comment = "valid manifest copy - diff");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Put the file referenced by both backups into the full backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/PG_VERSION", fileContents);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_ERROR(cmdVerify(), RuntimeError, "2 fatal errors encountered, see log for details");
// The error for the referenced file is logged twice because it is checked again by the second backup since the first backup
// verification had not yet completed before the second backup verification began
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
"P00 WARN: no archives exist in the repo\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"P01 ERROR: [028]: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 ERROR: [028]: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
"P00 INFO: Results:\n"
" archiveId: none found\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20181119-152900F, status: invalid, total files checked: 1, total valid files: 0\n"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 1, total valid files: 0"
"\n"
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("valid backup, prior backup verification complete - referenced file not checked");
// Set process max to 1 and add more files to check so first backup completes before second is checked
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
hrnCfgArgRawZ(argList, cfgOptProcessMax, "1");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
String *manifestContent = strNewFmt(
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
"pg_data/base/1/555_init={\"checksum\":\"%s\",\"master\":false,\"size\":1,\"timestamp\":1565282114}\n"
"pg_data/base/1/555_init.1={\"master\":false,\"size\":0,\"timestamp\":1565282114}\n"
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
strZ(fileChecksum));
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE, strZ(manifestContent),
.comment = "valid manifest - full");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestContent),
.comment = "valid manifest copy - full");
HRN_STORAGE_PUT_Z(
storageRepoWrite(),
STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/base/1/555_init", fileContents, .comment = "invalid size");
// Diff manifest
manifestContent = strNewZ(
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
"\n"
"[target:file]\n"
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"master\":true,"
"\"reference\":\"20181119-152900F\",\"size\":4,\"timestamp\":1565282114}\n"
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT);
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE,
strZ(manifestContent), .comment = "valid manifest - diff");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
strZ(manifestContent), .comment = "valid manifest copy - diff");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create valid full backup and valid diff backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
manifestContent = strNewFmt(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
"\n"
"[backup:db]\n"
TEST_BACKUP_DB2_11
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
"\n"
"[target:file]\n"
"pg_data/validfile={\"checksum\":\"%s\",\"master\":true,\"size\":%u,\"timestamp\":1565282114}\n"
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
strZ(fileChecksum), (unsigned int)fileSize);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20201119-163000F/" BACKUP_MANIFEST_FILE, strZ(manifestContent),
.comment = "valid manifest - full");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20201119-163000F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestContent),
.comment = "valid manifest copy - full");
HRN_STORAGE_PUT_Z(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20201119-163000F/pg_data/validfile", fileContents, .comment = "valid file");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create WAL file with just header info and small WAL size
Buffer *walBuffer = bufNew((size_t)(1024 * 1024));
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
hrnPgWalToBuffer(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
(PgWal){.version = PG_VERSION_11, .systemId = 6626363367545678089, .size = 1024 * 1024}, walBuffer);
const char *walBufferSha1 = strZ(bufHex(cryptoHashOne(HASH_TYPE_SHA1_STR, walBuffer)));
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoWrite(),
strZ(strNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000000/000000020000000000000001-%s", walBufferSha1)), walBuffer,
.comment = "valid WAL");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_ERROR(cmdVerify(), RuntimeError, "3 fatal errors encountered, see log for details");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
"P01 ERROR: [028]: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 ERROR: [028]: invalid size '20181119-152900F/pg_data/base/1/555_init'\n"
"P01 ERROR: [028]: file missing '20181119-152900F/pg_data/base/1/555_init.1'\n"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
"P00 INFO: Results:\n"
" archiveId: 11-2, total WAL checked: 1, total valid WAL: 1\n"
" missing: 0, checksum invalid: 0, size invalid: 0, other: 0\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20181119-152900F, status: invalid, total files checked: 3, total valid files: 0\n"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
" missing: 1, checksum invalid: 1, size invalid: 1, other: 0\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 1, total valid files: 0\n"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20201119-163000F, status: valid, total files checked: 1, total valid files: 1\n"
" missing: 0, checksum invalid: 0, size invalid: 0, other: 0");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*(). When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed. Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
FUNCTION_HARNESS_RETURN_VOID();
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
Reference in New Issue Copy Permalink