db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-07-05 00:28:52 +02:00
Code Issues Releases Activity
Files
a96837d2eda4b7497e74954b24d9323f33465c50
pgbackrest/test/src/module/command/verifyTest.c

1917 lines
118 KiB
C
Raw Normal View History

Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
/***********************************************************************************************************************************
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
Test Verify Command
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
***********************************************************************************************************************************/
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
#include "common/io/bufferRead.h"
#include "postgres/interface.h"
#include "postgres/version.h"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
#include "storage/posix/storage.h"
#include "common/harnessConfig.h"
#include "common/harnessInfo.h"
Move PostgreSQL version interface test functions to a test harness. Some version interface test functions were integrated into the core code because they relied on the PostgreSQL versioned interface. Even though they were compiled out for production builds they cluttered the core code and made it harder to determine what was required by core. Create a PostgreSQL version interface in a test harness to contain these functions. This does require some duplication but the cleaner core code seems a good tradeoff. It is possible for some of this code to be auto-generated but since it is only updated once per year the matter is not pressing.
2021-05-17 07:20:28 -04:00
#include "common/harnessPostgres.h"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
#include "common/harnessPq.h"
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
#include "common/harnessProtocol.h"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
/***********************************************************************************************************************************
Test Run
***********************************************************************************************************************************/
Add missing static keywords in test modules.
2021-10-18 12:22:48 -04:00
static void
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
testRun(void)
{
FUNCTION_HARNESS_VOID();
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Create storage
Storage *storageTest = storagePosixNewP(TEST_PATH_STR, .write = true);
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
// Install local command handler shim
static const ProtocolServerHandler testLocalHandlerList[] = {PROTOCOL_SERVER_HANDLER_VERIFY_LIST};
Add LENGTH_OF() macro. Determining the length of arrays that could be calculated at compile time was a bit piecemeal, with special macros used sometimes and with the math done directly other times. This macro makes the task easier, uses less space, and automatically adjusts when the type changes.
2022-04-07 19:00:15 -04:00
hrnProtocolLocalShimInstall(testLocalHandlerList, LENGTH_OF(testLocalHandlerList));
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
StringList *argListBase = strLstNew();
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
hrnCfgArgRawZ(argListBase, cfgOptStanza, "db");
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgRawZ(argListBase, cfgOptRepoPath, TEST_PATH "/repo");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
const char *fileContents = "acefile";
uint64_t fileSize = 7;
Store manifest checksums in memory more efficiently. Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions. There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
const Buffer *fileChecksum = bufNewDecode(encodingHex, STRDEF("d1cd8a7d11daa26814b93eb604e1d49ab4b43770"));
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
#define TEST_BACKUP_DB1_94 \
"db-catalog-version=201409291\n" \
"db-control-version=942\n" \
"db-id=1\n" \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_94_Z "\n" \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"db-version=\"9.4\"\n"
#define TEST_BACKUP_DB2_11 \
"db-catalog-version=201707211\n" \
"db-control-version=1100\n" \
"db-id=2\n" \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_11_Z "\n" \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"db-version=\"11\"\n"
#define TEST_BACKUP_DB1_CURRENT_FULL1 \
"20181119-152138F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.28dev\"," \
"\"backup-archive-start\":\"000000010000000000000002\",\"backup-archive-stop\":\"000000010000000000000002\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1482182846,\"backup-timestamp-stop\":1482182861,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB1_CURRENT_FULL2 \
"20181119-152800F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB1_CURRENT_FULL3 \
"20181119-152900F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000010000000000000004\",\"backup-archive-stop\":\"000000010000000000000004\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB1_HISTORY \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-catalog-version\":201409291,\"db-control-version\":942,\"db-system-id\":" HRN_PG_SYSTEMID_94_Z "," \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"\"db-version\":\"9.4\"}"
#define TEST_BACKUP_DB2_HISTORY \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"2={\"db-catalog-version\":201707211,\"db-control-version\":1100,\"db-system-id\":" HRN_PG_SYSTEMID_11_Z "," \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"\"db-version\":\"11\"}"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_BACKUP_INFO_MULTI_HISTORY_BASE \
"[backup:current]\n" \
TEST_BACKUP_DB1_CURRENT_FULL1 \
TEST_BACKUP_DB1_CURRENT_FULL2 \
TEST_BACKUP_DB1_CURRENT_FULL3 \
"\n" \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_BACKUP_DB2_HISTORY
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_ARCHIVE_INFO_BASE \
"[db]\n" \
"db-id=1\n" \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_94_Z "\n" \
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"db-version=\"9.4\"\n" \
"\n" \
"[db:history]\n" \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_94_Z ",\"db-version\":\"9.4\"}"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE \
"[db]\n" \
"db-id=2\n" \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_11_Z "\n" \
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"db-version=\"11\"\n" \
"\n" \
"[db:history]\n" \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_94_Z ",\"db-version\":\"9.4\"}\n" \
"2={\"db-id\":" HRN_PG_SYSTEMID_11_Z ",\"db-version\":\"11\"}"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
#define TEST_MANIFEST_HEADER \
"[backup]\n" \
"backup-label=null\n" \
"backup-timestamp-copy-start=0\n" \
"backup-timestamp-start=0\n" \
"backup-timestamp-stop=0\n" \
"backup-type=\"full\"\n"
#define TEST_MANIFEST_DB_94 \
"\n" \
"[backup:db]\n" \
"db-catalog-version=201409291\n" \
"db-control-version=942\n" \
"db-id=1\n" \
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_94_Z "\n" \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"db-version=\"9.4\"\n"
#define TEST_MANIFEST_OPTION_ALL \
"\n" \
"[backup:option]\n" \
"option-archive-check=false\n" \
"option-archive-copy=false\n" \
"option-checksum-page=false\n" \
"option-compress=false\n" \
"option-compress-type=\"none\"\n" \
"option-hardlink=false\n" \
"option-online=false\n"
#define TEST_MANIFEST_OPTION_ARCHIVE_TRUE \
"\n" \
"[backup:option]\n" \
"option-archive-check=true\n" \
"option-archive-copy=true\n"
#define TEST_MANIFEST_TARGET \
"\n" \
"[backup:target]\n" \
"pg_data={\"path\":\"/pg/base\",\"type\":\"path\"}\n"
#define TEST_MANIFEST_DB \
"\n" \
"[db]\n" \
"postgres={\"db-id\":12173,\"db-last-system-id\":12168}\n"
#define TEST_MANIFEST_FILE \
"\n" \
"[target:file]\n" \
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"size\":4,\"timestamp\":1565282114}\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
#define TEST_MANIFEST_FILE_DEFAULT \
"\n" \
"[target:file:default]\n" \
"group=\"group1\"\n" \
"mode=\"0600\"\n" \
"user=\"user1\"\n"
#define TEST_MANIFEST_LINK \
"\n" \
"[target:link]\n" \
"pg_data/pg_stat={\"destination\":\"../pg_stat\"}\n"
#define TEST_MANIFEST_LINK_DEFAULT \
"\n" \
"[target:link:default]\n" \
"group=\"group1\"\n" \
"user=false\n"
#define TEST_MANIFEST_PATH \
"\n" \
"[target:path]\n" \
"pg_data={\"user\":\"user1\"}\n" \
#define TEST_MANIFEST_PATH_DEFAULT \
"\n" \
"[target:path:default]\n" \
"group=false\n" \
"mode=\"0700\"\n" \
"user=\"user1\"\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_INVALID_BACKREST_INFO \
"[backrest]\n" \
"backrest-checksum=\"BOGUS\"\n" \
"backrest-format=5\n" \
"backrest-version=\"2.28\"\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// *****************************************************************************************************************************
if (testBegin("verifyManifestFile()"))
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_BACKUP_LABEL_FULL "20181119-152138F"
Manifest *manifest = NULL;
unsigned int jobErrorTotal = 0;
VerifyBackupResult backupResult = {.backupLabel = strNewZ(TEST_BACKUP_LABEL_FULL)};
InfoPg *infoPg = NULL;
TEST_ASSIGN(
infoPg, infoArchivePg(infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_ARCHIVE_INFO_BASE)))),
"infoPg from archive.info");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("manifest.copy exists, no manifest main, manifest db version not in history, not current db");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
"\n"
"[backup:db]\n"
"db-catalog-version=201306121\n"
"db-control-version=937\n"
"db-id=1\n"
"db-system-id=" HRN_PG_SYSTEMID_94_Z "\n"
"db-version=\"9.3\"\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "manifest db section mismatch");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
harnessLogLevelSet(logLevelDetail);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set - pg version mismatch");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest.copy'"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" for read\n"
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
"P00 INFO: '20181119-152138F' may not be recoverable - PG data (id 1, version 9.3, system-id "
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
HRN_PG_SYSTEMID_94_Z ") is not in the backup.info history, skipping");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("rerun test with db-system-id invalid and no main");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE);
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
"\n"
"[backup:db]\n"
"db-catalog-version=201409291\n"
"db-control-version=942\n"
"db-id=1\n"
"db-system-id=0\n"
"db-version=\"9.4\"\n"
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "manifest copy - invalid system-id");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set - pg system-id mismatch");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest' for read\n"
"P00 DETAIL: 20181119-152138F/backup.manifest is missing or unusable, using copy\n"
"P00 INFO: '20181119-152138F' may not be recoverable - PG data (id 1, version 9.4, system-id 0) is not in the "
"backup.info history, skipping");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("rerun copy test with db-id invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
"\n"
"[backup:db]\n"
"db-catalog-version=201409291\n"
"db-control-version=942\n"
"db-id=0\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_94_Z "\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"db-version=\"9.4\"\n"
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "manifest copy - invalid db-id");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set - pg db-id mismatch");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest' for read\n"
"P00 DETAIL: 20181119-152138F/backup.manifest is missing or unusable, using copy\n"
"P00 INFO: '20181119-152138F' may not be recoverable - PG data (id 0, version 9.4, system-id "
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
HRN_PG_SYSTEMID_94_Z ") is not in the backup.info history, skipping");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("missing main manifest, errored copy");
backupResult.status = backupValid;
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
storageRepoWrite(), TEST_PATH "/repo/" STORAGE_PATH_BACKUP "/db/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
INFO_COPY_EXT, TEST_INVALID_BACKREST_INFO, .comment = "invalid manifest copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, false, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152138F/backup.manifest' for read\n"
"P00 DETAIL: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"<REPO:BACKUP>/20181119-152138F/backup.manifest.copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("current backup true");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
storageRepoWrite(), TEST_PATH "/repo/" STORAGE_PATH_BACKUP "/db/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE,
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_INVALID_BACKREST_INFO, .comment = "invalid manifest");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, true, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR(manifest, NULL, "manifest not set");
TEST_RESULT_UINT(backupResult.status, backupInvalid, "manifest unusable - backup invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"<REPO:BACKUP>/20181119-152138F/backup.manifest\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
"<REPO:BACKUP>/20181119-152138F/backup.manifest.copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write a valid manifest with a manifest copy that is invalid
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/" TEST_BACKUP_LABEL_FULL "/" BACKUP_MANIFEST_FILE,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_PATH_DEFAULT,
.comment = "valid manifest");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
backupResult.status = backupValid;
TEST_ASSIGN(manifest, verifyManifestFile(&backupResult, NULL, true, infoPg, &jobErrorTotal), "verify manifest");
TEST_RESULT_PTR_NE(manifest, NULL, "manifest set");
TEST_RESULT_UINT(backupResult.status, backupValid, "manifest usable");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
TEST_RESULT_LOG("P00 DETAIL: backup '20181119-152138F' manifest.copy does not match manifest");
harnessLogLevelReset();
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
}
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// *****************************************************************************************************************************
if (testBegin("verifyCreateArchiveIdRange()"))
{
VerifyWalRange *walRangeResult = NULL;
unsigned int errTotal = 0;
StringList *walFileList = strLstNew();
VerifyArchiveResult archiveResult =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.archiveId = strNewZ("9.4-1"),
Remove extra spaces.
2022-01-18 17:40:53 -05:00
.walRangeList = lstNewP(sizeof(VerifyWalRange), .comparator = lstComparatorStr),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
};
Remove extra spaces.
2022-01-18 17:40:53 -05:00
List *archiveIdResultList = lstNewP(sizeof(VerifyArchiveResult), .comparator = archiveIdComparator);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
lstAdd(archiveIdResultList, &archiveResult);
VerifyArchiveResult *archiveIdResult = lstGetLast(archiveIdResultList);
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("Single WAL");
archiveIdResult->pgWalInfo.size = PG_WAL_SEGMENT_SIZE_DEFAULT;
archiveIdResult->pgWalInfo.version = PG_VERSION_94;
strLstAddZ(walFileList, "000000020000000200000000-daa497dba64008db824607940609ba1cd7c6c501.gz");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 0, "no errors");
TEST_RESULT_UINT(lstSize(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList), 1, "single range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 0),
"get range");
TEST_RESULT_STR_Z(walRangeResult->start, "000000020000000200000000", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000000", "stop range");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("Duplicate WAL only - no range, all removed from list");
lstClear(archiveIdResult->walRangeList);
// Add a duplicate
strLstAddZ(walFileList, "000000020000000200000000");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 1, "duplicate WAL error");
TEST_RESULT_UINT(strLstSize(walFileList), 0, "all WAL removed from WAL file list");
TEST_RESULT_UINT(lstSize(archiveIdResult->walRangeList), 0, "no range");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
TEST_RESULT_LOG("P00 INFO: duplicate WAL '000000020000000200000000' for '9.4-1' exists, skipping");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
Remove support for PostgreSQL 9.0/9.1/9.2. Our new policy is to support ten versions of PostgreSQL, the five supported releases and the last five EOL releases. As of PostgreSQL 15, that means 9.0/9.1/9.2 are no longer supported by pgBackRest. Remove all logic associated with 9.0/9.1/9.2 and update the tests. Document the new support policy. Update InfoPg to read/write control versions for the history in backup.info, since we can no longer rely on the mappings being available. In theory this could have been an issue after removing 8.3/8.4 if anybody was using a version that old.
2022-12-20 12:20:47 +07:00
TEST_TITLE("duplicates at beginning and end of list are removed");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
errTotal = 0;
strLstAddZ(walFileList, "000000020000000100000000-daa497dba64008db824607940609ba1cd7c6c501.gz");
strLstAddZ(walFileList, "000000020000000100000000");
strLstAddZ(walFileList, "000000020000000100000000-aaaaaadba64008db824607940609ba1cd7c6c501");
strLstAddZ(walFileList, "0000000200000001000000FD-daa497dba64008db824607940609ba1cd7c6c501.gz");
strLstAddZ(walFileList, "0000000200000001000000FE-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz");
strLstAddZ(walFileList, "0000000200000001000000FF-daa497dba64008db824607940609ba1cd7c6c501");
strLstAddZ(walFileList, "000000020000000200000000");
strLstAddZ(walFileList, "000000020000000200000001");
strLstAddZ(walFileList, "000000020000000200000001");
TEST_RESULT_VOID(verifyCreateArchiveIdRange(archiveIdResult, walFileList, &errTotal), "create archiveId WAL range");
TEST_RESULT_UINT(errTotal, 2, "triplicate WAL error at beginning, duplicate WAL at end");
TEST_RESULT_UINT(strLstSize(walFileList), 4, "only duplicate WAL removed from WAL list");
TEST_RESULT_UINT(lstSize(archiveIdResultList), 1, "single archiveId result");
TEST_RESULT_UINT(lstSize(archiveIdResult->walRangeList), 1, "single range");
TEST_ASSIGN(
walRangeResult, (VerifyWalRange *)lstGet(((VerifyArchiveResult *)lstGet(archiveIdResultList, 0))->walRangeList, 0),
"get range");
TEST_RESULT_STR_Z(walRangeResult->start, "0000000200000001000000FD", "start range");
TEST_RESULT_STR_Z(walRangeResult->stop, "000000020000000200000000", "stop range");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 INFO: duplicate WAL '000000020000000100000000' for '9.4-1' exists, skipping\n"
"P00 INFO: duplicate WAL '000000020000000200000001' for '9.4-1' exists, skipping");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
// *****************************************************************************************************************************
if (testBegin("verifyPgHistory()"))
{
// Create backup.info
InfoBackup *backupInfo = NULL;
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_ASSIGN(
backupInfo, infoBackupNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_BACKUP_INFO_MULTI_HISTORY_BASE))),
"backup.info multi-history");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - missing history");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Create archive.info - history mismatch
InfoArchive *archiveInfo = NULL;
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_11_Z "\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"2={\"db-id\":" HRN_PG_SYSTEMID_11_Z ",\"db-version\":\"11\"}"))), "archive.info missing history");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - system id");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_11_Z "\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
"1={\"db-id\":6625592122879095777,\"db-version\":\"9.4\"}\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"2={\"db-id\":" HRN_PG_SYSTEMID_11_Z ",\"db-version\":\"11\"}"))), "archive.info history system id mismatch");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - version");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_11_Z "\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"1={\"db-id\":" HRN_PG_SYSTEMID_94_Z ",\"db-version\":\"9.5\"}\n"
"2={\"db-id\":" HRN_PG_SYSTEMID_11_Z ",\"db-version\":\"11\"}"))), "archive.info history version mismatch");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("history mismatch - id");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ASSIGN(
archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(
"[db]\n"
"db-id=2\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"db-system-id=" HRN_PG_SYSTEMID_11_Z "\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"db-version=\"11\"\n"
"\n"
"[db:history]\n"
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
"3={\"db-id\":" HRN_PG_SYSTEMID_94_Z ",\"db-version\":\"9.4\"}\n"
"2={\"db-id\":" HRN_PG_SYSTEMID_11_Z ",\"db-version\":\"11\"}"))), "archive.info history id mismatch");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_ERROR(
verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)), FormatError,
"archive and backup history lists do not match");
}
// *****************************************************************************************************************************
if (testBegin("verifySetBackupCheckArchive(), verifyLogInvalidResult(), verifyRender()"))
{
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifySetBackupCheckArchive()");
InfoBackup *backupInfo = NULL;
InfoArchive *archiveInfo = NULL;
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_ASSIGN(
backupInfo, infoBackupNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_BACKUP_INFO_MULTI_HISTORY_BASE))),
"backup.info multi-history");
TEST_ASSIGN(archiveInfo, infoArchiveNewLoad(ioBufferReadNew(harnessInfoChecksumZ(TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE))),
"archive.info multi-history");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
InfoPg *pgHistory = infoArchivePg(archiveInfo);
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
StringList *backupList = strLstNew();
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
strLstAddZ(backupList, "20181119-152138F");
strLstAddZ(backupList, "20181119-152900F");
StringList *archiveIdList = strLstComparatorSet(strLstNew(), archiveIdComparator);
strLstAddZ(archiveIdList, "9.4-1");
strLstAddZ(archiveIdList, "11-2");
unsigned int errTotal = 0;
// Add backup to end of list
strLstAddZ(backupList, "20181119-153000F");
strLstAddZ(archiveIdList, "12-3");
TEST_RESULT_STR_Z(
verifySetBackupCheckArchive(backupList, backupInfo, archiveIdList, pgHistory, &errTotal),
"20181119-153000F", "current backup, missing archive");
TEST_RESULT_UINT(errTotal, 1, "error logged");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
TEST_RESULT_LOG("P00 INFO: archiveIds '12-3' are not in the archive.info history list");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
errTotal = 0;
strLstAddZ(archiveIdList, "13-4");
TEST_RESULT_STR_Z(
verifySetBackupCheckArchive(backupList, backupInfo, archiveIdList, pgHistory, &errTotal),
"20181119-153000F", "test multiple archiveIds on disk not in archive.info");
TEST_RESULT_UINT(errTotal, 1, "error logged");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
TEST_RESULT_LOG("P00 INFO: archiveIds '12-3, 13-4' are not in the archive.info history list");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifyLogInvalidResult() - missing file");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_RESULT_UINT(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
verifyLogInvalidResult(STORAGE_REPO_ARCHIVE_STR, verifyFileMissing, 0, STRDEF("missingfilename")),
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
0, "file missing message");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG("P00 WARN: file missing 'missingfilename'");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifyRender() - missing file, empty invalidList");
Remove extra spaces.
2022-01-18 17:40:53 -05:00
List *archiveIdResultList = lstNewP(sizeof(VerifyArchiveResult), .comparator = archiveIdComparator);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
List *backupResultList = lstNewP(sizeof(VerifyBackupResult), .comparator = lstComparatorStr);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
VerifyArchiveResult archiveIdResult =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.archiveId = strNewZ("9.6-1"),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
.totalWalFile = 1,
Remove extra spaces.
2022-01-18 17:40:53 -05:00
.walRangeList = lstNewP(sizeof(VerifyWalRange), .comparator = lstComparatorStr),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
};
VerifyWalRange walRange =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.start = strNewZ("0"),
.stop = strNewZ("2"),
Remove extra spaces.
2022-01-18 17:40:53 -05:00
.invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
};
lstAdd(archiveIdResult.walRangeList, &walRange);
lstAdd(archiveIdResultList, &archiveIdResult);
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_STR_Z(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
verifyRender(archiveIdResultList, backupResultList, cfgOptionBool(cfgOptVerbose)),
"\n"
" archiveId: 9.6-1, total WAL checked: 1, total valid WAL: 0", "archive: no invalid file list");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
VerifyInvalidFile invalidFile =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.fileName = strNewZ("file"),
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
.reason = verifyFileMissing,
};
lstAdd(walRange.invalidFileList, &invalidFile);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
VerifyBackupResult backupResult =
{
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
.backupLabel = strNewZ("test-backup-label"),
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
.status = backupInvalid,
.totalFileVerify = 1,
Remove extra spaces.
2022-01-18 17:40:53 -05:00
.invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr),
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
};
lstAdd(backupResult.invalidFileList, &invalidFile);
lstAdd(backupResultList, &backupResult);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_STR_Z(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
verifyRender(archiveIdResultList, backupResultList, cfgOptionBool(cfgOptVerbose)),
"\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
" archiveId: 9.6-1, total WAL checked: 1, total valid WAL: 0\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" missing: 1\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
" backup: test-backup-label, status: invalid, total files checked: 1, total valid files: 0\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" missing: 1", "archive file missing, backup file missing, no text, no verbose");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("verifyAddInvalidWalFile() - file missing (coverage test)");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_VOID(
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
verifyAddInvalidWalFile(archiveIdResult.walRangeList, verifyFileMissing, STRDEF("test"), STRDEF("3")), "coverage test");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
// *****************************************************************************************************************************
if (testBegin("cmdVerify() - info files"))
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info invalid checksum, neither backup copy nor archive infos exist");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_INVALID_BACKREST_INFO, .comment = "invalid backup.info");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
harnessLogLevelSet(logLevelDetail);
// Redirect stdout to a file
int stdoutSave = dup(STDOUT_FILENO);
const String *stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS' "
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"<REPO:BACKUP>/backup.info\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info.copy' for read\n"
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info' for read\n"
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info.copy' for read\n"
"P00 INFO: stanza: db\n"
" status: error\n"
" No usable backup.info file\n"
" No usable archive.info file");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info invalid checksum, backup.info.copy valid, archive.info not exist, archive copy checksum invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_INVALID_BACKREST_INFO,
.comment = "invalid archive.info.copy");
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT,
"[backup:current]\n"
TEST_BACKUP_DB1_CURRENT_FULL1
"\n"
"[db]\n"
TEST_BACKUP_DB1_94
"\n"
"[db:history]\n"
TEST_BACKUP_DB1_HISTORY,
.comment = "valid backup.info.copy");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Redirect stdout to a file
stdoutSave = dup(STDOUT_FILENO);
stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS'"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
" <REPO:BACKUP>/backup.info\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info' for read\n"
"P00 DETAIL: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS'"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
" <REPO:ARCHIVE>/archive.info.copy\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 INFO: stanza: db\n"
" status: error\n"
" No usable archive.info file");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info and copy valid but checksum mismatch, archive.info checksum invalid, archive.info copy valid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_BACKUP_INFO_MULTI_HISTORY_BASE, .comment = "valid backup.info");
HRN_STORAGE_PUT_Z(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_INVALID_BACKREST_INFO, .comment = "invalid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_BASE, .comment = "valid archive.info.copy");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Redirect stdout to a file
stdoutSave = dup(STDOUT_FILENO);
stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: backup.info.copy does not match backup.info\n"
"P00 DETAIL: invalid checksum, actual 'e056f784a995841fd4e2802b809299b8db6803a2' but expected 'BOGUS'"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
" <REPO:ARCHIVE>/archive.info\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 INFO: stanza: db\n"
" status: error\n"
" backup info file and archive info file do not match\n"
" archive: id = 1, version = 9.4, system-id = 10000000000000090400\n"
" backup : id = 2, version = 11, system-id = 10000000000000110000\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
" HINT: this may be a symptom of repository corruption!");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info and copy valid and checksums match, archive.info and copy valid, but checksum mismatch");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_BACKUP_INFO_MULTI_HISTORY_BASE,
.comment = "valid backup.info.copy");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_VOID(cmdVerify(), "usable backup and archive info files");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: archive.info.copy does not match archive.info");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info valid, copy invalid, archive.info valid, copy invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, .comment = "remove backup.info.copy");
HRN_STORAGE_REMOVE(storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, .comment = "remove archive.info.copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_VOID(cmdVerify(), "usable backup and archive info files");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info.copy' for read\n"
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/archive/db/archive.info.copy' for read");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("backup.info and copy missing, archive.info and copy valid");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
hrnCfgArgRawZ(argList, cfgOptVerbose, "y");
HRN_CFG_LOAD(cfgCmdVerify, argList);
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(storageRepoWrite(), INFO_BACKUP_PATH_FILE);
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid and matching archive.info.copy");
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Redirect stdout to a file
stdoutSave = dup(STDOUT_FILENO);
stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info' for read\n"
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/backup.info.copy' for read\n"
"P00 INFO: stanza: db\n"
" status: error\n"
" No usable backup.info file");
harnessLogLevelReset();
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
// *****************************************************************************************************************************
Add local process shim. Run the local process inside a forked child process instead of exec'ing it. This allows coverage to accumulate in the local process rather than needing to test the local protocol functions directly, resulting in better end-to-end testing and less test duplication. Another advantage is that the pgbackrest binary does not need to be built for the test. The backup, restore, and verify command tests have been updated to use the new shim for coverage.
2021-05-21 12:45:00 -04:00
if (testBegin("verifyFile()"))
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_TITLE("zero-sized file in archive");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
String *filePathName = strNewZ(STORAGE_REPO_ARCHIVE "/testfile");
HRN_STORAGE_PUT_EMPTY(storageRepoWrite(), strZ(filePathName));
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
TEST_RESULT_UINT(
Store manifest checksums in memory more efficiently. Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions. There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
verifyFile(filePathName, 0, NULL, compressTypeNone, HASH_TYPE_SHA1_ZERO_BUF, 0, NULL), verifyOk, "file ok");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("file size invalid in archive");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(storageRepoWrite(), strZ(filePathName), fileContents);
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
TEST_RESULT_UINT(
verifyFile(filePathName, 0, NULL, compressTypeNone, fileChecksum, 0, NULL), verifySizeInvalid, "file size invalid");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("file missing in archive");
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_UINT(
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
verifyFile(strNewFmt(STORAGE_REPO_ARCHIVE "/missingFile"), 0, NULL, compressTypeNone, fileChecksum, 0, NULL),
verifyFileMissing, "file missing");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("encrypted/compressed file in backup");
// Create a compressed encrypted repo file in backup
More efficient memory allocation for Strings and String Variants. The vast majority of Strings are never modified so for most cases allocate memory for the string with the object. This results in one allocation in most cases instead of two. Use strNew() if strCat*() functions are needed. Update varNewStr() in the same way since String Variants can never be modified. This results in one allocation in all cases instead of three. Also update varNewStrZ() to use STR() instead of strNewZ() to save two more allocations.
2021-10-07 19:43:28 -04:00
filePathName = strCatZ(strNew(), STORAGE_REPO_BACKUP "/testfile");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
storageRepoWrite(), strZ(filePathName), fileContents, .compressType = compressTypeGz, .cipherType = cipherTypeAes256Cbc,
.cipherPass = "pass");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
strCatZ(filePathName, ".gz");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_UINT(
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
verifyFile(filePathName, 0, NULL, compressTypeGz, fileChecksum, fileSize, STRDEF("pass")),
verifyOk, "file encrypted compressed ok");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
TEST_RESULT_UINT(
verifyFile(
Store manifest checksums in memory more efficiently. Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions. There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
filePathName, 0, NULL, compressTypeGz, bufNewDecode(encodingHex, STRDEF("aa")), fileSize, STRDEF("pass")),
verifyChecksumMismatch, "file encrypted compressed checksum mismatch");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
// *****************************************************************************************************************************
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
if (testBegin("cmdVerify(), verifyProcess() - errors"))
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
{
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
//--------------------------------------------------------------------------------------------------------------------------
Make --repo optional for remaining commands except stanza-delete. Some commands (repo-*, verify) still required the --repo option but it makes sense to give them the same treatment as backup and simply use the first repo when one is not specified. This leaves stanza-delete as the only remaining command that requires --repo. This is by design to enhance safe usage.
2021-03-03 09:21:06 -05:00
// Load Parameters with multi-repo
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
StringList *argList = strLstDup(argListBase);
Add defines for many test*() getter functions. A define was already added for TEST_PATH but it was not widely used. Replace all occurrences of testPath() with TEST_PATH in the tests. Replace testUser() with TEST_USER, testGroup() with TEST_GROUP, testRepoPath() with HRN_PATH_REPO, testDataPath() with HRN_PATH, testProjectExe() with TEST_PROJECT_EXE, and testScale() with TEST_SCALE. Replace {[path]}, {[user]}, {[group]}, etc. with defines and remove hrnReplaceKey(). This is better than having two ways to deal with replacements. In some cases the original test*() getters were kept because they are used by the harness, which does not have access to the new defines. Move them to harnessTest.intern.h to indicate that the tests should no longer use them.
2021-05-22 09:30:54 -04:00
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 4, TEST_PATH "/repo4");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Store valid archive/backup info files
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("valid info files, WAL files present, no backups");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
#define TEST_NO_CURRENT_BACKUP \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_NO_CURRENT_BACKUP, .comment = "no current backups");
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_NO_CURRENT_BACKUP, .comment = "no current backups copy");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Create WAL file with just header info and small WAL size
Buffer *walBuffer = bufNew((size_t)(1024 * 1024));
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .size = 1024 * 1024}, walBuffer);
Add hex encode/decoding to decode module. This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
const char *walBufferSha1 = strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha1, walBuffer)));
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-%s", walBufferSha1), walBuffer,
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
.comment = "valid WAL");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-bad817043007aa2100c44c712bcb456db705dab9",
walBuffer, .comment = "duplicate WAL");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log detail level to capture ranges (there should be none)
harnessLogLevelSet(logLevelDetail);
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Redirect stdout to a file
int stdoutSave = dup(STDOUT_FILENO);
const String *stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: no backups exist in the repo\n"
"P00 INFO: duplicate WAL '000000020000000700000FFE' for '11-2' exists, skipping\n"
"P00 DETAIL: path '11-2/0000000200000007' does not contain any valid WAL to be processed\n"
"P00 INFO: stanza: db\n"
" status: error\n"
" archiveId: 11-2, total WAL checked: 2, total valid WAL: 0");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
harnessLogLevelReset();
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_REMOVE(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-bad817043007aa2100c44c712bcb456db705dab9",
.comment = "remove duplicate WAL");
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/9.4-1", .comment = "empty path for old archiveId");
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_ARCHIVE "/11-2/0000000100000000", .comment = "empty timeline path");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4",
walBuffer, .compressType = compressTypeGz, .comment = "first WAL compressed - but checksum failure");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(
STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFF-%s",
Add hex encode/decoding to decode module. This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha1, BUFSTRDEF("invalidsize"))))),
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
BUFSTRDEF("invalidsize"), .comment = "WAL - invalid size");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000008/000000020000000800000000-%s", walBufferSha1), walBuffer,
.comment = "WAL - continue range");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log detail level to capture ranges
harnessLogLevelSet(logLevelDetail);
// Test verifyProcess directly
TEST_RESULT_STR_Z(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
verifyProcess(cfgOptionBool(cfgOptVerbose)),
"stanza: db\n"
"status: error\n"
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
" archiveId: 11-2, total WAL checked: 4, total valid WAL: 2\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" checksum invalid: 1, size invalid: 1", "verifyProcess() no text, no verbose");
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: no backups exist in the repo\n"
"P00 DETAIL: archive path '9.4-1' is empty\n"
"P00 DETAIL: path '11-2/0000000100000000' does not contain any valid WAL to be processed\n"
"P01 INFO: invalid checksum "
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P01 INFO: invalid size "
Rename strNew() to strNewZ() and add parameter-less strNew(). Replace all instances of strNew("") with strNew() and use strNewZ() for non-empty zero-terminated strings. Besides saving a useless parameter, this will allow smarter memory allocation in a future commit by signaling intent, in general, to append or not. In the tests use STRDEF() or VARSTRDEF() where more appropriate rather than blindly replacing with strNewZ(). Also replace strLstAdd() with strLstAddZ() where appropriate for the same reason.
2021-05-21 17:36:43 -04:00
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFD, wal stop: 000000020000000800000000");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
harnessLogLevelReset();
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("no text output, verbose, with verify failures");
hrnCfgArgRawZ(argList, cfgOptVerbose, "y");
HRN_CFG_LOAD(cfgCmdVerify, argList);
// Verify text output, verbose, with verify failures
TEST_RESULT_STR_Z(
verifyProcess(cfgOptionBool(cfgOptVerbose)),
"stanza: db\n"
"status: error\n"
" archiveId: 9.4-1, total WAL checked: 0, total valid WAL: 0\n"
" archiveId: 11-2, total WAL checked: 4, total valid WAL: 2\n"
" missing: 0, checksum invalid: 1, size invalid: 1, other: 0\n"
" backup: none found" , "verbose, with failures");
TEST_RESULT_LOG(
"P01 INFO: invalid checksum "
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
"P01 INFO: invalid size "
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("text output, verbose, with verify failures");
hrnCfgArgRawZ(argList, cfgOptOutput, "text");
HRN_CFG_LOAD(cfgCmdVerify, argList);
// Verify text output, verbose, with verify failures
TEST_RESULT_STR_Z(
verifyProcess(cfgOptionBool(cfgOptVerbose)),
"stanza: db\n"
"status: error\n"
" archiveId: 9.4-1, total WAL checked: 0, total valid WAL: 0\n"
" archiveId: 11-2, total WAL checked: 4, total valid WAL: 2\n"
" missing: 0, checksum invalid: 1, size invalid: 1, other: 0\n"
" backup: none found", "verify text output, verbose, with verify failures");
TEST_RESULT_LOG(
"P01 INFO: invalid checksum "
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
"P01 INFO: invalid size "
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("valid info files, start next timeline");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// Load Parameters - single default repo
argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000008/000000020000000800000002-%s", walBufferSha1),
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
walBuffer, .comment = "WAL - starts next range");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000300000000/000000030000000000000000-%s", walBufferSha1),
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
walBuffer, .comment = "WAL - starts next timeline");
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000300000000/000000030000000000000001-%s", walBufferSha1),
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
walBuffer, .comment = "WAL - end next timeline");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log level to errors only
harnessLogLevelSet(logLevelError);
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
TEST_RESULT_BOOL(cfgOptionBool(cfgOptVerbose), false, "verbose is false");
TEST_RESULT_BOOL((cfgOptionStrId(cfgOptOutput) == CFGOPTVAL_OUTPUT_TEXT), false, "text is false");
// Redirect stdout to a file
stdoutSave = dup(STDOUT_FILENO);
stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
TEST_RESULT_LOG("");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("none output, with verify failures");
hrnCfgArgRawZ(argList, cfgOptOutput, "none");
HRN_CFG_LOAD(cfgCmdVerify, argList);
// Verify none output
TEST_RESULT_STR_Z(
verifyProcess(cfgOptionBool(cfgOptVerbose)),
"stanza: db\n"
"status: error\n"
" archiveId: 11-2, total WAL checked: 7, total valid WAL: 5\n"
" checksum invalid: 1, size invalid: 1", "verify none output");
TEST_RESULT_LOG("");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
//--------------------------------------------------------------------------------------------------------------------------
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_TITLE("valid info files - various archive/backup errors");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
// Load Parameters - single non-default repo
argList = strLstNew();
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 2, TEST_PATH "/repo");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
hrnCfgArgRawZ(argList, cfgOptStanza, "db");
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
hrnCfgArgRawZ(argList, cfgOptRepo, "2");
Add jobRetry parameter to HRN_CFG_LOAD(). Allow the default of 0 to be overridden to test retry behavior for commands.
2021-10-01 17:15:36 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList, .jobRetry = 1);
Partial multi-repository implementation. Multi-repository implementations for the archive-push, check, info, stanza-create, stanza-upgrade, and stanza-delete commands. Multi-repo configuration is disabled so there should be no behavioral changes between these commands and their current single-repo implementations. Multi-repo documentation and integration tests are still in the multi-repo development branch. All unit tests work as multi-repo since they are able to bypass the configuration restrictions.
2021-01-21 15:21:50 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
STORAGE_REPO_ARCHIVE "/11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9",
walBuffer, .modeFile = 0200, .comment = "WAL - file not readable");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152800F", .comment = "prior backup path missing manifests");
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152810F/" BACKUP_MANIFEST_FILE,
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_DB,
.comment = "manifest without target files");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create full backup with files
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/PG_VERSION", "BOGUS",
.comment = "put checksum-error backup file");
HRN_STORAGE_PUT_EMPTY(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/testzero",
.comment = "put zero-size backup file");
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/testvalid", fileContents,
.comment = "put valid file");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write manifests for full backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
String *manifestContent = strNewFmt(
"[backup]\n"
"backup-label=\"20181119-152900F\"\n"
"backup-timestamp-copy-start=0\n"
"backup-timestamp-start=0\n"
"backup-timestamp-stop=0\n"
"backup-type=\"full\"\n"
"\n"
"[backup:db]\n"
TEST_BACKUP_DB2_11
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/testvalid={\"checksum\":\"%s\",\"size\":7,\"timestamp\":1565282114}\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"pg_data/testzero={\"repo-size\":20,\"size\":0,\"timestamp\":1601405663}\n"
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
Store manifest checksums in memory more efficiently. Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions. There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
strZ(strNewEncode(encodingHex, fileChecksum)));
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE, strZ(manifestContent),
.comment = "valid manifest");
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
strZ(manifestContent), .comment = "valid manifest copy");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
// Create a manifest for the dependent that has references
manifestContent = strNewFmt(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"[backup]\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"backup-label=\"20181119-152900F_20181119-152909D\"\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"backup-timestamp-copy-start=0\n"
"backup-timestamp-start=0\n"
"backup-timestamp-stop=0\n"
"backup-type=\"diff\"\n"
"\n"
"[backup:db]\n"
TEST_BACKUP_DB2_11
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
"\n"
"[target:file]\n"
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\","
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"\"reference\":\"20181119-152900F\",\"size\":4,\"timestamp\":1565282114}\n"
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/testfile={\"checksum\":\"%s\",\"reference\":\"20181119-152900F\",\"size\":7,\"timestamp\":1565282114}\n"
"pg_data/testfile2={\"checksum\":\"%s\",\"size\":7,\"timestamp\":1565282114}\n"
Add repository checksum to make verify and resume more efficient. Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data. This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum. Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
"pg_data/repochk={\"checksum\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"rck\":\"%s\",\"repo-size\":7"
",\"size\":3,\"timestamp\":1565282114}\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"pg_data/testmissing="
"{\"checksum\":\"123473f470864e067ee3a22e64b47b0a1c356abc\",\"size\":7,\"timestamp\":1565282114}\n"
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/testother={\"checksum\":\"%s\",\"reference\":\"UNPROCESSEDBACKUP\",\"size\":7,\"timestamp\":1565282114}\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
Store manifest checksums in memory more efficiently. Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions. There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
strZ(strNewEncode(encodingHex, fileChecksum)), strZ(strNewEncode(encodingHex, fileChecksum)),
Add repository checksum to make verify and resume more efficient. Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data. This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum. Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
strZ(strNewEncode(encodingHex, fileChecksum)), strZ(strNewEncode(encodingHex, fileChecksum)));
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write manifests for dependent backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE,
strZ(manifestContent), .comment = "manifest to dependent");
HRN_INFO_PUT(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
strZ(manifestContent), .comment = "manifest copy to dependent");
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0),
STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/pg_data/testfile2", fileContents,
.comment = "put valid file to dependent");
Add repository checksum to make verify and resume more efficient. Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data. This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum. Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0),
STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/pg_data/repochk", fileContents,
.comment = "put valid file to dependent");
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
// Create an unprocessed backup label with a file that will be referenced in the dependent manifest
HRN_STORAGE_PUT_Z(
storageRepoIdxWrite(0),
STORAGE_REPO_BACKUP "/UNPROCESSEDBACKUP/pg_data/testother", fileContents, .modeFile = 0200,
.comment = "put unreadable file to unprocessed backup");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create in-progress backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PATH_CREATE(
storageRepoIdxWrite(0), STORAGE_REPO_BACKUP "/20181119-153000F",
.comment = "create empty backup path for newest backup so in-progress");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
// Set log level to capture ranges
harnessLogLevelSet(logLevelDetail);
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
TEST_RESULT_BOOL(cfgOptionBool(cfgOptVerbose), false, "verbose is false");
TEST_RESULT_BOOL((cfgOptionStrId(cfgOptOutput) == CFGOPTVAL_OUTPUT_TEXT), false, "text is false");
// Redirect stdout to a file
stdoutSave = dup(STDOUT_FILENO);
stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: archive path '9.4-1' is empty\n"
"P00 DETAIL: path '11-2/0000000100000000' does not contain any valid WAL to be processed\n"
"P01 INFO: invalid checksum "
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P01 INFO: invalid size "
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P01 INFO: invalid result "
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9: [41] raised from "
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
"local-1 shim protocol: unable to open file '" TEST_PATH "/repo/archive/db/"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
"11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9' for read:"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
" [13] Permission denied\n"
Report original error and retries on local job failure. The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync. Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00
" [FileOpenError] on retry after 0ms\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152800F/backup.manifest' for read\n"
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152800F/backup.manifest.copy'"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" for read\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 DETAIL: manifest missing for '20181119-152800F' - backup may have expired\n"
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-152810F/backup.manifest.copy'"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" for read\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 INFO: backup '20181119-152810F' manifest does not contain any target files to verify\n"
"P01 INFO: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 INFO: file missing '20181119-152900F_20181119-152909D/pg_data/testmissing'\n"
"P00 DETAIL: unable to open missing file '" TEST_PATH "/repo/backup/db/20181119-153000F/backup.manifest' for read\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"P00 INFO: backup '20181119-153000F' appears to be in progress, skipping\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P01 INFO: invalid result UNPROCESSEDBACKUP/pg_data/testother: [41] raised from local-1 shim protocol:"
Remove TEST_PATH_REPO, TEST_PATH_PG, and TEST_PATH_SPOOL constants. It seems better to use TEST_PATH in combination with a constant string rather than have a number of different path constants. This improves readability and reduces confusion about which constant should be used.
2021-06-28 13:28:28 -04:00
" unable to open file '" TEST_PATH "/repo/backup/db/UNPROCESSEDBACKUP/pg_data/testother' for read: [13]"
Replace defines with string constants in unit tests. For tests already updated as part of the macro-replacement effort, the output tests (TEST_ERROR, TEST_RESULT_LOG, TEST_STORAGE_LIST and TEST_RESULT_STR) have been simplified for readability to remove all but the TEST_PATH constants. The ongoing macro-replacement effort will include these changes. Updated: expireTest, stanzaTest, checkTest, infoTest, verifyTest (infoArchive and infoBackup had no changes).
2021-06-28 10:58:27 -04:00
" Permission denied\n"
Report original error and retries on local job failure. The local process will retry jobs (e.g. backup file) but after a certain number of failures gives up. Previously, the last error was reported but generally the first error is far more valuable. The last error is likely to be a cascade failure such as the protocol being out of sync. Report the first error (and stack trace) and append the retry errors to the first error without stack trace information.
2021-10-05 09:00:16 -04:00
" [FileOpenError] on retry after 0ms\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFD, wal stop: 000000020000000800000000\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000800000002, wal stop: 000000020000000800000003\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000030000000000000000, wal stop: 000000030000000000000001\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P00 INFO: stanza: db\n"
" status: error\n"
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
" archiveId: 11-2, total WAL checked: 8, total valid WAL: 5\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" checksum invalid: 1, size invalid: 1, other: 1\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
" backup: 20181119-152800F, status: manifest missing, total files checked: 0, total valid files: 0\n"
" backup: 20181119-152810F, status: invalid, total files checked: 0, total valid files: 0\n"
" backup: 20181119-152900F, status: invalid, total files checked: 3, total valid files: 2\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" checksum invalid: 1\n"
Add repository checksum to make verify and resume more efficient. Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data. This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum. Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 6,"
" total valid files: 3\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" missing: 1, checksum invalid: 1, other: 1");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
harnessLogLevelReset();
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("text output, not verbose, with verify failures");
hrnCfgArgRawZ(argList, cfgOptOutput, "text");
HRN_CFG_LOAD(cfgCmdVerify, argList);
// Verify text output, not verbose, with failures
TEST_RESULT_STR_Z(
verifyProcess(cfgOptionBool(cfgOptVerbose)),
"stanza: db\n"
"status: error\n"
" archiveId: 11-2, total WAL checked: 8, total valid WAL: 5\n"
" checksum invalid: 1, size invalid: 1, other: 1\n"
" backup: 20181119-152800F, status: manifest missing, total files checked: 0, total valid files: 0\n"
" backup: 20181119-152810F, status: invalid, total files checked: 0, total valid files: 0\n"
" backup: 20181119-152900F, status: invalid, total files checked: 3, total valid files: 2\n"
" checksum invalid: 1\n"
Add repository checksum to make verify and resume more efficient. Calculate a checksum of the data stored in the repository when a file is transformed (e.g. compressed). This allows resume and verify to operate without needing to decompress/decrypt the data. This can also be used to verify more complex formats such as block incremental and allow backups from the repository without needing to decompress the data to verify the checksum. Add some basic encrypted tests to maintain coverage. These will be expanded in a future commit.
2022-12-22 09:26:26 +07:00
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 6, total valid files: 3\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" missing: 1, checksum invalid: 1, other: 1", "verify text output, not verbose, with verify failures");
TEST_RESULT_LOG(
"P01 INFO: invalid checksum "
"'11-2/0000000200000007/000000020000000700000FFD-a6e1a64f0813352bc2e97f116a1800377e17d2e4.gz'\n"
"P01 INFO: invalid size "
"'11-2/0000000200000007/000000020000000700000FFF-ee161f898c9012dd0c28b3fd1e7140b9cf411306'\n"
"P01 INFO: invalid result "
"11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9: [41] raised from "
"local-1 shim protocol: unable to open file '" TEST_PATH "/repo/archive/db/"
"11-2/0000000200000008/000000020000000800000003-656817043007aa2100c44c712bcb456db705dab9' for read:"
" [13] Permission denied\n"
"P00 INFO: backup '20181119-152810F' manifest does not contain any target files to verify\n"
"P01 INFO: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 INFO: file missing '20181119-152900F_20181119-152909D/pg_data/testmissing'\n"
"P00 INFO: backup '20181119-153000F' appears to be in progress, skipping\n"
"P01 INFO: invalid result UNPROCESSEDBACKUP/pg_data/testother: [41] raised from local-1 shim protocol:"
" unable to open file '" TEST_PATH "/repo/backup/db/UNPROCESSEDBACKUP/pg_data/testother' for read: [13]"
" Permission denied");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
}
// *****************************************************************************************************************************
if (testBegin("cmdVerify()"))
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
#define TEST_BACKUP_DB1_CURRENT_FULL3_DIFF1 \
"20181119-152900F_20181119-152909D={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000010000000000000006\",\"backup-archive-stop\":\"000000010000000000000007\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB2_CURRENT_FULL1 \
"20201119-163000F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000020000000000000001\",\"backup-archive-stop\":\"000000020000000000000001\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":2,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("prior backup verification incomplete - referenced file checked");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
#define TEST_BACKUP_INFO \
"[backup:current]\n" \
TEST_BACKUP_DB1_CURRENT_FULL3 \
TEST_BACKUP_DB1_CURRENT_FULL3_DIFF1 \
TEST_BACKUP_DB2_CURRENT_FULL1 \
"\n" \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_BACKUP_INFO);
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_BACKUP_INFO);
// Create valid full backup for DB1
#define TEST_MANIFEST_FULL_DB1 \
TEST_MANIFEST_HEADER \
TEST_MANIFEST_DB_94 \
TEST_MANIFEST_OPTION_ALL \
TEST_MANIFEST_TARGET \
TEST_MANIFEST_DB \
TEST_MANIFEST_FILE \
TEST_MANIFEST_FILE_DEFAULT \
TEST_MANIFEST_LINK \
TEST_MANIFEST_LINK_DEFAULT \
TEST_MANIFEST_PATH \
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_PATH_DEFAULT
// Write manifests for full backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE, TEST_MANIFEST_FULL_DB1,
.comment = "valid manifest - full");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, TEST_MANIFEST_FULL_DB1,
.comment = "valid manifest copy - full");
// Create valid diff backup for DB1
#define TEST_MANIFEST_DIFF_DB1 \
TEST_MANIFEST_HEADER \
TEST_MANIFEST_DB_94 \
TEST_MANIFEST_OPTION_ALL \
TEST_MANIFEST_TARGET \
TEST_MANIFEST_DB \
"\n" \
"[target:file]\n" \
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"reference\":\"20181119-152900F\"" \
",\"size\":4,\"timestamp\":1565282114}\n" \
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_FILE_DEFAULT \
TEST_MANIFEST_LINK \
TEST_MANIFEST_LINK_DEFAULT \
TEST_MANIFEST_PATH \
TEST_MANIFEST_PATH_DEFAULT
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Write manifests for diff backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE,
TEST_MANIFEST_DIFF_DB1, .comment = "valid manifest - diff");
HRN_INFO_PUT(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
TEST_MANIFEST_DIFF_DB1, .comment = "valid manifest copy - diff");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Put the file referenced by both backups into the full backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT_Z(storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/PG_VERSION", fileContents);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Redirect stdout to a file
int stdoutSave = dup(STDOUT_FILENO);
const String *stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// The error for the referenced file is logged twice because it is checked again by the second backup since the first backup
// verification had not yet completed before the second backup verification began
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P01 INFO: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 INFO: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P00 INFO: stanza: db\n"
" status: error\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20181119-152900F, status: invalid, total files checked: 1, total valid files: 0\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" checksum invalid: 1\n"
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 1,"
" total valid files: 0\n"
" checksum invalid: 1");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("valid backup, prior backup verification complete - referenced file not checked");
// Set process max to 1 and add more files to check so first backup completes before second is checked
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
hrnCfgArgRawZ(argList, cfgOptProcessMax, "1");
Replace harnessCfgLoad*() functions with HRN_CFG_LOAD() macro. HRN_CFG_LOAD() handles the majority of test configuration loads and has various options for special cases. It was not clear when to use harnessCfgLoadRaw() vs harnessCfgLoad(). Now "raw" functionality is granular and enabled by parameters, e.g. noStd.
2021-06-01 09:03:44 -04:00
HRN_CFG_LOAD(cfgCmdVerify, argList);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
String *manifestContent = strNewFmt(
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
TEST_MANIFEST_FILE
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/base/1/555_init={\"checksum\":\"%s\",\"size\":1,\"timestamp\":1565282114}\n"
"pg_data/base/1/555_init.1={\"size\":0,\"timestamp\":1565282114}\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
Store manifest checksums in memory more efficiently. Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions. There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
strZ(strNewEncode(encodingHex, fileChecksum)));
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE, strZ(manifestContent),
.comment = "valid manifest - full");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestContent),
.comment = "valid manifest copy - full");
HRN_STORAGE_PUT_Z(
storageRepoWrite(),
STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/base/1/555_init", fileContents, .comment = "invalid size");
// Diff manifest
manifestContent = strNewZ(
TEST_MANIFEST_HEADER
TEST_MANIFEST_DB_94
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
"\n"
"[target:file]\n"
Remove primary flag from manifest. This flag was only being used by the backup command after manifestNewBuild() and had no other uses. There was a time when it was important for integration testing but the unit tests now fulfill this role. Since backup is the only code concerned with the primary flag, move the code into the backup module. We don't have any cross-version testing but this change was tested manually with the most recent version of pgBackRest to make sure it was tolerant of the missing primary info. When an older version of pgBackRest loads a newer manifest the primary flag will always be set to false, which is fine since it is not used.
2022-01-20 14:01:10 -05:00
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"reference\":\"20181119-152900F\""
",\"size\":4,\"timestamp\":1565282114}\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT);
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE,
strZ(manifestContent), .comment = "valid manifest - diff");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
strZ(manifestContent), .comment = "valid manifest copy - diff");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create valid full backup and valid diff backup
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
manifestContent = strNewFmt(
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_HEADER
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
"backup-bundle=true\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
"\n"
"[backup:db]\n"
TEST_BACKUP_DB2_11
TEST_MANIFEST_OPTION_ALL
TEST_MANIFEST_TARGET
TEST_MANIFEST_DB
"\n"
"[target:file]\n"
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
"pg_data/validfile={\"bni\":1,\"bno\":3,\"checksum\":\"%s\",\"size\":%u,\"timestamp\":1565282114}\n"
"pg_data/zerofile={\"size\":0,\"timestamp\":1565282114}\n"
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
TEST_MANIFEST_FILE_DEFAULT
TEST_MANIFEST_LINK
TEST_MANIFEST_LINK_DEFAULT
TEST_MANIFEST_PATH
TEST_MANIFEST_PATH_DEFAULT,
Store manifest checksums in memory more efficiently. Manifest checksums were stored as hex-encoded strings due to legacy compatibility with Perl. Storing the checksums as binary in memory uses half the space and avoids many conversions. There is no change to the on-disk manifest format which stores the checksum as a hex-encoded string.
2022-12-20 16:35:27 +07:00
strZ(strNewEncode(encodingHex, fileChecksum)), (unsigned int)fileSize);
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20201119-163000F/" BACKUP_MANIFEST_FILE, strZ(manifestContent),
.comment = "valid manifest - full");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20201119-163000F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestContent),
.comment = "valid manifest copy - full");
HRN_STORAGE_PUT_Z(
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
storageRepoWrite(), STORAGE_REPO_BACKUP "/20201119-163000F/bundle/1", zNewFmt("XXX%s", fileContents),
Bundle files in the repository during backup. Bundle (combine) smaller files during backup to reduce the number of files written to the repository (enable with --bundle). Reducing the number of files is a benefit on all file systems, but especially so on object stores such as S3 that have a high file creation cost. Another benefit is that zero-length files are only stored as metadata in the manifest. Files are batched up to bundle-size and then compressed/encrypted individually and stored sequentially in the bundle. The bundle id and offset of each file is stored in the manifest so files can be retrieved randomly without needing to read the entire bundle. Files are ordered by timestamp descending when being assigned to bundles to reduce the amount of random access that needs to be done. The idea is that bundles with older files can be read in their entirety on restore and only bundles with newer files will get fragmented. Bundles are a custom format with metadata stored in the manifest. Tar was considered but it is too limited a format, the major issue being that the size of the file must be known in advance and that is very contrary to how pgBackRest works, especially once we introduce page-level incremental backups. Bundles are stored numbered in the bundle directory. Some files may still end up in pg_data if they are added after the backup is complete. backup_label is an example. Currently, only the backup command works in batches. The restore and verify commands use the offsets to pull individual files out of the bundle. It seems better to finalize how this is going to work before optimizing the other commands. Even as is, this is a major step forward, and all commands function with bundling. One caveat: resume is currently not supported when bundle is enabled.
2022-02-14 13:24:14 -06:00
.comment = "valid file");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
// Create WAL file with just header info and small WAL size
Buffer *walBuffer = bufNew((size_t)(1024 * 1024));
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
Autogenerate test system identifiers. hrnPgControlToBuffer() and hrnPgWalToBuffer() now generate the system id based on the version of Postgres. If a value less than 100 is specified for systemId then it will be added to the default system id so there can be multiple ids for a single version of PostgreSQL. Add constants to represent version system ids in tests. These will eventually be auto-generated. This changes some checksums and we no longer have big-endian tests systems, so X those checksums out so it is obvious they are no longer valid.
2021-11-30 08:28:36 -05:00
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .size = 1024 * 1024}, walBuffer);
Add hex encode/decoding to decode module. This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
const char *walBufferSha1 = strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha1, walBuffer)));
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
HRN_STORAGE_PUT(
storageRepoWrite(),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000000/000000020000000000000001-%s", walBufferSha1), walBuffer,
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
.comment = "valid WAL");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Redirect stdout to a file
stdoutSave = dup(STDOUT_FILENO);
stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
Add backup verification to internal verify command. This is phase 2 of verify command development (phase 1 was processing the archives and phase 3 will be reconciling the archives and backups). In this phase the backups are verified by verifying each file listed in the manifest for the backup and creating a result set with the list of invalid files, if any. A summary is then rendered. Unit tests have been added and duplicate tests have been removed.
2021-01-26 11:21:36 -05:00
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile), "", .remove = true);
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
TEST_RESULT_LOG(
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
"P01 INFO: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 INFO: invalid size '20181119-152900F/pg_data/base/1/555_init'\n"
"P01 INFO: file missing '20181119-152900F/pg_data/base/1/555_init.1'\n"
"P00 INFO: stanza: db\n"
" status: error\n"
Update command/verify test to use standard patterns.
2021-06-23 18:09:46 -04:00
" backup: 20181119-152900F, status: invalid, total files checked: 3, total valid files: 0\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" missing: 1, checksum invalid: 1, size invalid: 1\n"
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 1,"
" total valid files: 0\n"
" checksum invalid: 1");
}
// *****************************************************************************************************************************
if (testBegin("cmdVerify() verbose text"))
{
// Load Parameters
StringList *argList = strLstDup(argListBase);
hrnCfgArgRawZ(argList, cfgOptOutput, "text");
hrnCfgArgRawZ(argList, cfgOptVerbose, "y");
HRN_CFG_LOAD(cfgCmdVerify, argList);
#define TEST_BACKUP_DB1_CURRENT_FULL3_DIFF1 \
"20181119-152900F_20181119-152909D={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000010000000000000006\",\"backup-archive-stop\":\"000000010000000000000007\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
#define TEST_BACKUP_DB2_CURRENT_FULL1 \
"20201119-163000F={" \
"\"backrest-format\":5,\"backrest-version\":\"2.08dev\"," \
"\"backup-archive-start\":\"000000020000000000000001\",\"backup-archive-stop\":\"000000020000000000000001\"," \
"\"backup-info-repo-size\":2369186,\"backup-info-repo-size-delta\":2369186," \
"\"backup-info-size\":20162900,\"backup-info-size-delta\":20162900," \
"\"backup-timestamp-start\":1542640898,\"backup-timestamp-stop\":1542640911,\"backup-type\":\"full\"," \
"\"db-id\":2,\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":true,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("prior backup verification incomplete - referenced file checked verbose, text output");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
#define TEST_BACKUP_INFO \
"[backup:current]\n" \
TEST_BACKUP_DB1_CURRENT_FULL3 \
TEST_BACKUP_DB1_CURRENT_FULL3_DIFF1 \
TEST_BACKUP_DB2_CURRENT_FULL1 \
"\n" \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_BACKUP_INFO);
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_BACKUP_INFO);
// Create valid full backup for DB1
#define TEST_MANIFEST_FULL_DB1 \
TEST_MANIFEST_HEADER \
TEST_MANIFEST_DB_94 \
TEST_MANIFEST_OPTION_ALL \
TEST_MANIFEST_TARGET \
TEST_MANIFEST_DB \
TEST_MANIFEST_FILE \
TEST_MANIFEST_FILE_DEFAULT \
TEST_MANIFEST_LINK \
TEST_MANIFEST_LINK_DEFAULT \
TEST_MANIFEST_PATH \
TEST_MANIFEST_PATH_DEFAULT
// Write manifests for full backup
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE, TEST_MANIFEST_FULL_DB1,
.comment = "valid manifest - full");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, TEST_MANIFEST_FULL_DB1,
.comment = "valid manifest copy - full");
// Create valid diff backup for DB1
#define TEST_MANIFEST_DIFF_DB1 \
TEST_MANIFEST_HEADER \
TEST_MANIFEST_DB_94 \
TEST_MANIFEST_OPTION_ALL \
TEST_MANIFEST_TARGET \
TEST_MANIFEST_DB \
"\n" \
"[target:file]\n" \
"pg_data/PG_VERSION={\"checksum\":\"184473f470864e067ee3a22e64b47b0a1c356f29\",\"reference\":\"20181119-152900F\"" \
",\"size\":4,\"timestamp\":1565282114}\n" \
TEST_MANIFEST_FILE_DEFAULT \
TEST_MANIFEST_LINK \
TEST_MANIFEST_LINK_DEFAULT \
TEST_MANIFEST_PATH \
TEST_MANIFEST_PATH_DEFAULT
// Write manifests for diff backup
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE,
TEST_MANIFEST_DIFF_DB1, .comment = "valid manifest - diff");
HRN_INFO_PUT(
storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F_20181119-152909D/" BACKUP_MANIFEST_FILE INFO_COPY_EXT,
TEST_MANIFEST_DIFF_DB1, .comment = "valid manifest copy - diff");
// Put the file referenced by both backups into the full backup
HRN_STORAGE_PUT_Z(storageRepoWrite(), STORAGE_REPO_BACKUP "/20181119-152900F/pg_data/PG_VERSION", fileContents);
// Redirect stdout to a file
int stdoutSave = dup(STDOUT_FILENO);
const String *stdoutFile = STRDEF(TEST_PATH "/stdout.info");
THROW_ON_SYS_ERROR(freopen(strZ(stdoutFile), "w", stdout) == NULL, FileWriteError, "unable to reopen stdout");
// Not in a test wrapper to avoid writing to stdout
cmdVerify();
// Restore normal stdout
dup2(stdoutSave, STDOUT_FILENO);
// Check output of verify command stored in file
TEST_STORAGE_GET(storageTest, strZ(stdoutFile),
"stanza: db\n"
"status: error\n"
" archiveId: none found\n"
" backup: 20181119-152900F, status: invalid, total files checked: 1, total valid files: 0\n"
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0\n"
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 1, total valid files: 0\n"
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0\n", .remove = true);
// The error for the referenced file is logged twice because it is checked again by the second backup since the first backup
// verification had not yet completed before the second backup verification began
TEST_RESULT_LOG(
"P01 INFO: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P01 INFO: invalid checksum '20181119-152900F/pg_data/PG_VERSION'\n"
"P00 INFO: stanza: db\n"
" status: error\n"
" archiveId: none found\n"
" backup: 20181119-152900F, status: invalid, total files checked: 1, total valid files: 0\n"
Replace hrnLogResult() with TEST_RESULT_LOG/_FMT(). The macros provide more information when there is an error and may be updated in the future without changing the test code.
2021-05-22 14:09:45 -04:00
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0\n"
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
" backup: 20181119-152900F_20181119-152909D, status: invalid, total files checked: 1,"
" total valid files: 0\n"
" missing: 0, checksum invalid: 1, size invalid: 0, other: 0");
}
// *****************************************************************************************************************************
if (testBegin("verifyProcess(), none output, not verbose, no failures"))
{
//--------------------------------------------------------------------------------------------------------------------------
// Load Parameters with multi-repo
StringList *argList = strLstDup(argListBase);
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 4, TEST_PATH "/repo4");
hrnCfgArgRawZ(argList, cfgOptOutput, "none");
HRN_CFG_LOAD(cfgCmdVerify, argList);
// Store valid archive/backup info files
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("none output, not verbose, with no verify failures");
#define TEST_NO_CURRENT_BACKUP \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_NO_CURRENT_BACKUP, .comment = "no current backups");
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_NO_CURRENT_BACKUP, .comment = "no current backups copy");
// Create WAL file with just header info and small WAL size
Buffer *walBuffer = bufNew((size_t)(1024 * 1024));
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .size = 1024 * 1024}, walBuffer);
Add hex encode/decoding to decode module. This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
const char *walBufferSha1 = strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha1, walBuffer)));
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-%s", walBufferSha1), walBuffer,
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
.comment = "valid WAL");
harnessLogLevelSet(logLevelDetail);
// Verify text output with no verify errors
TEST_RESULT_STR_Z(verifyProcess(cfgOptionBool(cfgOptVerbose)), "", "verify none output, not verbose, with no failures");
TEST_RESULT_LOG(
"P00 DETAIL: no backups exist in the repo\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFE, wal stop: 000000020000000700000FFE");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("none output, verbose, with no verify failures");
hrnCfgArgRawZ(argList, cfgOptVerbose, "y");
HRN_CFG_LOAD(cfgCmdVerify, argList);
TEST_RESULT_STR_Z(
verifyProcess(cfgOptionBool(cfgOptVerbose)),
"stanza: db\n"
"status: ok\n"
" archiveId: 11-2, total WAL checked: 1, total valid WAL: 1\n"
" missing: 0, checksum invalid: 0, size invalid: 0, other: 0\n"
" backup: none found",
"verify none output, verbose, with no failures");
TEST_RESULT_LOG(
"P00 DETAIL: no backups exist in the repo\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFE, wal stop: 000000020000000700000FFE");
}
// *****************************************************************************************************************************
if (testBegin("verifyProcess(), text"))
{
//--------------------------------------------------------------------------------------------------------------------------
// Load Parameters with multi-repo
StringList *argList = strLstDup(argListBase);
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 4, TEST_PATH "/repo4");
hrnCfgArgRawZ(argList, cfgOptOutput, "text");
HRN_CFG_LOAD(cfgCmdVerify, argList);
// Store valid archive/backup info files
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("text output, not verbose, with no verify failures");
#define TEST_NO_CURRENT_BACKUP \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_NO_CURRENT_BACKUP, .comment = "no current backups");
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_NO_CURRENT_BACKUP, .comment = "no current backups copy");
// Create WAL file with just header info and small WAL size
Buffer *walBuffer = bufNew((size_t)(1024 * 1024));
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .size = 1024 * 1024}, walBuffer);
Add hex encode/decoding to decode module. This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
const char *walBufferSha1 = strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha1, walBuffer)));
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-%s", walBufferSha1), walBuffer,
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
.comment = "valid WAL");
harnessLogLevelSet(logLevelDetail);
// Verify text output with no verify errors
TEST_RESULT_STR_Z(verifyProcess(cfgOptionBool(cfgOptVerbose)), "", "verify text output, not verbose, with no failures");
TEST_RESULT_LOG(
"P00 DETAIL: no backups exist in the repo\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFE, wal stop: 000000020000000700000FFE");
}
// *****************************************************************************************************************************
if (testBegin("verifyProcess(), text, verbose"))
{
//--------------------------------------------------------------------------------------------------------------------------
// Load Parameters with multi-repo
StringList *argList = strLstDup(argListBase);
hrnCfgArgKeyRawZ(argList, cfgOptRepoPath, 4, TEST_PATH "/repo4");
hrnCfgArgRawZ(argList, cfgOptOutput, "text");
hrnCfgArgRawZ(argList, cfgOptVerbose, "y");
HRN_CFG_LOAD(cfgCmdVerify, argList);
// Store valid archive/backup info files
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE, .comment = "valid archive.info");
HRN_INFO_PUT(
storageRepoWrite(), INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT, TEST_ARCHIVE_INFO_MULTI_HISTORY_BASE,
.comment = "valid archive.info.copy");
//--------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("text output, verbose with no verify failures");
#define TEST_NO_CURRENT_BACKUP \
"[db]\n" \
TEST_BACKUP_DB2_11 \
"\n" \
"[db:history]\n" \
TEST_BACKUP_DB1_HISTORY \
"\n" \
TEST_BACKUP_DB2_HISTORY
HRN_INFO_PUT(storageRepoWrite(), INFO_BACKUP_PATH_FILE, TEST_NO_CURRENT_BACKUP, .comment = "no current backups");
HRN_INFO_PUT(
storageRepoWrite(), INFO_BACKUP_PATH_FILE INFO_COPY_EXT, TEST_NO_CURRENT_BACKUP, .comment = "no current backups copy");
// Create WAL file with just header info and small WAL size
Buffer *walBuffer = bufNew((size_t)(1024 * 1024));
bufUsedSet(walBuffer, bufSize(walBuffer));
memset(bufPtr(walBuffer), 0, bufSize(walBuffer));
hrnPgWalToBuffer((PgWal){.version = PG_VERSION_11, .size = 1024 * 1024}, walBuffer);
Add hex encode/decoding to decode module. This replaces the bufHex() function and also allows hex to be decoded.
2022-12-11 19:46:48 +07:00
const char *walBufferSha1 = strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha1, walBuffer)));
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
HRN_STORAGE_PUT(
storageRepoIdxWrite(0),
Add zNewFmt(). This replaces strZ(strNewFmt()), making the code simpler and reducing indentation.
2022-05-06 12:32:49 -04:00
zNewFmt(STORAGE_REPO_ARCHIVE "/11-2/0000000200000007/000000020000000700000FFE-%s", walBufferSha1), walBuffer,
Add verify output and verbose options. These options allow the user to control how the verify results will be output to the console and log.
2022-05-06 11:11:36 -04:00
.comment = "valid WAL");
harnessLogLevelSet(logLevelDetail);
// Verify text output with no verify errors
TEST_RESULT_STR_Z(
verifyProcess(cfgOptionBool(cfgOptVerbose)),
"stanza: db\n"
"status: ok\n"
" archiveId: 11-2, total WAL checked: 1, total valid WAL: 1\n"
" missing: 0, checksum invalid: 0, size invalid: 0, other: 0\n"
" backup: none found", "verify text output, verbose, with no failures");
TEST_RESULT_LOG(
"P00 DETAIL: no backups exist in the repo\n"
"P00 DETAIL: archiveId: 11-2, wal start: 000000020000000700000FFE, wal stop: 000000020000000700000FFE");
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
Rename FUNCTION_HARNESS_RESULT*() macros to FUNCTION_HARNESS_RETURN*(). When the FUNCTION_*_RESULT*() macros were renamed to FUNCTION_*_RETURN_*() in the core code the test harness macros were missed. Update them to make the naming consistent.
2021-03-10 18:42:22 -05:00
FUNCTION_HARNESS_RETURN_VOID();
Add internal verify command. Scan the WAL archive for missing or invalid files and build up ranges of WAL that will be used to verify backup integrity. A number of errors and warnings are currently emitted but they should not be considered authoritative (yet). The command is incomplete so is marked internal.
2020-09-22 11:57:38 -04:00
}
Reference in New Issue Copy Permalink