db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Code Issues Releases Activity

Auto-select backup for restore command --type=lsn.

Browse Source 
For PITR with --type=lsn, attempt to auto-select the appropriate backup set based on the --target LSN provided. Pick the most recent backup where backup-lsn-stop is less than or equal to the provided LSN.
This commit is contained in:
Reid Thompson 2022-04-05 11:59:12 -04:00 committed by GitHub
parent 08d9e269c6
commit d8d4132118
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 198 additions and 31 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
doc/xml/release.xml
View File

@ -17,6 +17,21 @@
<release date="XXXX-XX-XX" version="2.39dev" title="UNDER DEVELOPMENT">
<release-core-list>
<release-feature-list>
<release-item>
<commit subject="Refactor target type checking for clarity in restore module."/>
<commit subject="Auto-select backup for restore command --type=lsn.">
<github-pull-request id="1706"/>
</commit>
<release-item-contributor-list>
<release-item-contributor id="reid.thompson"/>
<release-item-reviewer id="stefan.fercot"/>
<release-item-reviewer id="david.steele"/>
</release-item-contributor-list>
<p>Auto-select backup for <cmd>restore</cmd> command <br-option>--type=lsn</br-option>.</p>
</release-item>
<release-item>
<github-issue id="1685"/>
<github-pull-request id="1691"/>

4
src/build/help/help.xml
View File

@ -2173,7 +2173,7 @@
<text>
<p>The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see <link page="user-guide" section="/quickstart/perform-restore">Quick Start - Restore a Backup</link>). The order in which the repositories are checked is dictated by the <file>pgbackrest.conf</file> (e.g. repo1 will be checked before repo2). To select from a specific repository, the <br-option>{[dash]}-repo</br-option> option can be passed (e.g. <br-option>{[dash]}-repo=1</br-option>). The <br-option>{[dash]}-set</br-option> option can be passed if a backup other than the latest is desired.</p>
<p>When PITR of <br-option>{[dash]}-type=time</br-option> is specified, then the target time must be specified with the <br-option>{[dash]}-target</br-option> option. If a backup is not specified via the <br-option>{[dash]}-set</br-option> option, then the configured repositories will be checked, in order, for a backup that contains the requested time. If no backup can be found, the latest backup from the first repository containing backups will be used. For other types of PITR, e.g. <id>xid</id>, the <br-option>{[dash]}-set</br-option> option must be provided if the target is prior to the latest backup. See <link page="user-guide" section="/pitr">Point-in-Time Recovery</link> for more details and examples.</p>
<p>When PITR of <br-option>{[dash]}-type=time</br-option> or <br-option>{[dash]}-type=lsn</br-option> is specified, then the target time or target lsn must be specified with the <br-option>{[dash]}-target</br-option> option. If a backup is not specified via the <br-option>{[dash]}-set</br-option> option, then the configured repositories will be checked, in order, for a backup that contains the requested time or lsn. If no matching backup is found, the latest backup from the first repository containing backups will be used for <br-option>{[dash]}-type=time</br-option> while no backup will be selected for <br-option>{[dash]}-type=lsn</br-option>. For other types of PITR, e.g. <id>xid</id>, the <br-option>{[dash]}-set</br-option> option must be provided if the target is prior to the latest backup. See <link page="user-guide" section="/pitr">Point-in-Time Recovery</link> for more details and examples.</p>
<p>Replication slots are not included per recommendation of <postgres/>. See <link url="https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-LOWLEVEL-BASE-BACKUP-DATA">Backing Up The Data Directory</link> in the <postgres/> documentation for more information.</p>
</text>
@ -2229,7 +2229,7 @@
<summary>Recovery target.</summary>
<text>
<p>Defines the recovery target when <br-option>--type</br-option> is <id>lsn</id>, <id>name</id>, <id>xid</id>, or <id>time</id>. If the target is prior to the latest backup and <br-option>--type</br-option> is not <id>time</id>, then use the <br-option>--set</br-option> option to specify the backup set.</p>
<p>Defines the recovery target when <br-option>--type</br-option> is <id>lsn</id>, <id>name</id>, <id>xid</id>, or <id>time</id>. If the target is prior to the latest backup and <br-option>--type</br-option> is not <id>time</id> or <id>lsn</id>, then use the <br-option>--set</br-option> option to specify the backup set.</p>
</text>
<example>2015-01-30 14:15:11 EST</example>

46
src/command/restore/restore.c
View File

@ -237,16 +237,23 @@ restoreBackupSet(void)
repoIdxMax = repoIdxMin;
}
// If the set option was not provided by the user but a time to recover was set, then we will need to search for a backup
// set that satisfies the time condition, else we will use the backup provided
// If the set option was not provided by the user but a target was set, then we will need to search for a backup set that
// satisfies the target condition, else we will use the backup provided
const String *backupSetRequested = NULL;
const StringId targetType = cfgOptionStrId(cfgOptType);
time_t targetTime = 0;
union
{
time_t time;
uint64_t lsn;
} target = {0};
if (cfgOptionSource(cfgOptSet) == cfgSourceDefault)
{
if (targetType == CFGOPTVAL_TYPE_TIME)
targetTime = getEpoch(cfgOptionStr(cfgOptTarget));
target.time = getEpoch(cfgOptionStr(cfgOptTarget));
else if (targetType == CFGOPTVAL_TYPE_LSN)
target.lsn = pgLsnFromStr(cfgOptionStr(cfgOptTarget));
}
else
backupSetRequested = cfgOptionStr(cfgOptSet);
@ -284,14 +291,14 @@ restoreBackupSet(void)
continue;
}
// If a backup set was not specified, then see if a time to recover was requested
// If a backup set was not specified, then see if a target was requested
if (backupSetRequested == NULL)
{
// Get the latest backup
InfoBackupData latestBackup = infoBackupData(infoBackup, infoBackupDataTotal(infoBackup) - 1);
// If the recovery type is time, attempt to determine the backup set
if (targetType == CFGOPTVAL_TYPE_TIME)
// If a target was requested, attempt to determine the backup set
if (targetType == CFGOPTVAL_TYPE_TIME || targetType == CFGOPTVAL_TYPE_LSN)
{
bool found = false;
@ -301,13 +308,25 @@ restoreBackupSet(void)
// Get the backup data
InfoBackupData backupData = infoBackupData(infoBackup, keyIdx);
// If the end of the backup is before the target time, then select this backup
if (backupData.backupTimestampStop < targetTime)
// If target is lsn and no backupLsnStop exists, exit this repo and log that backup may be manually selected
if (targetType == CFGOPTVAL_TYPE_LSN && !backupData.backupLsnStop)
{
LOG_WARN_FMT(
"%s reached backup from prior version missing required LSN info before finding a match -- backup"
" auto-select has been disabled for this repo\n"
"HINT: you may specify a backup to restore using the --set option.",
cfgOptionGroupName(cfgOptGrpRepo, repoIdx));
break;
}
// If the end of the backup is valid for the target, then select this backup
if ((targetType == CFGOPTVAL_TYPE_TIME && backupData.backupTimestampStop < target.time) ||
(targetType == CFGOPTVAL_TYPE_LSN && pgLsnFromStr(backupData.backupLsnStop) <= target.lsn))
{
found = true;
result = restoreBackupData(
backupData.backupLabel, repoIdx, infoPgCipherPass(infoBackupPg(infoBackup)));
result = restoreBackupData(backupData.backupLabel, repoIdx, infoPgCipherPass(infoBackupPg(infoBackup)));
break;
}
}
@ -346,10 +365,11 @@ restoreBackupSet(void)
{
if (backupSetRequested != NULL)
THROW_FMT(BackupSetInvalidError, "backup set %s is not valid", strZ(backupSetRequested));
else if (targetType == CFGOPTVAL_TYPE_TIME)
else if (targetType == CFGOPTVAL_TYPE_TIME || targetType == CFGOPTVAL_TYPE_LSN)
{
THROW_FMT(
BackupSetInvalidError, "unable to find backup set with stop time less than '%s'",
BackupSetInvalidError, "unable to find backup set with %s '%s'",
targetType == CFGOPTVAL_TYPE_LSN ? "lsn less than or equal to" : "stop time less than",
strZ(cfgOptionDisplay(cfgOptTarget)));
}
else

16
test/src/module/command/helpTest.c
View File

@ -177,13 +177,15 @@ testRun(void)
"specific repository, the --repo option can be passed (e.g. --repo=1). The --set\n"
"option can be passed if a backup other than the latest is desired.\n"
"\n"
"When PITR of --type=time is specified, then the target time must be specified\n"
"with the --target option. If a backup is not specified via the --set option,\n"
"then the configured repositories will be checked, in order, for a backup that\n"
"contains the requested time. If no backup can be found, the latest backup from\n"
"the first repository containing backups will be used. For other types of PITR,\n"
"e.g. xid, the --set option must be provided if the target is prior to the\n"
"latest backup. See Point-in-Time Recovery for more details and examples.\n"
"When PITR of --type=time or --type=lsn is specified, then the target time or\n"
"target lsn must be specified with the --target option. If a backup is not\n"
"specified via the --set option, then the configured repositories will be\n"
"checked, in order, for a backup that contains the requested time or lsn. If no\n"
"matching backup is found, the latest backup from the first repository\n"
"containing backups will be used for --type=time while no backup will be\n"
"selected for --type=lsn. For other types of PITR, e.g. xid, the --set option\n"
"must be provided if the target is prior to the latest backup. See Point-in-Time\n"
"Recovery for more details and examples.\n"
"\n"
"Replication slots are not included per recommendation of PostgreSQL. See\n"
"Backing Up The Data Directory in the PostgreSQL documentation for more\n"

148
test/src/module/command/restoreTest.c
View File

@ -34,6 +34,33 @@ Test data for backup.info
"\"db-version\":\"9.4\"}\n"
#define TEST_RESTORE_BACKUP_INFO \
"[backup:current]\n" \
"20161219-212741F={\"backrest-format\":5,\"backrest-version\":\"2.04\"," \
"\"backup-archive-start\":\"00000007000000000000001C\",\"backup-archive-stop\":\"00000007000000000000001C\"," \
"\"backup-info-repo-size\":3159776,\"backup-info-repo-size-delta\":3159776,\"backup-info-size\":26897030," \
"\"backup-info-size-delta\":26897030,\"backup-lsn-stop\":\"0/1C000101\",\"backup-timestamp-start\":1482182846," \
"\"backup-timestamp-stop\":1482182861,\"backup-type\":\"full\",\"db-id\":1,\"option-archive-check\":true," \
"\"option-archive-copy\":false,\"option-backup-standby\":false,\"option-checksum-page\":false,\"option-compress\":true," \
"\"option-hardlink\":false,\"option-online\":true}\n" \
"20161219-212741F_20161219-212803D={\"backrest-format\":5,\"backrest-version\":\"2.04\"," \
"\"backup-archive-start\":\"00000008000000000000001E\",\"backup-archive-stop\":\"00000008000000000000001E\"," \
"\"backup-info-repo-size\":3159811,\"backup-info-repo-size-delta\":15765,\"backup-info-size\":26897030," \
"\"backup-info-size-delta\":163866,\"backup-lsn-stop\":\"0/1E000101\",\"backup-prior\":\"20161219-212741F\"," \
"\"backup-reference\":[\"20161219-212741F\"],\"backup-timestamp-start\":1482182877,\"backup-timestamp-stop\":1482182883," \
"\"backup-type\":\"diff\",\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false," \
"\"option-backup-standby\":false,\"option-checksum-page\":false,\"option-compress\":true,\"option-hardlink\":false," \
"\"option-online\":true}\n" \
"20161219-212741F_20161219-212918I={\"backrest-format\":5,\"backrest-version\":\"2.04\"," \
"\"backup-archive-start\":null,\"backup-archive-stop\":null," \
"\"backup-info-repo-size\":3159811,\"backup-info-repo-size-delta\":15765,\"backup-info-size\":26897030," \
"\"backup-lsn-stop\":\"0/1E000105\",\"backup-info-size-delta\":163866,\"backup-prior\":\"20161219-212741F\"," \
"\"backup-reference\":[\"20161219-212741F\",\"20161219-212741F_20161219-212803D\"],\"backup-timestamp-start\":1482182884," \
"\"backup-timestamp-stop\":1482182985,\"backup-type\":\"incr\",\"db-id\":1,\"option-archive-check\":true," \
"\"option-archive-copy\":false,\"option-backup-standby\":false,\"option-checksum-page\":false,\"option-compress\":true," \
"\"option-hardlink\":false,\"option-online\":true}\n"
// To verify handling of missing backup-lsn-stop for --type=lsn --target=<lsn> backup
#define TEST_RESTORE_BACKUP_INFO1 \
"[backup:current]\n" \
"20161219-212741F={\"backrest-format\":5,\"backrest-version\":\"2.04\"," \
"\"backup-archive-start\":\"00000007000000000000001C\",\"backup-archive-stop\":\"00000007000000000000001C\"," \
@ -45,18 +72,19 @@ Test data for backup.info
"20161219-212741F_20161219-212803D={\"backrest-format\":5,\"backrest-version\":\"2.04\"," \
"\"backup-archive-start\":\"00000008000000000000001E\",\"backup-archive-stop\":\"00000008000000000000001E\"," \
"\"backup-info-repo-size\":3159811,\"backup-info-repo-size-delta\":15765,\"backup-info-size\":26897030," \
"\"backup-info-size-delta\":163866,\"backup-prior\":\"20161219-212741F\",\"backup-reference\":[\"20161219-212741F\"]," \
"\"backup-timestamp-start\":1482182877,\"backup-timestamp-stop\":1482182883,\"backup-type\":\"diff\",\"db-id\":1," \
"\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":false,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n" \
"\"backup-info-size-delta\":163866,\"backup-lsn-stop\":\"0/1E000101\",\"backup-prior\":\"20161219-212741F\"," \
"\"backup-reference\":[\"20161219-212741F\"],\"backup-timestamp-start\":1482182877,\"backup-timestamp-stop\":1482182883," \
"\"backup-type\":\"diff\",\"db-id\":1,\"option-archive-check\":true,\"option-archive-copy\":false," \
"\"option-backup-standby\":false,\"option-checksum-page\":false,\"option-compress\":true,\"option-hardlink\":false," \
"\"option-online\":true}\n" \
"20161219-212741F_20161219-212918I={\"backrest-format\":5,\"backrest-version\":\"2.04\"," \
"\"backup-archive-start\":null,\"backup-archive-stop\":null," \
"\"backup-info-repo-size\":3159811,\"backup-info-repo-size-delta\":15765,\"backup-info-size\":26897030," \
"\"backup-info-size-delta\":163866,\"backup-prior\":\"20161219-212741F\",\"backup-reference\":[\"20161219-212741F\"," \
"\"20161219-212741F_20161219-212803D\"]," \
"\"backup-timestamp-start\":1482182884,\"backup-timestamp-stop\":1482182985,\"backup-type\":\"incr\",\"db-id\":1," \
"\"option-archive-check\":true,\"option-archive-copy\":false,\"option-backup-standby\":false," \
"\"option-checksum-page\":false,\"option-compress\":true,\"option-hardlink\":false,\"option-online\":true}\n"
"\"backup-lsn-stop\":\"0/1E000105\",\"backup-info-size-delta\":163866,\"backup-prior\":\"20161219-212741F\"," \
"\"backup-reference\":[\"20161219-212741F\",\"20161219-212741F_20161219-212803D\"],\"backup-timestamp-start\":1482182884," \
"\"backup-timestamp-stop\":1482182985,\"backup-type\":\"incr\",\"db-id\":1,\"option-archive-check\":true," \
"\"option-archive-copy\":false,\"option-backup-standby\":false,\"option-checksum-page\":false,\"option-compress\":true," \
"\"option-hardlink\":false,\"option-online\":true}\n"
/***********************************************************************************************************************************
Test restores to be sure they match the manifest
@ -484,6 +512,108 @@ testRun(void)
TEST_RESULT_LOG(
"P00 WARN: repo1: [BackupSetInvalidError] no backup sets to restore\n"
"P00 WARN: repo2: [BackupSetInvalidError] no backup sets to restore");
// -------------------------------------------------------------------------------------------------------------------------
TEST_TITLE("target lsn");
// Match oldest backup on repo 2
argList = strLstNew();
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 1, repoPath2);
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 2, repoPath);
hrnCfgArgKeyRaw(argList, cfgOptPgPath, 1, pgPath);
hrnCfgArgRawZ(argList, cfgOptType, "lsn");
hrnCfgArgRawZ(argList, cfgOptTarget, "0/1C000101");
HRN_CFG_LOAD(cfgCmdRestore, argList);
// Write out backup.info with no current backups to repo1, with current backups to repo2
HRN_INFO_PUT(storageRepoIdxWrite(0), INFO_BACKUP_PATH_FILE, TEST_RESTORE_BACKUP_INFO_DB);
HRN_INFO_PUT(storageRepoIdxWrite(1), INFO_BACKUP_PATH_FILE, TEST_RESTORE_BACKUP_INFO "\n" TEST_RESTORE_BACKUP_INFO_DB);
TEST_ASSIGN(backupData, restoreBackupSet(), "get backup set for lsn 0/1C000101");
TEST_RESULT_STR_Z(backupData.backupSet, "20161219-212741F", "backup set found");
TEST_RESULT_UINT(backupData.repoIdx, 1, "backup set found, repo2");
TEST_RESULT_LOG("P00 WARN: repo1: [BackupSetInvalidError] no backup sets to restore");
// Switch repo paths and target lsn to match newer backup on repo1
argList = strLstNew();
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 1, repoPath);
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 2, repoPath2);
hrnCfgArgKeyRaw(argList, cfgOptPgPath, 1, pgPath);
hrnCfgArgRawZ(argList, cfgOptType, "lsn");
hrnCfgArgRawZ(argList, cfgOptTarget, "0/1E000105");
HRN_CFG_LOAD(cfgCmdRestore, argList);
TEST_ASSIGN(backupData, restoreBackupSet(), "get backup set for lsn 0/1E000105");
TEST_RESULT_STR_Z(backupData.backupSet, "20161219-212741F_20161219-212918I", "backup set found");
TEST_RESULT_UINT(backupData.repoIdx, 0, "backup set found, repo1");
// Log warning if missing backup-lsn-stop is found before finding a match
// Missing backup-lsn-stop in repo1, no backups in repo2, no qualifying auto-selectable backup in either repo
argList = strLstNew();
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 1, repoPath);
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 2, repoPath2);
hrnCfgArgKeyRaw(argList, cfgOptPgPath, 1, pgPath);
hrnCfgArgRawZ(argList, cfgOptType, "lsn");
hrnCfgArgRawZ(argList, cfgOptTarget, "0/1C000101");
// Re-write repo information with set missing backup-lsn-stop
HRN_INFO_PUT(storageRepoIdxWrite(0), INFO_BACKUP_PATH_FILE, TEST_RESTORE_BACKUP_INFO1 "\n" TEST_RESTORE_BACKUP_INFO_DB);
HRN_CFG_LOAD(cfgCmdRestore, argList);
TEST_ERROR(
restoreBackupSet(), BackupSetInvalidError, "unable to find backup set with lsn less than or equal to '0/1C000101'");
TEST_RESULT_LOG(
"P00 WARN: repo1 reached backup from prior version missing required LSN info before finding a match -- backup"
" auto-select has been disabled for this repo\n"
" HINT: you may specify a backup to restore using the --set option.\n"
"P00 WARN: repo2: [BackupSetInvalidError] no backup sets to restore");
// Log warning if missing backup-lsn-stop is found before finding a match
// Missing backup-lsn-stop in repo1, qualifying auto-selectable backup set in repo2
argList = strLstNew();
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 1, repoPath);
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 2, repoPath2);
hrnCfgArgKeyRaw(argList, cfgOptPgPath, 1, pgPath);
hrnCfgArgRawZ(argList, cfgOptType, "lsn");
hrnCfgArgRawZ(argList, cfgOptTarget, "0/1C000102");
// Write repo2 information with data required to find backup
HRN_INFO_PUT(storageRepoIdxWrite(1), INFO_BACKUP_PATH_FILE, TEST_RESTORE_BACKUP_INFO "\n" TEST_RESTORE_BACKUP_INFO_DB);
HRN_CFG_LOAD(cfgCmdRestore, argList);
TEST_ASSIGN(backupData, restoreBackupSet(), "get backup set for lsn 0/1C000102");
TEST_RESULT_STR_Z(backupData.backupSet, "20161219-212741F", "backup set found");
TEST_RESULT_UINT(backupData.repoIdx, 1, "backup set found, repo2");
TEST_RESULT_LOG(
"P00 WARN: repo1 reached backup from prior version missing required LSN info before finding a match -- backup"
" auto-select has been disabled for this repo\n"
" HINT: you may specify a backup to restore using the --set option.");
// No backups to search for qualifying backup set
argList = strLstNew();
hrnCfgArgRawZ(argList, cfgOptStanza, "test1");
hrnCfgArgKeyRaw(argList, cfgOptRepoPath, 2, repoPath);
hrnCfgArgKeyRaw(argList, cfgOptPgPath, 1, pgPath);
hrnCfgArgRawZ(argList, cfgOptType, "lsn");
hrnCfgArgRawZ(argList, cfgOptTarget, "0/1A000102");
// Write repo info with no current backups
HRN_INFO_PUT(storageRepoIdxWrite(0), INFO_BACKUP_PATH_FILE, TEST_RESTORE_BACKUP_INFO_DB);
HRN_CFG_LOAD(cfgCmdRestore, argList);
TEST_ERROR(
restoreBackupSet(), BackupSetInvalidError, "unable to find backup set with lsn less than or equal to '0/1A000102'");
TEST_RESULT_LOG(
"P00 WARN: repo2: [BackupSetInvalidError] no backup sets to restore");
}
// *****************************************************************************************************************************