mirror of
https://github.com/postgrespro/pg_probackup.git
synced 2025-03-30 23:04:31 +02:00
[Issue #68] the backup pinning
This commit is contained in:
Grigory Smolkin
parent
0cd9e35028
commit
c4b49c6abb
@ -38,6 +38,7 @@ Current version - 2.1.5
|
||||
* [Managing the Backup Catalog](#managing-the-backup-catalog)
|
||||
* [Viewing WAL Archive Information](#viewing-wal-archive-information)
|
||||
* [Configuring Backup Retention Policy](#configuring-backup-retention-policy)
|
||||
* [Pinning a Backup]
|
||||
* [Merging Backups](#merging-backups)
|
||||
* [Deleting Backups](#deleting-backups)
|
||||
|
||||
@ -49,6 +50,7 @@ Current version - 2.1.5
|
||||
* [add-instance](#add-instance)
|
||||
* [del-instance](#del-instance)
|
||||
* [set-config](#set-config)
|
||||
* [set-backup](#set-backup)
|
||||
* [show-config](#show-config)
|
||||
* [show](#show)
|
||||
* [backup](#backup)
|
||||
@ -63,6 +65,7 @@ Current version - 2.1.5
|
||||
* [Common Options](#common-options)
|
||||
* [Recovery Target Options](#recovery-target-options)
|
||||
* [Retention Options](#retention-options)
|
||||
* [Pinning Options](#pinning-options)
|
||||
* [Logging Options](#logging-options)
|
||||
* [Connection Options](#connection-options)
|
||||
* [Compression Options](#compression-options)
|
||||
@ -89,6 +92,8 @@ Current version - 2.1.5
|
||||
|
||||
`pg_probackup set-config -B backup_dir --instance instance_name [option...]`
|
||||
|
||||
`pg_probackup set-backup -B backup_dir --instance instance_name -i backup_id [option...]`
|
||||
|
||||
`pg_probackup show-config -B backup_dir --instance instance_name [--format=format]`
|
||||
|
||||
`pg_probackup show -B backup_dir [option...]`
|
||||
@ -775,6 +780,7 @@ start-time = '2017-05-16 12:57:29'
|
||||
end-time = '2017-05-16 12:57:31'
|
||||
recovery-xid = 597
|
||||
recovery-time = '2017-05-16 12:57:31'
|
||||
expire-time = '2020-05-16 12:57:31'
|
||||
data-bytes = 22288792
|
||||
wal-bytes = 16777216
|
||||
uncompressed-bytes = 39961833
|
||||
@ -794,6 +800,7 @@ Detailed output has additional attributes:
|
||||
- program-version — full version of pg_probackup binary used to create backup.
|
||||
- start-time — the backup starting time.
|
||||
- end-time — the backup ending time.
|
||||
- expire-time — if the backup was pinned, then until this point in time the backup cannot be removed by retention purge.
|
||||
- uncompressed-bytes — size of the data files before adding page headers and applying compression. You can evaluate the effectiveness of compression by comparing 'uncompressed-bytes' to 'data-bytes' if compression if used.
|
||||
- pgdata-bytes — size of the PostgreSQL cluster data files at the time of backup. You can evaluate the effectiveness of incremental backup by comparing 'pgdata-bytes' to 'uncompressed-bytes'.
|
||||
- recovery-xid — current transaction id at the moment of backup ending.
|
||||
@ -1171,6 +1178,42 @@ BACKUP INSTANCE 'node'
|
||||
|
||||
>NOTE: The Time field for the merged backup displays the time required for the merge.
|
||||
|
||||
#### Pinning a Backup
|
||||
|
||||
If you have the necessity to exclude certain backups from established retention policy then it is possible to pin a backup for an arbitrary amount of time. Example:
|
||||
|
||||
pg_probackup set-backup -B backup_dir --instance instance_name -i backup_id --ttl=30d
|
||||
|
||||
This command will set `expire-time` of specified backup to 30 days starting from backup `recovery-time` attribute. Basically 'expire-time = recovery-time + ttl'.
|
||||
|
||||
You can set `expire-time` explicitly using `--expire-time` option. Example:
|
||||
|
||||
pg_probackup set-backup -B backup_dir --instance instance_name -i backup_id --expire-time='2020-01-01 00:00:00+03'
|
||||
|
||||
Alternatively you can use the `--ttl` and `--expire-time` options with the [backup](#backup) command to pin newly created backup:
|
||||
|
||||
pg_probackup backup -B backup_dir --instance instance_name -b FULL --ttl=30d
|
||||
pg_probackup backup -B backup_dir --instance instance_name -b FULL --expire-time='2020-01-01 00:00:00+03'
|
||||
|
||||
You can determine the fact that backup is pinned and check due expire time by looking up 'expire-time' attribute in backup metadata via (show)[#show] command:
|
||||
|
||||
pg_probackup show --instance instance_name -i backup_id
|
||||
|
||||
Pinned backup has `expire-time` attribute:
|
||||
```
|
||||
...
|
||||
recovery-time = '2017-05-16 12:57:31'
|
||||
expire-time = '2020-01-01 00:00:00+03'
|
||||
data-bytes = 22288792
|
||||
...
|
||||
```
|
||||
|
||||
You can unpin a backup by setting `--ttl` option to zero using `set-backup` command. Example:
|
||||
|
||||
pg_probackup set-backup -B backup_dir --instance instance_name -i backup_id --ttl=0
|
||||
|
||||
Only pinned backups have `expire-time` attribute in backup metadata.
|
||||
|
||||
### Merging Backups
|
||||
|
||||
As you take more and more incremental backups, the total size of the backup catalog can substantially grow. To save disk space, you can merge incremental backups to their parent full backup by running the merge command, specifying the backup ID of the most recent incremental backup you would like to merge:
|
||||
@ -1267,6 +1310,15 @@ For all available settings, see the [Options](#options) section.
|
||||
|
||||
It is **not recommended** to edit pg_probackup.conf manually.
|
||||
|
||||
#### set-backup
|
||||
|
||||
pg_probackup set-backup -B backup_dir --instance instance_name -i backup_id
|
||||
{--ttl=ttl | --expire-time=time} [--help]
|
||||
|
||||
Sets the provided backup-specific settings into the backup.control configuration file, or modifies previously defined values.
|
||||
|
||||
For all available settings, see the section [Pinning Options](#pinning-options).
|
||||
|
||||
#### show-config
|
||||
|
||||
pg_probackup show-config -B backup_dir --instance instance_name [--format=plain|json]
|
||||
@ -1296,7 +1348,7 @@ For details on usage, see the sections [Managing the Backup Catalog](#managing-t
|
||||
[-w --no-password] [-W --password]
|
||||
[--archive-timeout=timeout] [--external-dirs=external_directory_path]
|
||||
[connection_options] [compression_options] [remote_options]
|
||||
[retention_options] [logging_options]
|
||||
[retention_options] [pinning_options] [logging_options]
|
||||
|
||||
Creates a backup copy of the PostgreSQL instance. The *backup_mode* option specifies the backup mode to use.
|
||||
|
||||
@ -1342,7 +1394,7 @@ Disables block-level checksum verification to speed up backup.
|
||||
--no-validate
|
||||
Skips automatic validation after successfull backup. You can use this flag if you validate backups regularly and would like to save time when running backup operations.
|
||||
|
||||
Additionally [Connection Options](#connection-options), [Retention Options](#retention-options), [Remote Mode Options](#remote-mode-options), [Compression Options](#compression-options), [Logging Options](#logging-options) and [Common Options](#common-options) can be used.
|
||||
Additionally [Connection Options](#connection-options), [Retention Options](#retention-options), [Pinning Options](#pinning-options), [Remote Mode Options](#remote-mode-options), [Compression Options](#compression-options), [Logging Options](#logging-options) and [Common Options](#common-options) can be used.
|
||||
|
||||
For details on usage, see the section [Creating a Backup](#creating-a-backup).
|
||||
|
||||
@ -1565,6 +1617,18 @@ Merges the oldest incremental backup that satisfies the requirements of retentio
|
||||
--dry-run
|
||||
Displays the current status of all the available backups, without deleting or merging expired backups, if any.
|
||||
|
||||
##### Pinning Options
|
||||
|
||||
You can use these options together with [backup](#backup) and [set-delete](#set-backup) commands.
|
||||
|
||||
For details on backup pinning, see the section [Pinning a Backup](#pinning-a-backup).
|
||||
|
||||
--ttl=ttl
|
||||
Specifies the amount of time the backup should be pinned. Must be a positive integer. The zero value unpin already pinned backup. Supported units: ms, s, min, h, d (s by default). Example: `--ttl=30d`.
|
||||
|
||||
--expire-time=time
|
||||
Specifies the timestamp up to which the backup will stay pinned. Must be a ISO-8601 complaint timestamp. Example: `--expire-time='2020-01-01 00:00:00+03'`
|
||||
|
||||
#### Logging Options
|
||||
|
||||
You can use these options with any command.
|
||||
|
||||
12
src/backup.c
12
src/backup.c
@ -695,7 +695,8 @@ pgdata_basic_setup(ConnectionOptions conn_opt, PGNodeInfo *nodeInfo)
|
||||
* Entry point of pg_probackup BACKUP subcommand.
|
||||
*/
|
||||
int
|
||||
do_backup(time_t start_time, bool no_validate)
|
||||
do_backup(time_t start_time, bool no_validate,
|
||||
pgSetBackupParams *set_backup_params)
|
||||
{
|
||||
PGconn *backup_conn = NULL;
|
||||
PGNodeInfo nodeInfo;
|
||||
@ -801,6 +802,15 @@ do_backup(time_t start_time, bool no_validate)
|
||||
current.status = BACKUP_STATUS_DONE;
|
||||
write_backup(¤t);
|
||||
|
||||
/* Pin backup if requested */
|
||||
if (set_backup_params &&
|
||||
(set_backup_params->ttl > 0 ||
|
||||
set_backup_params->expire_time > 0))
|
||||
{
|
||||
if (!pin_backup(¤t, set_backup_params))
|
||||
elog(ERROR, "Failed to pin the backup %s", base36enc(current.backup_id));
|
||||
}
|
||||
|
||||
if (!no_validate)
|
||||
pgBackupValidate(¤t, NULL);
|
||||
|
||||
|
||||
@ -987,6 +987,81 @@ get_oldest_backup(timelineInfo *tlinfo)
|
||||
return oldest_backup;
|
||||
}
|
||||
|
||||
/*
|
||||
* Overwrite backup metadata.
|
||||
*/
|
||||
void
|
||||
do_set_backup(const char *instance_name, time_t backup_id,
|
||||
pgSetBackupParams *set_backup_params)
|
||||
{
|
||||
pgBackup *target_backup = NULL;
|
||||
parray *backup_list = NULL;
|
||||
|
||||
if (!set_backup_params)
|
||||
elog(ERROR, "Nothing to set by 'set-backup' command");
|
||||
|
||||
backup_list = catalog_get_backup_list(instance_name, backup_id);
|
||||
if (parray_num(backup_list) != 1)
|
||||
elog(ERROR, "Failed to find backup %s", base36enc(backup_id));
|
||||
|
||||
target_backup = (pgBackup *) parray_get(backup_list, 0);
|
||||
|
||||
if (!pin_backup(target_backup, set_backup_params))
|
||||
elog(ERROR, "Failed to pin the backup %s", base36enc(backup_id));
|
||||
}
|
||||
|
||||
/*
|
||||
* Set 'expire-time' attribute based on set_backup_params, or unpin backup
|
||||
* if ttl is equal to zero.
|
||||
*/
|
||||
bool
|
||||
pin_backup(pgBackup *target_backup, pgSetBackupParams *set_backup_params)
|
||||
{
|
||||
|
||||
if (target_backup->recovery_time <= 0)
|
||||
elog(ERROR, "Failed to set 'expire-time' for backup %s: invalid 'recovery-time'",
|
||||
base36enc(target_backup->backup_id));
|
||||
|
||||
/* Pin comes from ttl */
|
||||
if (set_backup_params->ttl > 0)
|
||||
target_backup->expire_time = target_backup->recovery_time + set_backup_params->ttl;
|
||||
/* Unpin backup */
|
||||
else if (set_backup_params->ttl == 0)
|
||||
{
|
||||
/* If backup was not pinned in the first place,
|
||||
* then there is nothing to unpin.
|
||||
*/
|
||||
if (target_backup->expire_time == 0)
|
||||
{
|
||||
elog(WARNING, "Backup %s is not pinned, nothing to unpin",
|
||||
base36enc(target_backup->start_time));
|
||||
return false;
|
||||
}
|
||||
target_backup->expire_time = 0;
|
||||
}
|
||||
/* Pin comes from expire-time */
|
||||
else if (set_backup_params->expire_time > 0)
|
||||
target_backup->expire_time = set_backup_params->expire_time;
|
||||
else
|
||||
return false;
|
||||
|
||||
/* Update backup.control */
|
||||
write_backup(target_backup);
|
||||
|
||||
if (set_backup_params->ttl > 0 || set_backup_params->expire_time > 0)
|
||||
{
|
||||
char expire_timestamp[100];
|
||||
|
||||
time2iso(expire_timestamp, lengthof(expire_timestamp), target_backup->expire_time);
|
||||
elog(INFO, "Backup %s is pinned until '%s'", base36enc(target_backup->start_time),
|
||||
expire_timestamp);
|
||||
}
|
||||
else
|
||||
elog(INFO, "Backup %s is unpinned", base36enc(target_backup->start_time));
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/*
|
||||
* Write information about backup.in to stream "out".
|
||||
*/
|
||||
@ -1041,6 +1116,11 @@ pgBackupWriteControl(FILE *out, pgBackup *backup)
|
||||
time2iso(timestamp, lengthof(timestamp), backup->recovery_time);
|
||||
fio_fprintf(out, "recovery-time = '%s'\n", timestamp);
|
||||
}
|
||||
if (backup->expire_time > 0)
|
||||
{
|
||||
time2iso(timestamp, lengthof(timestamp), backup->expire_time);
|
||||
fio_fprintf(out, "expire-time = '%s'\n", timestamp);
|
||||
}
|
||||
|
||||
/*
|
||||
* Size of PGDATA directory. The size does not include size of related
|
||||
@ -1284,6 +1364,7 @@ readBackupControlFile(const char *path)
|
||||
{'t', 0, "end-time", &backup->end_time, SOURCE_FILE_STRICT},
|
||||
{'U', 0, "recovery-xid", &backup->recovery_xid, SOURCE_FILE_STRICT},
|
||||
{'t', 0, "recovery-time", &backup->recovery_time, SOURCE_FILE_STRICT},
|
||||
{'t', 0, "expire-time", &backup->expire_time, SOURCE_FILE_STRICT},
|
||||
{'I', 0, "data-bytes", &backup->data_bytes, SOURCE_FILE_STRICT},
|
||||
{'I', 0, "wal-bytes", &backup->wal_bytes, SOURCE_FILE_STRICT},
|
||||
{'I', 0, "uncompressed-bytes", &backup->uncompressed_bytes, SOURCE_FILE_STRICT},
|
||||
@ -1529,6 +1610,7 @@ pgBackupInit(pgBackup *backup)
|
||||
backup->end_time = (time_t) 0;
|
||||
backup->recovery_xid = 0;
|
||||
backup->recovery_time = (time_t) 0;
|
||||
backup->expire_time = (time_t) 0;
|
||||
|
||||
backup->data_bytes = BYTES_INVALID;
|
||||
backup->wal_bytes = BYTES_INVALID;
|
||||
|
||||
26
src/delete.c
26
src/delete.c
@ -292,6 +292,20 @@ do_retention_internal(parray *backup_list, parray *to_keep_list, parray *to_purg
|
||||
* FULL CORRUPT out of retention
|
||||
*/
|
||||
|
||||
/* Save backup from purge if backup is pinned and
|
||||
* expire date is not yet due.
|
||||
*/
|
||||
if ((backup->expire_time > 0) &&
|
||||
(backup->expire_time > current_time))
|
||||
{
|
||||
char expire_timestamp[100];
|
||||
time2iso(expire_timestamp, lengthof(expire_timestamp), backup->expire_time);
|
||||
|
||||
elog(LOG, "Backup %s is pinned until '%s', retain",
|
||||
base36enc(backup->start_time), expire_timestamp);
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Add backup to purge_list */
|
||||
elog(VERBOSE, "Mark backup %s for purge.", base36enc(backup->start_time));
|
||||
parray_append(to_purge_list, backup);
|
||||
@ -355,6 +369,7 @@ do_retention_internal(parray *backup_list, parray *to_keep_list, parray *to_purg
|
||||
for (i = 0; i < parray_num(backup_list); i++)
|
||||
{
|
||||
char *action = "Active";
|
||||
uint32 pinning_window = 0;
|
||||
|
||||
pgBackup *backup = (pgBackup *) parray_get(backup_list, i);
|
||||
|
||||
@ -364,7 +379,11 @@ do_retention_internal(parray *backup_list, parray *to_keep_list, parray *to_purg
|
||||
if (backup->recovery_time == 0)
|
||||
actual_window = 0;
|
||||
else
|
||||
actual_window = (current_time - backup->recovery_time)/(60 * 60 * 24);
|
||||
actual_window = (current_time - backup->recovery_time)/(3600 * 24);
|
||||
|
||||
/* For pinned backups show expire date */
|
||||
if (backup->expire_time > 0 && backup->expire_time > backup->recovery_time)
|
||||
pinning_window = (backup->expire_time - backup->recovery_time)/(3600 * 24);
|
||||
|
||||
/* TODO: add ancestor(chain full backup) ID */
|
||||
elog(INFO, "Backup %s, mode: %s, status: %s. Redundancy: %i/%i, Time Window: %ud/%ud. %s",
|
||||
@ -373,7 +392,8 @@ do_retention_internal(parray *backup_list, parray *to_keep_list, parray *to_purg
|
||||
status2str(backup->status),
|
||||
cur_full_backup_num,
|
||||
instance_config.retention_redundancy,
|
||||
actual_window, instance_config.retention_window,
|
||||
actual_window,
|
||||
pinning_window ? pinning_window : instance_config.retention_window,
|
||||
action);
|
||||
|
||||
if (backup->backup_mode == BACKUP_MODE_FULL)
|
||||
@ -576,7 +596,7 @@ do_retention_purge(parray *to_keep_list, parray *to_purge_list)
|
||||
{
|
||||
|
||||
/* We must not delete this backup, evict it from purge list */
|
||||
elog(LOG, "Retain backup %s from purge because his "
|
||||
elog(LOG, "Retain backup %s because his "
|
||||
"descendant %s is guarded by retention",
|
||||
base36enc(delete_backup->start_time), keeped_backup_id);
|
||||
|
||||
|
||||
31
src/help.c
31
src/help.c
@ -16,6 +16,7 @@ static void help_validate(void);
|
||||
static void help_show(void);
|
||||
static void help_delete(void);
|
||||
static void help_merge(void);
|
||||
static void help_set_backup(void);
|
||||
static void help_set_config(void);
|
||||
static void help_show_config(void);
|
||||
static void help_add_instance(void);
|
||||
@ -41,6 +42,8 @@ help_command(char *command)
|
||||
help_delete();
|
||||
else if (strcmp(command, "merge") == 0)
|
||||
help_merge();
|
||||
else if (strcmp(command, "set-backup") == 0)
|
||||
help_set_backup();
|
||||
else if (strcmp(command, "set-config") == 0)
|
||||
help_set_config();
|
||||
else if (strcmp(command, "show-config") == 0)
|
||||
@ -101,6 +104,10 @@ help_pg_probackup(void)
|
||||
printf(_(" [--archive-port=port] [--archive-user=username]\n"));
|
||||
printf(_(" [--help]\n"));
|
||||
|
||||
printf(_("\n %s set-backup -B backup-path --instance=instance_name\n"), PROGRAM_NAME);
|
||||
printf(_(" -i backup-id [--ttl] [--expire-time]\n"));
|
||||
printf(_(" [--help]\n"));
|
||||
|
||||
printf(_("\n %s show-config -B backup-path --instance=instance_name\n"), PROGRAM_NAME);
|
||||
printf(_(" [--format=format]\n"));
|
||||
printf(_(" [--help]\n"));
|
||||
@ -130,8 +137,10 @@ help_pg_probackup(void)
|
||||
printf(_(" [--remote-proto] [--remote-host]\n"));
|
||||
printf(_(" [--remote-port] [--remote-path] [--remote-user]\n"));
|
||||
printf(_(" [--ssh-options]\n"));
|
||||
printf(_(" [--ttl] [--expire-time]\n"));
|
||||
printf(_(" [--help]\n"));
|
||||
|
||||
|
||||
printf(_("\n %s restore -B backup-path --instance=instance_name\n"), PROGRAM_NAME);
|
||||
printf(_(" [-D pgdata-path] [-i backup-id] [-j num-threads]\n"));
|
||||
printf(_(" [--recovery-target-time=time|--recovery-target-xid=xid\n"));
|
||||
@ -262,7 +271,8 @@ help_backup(void)
|
||||
printf(_(" [-w --no-password] [-W --password]\n"));
|
||||
printf(_(" [--remote-proto] [--remote-host]\n"));
|
||||
printf(_(" [--remote-port] [--remote-path] [--remote-user]\n"));
|
||||
printf(_(" [--ssh-options]\n\n"));
|
||||
printf(_(" [--ssh-options]\n"));
|
||||
printf(_(" [--ttl] [--expire-time]\n\n"));
|
||||
|
||||
printf(_(" -B, --backup-path=backup-path location of the backup storage area\n"));
|
||||
printf(_(" -b, --backup-mode=backup-mode backup mode=FULL|PAGE|DELTA|PTRACK\n"));
|
||||
@ -314,6 +324,12 @@ help_backup(void)
|
||||
printf(_(" number of days of recoverability; 0 disables; (default: 0)\n"));
|
||||
printf(_(" --dry-run perform a trial run without any changes\n"));
|
||||
|
||||
printf(_("\n Pinning options:\n"));
|
||||
printf(_(" --ttl=ttl pin backup for specified amount of time; 0 unpin\n"));
|
||||
printf(_(" available units: 'ms', 's', 'min', 'h', 'd' (default: s)\n"));
|
||||
printf(_(" --expire-time=time pin backup until specified time stamp\n"));
|
||||
printf(_(" (example: --expire-time='2024-01-01 00:00:00+03')\n"));
|
||||
|
||||
printf(_("\n Compression options:\n"));
|
||||
printf(_(" --compress alias for --compress-algorithm='zlib' and --compress-level=1\n"));
|
||||
printf(_(" --compress-algorithm=compress-algorithm\n"));
|
||||
@ -659,6 +675,19 @@ help_merge(void)
|
||||
printf(_(" available units: 'ms', 's', 'min', 'h', 'd' (default: min)\n\n"));
|
||||
}
|
||||
|
||||
static void
|
||||
help_set_backup(void)
|
||||
{
|
||||
printf(_("\n%s set-backup -B backup-path --instance=instance_name\n"), PROGRAM_NAME);
|
||||
printf(_(" -i backup-id\n"));
|
||||
printf(_(" [--ttl] [--expire-time]\n\n"));
|
||||
|
||||
printf(_(" --ttl=ttl pin backup for specified amount of time; 0 unpin\n"));
|
||||
printf(_(" available units: 'ms', 's', 'min', 'h', 'd' (default: s)\n"));
|
||||
printf(_(" --expire-time=time pin backup until specified time stamp\n"));
|
||||
printf(_(" (example: --expire-time='2024-01-01 00:00:00+03')\n"));
|
||||
}
|
||||
|
||||
static void
|
||||
help_set_config(void)
|
||||
{
|
||||
|
||||
@ -42,6 +42,7 @@ typedef enum ProbackupSubcmd
|
||||
MERGE_CMD,
|
||||
SHOW_CMD,
|
||||
SET_CONFIG_CMD,
|
||||
SET_BACKUP_CMD,
|
||||
SHOW_CONFIG_CMD,
|
||||
CHECKDB_CMD
|
||||
} ProbackupSubcmd;
|
||||
@ -128,6 +129,11 @@ static bool file_overwrite = false;
|
||||
ShowFormat show_format = SHOW_PLAIN;
|
||||
bool show_archive = false;
|
||||
|
||||
/* set-backup options */
|
||||
int64 ttl = -1;
|
||||
static char *expire_time_string = NULL;
|
||||
static pgSetBackupParams *set_backup_params = NULL;
|
||||
|
||||
/* current settings */
|
||||
pgBackup current;
|
||||
static ProbackupSubcmd backup_subcmd = NO_CMD;
|
||||
@ -205,6 +211,9 @@ static ConfigOption cmd_options[] =
|
||||
/* show options */
|
||||
{ 'f', 153, "format", opt_show_format, SOURCE_CMD_STRICT },
|
||||
{ 'b', 161, "archive", &show_archive, SOURCE_CMD_STRICT },
|
||||
/* set-backup options */
|
||||
{ 'I', 170, "ttl", &ttl, SOURCE_CMD_STRICT, SOURCE_DEFAULT, 0, OPTION_UNIT_S, option_get_value},
|
||||
{ 's', 171, "expire-time", &expire_time_string, SOURCE_CMD_STRICT },
|
||||
|
||||
/* options for backward compatibility */
|
||||
{ 's', 136, "time", &target_time, SOURCE_CMD_STRICT },
|
||||
@ -300,6 +309,8 @@ main(int argc, char *argv[])
|
||||
backup_subcmd = SHOW_CMD;
|
||||
else if (strcmp(argv[1], "set-config") == 0)
|
||||
backup_subcmd = SET_CONFIG_CMD;
|
||||
else if (strcmp(argv[1], "set-backup") == 0)
|
||||
backup_subcmd = SET_BACKUP_CMD;
|
||||
else if (strcmp(argv[1], "show-config") == 0)
|
||||
backup_subcmd = SHOW_CONFIG_CMD;
|
||||
else if (strcmp(argv[1], "checkdb") == 0)
|
||||
@ -361,7 +372,9 @@ main(int argc, char *argv[])
|
||||
backup_subcmd == RESTORE_CMD ||
|
||||
backup_subcmd == VALIDATE_CMD ||
|
||||
backup_subcmd == DELETE_CMD ||
|
||||
backup_subcmd == MERGE_CMD)
|
||||
backup_subcmd == MERGE_CMD ||
|
||||
backup_subcmd == SET_CONFIG_CMD ||
|
||||
backup_subcmd == SET_BACKUP_CMD)
|
||||
{
|
||||
int i,
|
||||
len = 0,
|
||||
@ -588,6 +601,7 @@ main(int argc, char *argv[])
|
||||
backup_subcmd != VALIDATE_CMD &&
|
||||
backup_subcmd != DELETE_CMD &&
|
||||
backup_subcmd != MERGE_CMD &&
|
||||
backup_subcmd != SET_BACKUP_CMD &&
|
||||
backup_subcmd != SHOW_CMD)
|
||||
elog(ERROR, "Cannot use -i (--backup-id) option together with the \"%s\" command",
|
||||
command_name);
|
||||
@ -658,6 +672,32 @@ main(int argc, char *argv[])
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Parse set-backup options into set_backup_params structure.
|
||||
*/
|
||||
if (backup_subcmd == SET_BACKUP_CMD || backup_subcmd == BACKUP_CMD)
|
||||
{
|
||||
time_t expire_time = 0;
|
||||
|
||||
if (expire_time_string && ttl >= 0)
|
||||
elog(ERROR, "You cannot specify '--expire-time' and '--ttl' options together");
|
||||
|
||||
/* Parse string to seconds */
|
||||
if (expire_time_string)
|
||||
{
|
||||
if (!parse_time(expire_time_string, &expire_time, false))
|
||||
elog(ERROR, "Invalid value for '--expire-time' option: '%s'",
|
||||
expire_time_string);
|
||||
}
|
||||
|
||||
if (expire_time > 0 || ttl >= 0)
|
||||
{
|
||||
set_backup_params = pgut_new(pgSetBackupParams);
|
||||
set_backup_params->ttl = ttl;
|
||||
set_backup_params->expire_time = expire_time;
|
||||
}
|
||||
}
|
||||
|
||||
/* sanity */
|
||||
if (backup_subcmd == VALIDATE_CMD && restore_params->no_validate)
|
||||
elog(ERROR, "You cannot specify \"--no-validate\" option with the \"%s\" command",
|
||||
@ -694,7 +734,7 @@ main(int argc, char *argv[])
|
||||
elog(ERROR, "required parameter not specified: BACKUP_MODE "
|
||||
"(-b, --backup-mode)");
|
||||
|
||||
return do_backup(start_time, no_validate);
|
||||
return do_backup(start_time, no_validate, set_backup_params);
|
||||
}
|
||||
case RESTORE_CMD:
|
||||
return do_restore_or_validate(current.backup_id,
|
||||
@ -738,6 +778,11 @@ main(int argc, char *argv[])
|
||||
case SET_CONFIG_CMD:
|
||||
do_set_config(false);
|
||||
break;
|
||||
case SET_BACKUP_CMD:
|
||||
if (!backup_id_string)
|
||||
elog(ERROR, "You must specify parameter (-i, --backup-id) for 'set-backup' command");
|
||||
do_set_backup(instance_name, current.backup_id, set_backup_params);
|
||||
break;
|
||||
case CHECKDB_CMD:
|
||||
do_checkdb(need_amcheck,
|
||||
instance_config.conn_opt, instance_config.pgdata);
|
||||
|
||||
@ -310,6 +310,7 @@ struct pgBackup
|
||||
time_t recovery_time; /* Earliest moment for which you can restore
|
||||
* the state of the database cluster using
|
||||
* this backup */
|
||||
time_t expire_time; /* Backup expiration date */
|
||||
TransactionId recovery_xid; /* Earliest xid for which you can restore
|
||||
* the state of the database cluster using
|
||||
* this backup */
|
||||
@ -390,6 +391,18 @@ typedef struct pgRestoreParams
|
||||
parray *partial_db_list;
|
||||
} pgRestoreParams;
|
||||
|
||||
/* Options needed for set-backup command */
|
||||
typedef struct pgSetBackupParams
|
||||
{
|
||||
int64 ttl; /* amount of time backup must be pinned
|
||||
* -1 - do nothing
|
||||
* 0 - disable pinning
|
||||
*/
|
||||
time_t expire_time; /* Point in time before which backup
|
||||
* must be pinned.
|
||||
*/
|
||||
} pgSetBackupParams;
|
||||
|
||||
typedef struct
|
||||
{
|
||||
const char *from_root;
|
||||
@ -572,7 +585,8 @@ extern char** commands_args;
|
||||
extern const char *pgdata_exclude_dir[];
|
||||
|
||||
/* in backup.c */
|
||||
extern int do_backup(time_t start_time, bool no_validate);
|
||||
extern int do_backup(time_t start_time, bool no_validate,
|
||||
pgSetBackupParams *set_backup_params);
|
||||
extern void do_checkdb(bool need_amcheck, ConnectionOptions conn_opt,
|
||||
char *pgdata);
|
||||
extern BackupMode parse_backup_mode(const char *value);
|
||||
@ -668,6 +682,10 @@ extern pgBackup *catalog_get_last_data_backup(parray *backup_list,
|
||||
TimeLineID tli,
|
||||
time_t current_start_time);
|
||||
extern parray *catalog_get_timelines(InstanceConfig *instance);
|
||||
extern void do_set_backup(const char *instance_name, time_t backup_id,
|
||||
pgSetBackupParams *set_backup_params);
|
||||
extern bool pin_backup(pgBackup *target_backup,
|
||||
pgSetBackupParams *set_backup_params);
|
||||
extern void pgBackupWriteControl(FILE *out, pgBackup *backup);
|
||||
extern void write_backup_filelist(pgBackup *backup, parray *files,
|
||||
const char *root, parray *external_list);
|
||||
|
||||
@ -374,6 +374,12 @@ print_backup_json_object(PQExpBuffer buf, pgBackup *backup)
|
||||
json_add_value(buf, "recovery-time", timestamp, json_level, true);
|
||||
}
|
||||
|
||||
if (backup->expire_time > 0)
|
||||
{
|
||||
time2iso(timestamp, lengthof(timestamp), backup->expire_time);
|
||||
json_add_value(buf, "expire-time", timestamp, json_level, true);
|
||||
}
|
||||
|
||||
if (backup->data_bytes != BYTES_INVALID)
|
||||
{
|
||||
json_add_key(buf, "data-bytes", json_level);
|
||||
|
||||
@ -1095,6 +1095,8 @@ parse_uint64(const char *value, uint64 *result, int flags)
|
||||
*
|
||||
* If utc_default is true, then if timezone offset isn't specified tz will be
|
||||
* +00:00.
|
||||
*
|
||||
* TODO: '0' converted into '2000-01-01 00:00:00'. Example: set-backup --expire-time=0
|
||||
*/
|
||||
bool
|
||||
parse_time(const char *value, time_t *result, bool utc_default)
|
||||
@ -1202,7 +1204,7 @@ parse_time(const char *value, time_t *result, bool utc_default)
|
||||
&tm.tm_hour, &tm.tm_min, &tm.tm_sec, junk);
|
||||
free(tmp);
|
||||
|
||||
if (i < 1 || 6 < i)
|
||||
if (i < 3 || i > 6)
|
||||
return false;
|
||||
|
||||
/* adjust year */
|
||||
|
||||
@ -6,7 +6,7 @@ from . import init, merge, option, show, compatibility, \
|
||||
retention, pgpro560, pgpro589, pgpro2068, false_positive, replica, \
|
||||
compression, page, ptrack, archive, exclude, cfs_backup, cfs_restore, \
|
||||
cfs_validate_backup, auth_test, time_stamp, snapfs, logging, \
|
||||
locking, remote, external, config, checkdb
|
||||
locking, remote, external, config, checkdb, set_backup
|
||||
|
||||
|
||||
def load_tests(loader, tests, pattern):
|
||||
@ -48,6 +48,7 @@ def load_tests(loader, tests, pattern):
|
||||
suite.addTests(loader.loadTestsFromModule(replica))
|
||||
suite.addTests(loader.loadTestsFromModule(restore))
|
||||
suite.addTests(loader.loadTestsFromModule(retention))
|
||||
suite.addTests(loader.loadTestsFromModule(set_backup))
|
||||
suite.addTests(loader.loadTestsFromModule(show))
|
||||
suite.addTests(loader.loadTestsFromModule(snapfs))
|
||||
suite.addTests(loader.loadTestsFromModule(time_stamp))
|
||||
|
||||
@ -737,6 +737,22 @@ class ProbackupTest(object):
|
||||
|
||||
return self.run_pb(cmd + options, old_binary=old_binary)
|
||||
|
||||
def set_backup(self, backup_dir, instance, backup_id=False,
|
||||
old_binary=False, options=[]):
|
||||
|
||||
cmd = [
|
||||
'set-backup',
|
||||
'-B', backup_dir
|
||||
]
|
||||
|
||||
if instance:
|
||||
cmd = cmd + ['--instance={0}'.format(instance)]
|
||||
|
||||
if backup_id:
|
||||
cmd = cmd + ['-i', backup_id]
|
||||
|
||||
return self.run_pb(cmd + options, old_binary=old_binary)
|
||||
|
||||
def del_instance(self, backup_dir, instance, old_binary=False):
|
||||
|
||||
return self.run_pb([
|
||||
@ -788,7 +804,7 @@ class ProbackupTest(object):
|
||||
|
||||
def checkdb_node(
|
||||
self, backup_dir=False, instance=False, data_dir=False,
|
||||
options=[], async=False, gdb=False, old_binary=False
|
||||
options=[], asynchronous=False, gdb=False, old_binary=False
|
||||
):
|
||||
|
||||
cmd_list = ["checkdb"]
|
||||
@ -802,7 +818,7 @@ class ProbackupTest(object):
|
||||
if data_dir:
|
||||
cmd_list += ["-D", data_dir]
|
||||
|
||||
return self.run_pb(cmd_list + options, async, gdb, old_binary)
|
||||
return self.run_pb(cmd_list + options, asynchronous, gdb, old_binary)
|
||||
|
||||
def merge_backup(
|
||||
self, backup_dir, instance, backup_id, asynchronous=False,
|
||||
|
||||
@ -693,7 +693,6 @@ class RetentionTest(ProbackupTest, unittest.TestCase):
|
||||
|
||||
page_id_b3 = self.backup_node(
|
||||
backup_dir, 'node', node, backup_type='page')
|
||||
|
||||
pgdata_b3 = self.pgdata_content(node.data_dir)
|
||||
|
||||
pgbench = node.pgbench(options=['-t', '10', '-c', '2'])
|
||||
|
||||
241
tests/set_backup.py
Normal file
241
tests/set_backup.py
Normal file
@ -0,0 +1,241 @@
|
||||
import unittest
|
||||
import subprocess
|
||||
import os
|
||||
from .helpers.ptrack_helpers import ProbackupTest, ProbackupException
|
||||
from sys import exit
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
module_name = 'set_backup'
|
||||
|
||||
|
||||
class SetBackupTest(ProbackupTest, unittest.TestCase):
|
||||
|
||||
# @unittest.expectedFailure
|
||||
# @unittest.skip("skip")
|
||||
def test_set_backup_sanity(self):
|
||||
"""general sanity for set-backup command"""
|
||||
fname = self.id().split('.')[3]
|
||||
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
|
||||
node = self.make_simple_node(
|
||||
base_dir=os.path.join(module_name, fname, 'node'),
|
||||
set_replication=True,
|
||||
initdb_params=['--data-checksums'])
|
||||
|
||||
self.init_pb(backup_dir)
|
||||
self.add_instance(backup_dir, 'node', node)
|
||||
node.slow_start()
|
||||
|
||||
backup_id = self.backup_node(
|
||||
backup_dir, 'node', node, options=['--stream'])
|
||||
|
||||
recovery_time = self.show_pb(
|
||||
backup_dir, 'node', backup_id=backup_id)['recovery-time']
|
||||
|
||||
expire_time_1 = "{:%Y-%m-%d %H:%M:%S}".format(
|
||||
datetime.now() + timedelta(days=5))
|
||||
|
||||
try:
|
||||
self.set_backup(backup_dir, False, options=['--ttl=30d'])
|
||||
# we should die here because exception is what we expect to happen
|
||||
self.assertEqual(
|
||||
1, 0,
|
||||
"Expecting Error because of missing instance. "
|
||||
"\n Output: {0} \n CMD: {1}".format(
|
||||
repr(self.output), self.cmd))
|
||||
except ProbackupException as e:
|
||||
self.assertIn(
|
||||
'ERROR: required parameter not specified: --instance',
|
||||
e.message,
|
||||
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
|
||||
repr(e.message), self.cmd))
|
||||
|
||||
try:
|
||||
self.set_backup(
|
||||
backup_dir, 'node',
|
||||
options=[
|
||||
"--ttl=30d",
|
||||
"--expire-time='{0}'".format(expire_time_1)])
|
||||
# we should die here because exception is what we expect to happen
|
||||
self.assertEqual(
|
||||
1, 0,
|
||||
"Expecting Error because options cannot be mixed. "
|
||||
"\n Output: {0} \n CMD: {1}".format(
|
||||
repr(self.output), self.cmd))
|
||||
except ProbackupException as e:
|
||||
self.assertIn(
|
||||
"ERROR: You cannot specify '--expire-time' "
|
||||
"and '--ttl' options together",
|
||||
e.message,
|
||||
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
|
||||
repr(e.message), self.cmd))
|
||||
|
||||
try:
|
||||
self.set_backup(backup_dir, 'node', options=["--ttl=30d"])
|
||||
# we should die here because exception is what we expect to happen
|
||||
self.assertEqual(
|
||||
1, 0,
|
||||
"Expecting Error because of missing backup_id. "
|
||||
"\n Output: {0} \n CMD: {1}".format(
|
||||
repr(self.output), self.cmd))
|
||||
except ProbackupException as e:
|
||||
self.assertIn(
|
||||
"ERROR: You must specify parameter (-i, --backup-id) "
|
||||
"for 'set-backup' command",
|
||||
e.message,
|
||||
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
|
||||
repr(e.message), self.cmd))
|
||||
|
||||
self.set_backup(
|
||||
backup_dir, 'node', backup_id, options=["--ttl=30d"])
|
||||
|
||||
actual_expire_time = self.show_pb(
|
||||
backup_dir, 'node', backup_id=backup_id)['expire-time']
|
||||
|
||||
self.assertNotEqual(expire_time_1, actual_expire_time)
|
||||
|
||||
expire_time_2 = "{:%Y-%m-%d %H:%M:%S}".format(
|
||||
datetime.now() + timedelta(days=6))
|
||||
|
||||
self.set_backup(
|
||||
backup_dir, 'node', backup_id,
|
||||
options=["--expire-time={0}".format(expire_time_2)])
|
||||
|
||||
actual_expire_time = self.show_pb(
|
||||
backup_dir, 'node', backup_id=backup_id)['expire-time']
|
||||
|
||||
self.assertIn(expire_time_2, actual_expire_time)
|
||||
|
||||
# unpin backup
|
||||
self.set_backup(
|
||||
backup_dir, 'node', backup_id, options=["--ttl=0"])
|
||||
|
||||
attr_list = self.show_pb(
|
||||
backup_dir, 'node', backup_id=backup_id)
|
||||
|
||||
self.assertNotIn('expire-time', attr_list)
|
||||
|
||||
self.set_backup(
|
||||
backup_dir, 'node', backup_id, options=["--expire-time={0}".format(recovery_time)])
|
||||
|
||||
# parse string to datetime object
|
||||
#new_expire_time = datetime.strptime(new_expire_time, '%Y-%m-%d %H:%M:%S%z')
|
||||
|
||||
# Clean after yourself
|
||||
self.del_test_dir(module_name, fname)
|
||||
|
||||
# @unittest.skip("skip")
|
||||
# @unittest.expectedFailure
|
||||
def test_retention_redundancy_pinning(self):
|
||||
""""""
|
||||
fname = self.id().split('.')[3]
|
||||
node = self.make_simple_node(
|
||||
base_dir=os.path.join(module_name, fname, 'node'),
|
||||
initdb_params=['--data-checksums'])
|
||||
|
||||
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
|
||||
self.init_pb(backup_dir)
|
||||
self.add_instance(backup_dir, 'node', node)
|
||||
self.set_archiving(backup_dir, 'node', node)
|
||||
node.slow_start()
|
||||
|
||||
with open(os.path.join(
|
||||
backup_dir, 'backups', 'node',
|
||||
"pg_probackup.conf"), "a") as conf:
|
||||
conf.write("retention-redundancy = 1\n")
|
||||
|
||||
self.set_config(
|
||||
backup_dir, 'node', options=['--retention-redundancy=1'])
|
||||
|
||||
# Make backups to be purged
|
||||
full_id = self.backup_node(backup_dir, 'node', node)
|
||||
page_id = self.backup_node(
|
||||
backup_dir, 'node', node, backup_type="page")
|
||||
# Make backups to be keeped
|
||||
self.backup_node(backup_dir, 'node', node)
|
||||
self.backup_node(backup_dir, 'node', node, backup_type="page")
|
||||
|
||||
self.assertEqual(len(self.show_pb(backup_dir, 'node')), 4)
|
||||
|
||||
self.set_backup(
|
||||
backup_dir, 'node', page_id, options=['--ttl=5d'])
|
||||
|
||||
# Purge backups
|
||||
log = self.delete_expired(
|
||||
backup_dir, 'node',
|
||||
options=['--delete-expired', '--log-level-console=LOG'])
|
||||
self.assertEqual(len(self.show_pb(backup_dir, 'node')), 4)
|
||||
|
||||
self.assertIn('Time Window: 0d/5d', log)
|
||||
self.assertIn(
|
||||
'LOG: Backup {0} is pinned until'.format(page_id),
|
||||
log)
|
||||
self.assertIn(
|
||||
'LOG: Retain backup {0} because his descendant '
|
||||
'{1} is guarded by retention'.format(full_id, page_id),
|
||||
log)
|
||||
|
||||
# Clean after yourself
|
||||
self.del_test_dir(module_name, fname)
|
||||
|
||||
# @unittest.skip("skip")
|
||||
def test_retention_window_pinning(self):
|
||||
"""purge all backups using window-based retention policy"""
|
||||
fname = self.id().split('.')[3]
|
||||
node = self.make_simple_node(
|
||||
base_dir=os.path.join(module_name, fname, 'node'),
|
||||
initdb_params=['--data-checksums'])
|
||||
|
||||
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
|
||||
self.init_pb(backup_dir)
|
||||
self.add_instance(backup_dir, 'node', node)
|
||||
self.set_archiving(backup_dir, 'node', node)
|
||||
node.slow_start()
|
||||
|
||||
# take FULL BACKUP
|
||||
backup_id_1 = self.backup_node(backup_dir, 'node', node)
|
||||
page1 = self.backup_node(
|
||||
backup_dir, 'node', node, backup_type='page')
|
||||
|
||||
# Take second FULL BACKUP
|
||||
backup_id_2 = self.backup_node(backup_dir, 'node', node)
|
||||
page2 = self.backup_node(
|
||||
backup_dir, 'node', node, backup_type='page')
|
||||
|
||||
# Take third FULL BACKUP
|
||||
backup_id_3 = self.backup_node(backup_dir, 'node', node)
|
||||
page2 = self.backup_node(
|
||||
backup_dir, 'node', node, backup_type='page')
|
||||
|
||||
backups = os.path.join(backup_dir, 'backups', 'node')
|
||||
for backup in os.listdir(backups):
|
||||
if backup == 'pg_probackup.conf':
|
||||
continue
|
||||
with open(
|
||||
os.path.join(
|
||||
backups, backup, "backup.control"), "a") as conf:
|
||||
conf.write("recovery_time='{:%Y-%m-%d %H:%M:%S}'\n".format(
|
||||
datetime.now() - timedelta(days=3)))
|
||||
|
||||
self.set_backup(
|
||||
backup_dir, 'node', page1, options=['--ttl=30d'])
|
||||
|
||||
# Purge backups
|
||||
out = self.delete_expired(
|
||||
backup_dir, 'node',
|
||||
options=[
|
||||
'--log-level-console=LOG',
|
||||
'--retention-window=1',
|
||||
'--delete-expired'])
|
||||
|
||||
self.assertEqual(len(self.show_pb(backup_dir, 'node')), 2)
|
||||
|
||||
self.assertIn(
|
||||
'LOG: Backup {0} is pinned until'.format(page1), out)
|
||||
|
||||
self.assertIn(
|
||||
'LOG: Retain backup {0} because his descendant '
|
||||
'{1} is guarded by retention'.format(backup_id_1, page1),
|
||||
out)
|
||||
|
||||
# Clean after yourself
|
||||
self.del_test_dir(module_name, fname)
|
||||
Loading…
x
Reference in New Issue
Block a user