db/pg_probackup
1
0
Fork 0
mirror of https://github.com/postgrespro/pg_probackup.git synced 2026-06-21 01:34:15 +02:00
Code Issues Releases Activity

Merge branch 'release_2_4' into issue_201

Browse Source
This commit is contained in:
Grigory Smolkin
2020-05-21 21:15:44 +03:00
parent b949231b93 0f58b2dcf4
commit b5ce80c694
33 changed files with 1020 additions and 842 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -2
View File
@@ -1,4 +1,5 @@
[![Build Status](https://travis-ci.com/postgrespro/pg_probackup.svg?branch=master)](https://travis-ci.com/postgrespro/pg_probackup)
[![GitHub release](https://img.shields.io/github/v/release/postgrespro/pg_probackup?include_prereleases)](https://github.com/postgrespro/pg_probackup/releases/latest)
# pg_probackup
@@ -49,10 +50,11 @@ Regardless of the chosen backup type, all backups taken with `pg_probackup` supp
`pg_probackup` currently has the following limitations:
* The server from which the backup was taken and the restored server must be compatible by the [block_size](https://postgrespro.com/docs/postgresql/current/runtime-config-preset#GUC-BLOCK-SIZE) and [wal_block_size](https://postgrespro.com/docs/postgresql/current/runtime-config-preset#GUC-WAL-BLOCK-SIZE) parameters and have the same major release number.
* Remote backup via ssh on Windows currently is not supported.
* When running remote operations via ssh, remote and local pg_probackup versions must be the same.
## Current release
[2.3.1](https://github.com/postgrespro/pg_probackup/releases/tag/2.3.1)
[2.3.3](https://github.com/postgrespro/pg_probackup/releases/tag/2.3.3)
## Documentation
@@ -60,7 +62,7 @@ Documentation can be found at [github](https://postgrespro.github.io/pg_probacku
## Installation and Setup
### Windows Installation
Installers are available in release **assets**. [Latests](https://github.com/postgrespro/pg_probackup/releases/latest).
Installers are available in release **assets**. [Latests](https://github.com/postgrespro/pg_probackup/releases/2.3.3).
### Linux Installation
#### pg_probackup for vanilla PostgreSQL
doc/pgprobackup.xml
+30 -14
View File
@@ -740,6 +740,16 @@ ALTER ROLE backup WITH REPLICATION;
DELTA, and PTRACK backups in the
<link linkend="pbk-stream-mode">STREAM</link> WAL mode.
</para>
<note>
<para>
If you are planning to rely on
<ulink url="https://postgrespro.com/docs/postgresql/current/libpq-pgpass.html">.pgpass</ulink>
for authentication when running backup in STREAM mode,
then .pgpass must contain credentials for <literal>replication</literal> database,
used to establish connection via replication protocol. Example:
pghost:5432:replication:backup_user:my_strong_password
</para>
</note>
</refsect2>
<refsect2 id="pbk-setting-up-continuous-wal-archiving">
<title>Setting up Continuous WAL Archiving</title>
@@ -1158,12 +1168,12 @@ CREATE EXTENSION ptrack;
</listitem>
<listitem>
<para>
To enable tracking page updates, set <varname>ptrack_map_size</varname>
To enable tracking page updates, set <varname>ptrack.map_size</varname>
parameter to a positive integer and restart the server.
</para>
<para>
For optimal performance, it is recommended to set
<varname>ptrack_map_size</varname> to
<varname>ptrack.map_size</varname> to
<literal><replaceable>N</replaceable> / 1024</literal>, where
<replaceable>N</replaceable> is the size of the
<productname>PostgreSQL</productname> cluster, in MB. If you set this
@@ -1171,7 +1181,7 @@ CREATE EXTENSION ptrack;
together, which leads to false-positive results when tracking changed
blocks and increases the incremental backup size as unchanged blocks
can also be copied into the incremental backup.
Setting <varname>ptrack_map_size</varname> to a higher value
Setting <varname>ptrack.map_size</varname> to a higher value
does not affect PTRACK operation. The maximum allowed value is 1024.
</para>
</listitem>
@@ -1191,11 +1201,11 @@ GRANT EXECUTE ON FUNCTION pg_ptrack_get_block(oid, oid, oid, bigint) TO backup;
<note>
<para>
If you change the <varname>ptrack_map_size</varname> parameter value,
If you change the <varname>ptrack.map_size</varname> parameter value,
the previously created PTRACK map file is cleared,
and tracking newly changed blocks starts from scratch. Thus, you have
to retake a full backup before taking incremental PTRACK backups after
changing <varname>ptrack_map_size</varname>.
changing <varname>ptrack.map_size</varname>.
</para>
</note>
@@ -2407,6 +2417,12 @@ primary_conninfo = 'user=backup passfile=/var/lib/pgsql/.pgpass port=5432 sslmod
<literal>note</literal> — text note attached to backup.
</para>
</listitem>
<listitem>
<para>
<literal>content-crc</literal> — CRC32 checksum of <literal>backup_content.control</literal> file.
It is used to detect corruption of backup metainformation.
</para>
</listitem>
</itemizedlist>
</para>
<para>
@@ -4039,10 +4055,10 @@ pg_probackup archive-push -B <replaceable>backup_dir</replaceable> --instance <r
corruption.
</para>
<para>
To speed up archiving, you can specify the <option>-j</option> option
to run <command>archive-push</command> on multiple threads.
If you provide the <option>--batch-size</option> option, WAL files
will be copied in batches of the specified size.
To speed up archiving, you can specify the <option>--batch-size</option> option
to copy WAL segments in batches of the specified size.
If <option>--batch-size</option> option is used, then you can also specify
the <option>-j</option> option to copy the batch of WAL segments on multiple threads.
</para>
<para>
WAL segments copied to the archive are synced to disk unless
@@ -4080,10 +4096,10 @@ pg_probackup archive-get -B <replaceable>backup_dir</replaceable> --instance <re
</para>
<para>
To speed up recovery, you can specify the <option>-j</option> option
to run <command>archive-get</command> on multiple threads.
If you provide the <option>--batch-size</option> option, WAL segments
will be copied in batches of the specified size.
To speed up recovery, you can specify the <option>--batch-size</option> option
to copy WAL segments in batches of the specified size.
If <option>--batch-size</option> option is used, then you can also specify
the <option>-j</option> option to copy the batch of WAL segments on multiple threads.
</para>
<para>
@@ -4708,7 +4724,7 @@ pg_probackup archive-get -B <replaceable>backup_dir</replaceable> --instance <re
<term><option>--password</option></term>
<listitem>
<para>
Forces a password prompt.
Forces a password prompt. (Deprecated)
</para>
</listitem>
</varlistentry>
src/archive.c
+2 -2
View File
@@ -945,7 +945,7 @@ setup_push_filelist(const char *archive_status_dir, const char *first_file,
/* get list of files from archive_status */
status_files = parray_new();
dir_list_file(status_files, archive_status_dir, false, false, false, 0, FIO_DB_HOST);
parray_qsort(status_files, pgFileComparePath);
parray_qsort(status_files, pgFileCompareName);
for (i = 0; i < parray_num(status_files); i++)
{
@@ -1109,7 +1109,7 @@ do_archive_get(InstanceConfig *instance, const char *prefetch_dir_arg,
{
/* discard prefetch */
// n_fetched = 0;
rmtree(prefetch_dir, false);
pgut_rmtree(prefetch_dir, false, false);
}
}
else
src/backup.c
+62 -79
View File
@@ -149,7 +149,6 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
parray *external_dirs = NULL;
parray *database_map = NULL;
pgFile *pg_control = NULL;
PGconn *master_conn = NULL;
PGconn *pg_startbackup_conn = NULL;
@@ -192,7 +191,7 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
{
/* try to setup multi-timeline backup chain */
elog(WARNING, "Valid backup on current timeline %u is not found, "
"try to look up on previous timelines",
"trying to look up on previous timelines",
current.tli);
tli_list = catalog_get_timelines(&instance_config);
@@ -217,8 +216,6 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
if (prev_backup)
{
char prev_backup_filelist_path[MAXPGPATH];
if (parse_program_version(prev_backup->program_version) > parse_program_version(PROGRAM_VERSION))
elog(ERROR, "pg_probackup binary version is %s, but backup %s version is %s. "
"pg_probackup do not guarantee to be forward compatible. "
@@ -227,10 +224,8 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
elog(INFO, "Parent backup: %s", base36enc(prev_backup->start_time));
join_path_components(prev_backup_filelist_path, prev_backup->root_dir,
DATABASE_FILE_LIST);
/* Files of previous backup needed by DELTA backup */
prev_backup_filelist = dir_read_file_list(NULL, NULL, prev_backup_filelist_path, FIO_BACKUP_HOST);
prev_backup_filelist = get_backup_filelist(prev_backup, true);
/* If lsn is not NULL, only pages with higher lsn will be copied. */
prev_backup_start_lsn = prev_backup->start_lsn;
@@ -375,6 +370,12 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
if (file->external_dir_num != 0)
continue;
if (S_ISDIR(file->mode))
{
current.pgdata_bytes += 4096;
continue;
}
current.pgdata_bytes += file->size;
}
@@ -392,7 +393,7 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
* Sorted array is used at least in parse_filelist_filenames(),
* extractPageMap(), make_pagemap_from_ptrack().
*/
parray_qsort(backup_files_list, pgFileComparePath);
parray_qsort(backup_files_list, pgFileCompareRelPathWithExternal);
/* Extract information about files in backup_list parsing their names:*/
parse_filelist_filenames(backup_files_list, instance_config.pgdata);
@@ -434,10 +435,11 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
/*
* Build the page map from ptrack information.
*/
if (nodeInfo->ptrack_version_num == 20)
if (nodeInfo->ptrack_version_num >= 20)
make_pagemap_from_ptrack_2(backup_files_list, backup_conn,
nodeInfo->ptrack_schema,
prev_backup_start_lsn);
nodeInfo->ptrack_schema,
nodeInfo->ptrack_version_num,
prev_backup_start_lsn);
else if (nodeInfo->ptrack_version_num == 15 ||
nodeInfo->ptrack_version_num == 16 ||
nodeInfo->ptrack_version_num == 17)
@@ -466,26 +468,18 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
if (S_ISDIR(file->mode))
{
char dirpath[MAXPGPATH];
char *dir_name;
if (file->external_dir_num)
dir_name = GetRelativePath(file->path,
parray_get(external_dirs,
file->external_dir_num - 1));
else
dir_name = GetRelativePath(file->path, instance_config.pgdata);
elog(VERBOSE, "Create directory \"%s\"", dir_name);
if (file->external_dir_num)
{
char temp[MAXPGPATH];
snprintf(temp, MAXPGPATH, "%s%d", external_prefix,
file->external_dir_num);
join_path_components(dirpath, temp, dir_name);
join_path_components(dirpath, temp, file->rel_path);
}
else
join_path_components(dirpath, database_path, dir_name);
join_path_components(dirpath, database_path, file->rel_path);
elog(VERBOSE, "Create directory '%s'", dirpath);
fio_mkdir(dirpath, DIR_PERMISSION, FIO_BACKUP_HOST);
}
@@ -501,7 +495,7 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
/* write initial backup_content.control file and update backup.control */
write_backup_filelist(&current, backup_files_list,
instance_config.pgdata, external_dirs);
instance_config.pgdata, external_dirs, true);
write_backup(&current, true);
/* init thread args with own file lists */
@@ -588,16 +582,14 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
*/
if (current.from_replica && !exclusive_backup)
{
char pg_control_path[MAXPGPATH];
snprintf(pg_control_path, sizeof(pg_control_path), "%s/%s",
instance_config.pgdata, XLOG_CONTROL_FILE);
pgFile *pg_control = NULL;
for (i = 0; i < parray_num(backup_files_list); i++)
{
pgFile *tmp_file = (pgFile *) parray_get(backup_files_list, i);
if (strcmp(tmp_file->path, pg_control_path) == 0)
if (tmp_file->external_dir_num == 0 &&
(strcmp(tmp_file->rel_path, XLOG_CONTROL_FILE) == 0))
{
pg_control = tmp_file;
break;
@@ -606,7 +598,7 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
if (!pg_control)
elog(ERROR, "Failed to find file \"%s\" in backup filelist.",
pg_control_path);
XLOG_CONTROL_FILE);
set_min_recovery_point(pg_control, database_path, current.stop_lsn);
}
@@ -634,21 +626,23 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
join_path_components(wal_full_path, pg_xlog_path, file->rel_path);
if (S_ISREG(file->mode))
{
file->crc = pgFileGetCRC(wal_full_path, true, false);
file->write_size = file->size;
}
/* Remove file path root prefix*/
if (strstr(file->path, database_path) == file->path)
{
char *ptr = file->path;
if (!S_ISREG(file->mode))
continue;
file->path = pgut_strdup(GetRelativePath(ptr, database_path));
file->rel_path = pgut_strdup(file->path);
free(ptr);
}
file->crc = pgFileGetCRC(wal_full_path, true, false);
file->write_size = file->size;
/* overwrite rel_path, because now it is relative to
* /backup_dir/backups/instance_name/backup_id/database/pg_xlog/
*/
pg_free(file->rel_path);
file->rel_path = pgut_strdup(GetRelativePath(wal_full_path, database_path));
file->name = last_dir_separator(file->rel_path);
/* Now it is relative to /backup_dir/backups/instance_name/backup_id/database/ */
}
/* Add xlog files into the list of backed up files */
parray_concat(backup_files_list, xlog_files_list);
parray_free(xlog_files_list);
@@ -665,7 +659,7 @@ do_backup_instance(PGconn *backup_conn, PGNodeInfo *nodeInfo, bool no_sync)
/* Print the list of files to backup catalog */
write_backup_filelist(&current, backup_files_list, instance_config.pgdata,
external_dirs);
external_dirs, true);
/* update backup control file to update size info */
write_backup(&current, true);
@@ -882,15 +876,10 @@ do_backup(time_t start_time, bool no_validate,
#endif
get_ptrack_version(backup_conn, &nodeInfo);
// elog(WARNING, "ptrack_version_num %d", ptrack_version_num);
// elog(WARNING, "ptrack_version_num %d", ptrack_version_num);
if (nodeInfo.ptrack_version_num > 0)
{
if (nodeInfo.ptrack_version_num >= 20)
nodeInfo.is_ptrack_enable = pg_ptrack_enable2(backup_conn);
else
nodeInfo.is_ptrack_enable = pg_ptrack_enable(backup_conn);
}
nodeInfo.is_ptrack_enable = pg_ptrack_enable(backup_conn, nodeInfo.ptrack_version_num);
if (current.backup_mode == BACKUP_MODE_DIFF_PTRACK)
{
@@ -1882,8 +1871,6 @@ pg_stop_backup(pgBackup *backup, PGconn *pg_startbackup_conn,
file->write_size = file->size;
file->uncompressed_size = file->size;
free(file->path);
file->path = strdup(PG_BACKUP_LABEL_FILE);
parray_append(backup_files_list, file);
}
}
@@ -1930,8 +1917,7 @@ pg_stop_backup(pgBackup *backup, PGconn *pg_startbackup_conn,
file->crc = pgFileGetCRC(tablespace_map, true, false);
file->write_size = file->size;
}
free(file->path);
file->path = strdup(PG_TABLESPACE_MAP_FILE);
parray_append(backup_files_list, file);
}
}
@@ -2072,15 +2058,15 @@ backup_files(void *arg)
if (arguments->thread_num == 1)
{
/* update backup_content.control every 10 seconds */
if ((difftime(time(NULL), prev_time)) > 10)
/* update backup_content.control every 60 seconds */
if ((difftime(time(NULL), prev_time)) > 60)
{
prev_time = time(NULL);
write_backup_filelist(&current, arguments->files_list, arguments->from_root,
arguments->external_dirs);
arguments->external_dirs, false);
/* update backup control file to update size info */
write_backup(&current, true);
prev_time = time(NULL);
}
}
@@ -2202,13 +2188,13 @@ parse_filelist_filenames(parray *files, const char *root)
while (i < parray_num(files))
{
pgFile *file = (pgFile *) parray_get(files, i);
char *relative;
// char *relative;
int sscanf_result;
relative = GetRelativePath(file->path, root);
// relative = GetRelativePath(file->rel_path, root);
if (S_ISREG(file->mode) &&
path_is_prefix_of_path(PG_TBLSPC_DIR, relative))
path_is_prefix_of_path(PG_TBLSPC_DIR, file->rel_path))
{
/*
* Found file in pg_tblspc/tblsOid/TABLESPACE_VERSION_DIRECTORY
@@ -2223,21 +2209,21 @@ parse_filelist_filenames(parray *files, const char *root)
* Check that the file is located under
* TABLESPACE_VERSION_DIRECTORY
*/
sscanf_result = sscanf(relative, PG_TBLSPC_DIR "/%u/%s/%u",
sscanf_result = sscanf(file->rel_path, PG_TBLSPC_DIR "/%u/%s/%u",
&tblspcOid, tmp_rel_path, &dbOid);
/* Yes, it is */
if (sscanf_result == 2 &&
strncmp(tmp_rel_path, TABLESPACE_VERSION_DIRECTORY,
strlen(TABLESPACE_VERSION_DIRECTORY)) == 0)
set_cfs_datafiles(files, root, relative, i);
set_cfs_datafiles(files, root, file->rel_path, i);
}
}
if (S_ISREG(file->mode) && file->tblspcOid != 0 &&
file->name && file->name[0])
{
if (strcmp(file->forkName, "init") == 0)
if (file->forkName == INIT)
{
/*
* Do not backup files of unlogged relations.
@@ -2288,7 +2274,6 @@ set_cfs_datafiles(parray *files, const char *root, char *relative, size_t i)
int p;
pgFile *prev_file;
char *cfs_tblspc_path;
char *relative_prev_file;
cfs_tblspc_path = strdup(relative);
if(!cfs_tblspc_path)
@@ -2300,22 +2285,21 @@ set_cfs_datafiles(parray *files, const char *root, char *relative, size_t i)
for (p = (int) i; p >= 0; p--)
{
prev_file = (pgFile *) parray_get(files, (size_t) p);
relative_prev_file = GetRelativePath(prev_file->path, root);
elog(VERBOSE, "Checking file in cfs tablespace %s", relative_prev_file);
elog(VERBOSE, "Checking file in cfs tablespace %s", prev_file->rel_path);
if (strstr(relative_prev_file, cfs_tblspc_path) != NULL)
if (strstr(prev_file->rel_path, cfs_tblspc_path) != NULL)
{
if (S_ISREG(prev_file->mode) && prev_file->is_datafile)
{
elog(VERBOSE, "Setting 'is_cfs' on file %s, name %s",
relative_prev_file, prev_file->name);
prev_file->rel_path, prev_file->name);
prev_file->is_cfs = true;
}
}
else
{
elog(VERBOSE, "Breaking on %s", relative_prev_file);
elog(VERBOSE, "Breaking on %s", prev_file->rel_path);
break;
}
}
@@ -2329,7 +2313,7 @@ set_cfs_datafiles(parray *files, const char *root, char *relative, size_t i)
void
process_block_change(ForkNumber forknum, RelFileNode rnode, BlockNumber blkno)
{
char *path;
// char *path;
char *rel_path;
BlockNumber blkno_inseg;
int segno;
@@ -2341,16 +2325,15 @@ process_block_change(ForkNumber forknum, RelFileNode rnode, BlockNumber blkno)
rel_path = relpathperm(rnode, forknum);
if (segno > 0)
path = psprintf("%s/%s.%u", instance_config.pgdata, rel_path, segno);
f.rel_path = psprintf("%s.%u", rel_path, segno);
else
path = psprintf("%s/%s", instance_config.pgdata, rel_path);
f.rel_path = rel_path;
pg_free(rel_path);
f.external_dir_num = 0;
f.path = path;
/* backup_files_list should be sorted before */
file_item = (pgFile **) parray_bsearch(backup_files_list, &f,
pgFileComparePath);
pgFileCompareRelPathWithExternal);
/*
* If we don't have any record of this file in the file map, it means
@@ -2370,7 +2353,7 @@ process_block_change(ForkNumber forknum, RelFileNode rnode, BlockNumber blkno)
pthread_mutex_unlock(&backup_pagemap_mutex);
}
pg_free(path);
pg_free(rel_path);
}
/*
src/catalog.c
+75 -87
View File
@@ -471,6 +471,9 @@ catalog_get_backup_list(const char *instance_name, time_t requested_backup_id)
backup->root_dir = pgut_strdup(data_path);
backup->database_dir = pgut_malloc(MAXPGPATH);
join_path_components(backup->database_dir, backup->root_dir, DATABASE_DIR);
/* TODO: save encoded backup id */
backup->backup_id = backup->start_time;
if (requested_backup_id != INVALID_BACKUP_ID
@@ -538,17 +541,17 @@ err_proc:
* TODO this function only used once. Is it really needed?
*/
parray *
get_backup_filelist(pgBackup *backup)
get_backup_filelist(pgBackup *backup, bool strict)
{
parray *files = NULL;
char backup_filelist_path[MAXPGPATH];
join_path_components(backup_filelist_path, backup->root_dir, DATABASE_FILE_LIST);
files = dir_read_file_list(NULL, NULL, backup_filelist_path, FIO_BACKUP_HOST);
files = dir_read_file_list(NULL, NULL, backup_filelist_path, FIO_BACKUP_HOST, backup->content_crc);
/* redundant sanity? */
if (!files)
elog(ERROR, "Failed to get filelist for backup %s", base36enc(backup->start_time));
elog(strict ? ERROR : WARNING, "Failed to get file list for backup %s", base36enc(backup->start_time));
return files;
}
@@ -841,6 +844,9 @@ pgBackupCreateDir(pgBackup *backup)
fio_mkdir(path, DIR_PERMISSION, FIO_BACKUP_HOST);
backup->root_dir = pgut_strdup(path);
backup->database_dir = pgut_malloc(MAXPGPATH);
join_path_components(backup->database_dir, backup->root_dir, DATABASE_DIR);
/* create directories for actual backup files */
for (i = 0; i < parray_num(subdirs); i++)
{
@@ -873,7 +879,7 @@ catalog_get_timelines(InstanceConfig *instance)
/* read all xlog files that belong to this archive */
sprintf(arclog_path, "%s/%s/%s", backup_path, "wal", instance->name);
dir_list_file(xlog_files_list, arclog_path, false, false, false, 0, FIO_BACKUP_HOST);
parray_qsort(xlog_files_list, pgFileComparePath);
parray_qsort(xlog_files_list, pgFileCompareName);
timelineinfos = parray_new();
tlinfo = NULL;
@@ -1759,6 +1765,9 @@ pgBackupWriteControl(FILE *out, pgBackup *backup)
if (backup->note)
fio_fprintf(out, "note = '%s'\n", backup->note);
if (backup->content_crc != 0)
fio_fprintf(out, "content-crc = %u\n", backup->content_crc);
}
/*
@@ -1770,40 +1779,39 @@ write_backup(pgBackup *backup, bool strict)
FILE *fp = NULL;
char path[MAXPGPATH];
char path_temp[MAXPGPATH];
int errno_temp;
char buf[4096];
join_path_components(path, backup->root_dir, BACKUP_CONTROL_FILE);
snprintf(path_temp, sizeof(path_temp), "%s.tmp", path);
fp = fio_fopen(path_temp, PG_BINARY_W, FIO_BACKUP_HOST);
fp = fopen(path_temp, PG_BINARY_W);
if (fp == NULL)
elog(ERROR, "Cannot open configuration file \"%s\": %s",
elog(ERROR, "Cannot open control file \"%s\": %s",
path_temp, strerror(errno));
if (chmod(path_temp, FILE_PERMISSION) == -1)
elog(ERROR, "Cannot change mode of \"%s\": %s", path_temp,
strerror(errno));
setvbuf(fp, buf, _IOFBF, sizeof(buf));
pgBackupWriteControl(fp, backup);
if (fio_fclose(fp))
{
errno_temp = errno;
fio_unlink(path_temp, FIO_BACKUP_HOST);
if (fflush(fp) != 0)
elog(ERROR, "Cannot flush control file \"%s\": %s",
path_temp, strerror(errno));
if (!strict && errno_temp == ENOSPC)
return;
if (fsync(fileno(fp)) < 0)
elog(ERROR, "Cannot sync control file \"%s\": %s",
path_temp, strerror(errno));
elog(ERROR, "Cannot write configuration file \"%s\": %s",
path_temp, strerror(errno_temp));
}
if (fclose(fp) != 0)
elog(ERROR, "Cannot close control file \"%s\": %s",
path_temp, strerror(errno));
if (fio_rename(path_temp, path, FIO_BACKUP_HOST) < 0)
{
errno_temp = errno;
fio_unlink(path_temp, FIO_BACKUP_HOST);
elog(ERROR, "Cannot rename configuration file \"%s\" to \"%s\": %s",
path_temp, path, strerror(errno_temp));
}
if (rename(path_temp, path) < 0)
elog(ERROR, "Cannot rename file \"%s\" to \"%s\": %s",
path_temp, path, strerror(errno));
}
/*
@@ -1811,39 +1819,42 @@ write_backup(pgBackup *backup, bool strict)
*/
void
write_backup_filelist(pgBackup *backup, parray *files, const char *root,
parray *external_list)
parray *external_list, bool sync)
{
FILE *out;
char path[MAXPGPATH];
char path_temp[MAXPGPATH];
int errno_temp;
char control_path[MAXPGPATH];
char control_path_temp[MAXPGPATH];
size_t i = 0;
#define BUFFERSZ 1024*1024
char *buf;
size_t write_len = 0;
int64 backup_size_on_disk = 0;
int64 uncompressed_size_on_disk = 0;
int64 wal_size_on_disk = 0;
join_path_components(path, backup->root_dir, DATABASE_FILE_LIST);
snprintf(path_temp, sizeof(path_temp), "%s.tmp", path);
join_path_components(control_path, backup->root_dir, DATABASE_FILE_LIST);
snprintf(control_path_temp, sizeof(control_path_temp), "%s.tmp", control_path);
out = fio_fopen(path_temp, PG_BINARY_W, FIO_BACKUP_HOST);
out = fopen(control_path_temp, PG_BINARY_W);
if (out == NULL)
elog(ERROR, "Cannot open file list \"%s\": %s", path_temp,
elog(ERROR, "Cannot open file list \"%s\": %s", control_path_temp,
strerror(errno));
if (chmod(control_path_temp, FILE_PERMISSION) == -1)
elog(ERROR, "Cannot change mode of \"%s\": %s", control_path_temp,
strerror(errno));
buf = pgut_malloc(BUFFERSZ);
setvbuf(out, buf, _IOFBF, BUFFERSZ);
if (sync)
INIT_FILE_CRC32(true, backup->content_crc);
/* print each file in the list */
while(i < parray_num(files))
for (i = 0; i < parray_num(files); i++)
{
pgFile *file = (pgFile *) parray_get(files, i);
char *path = file->path; /* for streamed WAL files */
char line[BLCKSZ];
int len = 0;
i++;
int len = 0;
char line[BLCKSZ];
pgFile *file = (pgFile *) parray_get(files, i);
if (S_ISDIR(file->mode))
{
@@ -1867,20 +1878,12 @@ write_backup_filelist(pgBackup *backup, parray *files, const char *root,
}
}
/* for files from PGDATA and external files use rel_path
* streamed WAL files has rel_path relative not to "database/"
* but to "database/pg_wal", so for them use path.
*/
if ((root && strstr(path, root) == path) ||
(file->external_dir_num && external_list))
path = file->rel_path;
len = sprintf(line, "{\"path\":\"%s\", \"size\":\"" INT64_FORMAT "\", "
"\"mode\":\"%u\", \"is_datafile\":\"%u\", "
"\"is_cfs\":\"%u\", \"crc\":\"%u\", "
"\"compress_alg\":\"%s\", \"external_dir_num\":\"%d\", "
"\"dbOid\":\"%u\"",
path, file->write_size, file->mode,
file->rel_path, file->write_size, file->mode,
file->is_datafile ? 1 : 0,
file->is_cfs ? 1 : 0,
file->crc,
@@ -1897,51 +1900,32 @@ write_backup_filelist(pgBackup *backup, parray *files, const char *root,
if (file->n_blocks != BLOCKNUM_INVALID)
len += sprintf(line+len, ",\"n_blocks\":\"%i\"", file->n_blocks);
len += sprintf(line+len, "}\n");
sprintf(line+len, "}\n");
if (write_len + len >= BUFFERSZ)
{
/* write buffer to file */
if (fio_fwrite(out, buf, write_len) != write_len)
{
errno_temp = errno;
fio_unlink(path_temp, FIO_BACKUP_HOST);
elog(ERROR, "Cannot write file list \"%s\": %s",
path_temp, strerror(errno));
}
/* reset write_len */
write_len = 0;
}
if (sync)
COMP_FILE_CRC32(true, backup->content_crc, line, strlen(line));
memcpy(buf+write_len, line, len);
write_len += len;
fprintf(out, "%s", line);
}
/* write what is left in the buffer to file */
if (write_len > 0)
if (fio_fwrite(out, buf, write_len) != write_len)
{
errno_temp = errno;
fio_unlink(path_temp, FIO_BACKUP_HOST);
elog(ERROR, "Cannot write file list \"%s\": %s",
path_temp, strerror(errno));
}
if (sync)
FIN_FILE_CRC32(true, backup->content_crc);
if (fio_fflush(out) || fio_fclose(out))
{
errno_temp = errno;
fio_unlink(path_temp, FIO_BACKUP_HOST);
elog(ERROR, "Cannot write file list \"%s\": %s",
path_temp, strerror(errno));
}
if (fflush(out) != 0)
elog(ERROR, "Cannot flush file list \"%s\": %s",
control_path_temp, strerror(errno));
if (fio_rename(path_temp, path, FIO_BACKUP_HOST) < 0)
{
errno_temp = errno;
fio_unlink(path_temp, FIO_BACKUP_HOST);
elog(ERROR, "Cannot rename configuration file \"%s\" to \"%s\": %s",
path_temp, path, strerror(errno_temp));
}
if (sync && fsync(fileno(out)) < 0)
elog(ERROR, "Cannot sync file list \"%s\": %s",
control_path_temp, strerror(errno));
if (fclose(out) != 0)
elog(ERROR, "Cannot close file list \"%s\": %s",
control_path_temp, strerror(errno));
if (rename(control_path_temp, control_path) < 0)
elog(ERROR, "Cannot rename file \"%s\" to \"%s\": %s",
control_path_temp, control_path, strerror(errno));
/* use extra variable to avoid reset of previous data_bytes value in case of error */
backup->data_bytes = backup_size_on_disk;
@@ -2004,6 +1988,7 @@ readBackupControlFile(const char *path)
{'s', 0, "primary-conninfo", &backup->primary_conninfo, SOURCE_FILE_STRICT},
{'s', 0, "external-dirs", &backup->external_dir_str, SOURCE_FILE_STRICT},
{'s', 0, "note", &backup->note, SOURCE_FILE_STRICT},
{'u', 0, "content-crc", &backup->content_crc, SOURCE_FILE_STRICT},
{0}
};
@@ -2269,8 +2254,10 @@ pgBackupInit(pgBackup *backup)
backup->server_version[0] = '\0';
backup->external_dir_str = NULL;
backup->root_dir = NULL;
backup->database_dir = NULL;
backup->files = NULL;
backup->note = NULL;
backup->content_crc = 0;
}
@@ -2283,6 +2270,7 @@ pgBackupFree(void *backup)
pfree(b->primary_conninfo);
pfree(b->external_dir_str);
pfree(b->root_dir);
pfree(b->database_dir);
pfree(b->note);
pfree(backup);
}
src/checkdb.c
+1 -1
View File
@@ -217,7 +217,7 @@ do_block_validation(char *pgdata, uint32 checksum_version)
* 1 - create 'base'
* 2 - create 'base/1'
*/
parray_qsort(files_list, pgFileComparePath);
parray_qsort(files_list, pgFileCompareRelPathWithExternal);
/* Extract information about files in pgdata parsing their names:*/
parse_filelist_filenames(files_list, pgdata);
src/data.c
+26 -27
View File
@@ -408,15 +408,12 @@ prepare_page(ConnectionArgs *conn_arg,
}
}
/* Get page via ptrack interface from PostgreSQL shared buffer.
* We do this in following cases:
* 1. PTRACK backup of 1.x versions
* 2. During backup, regardless of backup mode, of PostgreSQL instance
* with ptrack support we encountered invalid page.
/*
* Get page via ptrack interface from PostgreSQL shared buffer.
* We do this only in the cases of PTRACK 1.x versions backup
*/
if ((backup_mode == BACKUP_MODE_DIFF_PTRACK
if (backup_mode == BACKUP_MODE_DIFF_PTRACK
&& (ptrack_version_num >= 15 && ptrack_version_num < 20))
|| !page_is_valid)
{
int rc = 0;
size_t page_size = 0;
@@ -440,7 +437,8 @@ prepare_page(ConnectionArgs *conn_arg,
memcpy(page, ptrack_page, BLCKSZ);
pg_free(ptrack_page);
/* UPD: It apprears that is possible to get zeroed page or page with invalid header
/*
* UPD: It apprears that is possible to get zeroed page or page with invalid header
* from shared buffer.
* Note, that getting page with wrong checksumm from shared buffer is
* acceptable.
@@ -462,7 +460,8 @@ prepare_page(ConnectionArgs *conn_arg,
from_fullpath, blknum, errormsg);
}
/* We must set checksum here, because it is outdated
/*
* We must set checksum here, because it is outdated
* in the block recieved from shared buffers.
*/
if (checksum_version)
@@ -1479,7 +1478,7 @@ check_data_file(ConnectionArgs *arguments, pgFile *file,
*/
if (errno == ENOENT)
{
elog(LOG, "File \"%s\" is not found", file->path);
elog(LOG, "File \"%s\" is not found", from_fullpath);
return true;
}
@@ -1489,7 +1488,7 @@ check_data_file(ConnectionArgs *arguments, pgFile *file,
}
if (file->size % BLCKSZ != 0)
elog(WARNING, "File: \"%s\", invalid file size %zu", file->path, file->size);
elog(WARNING, "File: \"%s\", invalid file size %zu", from_fullpath, file->size);
/*
* Compute expected number of blocks in the file.
@@ -1525,8 +1524,8 @@ check_data_file(ConnectionArgs *arguments, pgFile *file,
/* Valiate pages of datafile in backup one by one */
bool
check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
uint32 backup_version)
check_file_pages(pgFile *file, const char *fullpath, XLogRecPtr stop_lsn,
uint32 checksum_version, uint32 backup_version)
{
size_t read_len = 0;
bool is_valid = true;
@@ -1534,19 +1533,19 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
pg_crc32 crc;
bool use_crc32c = backup_version <= 20021 || backup_version >= 20025;
elog(VERBOSE, "Validate relation blocks for file \"%s\"", file->path);
elog(VERBOSE, "Validate relation blocks for file \"%s\"", fullpath);
in = fopen(file->path, PG_BINARY_R);
in = fopen(fullpath, PG_BINARY_R);
if (in == NULL)
{
if (errno == ENOENT)
{
elog(WARNING, "File \"%s\" is not found", file->path);
elog(WARNING, "File \"%s\" is not found", fullpath);
return false;
}
elog(ERROR, "Cannot open file \"%s\": %s",
file->path, strerror(errno));
fullpath, strerror(errno));
}
/* calc CRC of backup file */
@@ -1570,7 +1569,7 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
if (ferror(in))
elog(ERROR, "Cannot read header of block %u of \"%s\": %s",
blknum, file->path, strerror(errno));
blknum, fullpath, strerror(errno));
if (read_len != sizeof(header))
{
@@ -1579,10 +1578,10 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
else if (read_len != 0 && feof(in))
elog(WARNING,
"Odd size page found at block %u of \"%s\"",
blknum, file->path);
blknum, fullpath);
else
elog(WARNING, "Cannot read header of block %u of \"%s\": %s",
blknum, file->path, strerror(errno));
blknum, fullpath, strerror(errno));
return false;
}
@@ -1590,14 +1589,14 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
if (header.block == 0 && header.compressed_size == 0)
{
elog(VERBOSE, "Skip empty block of \"%s\"", file->path);
elog(VERBOSE, "Skip empty block of \"%s\"", fullpath);
continue;
}
if (header.block < blknum)
{
elog(WARNING, "Backup is broken at block %u of \"%s\"",
blknum, file->path);
blknum, fullpath);
return false;
}
@@ -1606,7 +1605,7 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
if (header.compressed_size == PageIsTruncated)
{
elog(LOG, "Block %u of \"%s\" is truncated",
blknum, file->path);
blknum, fullpath);
continue;
}
@@ -1617,7 +1616,7 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
if (read_len != MAXALIGN(header.compressed_size))
{
elog(WARNING, "Cannot read block %u of \"%s\" read %zu of %d",
blknum, file->path, read_len, header.compressed_size);
blknum, fullpath, read_len, header.compressed_size);
return false;
}
@@ -1637,7 +1636,7 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
&errormsg);
if (uncompressed_size < 0 && errormsg != NULL)
elog(WARNING, "An error occured during decompressing block %u of file \"%s\": %s",
blknum, file->path, errormsg);
blknum, fullpath, errormsg);
if (uncompressed_size != BLCKSZ)
{
@@ -1647,7 +1646,7 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
continue;
}
elog(WARNING, "Page of file \"%s\" uncompressed to %d bytes. != BLCKSZ",
file->path, uncompressed_size);
fullpath, uncompressed_size);
return false;
}
@@ -1693,7 +1692,7 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
if (crc != file->crc)
{
elog(WARNING, "Invalid CRC of backup file \"%s\": %X. Expected %X",
file->path, crc, file->crc);
fullpath, crc, file->crc);
is_valid = false;
}
src/delete.c
+13 -27
View File
@@ -753,7 +753,7 @@ delete_backup_files(pgBackup *backup)
dir_list_file(files, backup->root_dir, false, true, true, 0, FIO_BACKUP_HOST);
/* delete leaf node first */
parray_qsort(files, pgFileComparePathDesc);
parray_qsort(files, pgFileCompareRelPathWithExternalDesc);
num_files = parray_num(files);
for (i = 0; i < num_files; i++)
{
@@ -923,31 +923,35 @@ delete_walfiles_in_tli(XLogRecPtr keep_lsn, timelineInfo *tlinfo,
*/
if (purge_all || wal_file->segno < OldestToKeepSegNo)
{
char wal_fullpath[MAXPGPATH];
join_path_components(wal_fullpath, instance_config.arclog_path, wal_file->file.name);
/* save segment from purging */
if (instance_config.wal_depth >= 0 && wal_file->keep)
{
elog(VERBOSE, "Retain WAL segment \"%s\"", wal_file->file.path);
elog(VERBOSE, "Retain WAL segment \"%s\"", wal_fullpath);
continue;
}
/* unlink segment */
if (fio_unlink(wal_file->file.path, FIO_BACKUP_HOST) < 0)
if (fio_unlink(wal_fullpath, FIO_BACKUP_HOST) < 0)
{
/* Missing file is not considered as error condition */
if (errno != ENOENT)
elog(ERROR, "Could not remove file \"%s\": %s",
wal_file->file.path, strerror(errno));
wal_fullpath, strerror(errno));
}
else
{
if (wal_file->type == SEGMENT)
elog(VERBOSE, "Removed WAL segment \"%s\"", wal_file->file.path);
elog(VERBOSE, "Removed WAL segment \"%s\"", wal_fullpath);
else if (wal_file->type == TEMP_SEGMENT)
elog(VERBOSE, "Removed temp WAL segment \"%s\"", wal_file->file.path);
elog(VERBOSE, "Removed temp WAL segment \"%s\"", wal_fullpath);
else if (wal_file->type == PARTIAL_SEGMENT)
elog(VERBOSE, "Removed partial WAL segment \"%s\"", wal_file->file.path);
elog(VERBOSE, "Removed partial WAL segment \"%s\"", wal_fullpath);
else if (wal_file->type == BACKUP_HISTORY_FILE)
elog(VERBOSE, "Removed backup history file \"%s\"", wal_file->file.path);
elog(VERBOSE, "Removed backup history file \"%s\"", wal_fullpath);
}
wal_deleted = true;
@@ -961,7 +965,6 @@ int
do_delete_instance(void)
{
parray *backup_list;
parray *xlog_files_list;
int i;
int rc;
char instance_config_path[MAXPGPATH];
@@ -983,24 +986,7 @@ do_delete_instance(void)
parray_free(backup_list);
/* Delete all wal files. */
xlog_files_list = parray_new();
dir_list_file(xlog_files_list, arclog_path, false, false, false, 0, FIO_BACKUP_HOST);
for (i = 0; i < parray_num(xlog_files_list); i++)
{
pgFile *wal_file = (pgFile *) parray_get(xlog_files_list, i);
if (S_ISREG(wal_file->mode))
{
rc = unlink(wal_file->path);
if (rc != 0)
elog(WARNING, "Failed to remove file \"%s\": %s",
wal_file->path, strerror(errno));
}
}
/* Cleanup */
parray_walk(xlog_files_list, pgFileFree);
parray_free(xlog_files_list);
pgut_rmtree(arclog_path, false, true);
/* Delete backup instance config file */
join_path_components(instance_config_path, backup_instance_path, BACKUP_CATALOG_CONF_FILE);
src/dir.c
+66 -131
View File
@@ -124,8 +124,8 @@ typedef struct TablespaceCreatedList
static int pgCompareString(const void *str1, const void *str2);
static char dir_check_file(pgFile *file);
static void dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
bool follow_symlink,
static void dir_list_file_internal(parray *files, pgFile *parent, const char *parent_dir,
bool exclude, bool follow_symlink,
int external_dir_num, fio_location location);
static void opt_path_map(ConfigOption *opt, const char *arg,
TablespaceList *list, const char *type);
@@ -178,7 +178,7 @@ pgFileNew(const char *path, const char *rel_path, bool follow_symlink,
strerror(errno));
}
file = pgFileInit(path, rel_path);
file = pgFileInit(rel_path);
file->size = st.st_size;
file->mode = st.st_mode;
file->mtime = st.st_mtime;
@@ -188,7 +188,7 @@ pgFileNew(const char *path, const char *rel_path, bool follow_symlink,
}
pgFile *
pgFileInit(const char *path, const char *rel_path)
pgFileInit(const char *rel_path)
{
pgFile *file;
char *file_name = NULL;
@@ -196,20 +196,14 @@ pgFileInit(const char *path, const char *rel_path)
file = (pgFile *) pgut_malloc(sizeof(pgFile));
MemSet(file, 0, sizeof(pgFile));
file->forkName = pgut_malloc(MAXPGPATH);
file->forkName[0] = '\0';
file->path = pgut_strdup(path);
canonicalize_path(file->path);
file->rel_path = pgut_strdup(rel_path);
canonicalize_path(file->rel_path);
/* Get file name from the path */
file_name = pgut_strdup(last_dir_separator(file->path));
file_name = last_dir_separator(file->rel_path);
if (file_name == NULL)
file->name = file->path;
file->name = file->rel_path;
else
{
file_name++;
@@ -395,27 +389,12 @@ pgFileFree(void *file)
file_ptr = (pgFile *) file;
if (file_ptr->linked)
free(file_ptr->linked);
if (file_ptr->forkName)
free(file_ptr->forkName);
pfree(file_ptr->path);
pfree(file_ptr->linked);
pfree(file_ptr->rel_path);
pfree(file);
}
/* Compare two pgFile with their path in ascending order of ASCII code. */
int
pgFileComparePath(const void *f1, const void *f2)
{
pgFile *f1p = *(pgFile **)f1;
pgFile *f2p = *(pgFile **)f2;
return strcmp(f1p->path, f2p->path);
}
/* Compare two pgFile with their path in ascending order of ASCII code. */
int
pgFileMapComparePath(const void *f1, const void *f2)
@@ -436,30 +415,6 @@ pgFileCompareName(const void *f1, const void *f2)
return strcmp(f1p->name, f2p->name);
}
/*
* Compare two pgFile with their path and external_dir_num
* in ascending order of ASCII code.
*/
int
pgFileComparePathWithExternal(const void *f1, const void *f2)
{
pgFile *f1p = *(pgFile **)f1;
pgFile *f2p = *(pgFile **)f2;
int res;
res = strcmp(f1p->path, f2p->path);
if (!res)
{
if (f1p->external_dir_num > f2p->external_dir_num)
return 1;
else if (f1p->external_dir_num < f2p->external_dir_num)
return -1;
else
return 0;
}
return res;
}
/*
* Compare two pgFile with their relative path and external_dir_num in ascending
* order of ASСII code.
@@ -484,23 +439,6 @@ pgFileCompareRelPathWithExternal(const void *f1, const void *f2)
return res;
}
/* Compare two pgFile with their path in descending order of ASCII code. */
int
pgFileComparePathDesc(const void *f1, const void *f2)
{
return -pgFileComparePath(f1, f2);
}
/*
* Compare two pgFile with their path and external_dir_num
* in descending order of ASCII code.
*/
int
pgFileComparePathWithExternalDesc(const void *f1, const void *f2)
{
return -pgFileComparePathWithExternal(f1, f2);
}
/*
* Compare two pgFile with their rel_path and external_dir_num
* in descending order of ASCII code.
@@ -593,15 +531,15 @@ dir_list_file(parray *files, const char *root, bool exclude, bool follow_symlink
{
if (external_dir_num > 0)
elog(ERROR, " --external-dirs option \"%s\": directory or symbolic link expected",
file->path);
root);
else
elog(WARNING, "Skip \"%s\": unexpected file format", file->path);
elog(WARNING, "Skip \"%s\": unexpected file format", root);
return;
}
if (add_root)
parray_append(files, file);
dir_list_file_internal(files, file, exclude, follow_symlink,
dir_list_file_internal(files, file, root, exclude, follow_symlink,
external_dir_num, location);
if (!add_root)
@@ -661,7 +599,7 @@ dir_check_file(pgFile *file)
* If the directory name is in the exclude list, do not list the
* contents.
*/
else if (S_ISDIR(file->mode) && !in_tablespace)
else if (S_ISDIR(file->mode) && !in_tablespace && file->external_dir_num == 0)
{
/*
* If the item in the exclude list starts with '/', compare to
@@ -670,20 +608,10 @@ dir_check_file(pgFile *file)
*/
for (i = 0; pgdata_exclude_dir[i]; i++)
{
/* Full-path exclude*/
if (pgdata_exclude_dir[i][0] == '/')
/* relative path exclude */
if (strcmp(file->rel_path, pgdata_exclude_dir[i]) == 0)
{
if (strcmp(file->path, pgdata_exclude_dir[i]) == 0)
{
elog(VERBOSE, "Excluding directory content: %s",
file->name);
return CHECK_EXCLUDE_FALSE;
}
}
else if (strcmp(file->name, pgdata_exclude_dir[i]) == 0)
{
elog(VERBOSE, "Excluding directory content: %s",
file->name);
elog(VERBOSE, "Excluding directory content: %s", file->rel_path);
return CHECK_EXCLUDE_FALSE;
}
}
@@ -780,10 +708,23 @@ dir_check_file(pgFile *file)
if (fork_name)
{
/* Auxiliary fork of the relfile */
sscanf(file->name, "%u_%s", &(file->relOid), file->forkName);
if (strcmp(fork_name, "vm") == 0)
file->forkName = VM;
else if (strcmp(fork_name, "fsm") == 0)
file->forkName = FSM;
else if (strcmp(fork_name, "cfm") == 0)
file->forkName = CFM;
else if (strcmp(fork_name, "init") == 0)
file->forkName = INIT;
else if (strcmp(fork_name, "ptrack") == 0)
file->forkName = PTRACK;
/* Do not backup ptrack files */
if (strcmp(file->forkName, "ptrack") == 0)
if (file->forkName == PTRACK)
return CHECK_FALSE;
}
else
@@ -820,18 +761,18 @@ dir_check_file(pgFile *file)
* pgdata_exclude_dir.
*/
static void
dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
bool follow_symlink,
dir_list_file_internal(parray *files, pgFile *parent, const char *parent_dir,
bool exclude, bool follow_symlink,
int external_dir_num, fio_location location)
{
DIR *dir;
struct dirent *dent;
if (!S_ISDIR(parent->mode))
elog(ERROR, "\"%s\" is not a directory", parent->path);
elog(ERROR, "\"%s\" is not a directory", parent_dir);
/* Open directory and list contents */
dir = fio_opendir(parent->path, location);
dir = fio_opendir(parent_dir, location);
if (dir == NULL)
{
if (errno == ENOENT)
@@ -840,7 +781,7 @@ dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
return;
}
elog(ERROR, "Cannot open directory \"%s\": %s",
parent->path, strerror(errno));
parent_dir, strerror(errno));
}
errno = 0;
@@ -851,7 +792,7 @@ dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
char rel_child[MAXPGPATH];
char check_res;
join_path_components(child, parent->path, dent->d_name);
join_path_components(child, parent_dir, dent->d_name);
join_path_components(rel_child, parent->rel_path, dent->d_name);
file = pgFileNew(child, rel_child, follow_symlink, external_dir_num,
@@ -870,7 +811,7 @@ dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
/* skip hidden files and directories */
if (file->name[0] == '.')
{
elog(WARNING, "Skip hidden file: '%s'", file->path);
elog(WARNING, "Skip hidden file: '%s'", child);
pgFileFree(file);
continue;
}
@@ -881,7 +822,7 @@ dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
*/
if (!S_ISDIR(file->mode) && !S_ISREG(file->mode))
{
elog(WARNING, "Skip '%s': unexpected file format", file->path);
elog(WARNING, "Skip '%s': unexpected file format", child);
pgFileFree(file);
continue;
}
@@ -910,7 +851,7 @@ dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
* recursively.
*/
if (S_ISDIR(file->mode))
dir_list_file_internal(files, file, exclude, follow_symlink,
dir_list_file_internal(files, file, child, exclude, follow_symlink,
external_dir_num, location);
}
@@ -919,7 +860,7 @@ dir_list_file_internal(parray *files, pgFile *parent, bool exclude,
int errno_tmp = errno;
fio_closedir(dir);
elog(ERROR, "Cannot read directory \"%s\": %s",
parent->path, strerror(errno_tmp));
parent_dir, strerror(errno_tmp));
}
fio_closedir(dir);
}
@@ -1201,16 +1142,10 @@ read_tablespace_map(parray *files, const char *backup_dir)
file = pgut_new(pgFile);
memset(file, 0, sizeof(pgFile));
file->path = pgut_malloc(strlen(link_name) + 1);
strcpy(file->path, link_name);
file->name = file->path;
file->linked = pgut_malloc(strlen(path) + 1);
strcpy(file->linked, path);
/* follow the convention for pgFileFree */
file->name = pgut_strdup(link_name);
file->linked = pgut_strdup(path);
canonicalize_path(file->linked);
canonicalize_path(file->path);
parray_append(files, file);
}
@@ -1512,12 +1447,13 @@ bad_format:
*/
parray *
dir_read_file_list(const char *root, const char *external_prefix,
const char *file_txt, fio_location location)
const char *file_txt, fio_location location, pg_crc32 expected_crc)
{
FILE *fp;
parray *files;
char buf[MAXPGPATH * 2];
char stdio_buf[STDIO_BUFSIZE];
FILE *fp;
parray *files;
char buf[BLCKSZ];
char stdio_buf[STDIO_BUFSIZE];
pg_crc32 content_crc = 0;
fp = fio_open_stream(file_txt, location);
if (fp == NULL)
@@ -1529,10 +1465,11 @@ dir_read_file_list(const char *root, const char *external_prefix,
files = parray_new();
INIT_FILE_CRC32(true, content_crc);
while (fgets(buf, lengthof(buf), fp))
{
char path[MAXPGPATH];
char filepath[MAXPGPATH];
char linked[MAXPGPATH];
char compress_alg_string[MAXPGPATH];
int64 write_size,
@@ -1546,6 +1483,8 @@ dir_read_file_list(const char *root, const char *external_prefix,
dbOid; /* used for partial restore */
pgFile *file;
COMP_FILE_CRC32(true, content_crc, buf, strlen(buf));
get_control_value(buf, "path", path, NULL, true);
get_control_value(buf, "size", NULL, &write_size, true);
get_control_value(buf, "mode", NULL, &mode, true);
@@ -1556,20 +1495,7 @@ dir_read_file_list(const char *root, const char *external_prefix,
get_control_value(buf, "external_dir_num", NULL, &external_dir_num, false);
get_control_value(buf, "dbOid", NULL, &dbOid, false);
if (external_dir_num && external_prefix)
{
char temp[MAXPGPATH];
makeExternalDirPathByNum(temp, external_prefix, external_dir_num);
join_path_components(filepath, temp, path);
}
else if (root)
join_path_components(filepath, root, path);
else
strcpy(filepath, path);
file = pgFileInit(filepath, path);
file = pgFileInit(path);
file->write_size = (int64) write_size;
file->mode = (mode_t) mode;
file->is_datafile = is_datafile ? true : false;
@@ -1598,10 +1524,21 @@ dir_read_file_list(const char *root, const char *external_prefix,
parray_append(files, file);
}
FIN_FILE_CRC32(true, content_crc);
if (ferror(fp))
elog(ERROR, "Failed to read from file: \"%s\"", file_txt);
fio_close_stream(fp);
if (expected_crc != 0 &&
expected_crc != content_crc)
{
elog(WARNING, "Invalid CRC of backup control file '%s': %u. Expected: %u",
file_txt, content_crc, expected_crc);
return NULL;
}
return files;
}
@@ -1795,12 +1732,10 @@ write_database_map(pgBackup *backup, parray *database_map, parray *backup_files_
/* Add metadata to backup_content.control */
file = pgFileNew(database_map_path, DATABASE_MAP, true, 0,
FIO_BACKUP_HOST);
pfree(file->path);
file->path = pgut_strdup(DATABASE_MAP);
file->crc = pgFileGetCRC(database_map_path, true, false);
file->write_size = file->size;
file->uncompressed_size = file->read_size;
parray_append(backup_files_list, file);
}
src/fetch.c
+1 -1
View File
@@ -44,7 +44,7 @@ slurpFile(const char *datadir, const char *path, size_t *filesize, bool safe, fi
fullpath, strerror(errno));
}
if (fio_fstat(fd, &statbuf) < 0)
if (fio_stat(fullpath, &statbuf, true, location) < 0)
{
if (safe)
return NULL;
src/merge.c
+4 -8
View File
@@ -565,13 +565,9 @@ merge_chain(parray *parent_chain, pgBackup *full_backup, pgBackup *dest_backup)
*/
for (i = parray_num(parent_chain) - 1; i >= 0; i--)
{
char control_file[MAXPGPATH];
pgBackup *backup = (pgBackup *) parray_get(parent_chain, i);
join_path_components(control_file, backup->root_dir, DATABASE_FILE_LIST);
backup->files = dir_read_file_list(NULL, NULL, control_file, FIO_BACKUP_HOST);
backup->files = get_backup_filelist(backup, true);
parray_qsort(backup->files, pgFileCompareRelPathWithExternal);
/* Set MERGING status for every member of the chain */
@@ -729,7 +725,7 @@ merge_chain(parray *parent_chain, pgBackup *full_backup, pgBackup *dest_backup)
parray_qsort(result_filelist, pgFileCompareRelPathWithExternal);
write_backup_filelist(full_backup, result_filelist, full_database_dir, NULL);
write_backup_filelist(full_backup, result_filelist, full_database_dir, NULL, true);
write_backup(full_backup, true);
/* Delete FULL backup files, that do not exists in destination backup
@@ -893,7 +889,7 @@ merge_files(void *arg)
if (!pg_atomic_test_set_flag(&dest_file->lock))
continue;
tmp_file = pgFileInit(dest_file->rel_path, dest_file->rel_path);
tmp_file = pgFileInit(dest_file->rel_path);
tmp_file->mode = dest_file->mode;
tmp_file->is_datafile = dest_file->is_datafile;
tmp_file->is_cfs = dest_file->is_cfs;
@@ -1060,7 +1056,7 @@ remove_dir_with_files(const char *path)
char full_path[MAXPGPATH];
dir_list_file(files, path, true, true, true, 0, FIO_LOCAL_HOST);
parray_qsort(files, pgFileComparePathDesc);
parray_qsort(files, pgFileCompareRelPathWithExternalDesc);
for (i = 0; i < parray_num(files); i++)
{
pgFile *file = (pgFile *) parray_get(files, i);
src/pg_probackup.c
+9 -7
View File
@@ -343,15 +343,17 @@ main(int argc, char *argv[])
else if (strcmp(argv[1], "ssh") == 0)
launch_ssh(argv);
#endif
else if (strcmp(argv[1], "agent") == 0 && argc > 2)
else if (strcmp(argv[1], "agent") == 0)
{
remote_agent = argv[2];
if (strcmp(remote_agent, PROGRAM_VERSION) != 0)
/* 'No forward compatibility' sanity:
* /old/binary -> ssh execute -> /newer/binary agent version_num
* If we are executed as an agent for older binary, then exit with error
*/
if (argc > 2)
{
uint32 agent_version = parse_program_version(remote_agent);
elog(agent_version < AGENT_PROTOCOL_VERSION ? ERROR : WARNING,
"Agent version %s doesn't match master pg_probackup version %s",
PROGRAM_VERSION, remote_agent);
elog(ERROR, "Version mismatch, pg_probackup binary with version '%s' "
"is launched as an agent for pg_probackup binary with version '%s'",
PROGRAM_VERSION, argv[2]);
}
fio_communicate(STDIN_FILENO, STDOUT_FILENO);
return 0;
src/pg_probackup.h
+32 -20
View File
@@ -118,6 +118,15 @@ typedef enum CompressAlg
ZLIB_COMPRESS,
} CompressAlg;
typedef enum ForkName
{
VM,
FSM,
CFM,
INIT,
PTRACK
} ForkName;
#define INIT_FILE_CRC32(use_crc32c, crc) \
do { \
if (use_crc32c) \
@@ -154,21 +163,20 @@ typedef struct pgFile
int64 write_size; /* size of the backed-up file. BYTES_INVALID means
that the file existed but was not backed up
because not modified since last backup. */
int64 uncompressed_size; /* size of the backed-up file before compression
size_t uncompressed_size; /* size of the backed-up file before compression
* and adding block headers.
*/
/* we need int64 here to store '-1' value */
pg_crc32 crc; /* CRC value of the file, regular file only */
char *linked; /* path of the linked file */
bool is_datafile; /* true if the file is PostgreSQL data file */
char *path; /* absolute path of the file */
char *rel_path; /* relative path of the file */
Oid tblspcOid; /* tblspcOid extracted from path, if applicable */
Oid dbOid; /* dbOid extracted from path, if applicable */
Oid relOid; /* relOid extracted from path, if applicable */
char *forkName; /* forkName extracted from path, if applicable */
ForkName forkName; /* forkName extracted from path, if applicable */
int segno; /* Segment number for ptrack */
int n_blocks; /* size of the file in blocks, readed during DELTA backup */
BlockNumber n_blocks; /* size of the data file in blocks */
bool is_cfs; /* Flag to distinguish files compressed by CFS*/
bool is_database;
int external_dir_num; /* Number of external directory. 0 if not external */
@@ -229,8 +237,8 @@ typedef enum ShowFormat
#define BYTES_INVALID (-1) /* file didn`t changed since previous backup, DELTA backup do not rely on it */
#define FILE_NOT_FOUND (-2) /* file disappeared during backup */
#define BLOCKNUM_INVALID (-1)
#define PROGRAM_VERSION "2.3.1"
#define AGENT_PROTOCOL_VERSION 20301
#define PROGRAM_VERSION "2.3.3"
#define AGENT_PROTOCOL_VERSION 20303
typedef struct ConnectionOptions
@@ -391,9 +399,13 @@ struct pgBackup
* separated by ':' */
char *root_dir; /* Full path for root backup directory:
backup_path/instance_name/backup_id */
char *database_dir; /* Full path to directory with data files:
backup_path/instance_name/backup_id/database */
parray *files; /* list of files belonging to this backup
* must be populated explicitly */
char *note;
pg_crc32 content_crc;
};
/* Recovery target for restore and validate subcommands */
@@ -708,7 +720,7 @@ extern pgRecoveryTarget *parseRecoveryTargetOptions(
extern parray *get_dbOid_exclude_list(pgBackup *backup, parray *datname_list,
PartialRestoreType partial_restore_type);
extern parray *get_backup_filelist(pgBackup *backup);
extern parray *get_backup_filelist(pgBackup *backup, bool strict);
extern parray *read_timeline_history(const char *arclog_path, TimeLineID targetTLI);
/* in merge.c */
@@ -803,7 +815,7 @@ extern void pin_backup(pgBackup *target_backup,
extern void add_note(pgBackup *target_backup, char *note);
extern void pgBackupWriteControl(FILE *out, pgBackup *backup);
extern void write_backup_filelist(pgBackup *backup, parray *files,
const char *root, parray *external_list);
const char *root, parray *external_list, bool sync);
extern void pgBackupGetPath(const pgBackup *backup, char *path, size_t len,
const char *subdir);
@@ -867,7 +879,7 @@ extern void db_map_entry_free(void *map);
extern void print_file_list(FILE *out, const parray *files, const char *root,
const char *external_prefix, parray *external_list);
extern parray *dir_read_file_list(const char *root, const char *external_prefix,
const char *file_txt, fio_location location);
const char *file_txt, fio_location location, pg_crc32 expected_crc);
extern parray *make_external_directory_list(const char *colon_separated_dirs,
bool remap);
extern void free_dir_list(parray *list);
@@ -884,7 +896,7 @@ extern size_t pgFileSize(const char *path);
extern pgFile *pgFileNew(const char *path, const char *rel_path,
bool follow_symlink, int external_dir_num,
fio_location location);
extern pgFile *pgFileInit(const char *path, const char *rel_path);
extern pgFile *pgFileInit(const char *rel_path);
extern void pgFileDelete(pgFile *file, const char *full_path);
extern void pgFileFree(void *file);
@@ -892,14 +904,10 @@ extern void pgFileFree(void *file);
extern pg_crc32 pgFileGetCRC(const char *file_path, bool missing_ok, bool use_crc32c);
extern pg_crc32 pgFileGetCRCgz(const char *file_path, bool missing_ok, bool use_crc32c);
extern int pgFileCompareName(const void *f1, const void *f2);
extern int pgFileComparePath(const void *f1, const void *f2);
extern int pgFileMapComparePath(const void *f1, const void *f2);
extern int pgFileComparePathWithExternal(const void *f1, const void *f2);
extern int pgFileCompareName(const void *f1, const void *f2);
extern int pgFileCompareRelPathWithExternal(const void *f1, const void *f2);
extern int pgFileCompareRelPathWithExternalDesc(const void *f1, const void *f2);
extern int pgFileComparePathDesc(const void *f1, const void *f2);
extern int pgFileComparePathWithExternalDesc(const void *f1, const void *f2);
extern int pgFileCompareLinked(const void *f1, const void *f2);
extern int pgFileCompareSize(const void *f1, const void *f2);
extern int pgCompareOid(const void *f1, const void *f2);
@@ -934,7 +942,7 @@ extern void restore_non_data_file_internal(FILE *in, FILE *out, pgFile *file,
extern bool create_empty_file(fio_location from_location, const char *to_root,
fio_location to_location, pgFile *file);
extern bool check_file_pages(pgFile *file, XLogRecPtr stop_lsn,
extern bool check_file_pages(pgFile *file, const char *fullpath, XLogRecPtr stop_lsn,
uint32 checksum_version, uint32 backup_version);
/* parsexlog.c */
extern bool extractPageMap(const char *archivedir, uint32 wal_seg_size,
@@ -998,11 +1006,12 @@ extern void parse_filelist_filenames(parray *files, const char *root);
/* in ptrack.c */
extern void make_pagemap_from_ptrack_1(parray* files, PGconn* backup_conn);
extern void make_pagemap_from_ptrack_2(parray* files, PGconn* backup_conn,
const char *ptrack_schema, XLogRecPtr lsn);
const char *ptrack_schema,
int ptrack_version_num,
XLogRecPtr lsn);
extern void pg_ptrack_clear(PGconn *backup_conn, int ptrack_version_num);
extern void get_ptrack_version(PGconn *backup_conn, PGNodeInfo *nodeInfo);
extern bool pg_ptrack_enable(PGconn *backup_conn);
extern bool pg_ptrack_enable2(PGconn *backup_conn);
extern bool pg_ptrack_enable(PGconn *backup_conn, int ptrack_version_num);
extern bool pg_ptrack_get_and_clear_db(Oid dbOid, Oid tblspcOid, PGconn *backup_conn);
extern char *pg_ptrack_get_and_clear(Oid tablespace_oid,
Oid db_oid,
@@ -1010,7 +1019,8 @@ extern char *pg_ptrack_get_and_clear(Oid tablespace_oid,
size_t *result_size,
PGconn *backup_conn);
extern XLogRecPtr get_last_ptrack_lsn(PGconn *backup_conn, PGNodeInfo *nodeInfo);
extern parray * pg_ptrack_get_pagemapset(PGconn *backup_conn, const char *ptrack_schema, XLogRecPtr lsn);
extern parray * pg_ptrack_get_pagemapset(PGconn *backup_conn, const char *ptrack_schema,
int ptrack_version_num, XLogRecPtr lsn);
/* FIO */
extern int fio_send_pages(FILE* in, FILE* out, pgFile *file, XLogRecPtr horizonLsn,
@@ -1021,6 +1031,8 @@ extern int fio_send_pages(FILE* in, FILE* out, pgFile *file, XLogRecPtr horizonL
extern int fio_send_file_gz(const char *from_fullpath, const char *to_fullpath, FILE* out, int thread_num);
extern int fio_send_file(const char *from_fullpath, const char *to_fullpath, FILE* out, int thread_num);
extern bool pgut_rmtree(const char *path, bool rmtopdir, bool strict);
/* return codes for fio_send_pages() and fio_send_file() */
#define SEND_OK (0)
#define FILE_MISSING (-1)
src/ptrack.c
+49 -45
View File
@@ -24,7 +24,7 @@
/*
* Given a list of files in the instance to backup, build a pagemap for each
* data file that has ptrack. Result is saved in the pagemap field of pgFile.
* NOTE we rely on the fact that provided parray is sorted by file->path.
* NOTE we rely on the fact that provided parray is sorted by file->rel_path.
*/
void
make_pagemap_from_ptrack_1(parray *files, PGconn *backup_conn)
@@ -46,11 +46,6 @@ make_pagemap_from_ptrack_1(parray *files, PGconn *backup_conn)
*/
if (file->is_database)
{
char *filename = strrchr(file->path, '/');
Assert(filename != NULL);
filename++;
/*
* The function pg_ptrack_get_and_clear_db returns true
* if there was a ptrack_init file.
@@ -71,7 +66,7 @@ make_pagemap_from_ptrack_1(parray *files, PGconn *backup_conn)
file->dbOid == dbOid_with_ptrack_init)
{
/* ignore ptrack if ptrack_init exists */
elog(VERBOSE, "Ignoring ptrack because of ptrack_init for file: %s", file->path);
elog(VERBOSE, "Ignoring ptrack because of ptrack_init for file: %s", file->rel_path);
file->pagemap_isabsent = true;
continue;
}
@@ -106,7 +101,7 @@ make_pagemap_from_ptrack_1(parray *files, PGconn *backup_conn)
*/
if (start_addr > ptrack_nonparsed_size)
{
elog(VERBOSE, "Ptrack is missing for file: %s", file->path);
elog(VERBOSE, "Ptrack is missing for file: %s", file->rel_path);
file->pagemap_isabsent = true;
}
else
@@ -137,7 +132,7 @@ make_pagemap_from_ptrack_1(parray *files, PGconn *backup_conn)
* i.e. CREATE DATABASE
* - target relation was deleted.
*/
elog(VERBOSE, "Ptrack is missing for file: %s", file->path);
elog(VERBOSE, "Ptrack is missing for file: %s", file->rel_path);
file->pagemap_isabsent = true;
}
}
@@ -204,6 +199,8 @@ get_ptrack_version(PGconn *backup_conn, PGNodeInfo *nodeInfo)
nodeInfo->ptrack_version_num = 17;
else if (strcmp(ptrack_version_str, "2.0") == 0)
nodeInfo->ptrack_version_num = 20;
else if (strcmp(ptrack_version_str, "2.1") == 0)
nodeInfo->ptrack_version_num = 21;
else
elog(WARNING, "Update your ptrack to the version 1.5 or upper. Current version is %s",
ptrack_version_str);
@@ -215,19 +212,30 @@ get_ptrack_version(PGconn *backup_conn, PGNodeInfo *nodeInfo)
* Check if ptrack is enabled in target instance
*/
bool
pg_ptrack_enable(PGconn *backup_conn)
pg_ptrack_enable(PGconn *backup_conn, int ptrack_version_num)
{
PGresult *res_db;
bool result = false;
res_db = pgut_execute(backup_conn, "SHOW ptrack_enable", 0, NULL);
if (strcmp(PQgetvalue(res_db, 0, 0), "on") != 0)
if (ptrack_version_num < 20)
{
PQclear(res_db);
return false;
res_db = pgut_execute(backup_conn, "SHOW ptrack_enable", 0, NULL);
result = strcmp(PQgetvalue(res_db, 0, 0), "on") == 0;
}
else if (ptrack_version_num == 20)
{
res_db = pgut_execute(backup_conn, "SHOW ptrack_map_size", 0, NULL);
result = strcmp(PQgetvalue(res_db, 0, 0), "0") != 0;
}
else
{
res_db = pgut_execute(backup_conn, "SHOW ptrack.map_size", 0, NULL);
result = strcmp(PQgetvalue(res_db, 0, 0), "0") != 0 &&
strcmp(PQgetvalue(res_db, 0, 0), "-1") != 0;
}
PQclear(res_db);
return true;
return result;
}
@@ -457,7 +465,11 @@ get_last_ptrack_lsn(PGconn *backup_conn, PGNodeInfo *nodeInfo)
{
char query[128];
sprintf(query, "SELECT %s.pg_ptrack_control_lsn()", nodeInfo->ptrack_schema);
if (nodeInfo->ptrack_version_num == 20)
sprintf(query, "SELECT %s.pg_ptrack_control_lsn()", nodeInfo->ptrack_schema);
else
sprintf(query, "SELECT %s.ptrack_init_lsn()", nodeInfo->ptrack_schema);
res = pgut_execute(backup_conn, query, 0, NULL);
}
@@ -509,7 +521,7 @@ pg_ptrack_get_block(ConnectionArgs *arguments,
if (arguments->cancel_conn == NULL)
arguments->cancel_conn = PQgetCancel(arguments->conn);
//elog(LOG, "db %i pg_ptrack_get_block(%i, %i, %u)",dbOid, tblsOid, relOid, blknum);
// elog(LOG, "db %i pg_ptrack_get_block(%i, %i, %u)",dbOid, tblsOid, relOid, blknum);
if (ptrack_version_num < 20)
res = pgut_execute_parallel(arguments->conn,
@@ -524,7 +536,11 @@ pg_ptrack_get_block(ConnectionArgs *arguments,
if (!ptrack_schema)
elog(ERROR, "Schema name of ptrack extension is missing");
sprintf(query, "SELECT %s.pg_ptrack_get_block($1, $2, $3, $4)", ptrack_schema);
if (ptrack_version_num == 20)
sprintf(query, "SELECT %s.pg_ptrack_get_block($1, $2, $3, $4)", ptrack_schema);
else
elog(ERROR, "ptrack >= 2.1.0 does not support pg_ptrack_get_block()");
// sprintf(query, "SELECT %s.ptrack_get_block($1, $2, $3, $4)", ptrack_schema);
res = pgut_execute_parallel(arguments->conn,
arguments->cancel_conn,
@@ -564,30 +580,12 @@ pg_ptrack_get_block(ConnectionArgs *arguments,
* ----------------------------
*/
/*
* Check if ptrack is enabled in target instance
*/
bool
pg_ptrack_enable2(PGconn *backup_conn)
{
PGresult *res_db;
res_db = pgut_execute(backup_conn, "SHOW ptrack_map_size", 0, NULL);
if (strcmp(PQgetvalue(res_db, 0, 0), "0") == 0)
{
PQclear(res_db);
return false;
}
PQclear(res_db);
return true;
}
/*
* Fetch a list of changed files with their ptrack maps.
*/
parray *
pg_ptrack_get_pagemapset(PGconn *backup_conn, const char *ptrack_schema, XLogRecPtr lsn)
pg_ptrack_get_pagemapset(PGconn *backup_conn, const char *ptrack_schema,
int ptrack_version_num, XLogRecPtr lsn)
{
PGresult *res;
char lsn_buf[17 + 1];
@@ -602,8 +600,12 @@ pg_ptrack_get_pagemapset(PGconn *backup_conn, const char *ptrack_schema, XLogRec
if (!ptrack_schema)
elog(ERROR, "Schema name of ptrack extension is missing");
sprintf(query, "SELECT path, pagemap FROM %s.pg_ptrack_get_pagemapset($1) ORDER BY 1",
ptrack_schema);
if (ptrack_version_num == 20)
sprintf(query, "SELECT path, pagemap FROM %s.pg_ptrack_get_pagemapset($1) ORDER BY 1",
ptrack_schema);
else
sprintf(query, "SELECT path, pagemap FROM %s.ptrack_get_pagemapset($1) ORDER BY 1",
ptrack_schema);
res = pgut_execute(backup_conn, query, 1, (const char **) params);
pfree(params[0]);
@@ -645,16 +647,18 @@ pg_ptrack_get_pagemapset(PGconn *backup_conn, const char *ptrack_schema, XLogRec
*/
void
make_pagemap_from_ptrack_2(parray *files,
PGconn *backup_conn,
const char *ptrack_schema,
XLogRecPtr lsn)
PGconn *backup_conn,
const char *ptrack_schema,
int ptrack_version_num,
XLogRecPtr lsn)
{
parray *filemaps;
int file_i = 0;
page_map_entry *dummy_map = NULL;
/* Receive all available ptrack bitmaps at once */
filemaps = pg_ptrack_get_pagemapset(backup_conn, ptrack_schema, lsn);
filemaps = pg_ptrack_get_pagemapset(backup_conn, ptrack_schema,
ptrack_version_num, lsn);
if (filemaps != NULL)
parray_qsort(filemaps, pgFileMapComparePath);
src/restore.c
+3 -8
View File
@@ -494,7 +494,6 @@ restore_chain(pgBackup *dest_backup, parray *parent_chain,
const char *pgdata_path, bool no_sync)
{
int i;
char control_file[MAXPGPATH];
char timestamp[100];
parray *dest_files = NULL;
parray *external_dirs = NULL;
@@ -515,8 +514,7 @@ restore_chain(pgBackup *dest_backup, parray *parent_chain,
time2iso(timestamp, lengthof(timestamp), dest_backup->start_time);
elog(INFO, "Restoring the database from backup at %s", timestamp);
join_path_components(control_file, dest_backup->root_dir, DATABASE_FILE_LIST);
dest_files = dir_read_file_list(NULL, NULL, control_file, FIO_BACKUP_HOST);
dest_files = get_backup_filelist(dest_backup, true);
/* Lock backup chain and make sanity checks */
for (i = parray_num(parent_chain) - 1; i >= 0; i--)
@@ -550,10 +548,7 @@ restore_chain(pgBackup *dest_backup, parray *parent_chain,
/* populate backup filelist */
if (backup->start_time != dest_backup->start_time)
{
join_path_components(control_file, backup->root_dir, DATABASE_FILE_LIST);
backup->files = dir_read_file_list(NULL, NULL, control_file, FIO_BACKUP_HOST);
}
backup->files = get_backup_filelist(backup, true);
else
backup->files = dest_files;
@@ -1499,7 +1494,7 @@ get_dbOid_exclude_list(pgBackup *backup, parray *datname_list,
char database_map_path[MAXPGPATH];
parray *files = NULL;
files = get_backup_filelist(backup);
files = get_backup_filelist(backup, true);
/* look for 'database_map' file in backup_content.control */
for (i = 0; i < parray_num(files); i++)
src/show.c
+6 -1
View File
@@ -430,11 +430,16 @@ print_backup_json_object(PQExpBuffer buf, pgBackup *backup)
json_add_value(buf, "status", status2str(backup->status), json_level,
true);
if (backup->note){
if (backup->note)
json_add_value(buf, "note", backup->note,
json_level, true);
if (backup->content_crc != 0)
{
json_add_key(buf, "content-crc", json_level);
appendPQExpBuffer(buf, "%u", backup->content_crc);
}
json_add(buf, JT_END_OBJECT, &json_level);
}
src/utils/file.c
+23 -46
View File
@@ -185,6 +185,19 @@ static ssize_t fio_write_all(int fd, void const* buf, size_t size)
return offs;
}
/* Get version of remote agent */
int fio_get_agent_version(void)
{
fio_header hdr;
hdr.cop = FIO_AGENT_VERSION;
hdr.size = 0;
IO_CHECK(fio_write_all(fio_stdout, &hdr, sizeof(hdr)), sizeof(hdr));
IO_CHECK(fio_read_all(fio_stdin, &hdr, sizeof(hdr)), sizeof(hdr));
return hdr.arg;
}
/* Open input stream. Remote file is fetched to the in-memory buffer and then accessed through Linux fmemopen */
FILE* fio_open_stream(char const* path, fio_location location)
{
@@ -249,7 +262,8 @@ DIR* fio_opendir(char const* path, fio_location location)
mask = fio_fdset;
for (i = 0; (mask & 1) != 0; i++, mask >>= 1);
if (i == FIO_FDMAX) {
return NULL;
elog(ERROR, "Descriptor pool for remote files is exhausted, "
"probably too many remote directories are opened");
}
hdr.cop = FIO_OPENDIR;
hdr.handle = i;
@@ -264,6 +278,7 @@ DIR* fio_opendir(char const* path, fio_location location)
if (hdr.arg != 0)
{
errno = hdr.arg;
fio_fdset &= ~(1 << hdr.handle);
return NULL;
}
dir = (DIR*)(size_t)(i + 1);
@@ -335,9 +350,10 @@ int fio_open(char const* path, int mode, fio_location location)
mask = fio_fdset;
for (i = 0; (mask & 1) != 0; i++, mask >>= 1);
if (i == FIO_FDMAX) {
return -1;
}
if (i == FIO_FDMAX)
elog(ERROR, "Descriptor pool for remote files is exhausted, "
"probably too many remote files are opened");
hdr.cop = FIO_OPEN;
hdr.handle = i;
hdr.size = strlen(path) + 1;
@@ -355,6 +371,7 @@ int fio_open(char const* path, int mode, fio_location location)
if (hdr.arg != 0)
{
errno = hdr.arg;
fio_fdset &= ~(1 << hdr.handle);
return -1;
}
fd = i | FIO_PIPE_MARKER;
@@ -728,44 +745,6 @@ ssize_t fio_read(int fd, void* buf, size_t size)
}
}
/* Get information about stdio file */
int fio_ffstat(FILE* f, struct stat* st)
{
return fio_is_remote_file(f)
? fio_fstat(fio_fileno(f), st)
: fio_fstat(fileno(f), st);
}
/* Get information about file descriptor */
int fio_fstat(int fd, struct stat* st)
{
if (fio_is_remote_fd(fd))
{
fio_header hdr;
hdr.cop = FIO_FSTAT;
hdr.handle = fd & ~FIO_PIPE_MARKER;
hdr.size = 0;
IO_CHECK(fio_write_all(fio_stdout, &hdr, sizeof(hdr)), sizeof(hdr));
IO_CHECK(fio_read_all(fio_stdin, &hdr, sizeof(hdr)), sizeof(hdr));
Assert(hdr.cop == FIO_FSTAT);
IO_CHECK(fio_read_all(fio_stdin, st, sizeof(*st)), sizeof(*st));
if (hdr.arg != 0)
{
errno = hdr.arg;
return -1;
}
return 0;
}
else
{
return fstat(fd, st);
}
}
/* Get information about file */
int fio_stat(char const* path, struct stat* st, bool follow_symlink, fio_location location)
{
@@ -2065,11 +2044,9 @@ void fio_communicate(int in, int out)
if (hdr.size != 0)
IO_CHECK(fio_write_all(out, buf, hdr.size), hdr.size);
break;
case FIO_FSTAT: /* Get information about opened file */
hdr.size = sizeof(st);
hdr.arg = fstat(fd[hdr.handle], &st) < 0 ? errno : 0;
case FIO_AGENT_VERSION:
hdr.arg = AGENT_PROTOCOL_VERSION;
IO_CHECK(fio_write_all(out, &hdr, sizeof(hdr)), sizeof(hdr));
IO_CHECK(fio_write_all(out, &st, sizeof(st)), sizeof(st));
break;
case FIO_STAT: /* Get information about file with specified path */
hdr.size = sizeof(st);
src/utils/file.h
+3 -1
View File
@@ -27,7 +27,6 @@ typedef enum
FIO_READ,
FIO_LOAD,
FIO_STAT,
FIO_FSTAT,
FIO_SEND,
FIO_ACCESS,
FIO_OPENDIR,
@@ -47,6 +46,8 @@ typedef enum
/* messages for closing connection */
FIO_DISCONNECT,
FIO_DISCONNECTED,
/* message for compatibility check */
FIO_AGENT_VERSION
} fio_operations;
typedef enum
@@ -79,6 +80,7 @@ extern fio_location MyLocation;
extern void fio_redirect(int in, int out, int err);
extern void fio_communicate(int in, int out);
extern int fio_get_agent_version(void);
extern FILE* fio_fopen(char const* name, char const* mode, fio_location location);
extern size_t fio_fwrite(FILE* f, void const* buf, size_t size);
extern ssize_t fio_fwrite_compressed(FILE* f, void const* buf, size_t size, int compress_alg);
src/utils/pgut.c
+140
View File
@@ -43,6 +43,9 @@ static void on_interrupt(void);
static void on_cleanup(void);
static pqsigfunc oldhandler = NULL;
static char ** pgut_pgfnames(const char *path, bool strict);
static void pgut_pgfnames_cleanup(char **filenames);
void discard_response(PGconn *conn);
void
@@ -1062,3 +1065,140 @@ discard_response(PGconn *conn)
PQclear(res);
} while (res);
}
/*
* pgfnames
*
* return a list of the names of objects in the argument directory. Caller
* must call pgfnames_cleanup later to free the memory allocated by this
* function.
*/
char **
pgut_pgfnames(const char *path, bool strict)
{
DIR *dir;
struct dirent *file;
char **filenames;
int numnames = 0;
int fnsize = 200; /* enough for many small dbs */
dir = opendir(path);
if (dir == NULL)
{
elog(strict ? ERROR : WARNING, "could not open directory \"%s\": %m", path);
return NULL;
}
filenames = (char **) palloc(fnsize * sizeof(char *));
while (errno = 0, (file = readdir(dir)) != NULL)
{
if (strcmp(file->d_name, ".") != 0 && strcmp(file->d_name, "..") != 0)
{
if (numnames + 1 >= fnsize)
{
fnsize *= 2;
filenames = (char **) repalloc(filenames,
fnsize * sizeof(char *));
}
filenames[numnames++] = pstrdup(file->d_name);
}
}
if (errno)
{
elog(strict ? ERROR : WARNING, "could not read directory \"%s\": %m", path);
return NULL;
}
filenames[numnames] = NULL;
if (closedir(dir))
{
elog(strict ? ERROR : WARNING, "could not close directory \"%s\": %m", path);
return NULL;
}
return filenames;
}
/*
* pgfnames_cleanup
*
* deallocate memory used for filenames
*/
void
pgut_pgfnames_cleanup(char **filenames)
{
char **fn;
for (fn = filenames; *fn; fn++)
pfree(*fn);
pfree(filenames);
}
/* Shamelessly stolen from commom/rmtree.c */
bool
pgut_rmtree(const char *path, bool rmtopdir, bool strict)
{
bool result = true;
char pathbuf[MAXPGPATH];
char **filenames;
char **filename;
struct stat statbuf;
/*
* we copy all the names out of the directory before we start modifying
* it.
*/
filenames = pgut_pgfnames(path, strict);
if (filenames == NULL)
return false;
/* now we have the names we can start removing things */
for (filename = filenames; *filename; filename++)
{
snprintf(pathbuf, MAXPGPATH, "%s/%s", path, *filename);
if (lstat(pathbuf, &statbuf) != 0)
{
elog(strict ? ERROR : WARNING, "could not stat file or directory \"%s\": %m", pathbuf);
result = false;
break;
}
if (S_ISDIR(statbuf.st_mode))
{
/* call ourselves recursively for a directory */
if (!pgut_rmtree(pathbuf, true, strict))
{
result = false;
break;
}
}
else
{
if (unlink(pathbuf) != 0)
{
elog(strict ? ERROR : WARNING, "could not remove file or directory \"%s\": %m", pathbuf);
result = false;
break;
}
}
}
if (rmtopdir)
{
if (rmdir(path) != 0)
{
elog(strict ? ERROR : WARNING, "could not remove file or directory \"%s\": %m", path);
result = false;
}
}
pgut_pgfnames_cleanup(filenames);
return result;
}
src/utils/remote.c
+28 -10
View File
@@ -111,6 +111,7 @@ bool launch_agent(void)
int outfd[2];
int infd[2];
int errfd[2];
int agent_version;
ssh_argc = 0;
#ifdef WIN32
@@ -163,24 +164,24 @@ bool launch_agent(void)
}
}
if (needs_quotes(instance_config.remote.path) || needs_quotes(PROGRAM_NAME_FULL))
snprintf(cmd, sizeof(cmd), "\"%s\\%s\" agent %s",
instance_config.remote.path, probackup, PROGRAM_VERSION);
snprintf(cmd, sizeof(cmd), "\"%s\\%s\" agent",
instance_config.remote.path, probackup);
else
snprintf(cmd, sizeof(cmd), "%s\\%s agent %s",
instance_config.remote.path, probackup, PROGRAM_VERSION);
snprintf(cmd, sizeof(cmd), "%s\\%s agent",
instance_config.remote.path, probackup);
#else
if (needs_quotes(instance_config.remote.path) || needs_quotes(PROGRAM_NAME_FULL))
snprintf(cmd, sizeof(cmd), "\"%s/%s\" agent %s",
instance_config.remote.path, probackup, PROGRAM_VERSION);
snprintf(cmd, sizeof(cmd), "\"%s/%s\" agent",
instance_config.remote.path, probackup);
else
snprintf(cmd, sizeof(cmd), "%s/%s agent %s",
instance_config.remote.path, probackup, PROGRAM_VERSION);
snprintf(cmd, sizeof(cmd), "%s/%s agent",
instance_config.remote.path, probackup);
#endif
} else {
if (needs_quotes(PROGRAM_NAME_FULL))
snprintf(cmd, sizeof(cmd), "\"%s\" agent %s", PROGRAM_NAME_FULL, PROGRAM_VERSION);
snprintf(cmd, sizeof(cmd), "\"%s\" agent", PROGRAM_NAME_FULL);
else
snprintf(cmd, sizeof(cmd), "%s agent %s", PROGRAM_NAME_FULL, PROGRAM_VERSION);
snprintf(cmd, sizeof(cmd), "%s agent", PROGRAM_NAME_FULL);
}
#ifdef WIN32
@@ -228,5 +229,22 @@ bool launch_agent(void)
fio_redirect(infd[0], outfd[1], errfd[0]); /* write to stdout */
}
/* Make sure that remote agent has the same version
* TODO: we must also check PG version and fork edition
*/
agent_version = fio_get_agent_version();
if (agent_version != AGENT_PROTOCOL_VERSION)
{
char agent_version_str[1024];
sprintf(agent_version_str, "%d.%d.%d",
agent_version / 10000,
(agent_version / 100) % 100,
agent_version % 100);
elog(ERROR, "Remote agent version %s does not match local program version %s",
agent_version_str, PROGRAM_VERSION);
}
return true;
}
src/validate.c
+35 -22
View File
@@ -31,6 +31,7 @@ typedef struct
uint32 backup_version;
BackupMode backup_mode;
parray *dbOid_exclude_list;
const char *external_prefix;
/*
* Return value from the thread.
@@ -48,8 +49,7 @@ pgBackupValidate(pgBackup *backup, pgRestoreParams *params)
{
char base_path[MAXPGPATH];
char external_prefix[MAXPGPATH];
char path[MAXPGPATH];
parray *files;
parray *files = NULL;
bool corrupted = false;
bool validation_isok = true;
/* arrays with meta info for multi threaded validate */
@@ -110,8 +110,15 @@ pgBackupValidate(pgBackup *backup, pgRestoreParams *params)
join_path_components(base_path, backup->root_dir, DATABASE_DIR);
join_path_components(external_prefix, backup->root_dir, EXTERNAL_DIR);
join_path_components(path, backup->root_dir, DATABASE_FILE_LIST);
files = dir_read_file_list(base_path, external_prefix, path, FIO_BACKUP_HOST);
files = get_backup_filelist(backup, false);
if (!files)
{
elog(WARNING, "Backup %s file list is corrupted", base36enc(backup->start_time));
backup->status = BACKUP_STATUS_CORRUPT;
write_backup_status(backup, BACKUP_STATUS_CORRUPT, instance_name, true);
return;
}
// if (params && params->partial_db_list)
// dbOid_exclude_list = get_dbOid_exclude_list(backup, files, params->partial_db_list,
@@ -142,6 +149,7 @@ pgBackupValidate(pgBackup *backup, pgRestoreParams *params)
arg->stop_lsn = backup->stop_lsn;
arg->checksum_version = backup->checksum_version;
arg->backup_version = parse_program_version(backup->program_version);
arg->external_prefix = external_prefix;
// arg->dbOid_exclude_list = dbOid_exclude_list;
/* By default there are some error */
threads_args[i].ret = 1;
@@ -223,6 +231,7 @@ pgBackupValidateFiles(void *arg)
{
struct stat st;
pgFile *file = (pgFile *) parray_get(arguments->files, i);
char file_fullpath[MAXPGPATH];
if (interrupted || thread_interrupted)
elog(ERROR, "Interrupted during validate");
@@ -244,20 +253,12 @@ pgBackupValidateFiles(void *arg)
// continue;
//}
/*
* Currently we don't compute checksums for
* cfs_compressed data files, so skip them.
* TODO: investigate
*/
if (file->is_cfs)
continue;
if (!pg_atomic_test_set_flag(&file->lock))
continue;
if (progress)
elog(INFO, "Progress: (%d/%d). Validate file \"%s\"",
i + 1, num_files, file->path);
i + 1, num_files, file->rel_path);
/*
* Skip files which has no data, because they
@@ -270,7 +271,7 @@ pgBackupValidateFiles(void *arg)
{
/* It is illegal for file in FULL backup to have BYTES_INVALID */
elog(WARNING, "Backup file \"%s\" has invalid size. Possible metadata corruption.",
file->path);
file->rel_path);
arguments->corrupted = true;
break;
}
@@ -282,14 +283,24 @@ pgBackupValidateFiles(void *arg)
if (file->write_size == 0)
continue;
if (file->external_dir_num)
{
char temp[MAXPGPATH];
makeExternalDirPathByNum(temp, arguments->external_prefix, file->external_dir_num);
join_path_components(file_fullpath, temp, file->rel_path);
}
else
join_path_components(file_fullpath, arguments->base_path, file->rel_path);
/* TODO: it is redundant to check file existence using stat */
if (stat(file->path, &st) == -1)
if (stat(file_fullpath, &st) == -1)
{
if (errno == ENOENT)
elog(WARNING, "Backup file \"%s\" is not found", file->path);
elog(WARNING, "Backup file \"%s\" is not found", file_fullpath);
else
elog(WARNING, "Cannot stat backup file \"%s\": %s",
file->path, strerror(errno));
file_fullpath, strerror(errno));
arguments->corrupted = true;
break;
}
@@ -297,7 +308,7 @@ pgBackupValidateFiles(void *arg)
if (file->write_size != st.st_size)
{
elog(WARNING, "Invalid size of backup file \"%s\" : " INT64_FORMAT ". Expected %lu",
file->path, (unsigned long) st.st_size, file->write_size);
file_fullpath, (unsigned long) st.st_size, file->write_size);
arguments->corrupted = true;
break;
}
@@ -305,8 +316,10 @@ pgBackupValidateFiles(void *arg)
/*
* If option skip-block-validation is set, compute only file-level CRC for
* datafiles, otherwise check them block by block.
* Currently we don't compute checksums for
* cfs_compressed data files, so skip block validation for them.
*/
if (!file->is_datafile || skip_block_validation)
if (!file->is_datafile || skip_block_validation || file->is_cfs)
{
/*
* Pre 2.0.22 we use CRC-32C, but in newer version of pg_probackup we
@@ -326,14 +339,14 @@ pgBackupValidateFiles(void *arg)
!file->external_dir_num)
crc = get_pgcontrol_checksum(arguments->base_path);
else
crc = pgFileGetCRC(file->path,
crc = pgFileGetCRC(file_fullpath,
arguments->backup_version <= 20021 ||
arguments->backup_version >= 20025,
false);
if (crc != file->crc)
{
elog(WARNING, "Invalid CRC of backup file \"%s\" : %X. Expected %X",
file->path, crc, file->crc);
file_fullpath, crc, file->crc);
arguments->corrupted = true;
}
}
@@ -344,7 +357,7 @@ pgBackupValidateFiles(void *arg)
* check page headers, checksums (if enabled)
* and compute checksum of the file
*/
if (!check_file_pages(file, arguments->stop_lsn,
if (!check_file_pages(file, file_fullpath, arguments->stop_lsn,
arguments->checksum_version,
arguments->backup_version))
arguments->corrupted = true;
tests/archive.py
+4
View File
@@ -977,6 +977,10 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
replica.promote()
replica.safe_psql(
'postgres',
'CHECKPOINT')
master.pgbench_init(scale=10)
replica.pgbench_init(scale=10)
tests/backup.py
+215 -114
View File
@@ -147,10 +147,9 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"without valid full backup.\n Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertIn(
"ERROR: Valid backup on current timeline 1 is not found. "
"Create new FULL backup before an incremental one.",
e.message,
self.assertTrue(
"WARNING: Valid backup on current timeline 1 is not found" in e.message and
"ERROR: Create new full backup before an incremental one" in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
@@ -165,10 +164,9 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"without valid full backup.\n Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertIn(
"ERROR: Valid backup on current timeline 1 is not found. "
"Create new FULL backup before an incremental one.",
e.message,
self.assertTrue(
"WARNING: Valid backup on current timeline 1 is not found" in e.message and
"ERROR: Create new full backup before an incremental one" in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
@@ -176,6 +174,10 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.show_pb(backup_dir, 'node')[0]['status'],
"ERROR")
self.assertEqual(
self.show_pb(backup_dir, 'node')[1]['status'],
"ERROR")
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -315,10 +317,8 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_page_corruption_heal_via_ptrack_1(self):
def test_page_detect_corruption(self):
"""make node, corrupt some page, check that backup failed"""
if not self.ptrack:
return unittest.skip('Skipped because ptrack support is disabled')
fname = self.id().split('.')[3]
node = self.make_simple_node(
@@ -380,98 +380,6 @@ class BackupTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_page_corruption_heal_via_ptrack_2(self):
"""make node, corrupt some page, check that backup failed"""
if not self.ptrack:
return unittest.skip('Skipped because ptrack support is disabled')
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
ptrack_enable=True,
initdb_params=['--data-checksums'])
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
if node.major_version >= 12:
node.safe_psql(
"postgres",
"CREATE EXTENSION ptrack WITH SCHEMA pg_catalog")
self.backup_node(
backup_dir, 'node', node, backup_type="full",
options=["-j", "4", "--stream"])
node.safe_psql(
"postgres",
"create table t_heap as select 1 as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,1000) i")
node.safe_psql(
"postgres",
"CHECKPOINT;")
heap_path = node.safe_psql(
"postgres",
"select pg_relation_filepath('t_heap')").rstrip()
node.stop()
with open(os.path.join(node.data_dir, heap_path), "rb+", 0) as f:
f.seek(9000)
f.write(b"bla")
f.flush()
f.close
node.slow_start()
try:
self.backup_node(
backup_dir, 'node', node, backup_type="full",
options=["-j", "4", "--stream", '--log-level-console=LOG'])
# we should die here because exception is what we expect to happen
self.assertEqual(
1, 0,
"Expecting Error because of page "
"corruption in PostgreSQL instance.\n"
" Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
if self.remote:
self.assertTrue(
"WARNING: File" in e.message and
"try to fetch via shared buffer" in e.message and
"WARNING: page verification failed, "
"calculated checksum" in e.message and
"ERROR: query failed: "
"ERROR: invalid page in block" in e.message and
"query was: SELECT pg_catalog.pg_ptrack_get_block" in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
else:
self.assertTrue(
"LOG: File" in e.message and
"blknum" in e.message and
"have wrong checksum" in e.message and
"try to fetch via shared buffer" in e.message and
"WARNING: page verification failed, "
"calculated checksum" in e.message and
"ERROR: query failed: "
"ERROR: invalid page in block" in e.message and
"query was: SELECT pg_catalog.pg_ptrack_get_block" in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
self.assertTrue(
self.show_pb(backup_dir, 'node')[1]['status'] == 'ERROR',
"Backup Status should be ERROR")
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_backup_detect_corruption(self):
@@ -495,6 +403,10 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"postgres",
"create extension ptrack")
self.backup_node(
backup_dir, 'node', node,
backup_type="full", options=["-j", "4", "--stream"])
node.safe_psql(
"postgres",
"create table t_heap as select 1 as id, md5(i::text) as text, "
@@ -529,10 +441,6 @@ class BackupTest(ProbackupTest, unittest.TestCase):
node.slow_start()
# self.backup_node(
# backup_dir, 'node', node,
# backup_type="full", options=["-j", "4", "--stream"])
try:
self.backup_node(
backup_dir, 'node', node,
@@ -608,12 +516,11 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"\n Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertTrue(
'WARNING: page verification failed, '
'calculated checksum' in e.message and
'ERROR: query failed: ERROR: '
'invalid page in block 1 of relation' in e.message and
'ERROR: Data files transferring failed' in e.message,
self.assertIn(
'ERROR: Corruption detected in file "{0}", block 1: '
'page verification failed, calculated checksum'.format(
heap_fullpath),
e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(e.message), self.cmd))
@@ -1273,6 +1180,155 @@ class BackupTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_drop_rel_during_full_backup(self):
""""""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'])
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
for i in range(1, 512):
node.safe_psql(
"postgres",
"create table t_heap_{0} as select i"
" as id from generate_series(0,100) i".format(i))
node.safe_psql(
"postgres",
"VACUUM")
node.pgbench_init(scale=10)
relative_path_1 = node.safe_psql(
"postgres",
"select pg_relation_filepath('t_heap_1')").rstrip()
relative_path_2 = node.safe_psql(
"postgres",
"select pg_relation_filepath('t_heap_1')").rstrip()
absolute_path_1 = os.path.join(node.data_dir, relative_path_1)
absolute_path_2 = os.path.join(node.data_dir, relative_path_2)
# FULL backup
gdb = self.backup_node(
backup_dir, 'node', node,
options=['--stream', '--log-level-file=LOG', '--log-level-console=LOG', '--progress'],
gdb=True)
gdb.set_breakpoint('backup_files')
gdb.run_until_break()
# REMOVE file
for i in range(1, 512):
node.safe_psql(
"postgres",
"drop table t_heap_{0}".format(i))
node.safe_psql(
"postgres",
"CHECKPOINT")
node.safe_psql(
"postgres",
"CHECKPOINT")
# File removed, we can proceed with backup
gdb.continue_execution_until_exit()
pgdata = self.pgdata_content(node.data_dir)
#with open(os.path.join(backup_dir, 'log', 'pg_probackup.log')) as f:
# log_content = f.read()
# self.assertTrue(
# 'LOG: File "{0}" is not found'.format(absolute_path) in log_content,
# 'File "{0}" should be deleted but it`s not'.format(absolute_path))
node.cleanup()
self.restore_node(backup_dir, 'node', node)
# Physical comparison
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
self.del_test_dir(module_name, fname)
@unittest.skip("skip")
def test_drop_db_during_full_backup(self):
""""""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'])
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
for i in range(1, 2):
node.safe_psql(
"postgres",
"create database t_heap_{0}".format(i))
node.safe_psql(
"postgres",
"VACUUM")
# FULL backup
gdb = self.backup_node(
backup_dir, 'node', node, gdb=True,
options=[
'--stream', '--log-level-file=LOG',
'--log-level-console=LOG', '--progress'])
gdb.set_breakpoint('backup_files')
gdb.run_until_break()
# REMOVE file
for i in range(1, 2):
node.safe_psql(
"postgres",
"drop database t_heap_{0}".format(i))
node.safe_psql(
"postgres",
"CHECKPOINT")
node.safe_psql(
"postgres",
"CHECKPOINT")
# File removed, we can proceed with backup
gdb.continue_execution_until_exit()
pgdata = self.pgdata_content(node.data_dir)
#with open(os.path.join(backup_dir, 'log', 'pg_probackup.log')) as f:
# log_content = f.read()
# self.assertTrue(
# 'LOG: File "{0}" is not found'.format(absolute_path) in log_content,
# 'File "{0}" should be deleted but it`s not'.format(absolute_path))
node.cleanup()
self.restore_node(backup_dir, 'node', node)
# Physical comparison
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_drop_rel_during_backup_delta(self):
""""""
@@ -1288,6 +1344,8 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.set_archiving(backup_dir, 'node', node)
node.slow_start()
node.pgbench_init(scale=10)
node.safe_psql(
"postgres",
"create table t_heap as select i"
@@ -2511,7 +2569,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
@unittest.skip("skip")
def test_issue_132(self):
"""
https://github.com/postgrespro/pg_probackup/issues/132
@@ -2546,6 +2604,8 @@ class BackupTest(ProbackupTest, unittest.TestCase):
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
exit(1)
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -2801,3 +2861,44 @@ class BackupTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
@unittest.skip("skip")
def test_issue_203(self):
"""
https://github.com/postgrespro/pg_probackup/issues/203
"""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'autovacuum': 'off'})
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
with node.connect("postgres") as conn:
for i in range(1200000):
conn.execute(
"CREATE TABLE t_{0} as select 1".format(i))
conn.commit()
full_id = self.backup_node(
backup_dir, 'node', node, options=['--stream'])
pgdata = self.pgdata_content(node.data_dir)
node_restored = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
self.restore_node(backup_dir, 'node',
node_restored, data_dir=node_restored.data_dir)
pgdata_restored = self.pgdata_content(node_restored.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
self.del_test_dir(module_name, fname)
tests/config.py
+64
View File
@@ -3,6 +3,7 @@ import subprocess
import os
from .helpers.ptrack_helpers import ProbackupTest, ProbackupException
from sys import exit
from shutil import copyfile
module_name = 'config'
@@ -51,3 +52,66 @@ class ConfigTest(ProbackupTest, unittest.TestCase):
e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
# @unittest.expectedFailure
# @unittest.skip("skip")
def test_corrupt_backup_content(self):
"""corrupt backup_content.control"""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'])
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.slow_start()
full1_id = self.backup_node(backup_dir, 'node', node)
node.safe_psql(
'postgres',
'create table t1()')
fulle2_id = self.backup_node(backup_dir, 'node', node)
fulle1_conf_file = os.path.join(
backup_dir, 'backups','node', full1_id, 'backup_content.control')
fulle2_conf_file = os.path.join(
backup_dir, 'backups','node', fulle2_id, 'backup_content.control')
copyfile(fulle2_conf_file, fulle1_conf_file)
try:
self.validate_pb(backup_dir, 'node')
self.assertEqual(
1, 0,
"Expecting Error because pg_probackup.conf is missing. "
".\n Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertIn(
"WARNING: Invalid CRC of backup control file '{0}':".format(fulle1_conf_file),
e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
self.assertIn(
"WARNING: Failed to get file list for backup {0}".format(full1_id),
e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
self.assertIn(
"WARNING: Backup {0} file list is corrupted".format(full1_id),
e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
self.show_pb(backup_dir, 'node', full1_id)['status']
self.assertEqual(self.show_pb(backup_dir, 'node')[0]['status'], 'CORRUPT')
self.assertEqual(self.show_pb(backup_dir, 'node')[1]['status'], 'OK')
tests/delta.py
-157
View File
@@ -1050,163 +1050,6 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_delta_corruption_heal_via_ptrack(self):
"""make node, corrupt some page, check that backup failed"""
if not self.ptrack:
return unittest.skip('Skipped because ptrack support is disabled')
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'])
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
if node.major_version >= 12:
node.safe_psql(
"postgres",
"CREATE EXTENSION ptrack WITH SCHEMA pg_catalog")
self.backup_node(
backup_dir, 'node', node,
backup_type="full", options=["-j", "4", "--stream"])
node.safe_psql(
"postgres",
"create table t_heap as select 1 as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,1000) i")
node.safe_psql(
"postgres",
"CHECKPOINT;")
heap_path = node.safe_psql(
"postgres",
"select pg_relation_filepath('t_heap')").rstrip()
with open(os.path.join(node.data_dir, heap_path), "rb+", 0) as f:
f.seek(9000)
f.write(b"bla")
f.flush()
f.close
self.backup_node(
backup_dir, 'node', node,
backup_type="delta",
options=["-j", "4", "--stream", '--log-level-file=verbose'])
# open log file and check
with open(os.path.join(backup_dir, 'log', 'pg_probackup.log')) as f:
log_content = f.read()
self.assertIn('block 1, try to fetch via shared buffer', log_content)
self.assertIn('SELECT pg_catalog.pg_ptrack_get_block', log_content)
f.close
self.assertTrue(
self.show_pb(backup_dir, 'node')[1]['status'] == 'OK',
"Backup Status should be OK")
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_page_corruption_heal_via_ptrack(self):
"""make node, corrupt some page, check that backup failed"""
if not self.ptrack:
return unittest.skip('Skipped because ptrack support is disabled')
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'])
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
if node.major_version >= 12:
node.safe_psql(
"postgres",
"CREATE EXTENSION ptrack WITH SCHEMA pg_catalog")
self.backup_node(
backup_dir, 'node', node, backup_type="full",
options=["-j", "4", "--stream"])
node.safe_psql(
"postgres",
"create table t_heap as select 1 as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,1000) i")
node.safe_psql(
"postgres",
"CHECKPOINT;")
heap_path = node.safe_psql(
"postgres",
"select pg_relation_filepath('t_heap')").rstrip()
node.stop()
with open(os.path.join(node.data_dir, heap_path), "rb+", 0) as f:
f.seek(9000)
f.write(b"bla")
f.flush()
f.close
node.slow_start()
try:
self.backup_node(
backup_dir, 'node', node, backup_type="delta",
options=["-j", "4", "--stream", "--log-level-console=LOG"])
# we should die here because exception is what we expect to happen
self.assertEqual(
1, 0,
"Expecting Error because of page "
"corruption in PostgreSQL instance.\n"
" Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
if self.remote:
self.assertTrue(
"LOG: File" in e.message and
"try to fetch via shared buffer" in e.message and
"WARNING: page verification failed, "
"calculated checksum" in e.message and
"ERROR: query failed: "
"ERROR: invalid page in block" in e.message and
"query was: SELECT pg_catalog.pg_ptrack_get_block" in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
else:
self.assertTrue(
"WARNING: File" in e.message and
"blknum" in e.message and
"have wrong checksum" in e.message and
"try to fetch via shared buffer" in e.message and
"WARNING: page verification failed, "
"calculated checksum" in e.message and
"ERROR: query failed: "
"ERROR: invalid page in block" in e.message and
"query was: SELECT pg_catalog.pg_ptrack_get_block" in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
self.assertTrue(
self.show_pb(backup_dir, 'node')[1]['status'] == 'ERROR',
"Backup Status should be ERROR")
# Clean after yourself
self.del_test_dir(module_name, fname)
def test_delta_nullified_heap_page_backup(self):
"""
make node, take full backup, nullify some heap block,
tests/exclude.py
+82
View File
@@ -153,3 +153,85 @@ class ExcludeTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_exclude_log_dir(self):
"""
check that by default 'log' and 'pg_log' directories are not backed up
"""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
'logging_collector': 'on',
'log_filename': 'postgresql.log'})
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
self.backup_node(
backup_dir, 'node', node,
backup_type='full', options=['--stream'])
log_dir = node.safe_psql(
'postgres',
'show log_directory').rstrip()
node.cleanup()
self.restore_node(
backup_dir, 'node', node, options=["-j", "4"])
# check that PGDATA/log or PGDATA/pg_log do not exists
path = os.path.join(node.data_dir, log_dir)
log_file = os.path.join(path, 'postgresql.log')
self.assertTrue(os.path.exists(path))
self.assertFalse(os.path.exists(log_file))
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_exclude_log_dir_1(self):
"""
check that "--backup-pg-log" works correctly
"""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
'logging_collector': 'on',
'log_filename': 'postgresql.log'})
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.slow_start()
log_dir = node.safe_psql(
'postgres',
'show log_directory').rstrip()
self.backup_node(
backup_dir, 'node', node,
backup_type='full', options=['--stream', '--backup-pg-log'])
node.cleanup()
self.restore_node(
backup_dir, 'node', node, options=["-j", "4"])
# check that PGDATA/log or PGDATA/pg_log do not exists
path = os.path.join(node.data_dir, log_dir)
log_file = os.path.join(path, 'postgresql.log')
self.assertTrue(os.path.exists(path))
self.assertTrue(os.path.exists(log_file))
# Clean after yourself
self.del_test_dir(module_name, fname)
tests/expected/option_version.out
+1 -1
View File
@@ -1 +1 @@
pg_probackup 2.3.1
pg_probackup 2.3.3
tests/helpers/ptrack_helpers.py
+2 -1
View File
@@ -366,7 +366,8 @@ class ProbackupTest(object):
if ptrack_enable:
if node.major_version > 11:
options['ptrack_map_size'] = '128MB'
options['ptrack.map_size'] = '128'
options['shared_preload_libraries'] = 'ptrack'
else:
options['ptrack_enable'] = 'on'
tests/merge.py
+10 -11
View File
@@ -261,7 +261,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
node.slow_start()
# Fill with data
node.pgbench_init(scale=5)
node.pgbench_init(scale=10)
# Do compressed FULL backup
self.backup_node(backup_dir, "node", node, options=[
@@ -272,7 +272,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
self.assertEqual(show_backup["backup-mode"], "FULL")
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench = node.pgbench(options=['-T', '10', '-c', '1', '--no-vacuum'])
pgbench.wait()
# Do compressed DELTA backup
@@ -281,7 +281,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
options=['--compress', '--stream'])
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench = node.pgbench(options=['-T', '10', '-c', '1', '--no-vacuum'])
pgbench.wait()
# Do uncompressed PAGE backup
@@ -1530,8 +1530,8 @@ class MergeTest(ProbackupTest, unittest.TestCase):
gdb.set_breakpoint('write_backup_filelist')
gdb.run_until_break()
gdb.set_breakpoint('fio_fwrite')
gdb.continue_execution_until_break(2)
gdb.set_breakpoint('sprintf')
gdb.continue_execution_until_break(20)
gdb._execute('signal SIGKILL')
@@ -1621,8 +1621,8 @@ class MergeTest(ProbackupTest, unittest.TestCase):
gdb.set_breakpoint('write_backup_filelist')
gdb.run_until_break()
gdb.set_breakpoint('fio_fwrite')
gdb.continue_execution_until_break(2)
gdb.set_breakpoint('sprintf')
gdb.continue_execution_until_break(20)
gdb._execute('signal SIGKILL')
@@ -1991,7 +1991,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# Take FULL
self.backup_node(backup_dir, 'node', node)
node.pgbench_init(scale=3)
node.pgbench_init(scale=5)
# Take PAGE from future
backup_id = self.backup_node(
@@ -2011,11 +2011,10 @@ class MergeTest(ProbackupTest, unittest.TestCase):
os.path.join(backup_dir, 'backups', 'node', backup_id),
os.path.join(backup_dir, 'backups', 'node', new_id))
pgbench = node.pgbench(options=['-T', '3', '-c', '2', '--no-vacuum'])
pgbench = node.pgbench(options=['-T', '5', '-c', '1', '--no-vacuum'])
pgbench.wait()
backup_id = self.backup_node(
backup_dir, 'node', node, backup_type='page')
backup_id = self.backup_node(backup_dir, 'node', node, backup_type='page')
pgdata = self.pgdata_content(node.data_dir)
result = node.safe_psql(
tests/ptrack.py
+21 -13
View File
@@ -269,7 +269,8 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'checkpoint_timeout': '30s'})
'checkpoint_timeout': '30s',
'shared_preload_libraries': 'ptrack'})
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
@@ -336,16 +337,16 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# DISABLE PTRACK
if node.major_version >= 12:
node.safe_psql('postgres', "alter system set ptrack_map_size to 0")
node.safe_psql('postgres', "alter system set ptrack.map_size to 0")
else:
node.safe_psql('postgres', "alter system set ptrack_enable to off")
node.stop()
node.slow_start()
# ENABLE PTRACK
if node.major_version >= 12:
node.safe_psql('postgres', "alter system set ptrack_map_size to '128MB'")
node.safe_psql('postgres', "alter system set ptrack.map_size to '128'")
node.safe_psql('postgres', "alter system set shared_preload_libraries to 'ptrack'")
else:
node.safe_psql('postgres', "alter system set ptrack_enable to on")
node.stop()
@@ -412,7 +413,11 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
backup_dir, 'node', node, backup_type='ptrack',
options=['--stream'])
pgdata = self.pgdata_content(node.data_dir)
# TODO: what's the point in taking pgdata content, then taking
# backup, and the trying to compare those two? Backup issues a
# checkpoint, so it will modify pgdata with close to 100% chance.
if self.paranoia:
pgdata = self.pgdata_content(node.data_dir)
self.backup_node(
backup_dir, 'node', node, backup_type='ptrack',
@@ -426,8 +431,9 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
backup_dir, 'node', node_restored,
node_restored.data_dir, options=["-j", "4"])
pgdata_restored = self.pgdata_content(
node_restored.data_dir, ignore_ptrack=False)
if self.paranoia:
pgdata_restored = self.pgdata_content(
node_restored.data_dir, ignore_ptrack=False)
self.set_auto_conf(
node_restored, {'port': node_restored.port})
@@ -435,7 +441,8 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
node_restored.slow_start()
# Physical comparison
self.compare_pgdata(pgdata, pgdata_restored)
if self.paranoia:
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -630,6 +637,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
node.slow_start()
if node.major_version >= 12:
self.skipTest("skip --- we do not need ptrack_get_block for ptrack 2.*")
node.safe_psql(
"postgres",
"CREATE EXTENSION ptrack")
@@ -3841,7 +3849,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
@unittest.expectedFailure
# @unittest.expectedFailure
def test_ptrack_pg_resetxlog(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
@@ -3946,7 +3954,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
)
except ProbackupException as e:
self.assertIn(
'Insert error message',
'ERROR: LSN from ptrack_control 0/0 differs from STOP LSN of previous backup',
e.message,
'\n Unexpected Error Message: {0}\n'
' CMD: {1}'.format(repr(e.message), self.cmd))
@@ -4051,10 +4059,10 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
log_content = f.read()
self.assertIn(
'FATAL: incorrect checksum of file "{0}"'.format(ptrack_map),
'FATAL: ptrack init: incorrect checksum of file "{0}"'.format(ptrack_map),
log_content)
self.set_auto_conf(node, {'ptrack_map_size': '0'})
self.set_auto_conf(node, {'ptrack.map_size': '0'})
node.slow_start()
@@ -4082,7 +4090,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
node.stop(['-m', 'immediate', '-D', node.data_dir])
self.set_auto_conf(node, {'ptrack_map_size': '32'})
self.set_auto_conf(node, {'ptrack.map_size': '32', 'shared_preload_libraries': 'ptrack'})
node.slow_start()
tests/restore.py
+3 -4
View File
@@ -1747,7 +1747,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
# Take FULL
self.backup_node(backup_dir, 'node', node)
node.pgbench_init(scale=3)
node.pgbench_init(scale=5)
# pgbench = node.pgbench(options=['-T', '20', '-c', '2'])
# pgbench.wait()
@@ -1769,11 +1769,10 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
os.path.join(backup_dir, 'backups', 'node', backup_id),
os.path.join(backup_dir, 'backups', 'node', new_id))
pgbench = node.pgbench(options=['-T', '3', '-c', '2', '--no-vacuum'])
pgbench = node.pgbench(options=['-T', '7', '-c', '1', '--no-vacuum'])
pgbench.wait()
backup_id = self.backup_node(
backup_dir, 'node', node, backup_type='page')
backup_id = self.backup_node(backup_dir, 'node', node, backup_type='page')
pgdata = self.pgdata_content(node.data_dir)
node.cleanup()
travis/Dockerfile.in
+5 -1
View File
@@ -2,7 +2,11 @@ FROM ololobus/postgres-dev:stretch
USER root
RUN apt-get update
RUN apt-get -yq install python python-pip python-virtualenv
RUN apt-get -yq install python python-pip
# RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
# RUN python2 get-pip.py
RUN python2 -m pip install virtualenv
# Environment
ENV PG_MAJOR=${PG_VERSION} PG_BRANCH=${PG_BRANCH}
travis/run_tests.sh
+1 -1
View File
@@ -60,7 +60,7 @@ make USE_PGXS=1 top_srcdir=$PG_SRC install
# Setup python environment
echo "############### Setting up python env:"
virtualenv pyenv
python2 -m virtualenv pyenv
source pyenv/bin/activate
pip install testgres==1.8.2