db/pg_probackup
1
0
Fork 0
mirror of https://github.com/postgrespro/pg_probackup.git synced 2026-06-21 01:34:15 +02:00
Code Issues Releases Activity

Merge with master

Browse Source
This commit is contained in:
Konstantin Knizhnik
2019-01-11 19:08:39 +03:00
parent ae5aa3db73 5bcea32f31
commit fa146f5da0
52 changed files with 1791 additions and 935 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+28 -13
View File
@@ -5,11 +5,6 @@
The utility is compatible with:
* PostgreSQL 9.5, 9.6, 10, 11;
`PTRACK` backup support provided via following options:
* vanilla PostgreSQL compiled with ptrack patch. Currently there are patches for [PostgreSQL 9.6](https://gist.githubusercontent.com/gsmol/5b615c971dfd461c76ef41a118ff4d97/raw/e471251983f14e980041f43bea7709b8246f4178/ptrack_9.6.6_v1.5.patch) and [PostgreSQL 10](https://gist.githubusercontent.com/gsmol/be8ee2a132b88463821021fd910d960e/raw/de24f9499f4f314a4a3e5fae5ed4edb945964df8/ptrack_10.1_v1.5.patch)
* Postgres Pro Standard 9.5, 9.6, 10, 11
* Postgres Pro Enterprise 9.5, 9.6, 10
As compared to other backup solutions, `pg_probackup` offers the following benefits that can help you implement different backup strategies and deal with large amounts of data:
* Choosing between full and page-level incremental backups to speed up backup and recovery
* Implementing a single backup strategy for multi-server PostgreSQL clusters
@@ -34,12 +29,17 @@ Regardless of the chosen backup type, all backups taken with `pg_probackup` supp
* `Autonomous backups` include all the files required to restore the cluster to a consistent state at the time the backup was taken. Even if continuous archiving is not set up, the required WAL segments are included into the backup.
* `Archive backups` rely on continuous archiving. Such backups enable cluster recovery to an arbitrary point after the backup was taken (point-in-time recovery).
`PTRACK` backup support provided via following options:
* vanilla PostgreSQL compiled with ptrack patch. Currently there are patches for [PostgreSQL 9.6](https://gist.githubusercontent.com/gsmol/5b615c971dfd461c76ef41a118ff4d97/raw/e471251983f14e980041f43bea7709b8246f4178/ptrack_9.6.6_v1.5.patch) and [PostgreSQL 10](https://gist.githubusercontent.com/gsmol/be8ee2a132b88463821021fd910d960e/raw/de24f9499f4f314a4a3e5fae5ed4edb945964df8/ptrack_10.1_v1.5.patch)
* Postgres Pro Standard 9.5, 9.6, 10, 11
* Postgres Pro Enterprise 9.5, 9.6, 10
## Limitations
`pg_probackup` currently has the following limitations:
* Creating backups from a remote server is currently not supported.
* The server from which the backup was taken and the restored server must be compatible by the [block_size](https://postgrespro.com/docs/postgresql/current/runtime-config-preset#GUC-BLOCK-SIZE) and [wal_block_size](https://postgrespro.com/docs/postgresql/current/runtime-config-preset#GUC-WAL-BLOCK-SIZE) parameters and have the same major release number.
* Microsoft Windows operating system is not supported.
* Microsoft Windows operating system support is in beta stage.
* Configuration files outside of PostgreSQL data directory are not included into the backup and should be backed up separately.
## Installation and Setup
@@ -48,36 +48,51 @@ Regardless of the chosen backup type, all backups taken with `pg_probackup` supp
#DEB Ubuntu|Debian Packages
echo "deb [arch=amd64] http://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" > /etc/apt/sources.list.d/pg_probackup.list
wget -O - http://repo.postgrespro.ru/pg_probackup/keys/GPG-KEY-PG_PROBACKUP | apt-key add - && apt-get update
apt-get install pg-probackup-{10,9.6,9.5}
apt-get install pg-probackup-{11,10,9.6,9.5}
#DEB-SRC Packages
echo "deb-src [arch=amd64] http://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" >>\
/etc/apt/sources.list.d/pg_probackup.list
apt-get source pg-probackup-{10,9.6,9.5}
apt-get source pg-probackup-{11,10,9.6,9.5}
#RPM Centos Packages
rpm -ivh http://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-centos.noarch.rpm
yum install pg_probackup-{10,9.6,9.5}
yum install pg_probackup-{11,10,9.6,9.5}
#RPM RHEL Packages
rpm -ivh http://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-rhel.noarch.rpm
yum install pg_probackup-{10,9.6,9.5}
yum install pg_probackup-{11,10,9.6,9.5}
#RPM Oracle Linux Packages
rpm -ivh http://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-oraclelinux.noarch.rpm
yum install pg_probackup-{10,9.6,9.5}
yum install pg_probackup-{11,10,9.6,9.5}
#SRPM Packages
yumdownloader --source pg_probackup-{10,9.6,9.5}
yumdownloader --source pg_probackup-{11,10,9.6,9.5}
```
Once you have `pg_probackup` installed, complete [the setup](https://postgrespro.com/docs/postgrespro/current/app-pgprobackup.html#pg-probackup-install-and-setup).
## Building from source
### Linux
To compile `pg_probackup`, you must have a PostgreSQL installation and raw source tree. To install `pg_probackup`, execute this in the module's directory:
```shell
make USE_PGXS=1 PG_CONFIG=<path_to_pg_config> top_srcdir=<path_to_PostgreSQL_source_tree>
```
### Windows
Once you have `pg_probackup` installed, complete [the setup](https://postgrespro.com/docs/postgrespro/current/app-pgprobackup.html#pg-probackup-install-and-setup).
Currently pg_probackup can be build using only MSVC 2013.
Build PostgreSQL using [pgwininstall](https://github.com/postgrespro/pgwininstall) or [PostgreSQL instruction](https://www.postgresql.org/docs/10/install-windows-full.html) with MSVC 2013.
If zlib support is needed, src/tools/msvc/config.pl must contain path to directory with compiled zlib. [Example](https://gist.githubusercontent.com/gsmol/80989f976ce9584824ae3b1bfb00bd87/raw/240032950d4ac4801a79625dd00c8f5d4ed1180c/gistfile1.txt)
```shell
CALL "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall" amd64
SET PATH=%PATH%;C:\Perl64\bin
SET PATH=%PATH%;C:\msys64\usr\bin
gen_probackup_project.pl C:\path_to_postgresql_source_tree
```
## Documentation
src/archive.c
+2 -2
View File
@@ -27,7 +27,7 @@ do_archive_push(char *wal_file_path, char *wal_file_name, bool overwrite)
char backup_wal_file_path[MAXPGPATH];
char absolute_wal_file_path[MAXPGPATH];
char current_dir[MAXPGPATH];
int64 system_id;
uint64 system_id;
bool is_compress = false;
if (wal_file_name == NULL && wal_file_path == NULL)
@@ -50,7 +50,7 @@ do_archive_push(char *wal_file_path, char *wal_file_name, bool overwrite)
if(system_id != instance_config.system_identifier)
elog(ERROR, "Refuse to push WAL segment %s into archive. Instance parameters mismatch."
"Instance '%s' should have SYSTEM_ID = %ld instead of %ld",
"Instance '%s' should have SYSTEM_ID = " UINT64_FORMAT " instead of " UINT64_FORMAT,
wal_file_name, instance_name, instance_config.system_identifier,
system_id);
src/backup.c
+6 -1
View File
@@ -1682,6 +1682,11 @@ wait_wal_lsn(XLogRecPtr lsn, bool is_start_lsn, bool wait_prev_segment)
(uint32) (lsn >> 32), (uint32) lsn, wal_segment_path);
}
if (!stream_wal && is_start_lsn && try_count == 30)
elog(WARNING, "By default pg_probackup assume WAL delivery method to be ARCHIVE. "
"If continius archiving is not set up, use '--stream' option to make autonomous backup. "
"Otherwise check that continius archiving works correctly.");
if (timeout > 0 && try_count > timeout)
{
if (file_exists)
@@ -2226,9 +2231,9 @@ backup_files(void *arg)
struct stat buf;
pgFile *file = (pgFile *) parray_get(arguments->files_list, i);
elog(VERBOSE, "Copying file: \"%s\" ", file->path);
if (!pg_atomic_test_set_flag(&file->lock))
continue;
elog(VERBOSE, "Copying file: \"%s\" ", file->path);
/* check for interrupt */
if (interrupted)
src/catalog.c
-14
View File
@@ -816,20 +816,6 @@ pgBackupInit(pgBackup *backup)
backup->server_version[0] = '\0';
}
/*
* Copy backup metadata from **src** into **dst**.
*/
void
pgBackupCopy(pgBackup *dst, pgBackup *src)
{
pfree(dst->primary_conninfo);
memcpy(dst, src, sizeof(pgBackup));
if (src->primary_conninfo)
dst->primary_conninfo = pstrdup(src->primary_conninfo);
}
/* free pgBackup object */
void
pgBackupFree(void *backup)
src/data.c
+24 -11
View File
@@ -98,6 +98,8 @@ do_decompress(void* dst, size_t dst_size, void const* src, size_t src_size,
{
case NONE_COMPRESS:
case NOT_DEFINED_COMPRESS:
if (errormsg)
*errormsg = "Invalid compression algorithm";
return -1;
#ifdef HAVE_LIBZ
case ZLIB_COMPRESS:
@@ -230,8 +232,8 @@ read_page_from_file(pgFile *file, BlockNumber blknum,
return 0;
}
else
elog(WARNING, "File: %s, block %u, expected block size %d,"
"but read %lu, try again",
elog(WARNING, "File: %s, block %u, expected block size %u,"
"but read %zu, try again",
file->path, blknum, BLCKSZ, read_len);
}
@@ -380,7 +382,7 @@ prepare_page(backup_files_arg *arguments,
else if (page_size != BLCKSZ)
{
free(ptrack_page);
elog(ERROR, "File: %s, block %u, expected block size %d, but read %lu",
elog(ERROR, "File: %s, block %u, expected block size %d, but read %zu",
file->path, absolute_blknum, BLCKSZ, page_size);
}
else
@@ -572,7 +574,7 @@ backup_data_file(backup_files_arg* arguments,
if (file->size % BLCKSZ != 0)
{
fclose(in);
elog(ERROR, "File: %s, invalid file size %lu", file->path, file->size);
elog(ERROR, "File: %s, invalid file size %zu", file->path, file->size);
}
/*
@@ -784,7 +786,7 @@ restore_data_file(const char *to_path, pgFile *file, bool allow_truncate,
read_len = fread(compressed_page.data, 1,
MAXALIGN(header.compressed_size), in);
if (read_len != MAXALIGN(header.compressed_size))
elog(ERROR, "cannot read block %u of \"%s\" read %lu of %d",
elog(ERROR, "cannot read block %u of \"%s\" read %zu of %d",
blknum, file->path, read_len, header.compressed_size);
/*
@@ -824,6 +826,8 @@ restore_data_file(const char *to_path, pgFile *file, bool allow_truncate,
if (write_header)
{
/* We uncompressed the page, so its size is BLCKSZ */
header.compressed_size = BLCKSZ;
if (fio_fwrite(out, &header, sizeof(header)) != sizeof(header))
elog(ERROR, "cannot write header of block %u of \"%s\": %s",
blknum, file->path, strerror(errno));
@@ -1580,19 +1584,23 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
if (read_len == 0 && feof(in))
break; /* EOF found */
else if (read_len != 0 && feof(in))
elog(ERROR,
elog(WARNING,
"odd size page found at block %u of \"%s\"",
blknum, file->path);
else
elog(ERROR, "cannot read header of block %u of \"%s\": %s",
elog(WARNING, "cannot read header of block %u of \"%s\": %s",
blknum, file->path, strerror(errno_tmp));
return false;
}
COMP_FILE_CRC32(use_crc32c, crc, &header, read_len);
if (header.block < blknum)
elog(ERROR, "backup is broken at file->path %s block %u",
{
elog(WARNING, "backup is broken at file->path %s block %u",
file->path, blknum);
return false;
}
blknum = header.block;
@@ -1608,8 +1616,11 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
read_len = fread(compressed_page.data, 1,
MAXALIGN(header.compressed_size), in);
if (read_len != MAXALIGN(header.compressed_size))
elog(ERROR, "cannot read block %u of \"%s\" read %lu of %d",
{
elog(WARNING, "cannot read block %u of \"%s\" read %zu of %d",
blknum, file->path, read_len, header.compressed_size);
return false;
}
COMP_FILE_CRC32(use_crc32c, crc, compressed_page.data, read_len);
@@ -1636,11 +1647,13 @@ check_file_pages(pgFile *file, XLogRecPtr stop_lsn, uint32 checksum_version,
is_valid = false;
continue;
}
elog(ERROR, "page of file \"%s\" uncompressed to %d bytes. != BLCKSZ",
elog(WARNING, "page of file \"%s\" uncompressed to %d bytes. != BLCKSZ",
file->path, uncompressed_size);
return false;
}
if (validate_one_page(page.data, file, blknum,
stop_lsn, checksum_version) == PAGE_IS_FOUND_AND_NOT_VALID)
stop_lsn, checksum_version) == PAGE_IS_FOUND_AND_NOT_VALID)
is_valid = false;
}
else
src/delete.c
+7 -13
View File
@@ -256,6 +256,7 @@ delete_backup_files(pgBackup *backup)
char path[MAXPGPATH];
char timestamp[100];
parray *files;
size_t num_files;
/*
* If the backup was deleted already, there is nothing to do.
@@ -286,23 +287,16 @@ delete_backup_files(pgBackup *backup)
/* delete leaf node first */
parray_qsort(files, pgFileComparePathDesc);
for (i = 0; i < parray_num(files); i++)
num_files = parray_num(files);
for (i = 0; i < num_files; i++)
{
pgFile *file = (pgFile *) parray_get(files, i);
/* print progress */
elog(VERBOSE, "Delete file(%zd/%lu) \"%s\"", i + 1,
(unsigned long) parray_num(files), file->path);
if (progress)
elog(INFO, "Progress: (%zd/%zd). Process file \"%s\"",
i + 1, num_files, file->path);
if (remove(file->path))
{
if (errno == ENOENT)
elog(VERBOSE, "File \"%s\" is absent", file->path);
else
elog(ERROR, "Cannot remove \"%s\": %s", file->path,
strerror(errno));
return;
}
pgFileDelete(file);
}
parray_walk(files, pgFileFree);
src/dir.c
+74 -49
View File
@@ -120,7 +120,7 @@ typedef struct TablespaceCreatedList
static int BlackListCompare(const void *str1, const void *str2);
static bool dir_check_file(const char *root, pgFile *file);
static char dir_check_file(const char *root, pgFile *file);
static void dir_list_file_internal(parray *files, const char *root,
pgFile *parent, bool exclude,
bool omit_symlink, parray *black_list, fio_location location);
@@ -213,7 +213,7 @@ pgFileInit(const char *path)
strcpy(file->path, path); /* enough buffer size guaranteed */
/* Get file name from the path */
file_name = strrchr(file->path, '/');
file_name = last_dir_separator(file->path);
if (file_name == NULL)
file->name = file->path;
else
@@ -417,6 +417,7 @@ dir_list_file(parray *files, const char *root, bool exclude, bool omit_symlink,
while (fgets(buf, lengthof(buf), black_list_file) != NULL)
{
black_item[0] = '\0';
join_path_components(black_item, instance_config.pgdata, buf);
if (black_item[strlen(black_item) - 1] == '\n')
@@ -425,7 +426,7 @@ dir_list_file(parray *files, const char *root, bool exclude, bool omit_symlink,
if (black_item[0] == '#' || black_item[0] == '\0')
continue;
parray_append(black_list, black_item);
parray_append(black_list, pgut_strdup(black_item));
}
fio_close_stream(black_list_file);
@@ -445,8 +446,21 @@ dir_list_file(parray *files, const char *root, bool exclude, bool omit_symlink,
parray_append(files, file);
dir_list_file_internal(files, root, file, exclude, omit_symlink, black_list, location);
if (!add_root)
pgFileFree(file);
if (black_list)
{
parray_walk(black_list, pfree);
parray_free(black_list);
}
}
#define CHECK_FALSE 0
#define CHECK_TRUE 1
#define CHECK_EXCLUDE_FALSE 2
/*
* Check file or directory.
*
@@ -455,16 +469,21 @@ dir_list_file(parray *files, const char *root, bool exclude, bool omit_symlink,
* Skip files:
* - skip temp tables files
* - skip unlogged tables files
* Skip recursive tablespace content
* Set flags for:
* - database directories
* - datafiles
*/
static bool
static char
dir_check_file(const char *root, pgFile *file)
{
const char *rel_path;
int i;
int sscanf_res;
bool in_tablespace = false;
rel_path = GetRelativePath(file->path, root);
in_tablespace = path_is_prefix_of_path(PG_TBLSPC_DIR, rel_path);
/* Check if we need to exclude file by name */
if (S_ISREG(file->mode))
@@ -477,7 +496,7 @@ dir_check_file(const char *root, pgFile *file)
{
/* Skip */
elog(VERBOSE, "Excluding file: %s", file->name);
return false;
return CHECK_FALSE;
}
}
@@ -486,14 +505,14 @@ dir_check_file(const char *root, pgFile *file)
{
/* Skip */
elog(VERBOSE, "Excluding file: %s", file->name);
return false;
return CHECK_FALSE;
}
}
/*
* If the directory name is in the exclude list, do not list the
* contents.
*/
else if (S_ISDIR(file->mode))
else if (S_ISDIR(file->mode) && !in_tablespace)
{
/*
* If the item in the exclude list starts with '/', compare to
@@ -509,20 +528,18 @@ dir_check_file(const char *root, pgFile *file)
{
elog(VERBOSE, "Excluding directory content: %s",
file->name);
return false;
return CHECK_EXCLUDE_FALSE;
}
}
else if (strcmp(file->name, pgdata_exclude_dir[i]) == 0)
{
elog(VERBOSE, "Excluding directory content: %s",
file->name);
return false;
return CHECK_EXCLUDE_FALSE;
}
}
}
rel_path = GetRelativePath(file->path, root);
/*
* Do not copy tablespaces twice. It may happen if the tablespace is located
* inside the PGDATA.
@@ -538,14 +555,33 @@ dir_check_file(const char *root, pgFile *file)
* pg_tblspc/tblsOid/TABLESPACE_VERSION_DIRECTORY
*/
if (!path_is_prefix_of_path(PG_TBLSPC_DIR, rel_path))
return false;
return CHECK_FALSE;
sscanf_res = sscanf(rel_path, PG_TBLSPC_DIR "/%u/%s",
&tblspcOid, tmp_rel_path);
if (sscanf_res == 0)
return false;
return CHECK_FALSE;
}
if (path_is_prefix_of_path("global", rel_path))
if (in_tablespace)
{
char tmp_rel_path[MAXPGPATH];
sscanf_res = sscanf(rel_path, PG_TBLSPC_DIR "/%u/%[^/]/%u/",
&(file->tblspcOid), tmp_rel_path,
&(file->dbOid));
/*
* We should skip other files and directories rather than
* TABLESPACE_VERSION_DIRECTORY, if this is recursive tablespace.
*/
if (sscanf_res == 2 && strcmp(tmp_rel_path, TABLESPACE_VERSION_DIRECTORY) != 0)
return CHECK_FALSE;
if (sscanf_res == 3 && S_ISDIR(file->mode) &&
strcmp(tmp_rel_path, TABLESPACE_VERSION_DIRECTORY) == 0)
file->is_database = true;
}
else if (path_is_prefix_of_path("global", rel_path))
{
file->tblspcOid = GLOBALTABLESPACE_OID;
@@ -561,22 +597,10 @@ dir_check_file(const char *root, pgFile *file)
if (S_ISDIR(file->mode) && strcmp(file->name, "base") != 0)
file->is_database = true;
}
else if (path_is_prefix_of_path(PG_TBLSPC_DIR, rel_path))
{
char tmp_rel_path[MAXPGPATH];
sscanf_res = sscanf(rel_path, PG_TBLSPC_DIR "/%u/%[^/]/%u/",
&(file->tblspcOid), tmp_rel_path,
&(file->dbOid));
if (sscanf_res == 3 && S_ISDIR(file->mode) &&
strcmp(tmp_rel_path, TABLESPACE_VERSION_DIRECTORY) == 0)
file->is_database = true;
}
/* Do not backup ptrack_init files */
if (S_ISREG(file->mode) && strcmp(file->name, "ptrack_init") == 0)
return false;
return CHECK_FALSE;
/*
* Check files located inside database directories including directory
@@ -586,10 +610,10 @@ dir_check_file(const char *root, pgFile *file)
file->name && file->name[0])
{
if (strcmp(file->name, "pg_internal.init") == 0)
return false;
return CHECK_FALSE;
/* Do not backup temp files */
else if (file->name[0] == 't' && isdigit(file->name[1]))
return false;
return CHECK_FALSE;
else if (isdigit(file->name[0]))
{
char *fork_name;
@@ -604,14 +628,14 @@ dir_check_file(const char *root, pgFile *file)
/* Do not backup ptrack files */
if (strcmp(file->forkName, "ptrack") == 0)
return false;
return CHECK_FALSE;
}
else
{
len = strlen(file->name);
/* reloid.cfm */
if (len > 3 && strcmp(file->name + len - 3, "cfm") == 0)
return true;
return CHECK_TRUE;
sscanf_res = sscanf(file->name, "%u.%d.%s", &(file->relOid),
&(file->segno), suffix);
@@ -623,7 +647,7 @@ dir_check_file(const char *root, pgFile *file)
}
}
return true;
return CHECK_TRUE;
}
/*
@@ -658,6 +682,7 @@ dir_list_file_internal(parray *files, const char *root, pgFile *parent,
{
pgFile *file;
char child[MAXPGPATH];
char check_res;
join_path_components(child, parent->path, dent->d_name);
@@ -693,21 +718,24 @@ dir_list_file_internal(parray *files, const char *root, pgFile *parent,
continue;
}
/* We add the directory anyway */
if (S_ISDIR(file->mode))
parray_append(files, file);
if (exclude && !dir_check_file(root, file))
if (exclude)
{
if (S_ISREG(file->mode))
check_res = dir_check_file(root, file);
if (check_res == CHECK_FALSE)
{
/* Skip */
pgFileFree(file);
/* Skip */
continue;
continue;
}
else if (check_res == CHECK_EXCLUDE_FALSE)
{
/* We add the directory itself which content was excluded */
parray_append(files, file);
continue;
}
}
/* At least add the file */
if (S_ISREG(file->mode))
parray_append(files, file);
parray_append(files, file);
/*
* If the entry is a directory call dir_list_file_internal()
@@ -875,6 +903,7 @@ opt_tablespace_map(ConfigOption *opt, const char *arg)
char *dst_ptr;
const char *arg_ptr;
memset(cell, 0, sizeof(TablespaceListCell));
dst_ptr = dst = cell->old_dir;
for (arg_ptr = arg; *arg_ptr; arg_ptr++)
{
@@ -1217,12 +1246,8 @@ print_file_list(FILE *out, const parray *files, const char *root)
if (file->is_datafile)
fio_fprintf(out, ",\"segno\":\"%d\"", file->segno);
#ifndef WIN32
if (S_ISLNK(file->mode))
#else
if (pgwin32_is_junction(file->path))
#endif
fio_fprintf(out, ",\"linked\":\"%s\"", file->linked);
if (file->linked)
fprintf(out, ",\"linked\":\"%s\"", file->linked);
if (file->n_blocks != BLOCKNUM_INVALID)
fio_fprintf(out, ",\"n_blocks\":\"%i\"", file->n_blocks);
src/merge.c
+88 -58
View File
@@ -91,7 +91,8 @@ do_merge(time_t backup_id)
}
else
{
Assert(dest_backup);
if (dest_backup == NULL)
elog(ERROR, "Target backup %s was not found", base36enc(backup_id));
if (backup->start_time != prev_parent)
continue;
@@ -131,7 +132,6 @@ do_merge(time_t backup_id)
{
pgBackup *from_backup = (pgBackup *) parray_get(backups, i - 1);
full_backup = (pgBackup *) parray_get(backups, i);
merge_backups(full_backup, from_backup);
}
@@ -208,10 +208,7 @@ merge_backups(pgBackup *to_backup, pgBackup *from_backup)
*/
pgBackupGetPath(to_backup, control_file, lengthof(control_file),
DATABASE_FILE_LIST);
to_files = dir_read_file_list(from_database_path, /* Use from_database_path
* so root path will be
* equal with 'files' */
control_file);
to_files = dir_read_file_list(NULL, control_file);
/* To delete from leaf, sort in reversed order */
parray_qsort(to_files, pgFileComparePathDesc);
/*
@@ -219,7 +216,7 @@ merge_backups(pgBackup *to_backup, pgBackup *from_backup)
*/
pgBackupGetPath(from_backup, control_file, lengthof(control_file),
DATABASE_FILE_LIST);
files = dir_read_file_list(from_database_path, control_file);
files = dir_read_file_list(NULL, control_file);
/* sort by size for load balancing */
parray_qsort(files, pgFileCompareSize);
@@ -281,6 +278,16 @@ merge_backups(pgBackup *to_backup, pgBackup *from_backup)
* Update to_backup metadata.
*/
to_backup->status = BACKUP_STATUS_OK;
to_backup->parent_backup = INVALID_BACKUP_ID;
to_backup->start_lsn = from_backup->start_lsn;
to_backup->stop_lsn = from_backup->stop_lsn;
to_backup->recovery_time = from_backup->recovery_time;
to_backup->recovery_xid = from_backup->recovery_xid;
/*
* If one of the backups isn't "stream" backup then the target backup become
* non-stream backup too.
*/
to_backup->stream = to_backup->stream && from_backup->stream;
/* Compute summary of size of regular files in the backup */
to_backup->data_bytes = 0;
for (i = 0; i < parray_num(files); i++)
@@ -314,14 +321,25 @@ delete_source_backup:
/*
* Delete files which are not in from_backup file list.
*/
parray_qsort(files, pgFileComparePathDesc);
for (i = 0; i < parray_num(to_files); i++)
{
pgFile *file = (pgFile *) parray_get(to_files, i);
if (parray_bsearch(files, file, pgFileComparePathDesc) == NULL)
{
char to_file_path[MAXPGPATH];
char *prev_path;
/* We need full path, file object has relative path */
join_path_components(to_file_path, to_database_path, file->path);
prev_path = file->path;
file->path = to_file_path;
pgFileDelete(file);
elog(VERBOSE, "Deleted \"%s\"", file->path);
file->path = prev_path;
}
}
@@ -336,11 +354,7 @@ delete_source_backup:
/*
* Merging finished, now we can safely update ID of the destination backup.
*/
pgBackupCopy(to_backup, from_backup);
/* Correct metadata */
to_backup->backup_mode = BACKUP_MODE_FULL;
to_backup->status = BACKUP_STATUS_OK;
to_backup->parent_backup = INVALID_BACKUP_ID;
to_backup->start_time = from_backup->start_time;
write_backup(to_backup);
/* Cleanup */
@@ -371,11 +385,14 @@ merge_files(void *arg)
pgBackup *from_backup = argument->from_backup;
int i,
num_files = parray_num(argument->files);
int to_root_len = strlen(argument->to_root);
for (i = 0; i < num_files; i++)
{
pgFile *file = (pgFile *) parray_get(argument->files, i);
pgFile *to_file;
pgFile **res_file;
char from_file_path[MAXPGPATH];
char *prev_file_path;
if (!pg_atomic_test_set_flag(&file->lock))
continue;
@@ -384,17 +401,24 @@ merge_files(void *arg)
if (interrupted)
elog(ERROR, "Interrupted during merging backups");
/* Directories were created before */
if (S_ISDIR(file->mode))
continue;
if (progress)
elog(LOG, "Progress: (%d/%d). Process file \"%s\"",
elog(INFO, "Progress: (%d/%d). Process file \"%s\"",
i + 1, num_files, file->path);
res_file = parray_bsearch(argument->to_files, file,
pgFileComparePathDesc);
to_file = (res_file) ? *res_file : NULL;
/*
* Skip files which haven't changed since previous backup. But in case
* of DELTA backup we should consider n_blocks to truncate the target
* backup.
*/
if (file->write_size == BYTES_INVALID &&
file->n_blocks == -1)
if (file->write_size == BYTES_INVALID && file->n_blocks == -1)
{
elog(VERBOSE, "Skip merging file \"%s\", the file didn't change",
file->path);
@@ -403,43 +427,36 @@ merge_files(void *arg)
* If the file wasn't changed in PAGE backup, retreive its
* write_size from previous FULL backup.
*/
if (S_ISREG(file->mode))
if (to_file)
{
pgFile **res_file;
res_file = parray_bsearch(argument->to_files, file,
pgFileComparePathDesc);
if (res_file && *res_file)
{
file->compress_alg = (*res_file)->compress_alg;
file->write_size = (*res_file)->write_size;
file->crc = (*res_file)->crc;
}
file->compress_alg = to_file->compress_alg;
file->write_size = to_file->write_size;
file->crc = to_file->crc;
}
continue;
}
/* Directories were created before */
if (S_ISDIR(file->mode))
continue;
/* We need to make full path, file object has relative path */
join_path_components(from_file_path, argument->from_root, file->path);
prev_file_path = file->path;
file->path = from_file_path;
/*
* Move the file. We need to decompress it and compress again if
* necessary.
*/
elog(VERBOSE, "Moving file \"%s\", is_datafile %d, is_cfs %d",
elog(VERBOSE, "Merging file \"%s\", is_datafile %d, is_cfs %d",
file->path, file->is_database, file->is_cfs);
if (file->is_datafile && !file->is_cfs)
{
char to_path_tmp[MAXPGPATH]; /* Path of target file */
char to_file_path[MAXPGPATH]; /* Path of target file */
join_path_components(to_path_tmp, argument->to_root,
file->path + to_root_len + 1);
join_path_components(to_file_path, argument->to_root, prev_file_path);
/*
* We need more complicate algorithm if target file exists and it is
* We need more complicate algorithm if target file should be
* compressed.
*/
if (to_backup->compress_alg == PGLZ_COMPRESS ||
@@ -448,54 +465,58 @@ merge_files(void *arg)
char tmp_file_path[MAXPGPATH];
char *prev_path;
snprintf(tmp_file_path, MAXPGPATH, "%s_tmp", to_path_tmp);
snprintf(tmp_file_path, MAXPGPATH, "%s_tmp", to_file_path);
/* Start the magic */
/*
* Merge files:
* - decompress first file
* - decompress second file and merge with first decompressed file
* - if target file exists restore and decompress it to the temp
* path
* - decompress source file if necessary and merge it with the
* target decompressed file
* - compress result file
*/
elog(VERBOSE, "File is compressed, decompress to the temporary file \"%s\"",
tmp_file_path);
prev_path = file->path;
/*
* We need to decompress target file only if it exists.
* We need to decompress target file if it exists.
*/
if (fileExists(to_path_tmp, FIO_LOCAL_HOST))
if (to_file)
{
elog(VERBOSE, "Merge target and source files into the temporary path \"%s\"",
tmp_file_path);
/*
* file->path points to the file in from_root directory. But we
* need the file in directory to_root.
*/
file->path = to_path_tmp;
/* Decompress first/target file */
restore_data_file(tmp_file_path, file, false, false,
prev_path = to_file->path;
to_file->path = to_file_path;
/* Decompress target file into temporary one */
restore_data_file(tmp_file_path, to_file, false, false,
parse_program_version(to_backup->program_version));
file->path = prev_path;
to_file->path = prev_path;
}
/* Merge second/source file with first/target file */
else
elog(VERBOSE, "Restore source file into the temporary path \"%s\"",
tmp_file_path);
/* Merge source file with target file */
restore_data_file(tmp_file_path, file,
from_backup->backup_mode == BACKUP_MODE_DIFF_DELTA,
false,
parse_program_version(from_backup->program_version));
elog(VERBOSE, "Compress file and save it to the directory \"%s\"",
elog(VERBOSE, "Compress file and save it into the directory \"%s\"",
argument->to_root);
/* Again we need change path */
/* Again we need to change path */
prev_path = file->path;
file->path = tmp_file_path;
/* backup_data_file() requires file size to calculate nblocks */
file->size = pgFileSize(file->path);
/* Now we can compress the file */
backup_data_file(NULL, /* We shouldn't need 'arguments' here */
to_path_tmp, file,
to_file_path, file,
to_backup->start_lsn,
to_backup->backup_mode,
to_backup->compress_alg,
@@ -514,7 +535,7 @@ merge_files(void *arg)
else
{
/* We can merge in-place here */
restore_data_file(to_path_tmp, file,
restore_data_file(to_file_path, file,
from_backup->backup_mode == BACKUP_MODE_DIFF_DELTA,
true,
parse_program_version(from_backup->program_version));
@@ -523,8 +544,8 @@ merge_files(void *arg)
* We need to calculate write_size, restore_data_file() doesn't
* do that.
*/
file->write_size = pgFileSize(to_path_tmp);
file->crc = pgFileGetCRC(to_path_tmp, true, true, NULL, FIO_LOCAL_HOST);
file->write_size = pgFileSize(to_file_path);
file->crc = pgFileGetCRC(to_file_path, true, true, NULL, FIO_LOCAL_HOST);
}
}
else if (strcmp(file->name, "pg_control") == 0)
@@ -532,9 +553,18 @@ merge_files(void *arg)
else
copy_file(argument->from_root, argument->to_root, file, FIO_LOCAL_HOST);
/*
* We need to save compression algorithm type of the target backup to be
* able to restore in the future.
*/
file->compress_alg = to_backup->compress_alg;
if (file->write_size != BYTES_INVALID)
elog(LOG, "Moved file \"%s\": " INT64_FORMAT " bytes",
elog(LOG, "Merged file \"%s\": " INT64_FORMAT " bytes",
file->path, file->write_size);
/* Restore relative path */
file->path = prev_file_path;
}
/* Data files merging is successful */
src/parsexlog.c
+19 -7
View File
@@ -521,6 +521,7 @@ validate_wal(pgBackup *backup, const char *archivedir,
TimestampTz last_time = 0;
char last_timestamp[100],
target_timestamp[100];
XLogRecPtr last_lsn = InvalidXLogRecPtr;
bool all_wal = false;
char backup_xlog_path[MAXPGPATH];
@@ -587,6 +588,7 @@ validate_wal(pgBackup *backup, const char *archivedir,
/* We can restore at least up to the backup end */
time2iso(last_timestamp, lengthof(last_timestamp), backup->recovery_time);
last_xid = backup->recovery_xid;
last_lsn = backup->stop_lsn;
if ((TransactionIdIsValid(target_xid) && target_xid == last_xid)
|| (target_time != 0 && backup->recovery_time >= target_time)
@@ -610,6 +612,7 @@ validate_wal(pgBackup *backup, const char *archivedir,
timestamp_record = getRecordTimestamp(xlogreader, &last_time);
if (XLogRecGetXid(xlogreader) != InvalidTransactionId)
last_xid = XLogRecGetXid(xlogreader);
last_lsn = xlogreader->ReadRecPtr;
/* Check target xid */
if (TransactionIdIsValid(target_xid) && target_xid == last_xid)
@@ -618,14 +621,21 @@ validate_wal(pgBackup *backup, const char *archivedir,
break;
}
/* Check target time */
else if (target_time != 0 && timestamp_record && timestamptz_to_time_t(last_time) >= target_time)
else if (target_time != 0 && timestamp_record &&
timestamptz_to_time_t(last_time) >= target_time)
{
all_wal = true;
break;
}
/* If there are no target xid and target time */
/* Check target lsn */
else if (XRecOffIsValid(target_lsn) && last_lsn >= target_lsn)
{
all_wal = true;
break;
}
/* If there are no target xid, target time and target lsn */
else if (!TransactionIdIsValid(target_xid) && target_time == 0 &&
xlogreader->ReadRecPtr == backup->stop_lsn)
!XRecOffIsValid(target_lsn))
{
all_wal = true;
/* We don't stop here. We want to get last_xid and last_time */
@@ -640,15 +650,17 @@ validate_wal(pgBackup *backup, const char *archivedir,
/* There are all needed WAL records */
if (all_wal)
elog(INFO, "backup validation completed successfully on time %s and xid " XID_FMT,
last_timestamp, last_xid);
elog(INFO, "backup validation completed successfully on time %s, xid " XID_FMT " and LSN %X/%X",
last_timestamp, last_xid,
(uint32) (last_lsn >> 32), (uint32) last_lsn);
/* Some needed WAL records are absent */
else
{
PrintXLogCorruptionMsg(&private, WARNING);
elog(WARNING, "recovery can be done up to time %s and xid " XID_FMT,
last_timestamp, last_xid);
elog(WARNING, "recovery can be done up to time %s, xid " XID_FMT " and LSN %X/%X",
last_timestamp, last_xid,
(uint32) (last_lsn >> 32), (uint32) last_lsn);
if (target_time > 0)
time2iso(target_timestamp, lengthof(target_timestamp),
src/pg_probackup.c
+17 -16
View File
@@ -76,10 +76,10 @@ char *remote_agent;
bool is_remote_backup;
/* restore options */
static char *target_time;
static char *target_xid;
static char *target_lsn;
static char *target_inclusive;
static char *target_time = NULL;
static char *target_xid = NULL;
static char *target_lsn = NULL;
static char *target_inclusive = NULL;
static TimeLineID target_tli;
static bool target_immediate;
static char *target_name = NULL;
@@ -374,15 +374,6 @@ main(int argc, char *argv[])
elog(ERROR, "-B, --backup-path must be a path to directory");
}
/* command was initialized for a few commands */
if (command)
{
elog_file(INFO, "command: %s", command);
pfree(command);
command = NULL;
}
/* Option --instance is required for all commands except init and show */
if (backup_subcmd != INIT_CMD && backup_subcmd != SHOW_CMD &&
backup_subcmd != VALIDATE_CMD)
@@ -415,8 +406,9 @@ main(int argc, char *argv[])
}
/*
* Read options from env variables or from config file,
* unless we're going to set them via set-config.
* We read options from command line, now we need to read them from
* configuration file since we got backup path and instance name.
* For some commands an instance option isn't required, see above.
*/
if (instance_name)
{
@@ -433,6 +425,15 @@ main(int argc, char *argv[])
/* Initialize logger */
init_logger(backup_path, &instance_config.logger);
/* command was initialized for a few commands */
if (command)
{
elog_file(INFO, "command: %s", command);
pfree(command);
command = NULL;
}
/*
* We have read pgdata path from command line or from configuration file.
* Ensure that pgdata is an absolute path.
@@ -633,7 +634,7 @@ compress_init(void)
if (backup_subcmd == BACKUP_CMD || backup_subcmd == ARCHIVE_PUSH_CMD)
{
#ifndef HAVE_LIBZ
if (compress_alg == ZLIB_COMPRESS)
if (instance_config.compress_alg == ZLIB_COMPRESS)
elog(ERROR, "This build does not support zlib compression");
else
#endif
src/pg_probackup.h
-1
View File
@@ -491,7 +491,6 @@ extern void pgBackupGetPath2(const pgBackup *backup, char *path, size_t len,
const char *subdir1, const char *subdir2);
extern int pgBackupCreateDir(pgBackup *backup);
extern void pgBackupInit(pgBackup *backup);
extern void pgBackupCopy(pgBackup *dst, pgBackup *src);
extern void pgBackupFree(void *backup);
extern int pgBackupCompareId(const void *f1, const void *f2);
extern int pgBackupCompareIdDesc(const void *f1, const void *f2);
src/restore.c
+2 -2
View File
@@ -480,7 +480,7 @@ restore_backup(pgBackup *backup)
/* By default there are some error */
threads_args[i].ret = 1;
elog(LOG, "Start thread for num:%li", parray_num(files));
elog(LOG, "Start thread for num:%zu", parray_num(files));
pthread_create(&threads[i], NULL, restore_files, arg);
}
@@ -582,7 +582,7 @@ restore_files(void *arg)
rel_path = GetRelativePath(file->path,from_root);
if (progress)
elog(LOG, "Progress: (%d/%lu). Process file %s ",
elog(INFO, "Progress: (%d/%lu). Process file %s ",
i + 1, (unsigned long) parray_num(arguments->files), rel_path);
/*
src/utils/configuration.c
+1 -1
View File
@@ -1,4 +1,4 @@
/*-------------------------------------------------------------------------
/*-------------------------------------------------------------------------
*
* configuration.c: - function implementations to work with pg_probackup
* configurations.
src/utils/logger.c
+55 -38
View File
@@ -40,10 +40,10 @@ typedef enum
void pg_log(eLogType type, const char *fmt,...) pg_attribute_printf(2, 3);
static void elog_internal(int elevel, bool file_only, const char *fmt, va_list args)
pg_attribute_printf(3, 0);
static void elog_internal(int elevel, bool file_only, const char *message);
static void elog_stderr(int elevel, const char *fmt, ...)
pg_attribute_printf(2, 3);
static char *get_log_message(const char *fmt, va_list args) pg_attribute_printf(1, 0);
/* Functions to work with log files */
static void open_logfile(FILE **file, const char *filename_format);
@@ -74,7 +74,7 @@ init_logger(const char *root_path, LoggerConfig *config)
/* Set log path */
if (config->log_directory == NULL)
{
config->log_directory = palloc(MAXPGPATH);
config->log_directory = pgut_malloc(MAXPGPATH);
join_path_components(config->log_directory,
root_path, LOG_DIRECTORY_DEFAULT);
}
@@ -148,13 +148,11 @@ exit_if_necessary(int elevel)
* Actual implementation for elog() and pg_log().
*/
static void
elog_internal(int elevel, bool file_only, const char *fmt, va_list args)
elog_internal(int elevel, bool file_only, const char *message)
{
bool write_to_file,
write_to_error_log,
write_to_stderr;
va_list error_args,
std_args;
time_t log_time = (time_t) time(NULL);
char strfbuf[128];
@@ -165,22 +163,8 @@ elog_internal(int elevel, bool file_only, const char *fmt, va_list args)
write_to_stderr = elevel >= logger_config.log_level_console && !file_only;
pthread_lock(&log_file_mutex);
#ifdef WIN32
std_args = NULL;
error_args = NULL;
#endif
loggin_in_progress = true;
/* We need copy args only if we need write to error log file */
if (write_to_error_log)
va_copy(error_args, args);
/*
* We need copy args only if we need write to stderr. But do not copy args
* if we need to log only to stderr.
*/
if (write_to_stderr && write_to_file)
va_copy(std_args, args);
if (write_to_file || write_to_error_log)
strftime(strfbuf, sizeof(strfbuf), "%Y-%m-%d %H:%M:%S %Z",
localtime(&log_time));
@@ -203,8 +187,7 @@ elog_internal(int elevel, bool file_only, const char *fmt, va_list args)
fprintf(log_file, "%s: ", strfbuf);
write_elevel(log_file, elevel);
vfprintf(log_file, fmt, args);
fputc('\n', log_file);
fprintf(log_file, "%s\n", message);
fflush(log_file);
}
@@ -221,11 +204,8 @@ elog_internal(int elevel, bool file_only, const char *fmt, va_list args)
fprintf(error_log_file, "%s: ", strfbuf);
write_elevel(error_log_file, elevel);
vfprintf(error_log_file, fmt, error_args);
fputc('\n', error_log_file);
fprintf(error_log_file, "%s\n", message);
fflush(error_log_file);
va_end(error_args);
}
/*
@@ -235,15 +215,9 @@ elog_internal(int elevel, bool file_only, const char *fmt, va_list args)
if (write_to_stderr)
{
write_elevel(stderr, elevel);
if (write_to_file)
vfprintf(stderr, fmt, std_args);
else
vfprintf(stderr, fmt, args);
fputc('\n', stderr);
fflush(stderr);
if (write_to_file)
va_end(std_args);
fprintf(stderr, "%s\n", message);
fflush(stderr);
}
exit_if_necessary(elevel);
@@ -280,12 +254,44 @@ elog_stderr(int elevel, const char *fmt, ...)
exit_if_necessary(elevel);
}
/*
* Formats text data under the control of fmt and returns it in an allocated
* buffer.
*/
static char *
get_log_message(const char *fmt, va_list args)
{
size_t len = 256; /* initial assumption about buffer size */
for (;;)
{
char *result;
size_t newlen;
va_list copy_args;
result = (char *) pgut_malloc(len);
/* Try to format the data */
va_copy(copy_args, args);
newlen = pvsnprintf(result, len, fmt, copy_args);
va_end(copy_args);
if (newlen < len)
return result; /* success */
/* Release buffer and loop around to try again with larger len. */
pfree(result);
len = newlen;
}
}
/*
* Logs to stderr or to log file and exit if ERROR.
*/
void
elog(int elevel, const char *fmt, ...)
{
char *message;
va_list args;
/*
@@ -297,8 +303,11 @@ elog(int elevel, const char *fmt, ...)
return;
va_start(args, fmt);
elog_internal(elevel, false, fmt, args);
message = get_log_message(fmt, args);
va_end(args);
elog_internal(elevel, false, message);
pfree(message);
}
/*
@@ -307,6 +316,7 @@ elog(int elevel, const char *fmt, ...)
void
elog_file(int elevel, const char *fmt, ...)
{
char *message;
va_list args;
/*
@@ -317,8 +327,11 @@ elog_file(int elevel, const char *fmt, ...)
return;
va_start(args, fmt);
elog_internal(elevel, true, fmt, args);
message = get_log_message(fmt, args);
va_end(args);
elog_internal(elevel, true, message);
pfree(message);
}
/*
@@ -327,6 +340,7 @@ elog_file(int elevel, const char *fmt, ...)
void
pg_log(eLogType type, const char *fmt, ...)
{
char *message;
va_list args;
int elevel = INFO;
@@ -359,8 +373,11 @@ pg_log(eLogType type, const char *fmt, ...)
return;
va_start(args, fmt);
elog_internal(elevel, false, fmt, args);
message = get_log_message(fmt, args);
va_end(args);
elog_internal(elevel, false, message);
pfree(message);
}
/*
@@ -445,7 +462,7 @@ logfile_getname(const char *format, time_t timestamp)
logger_config.log_directory[0] == '\0')
elog_stderr(ERROR, "logging path is not set");
filename = (char *) palloc(MAXPGPATH);
filename = (char *) pgut_malloc(MAXPGPATH);
snprintf(filename, MAXPGPATH, "%s/", logger_config.log_directory);
src/utils/pgut.c
+4 -1
View File
@@ -704,7 +704,10 @@ on_interrupt(void)
/* Set interruped flag */
interrupted = true;
/* User promts password, call on_cleanup() byhand */
/*
* User promts password, call on_cleanup() byhand. Unless we do that we will
* get stuck forever until a user enters a password.
*/
if (in_password)
{
on_cleanup();
src/validate.c
+12 -4
View File
@@ -52,6 +52,13 @@ pgBackupValidate(pgBackup *backup)
validate_files_arg *threads_args;
int i;
/* Check backup version */
if (parse_program_version(backup->program_version) > parse_program_version(PROGRAM_VERSION))
elog(ERROR, "pg_probackup binary version is %s, but backup %s version is %s. "
"pg_probackup do not guarantee to be forward compatible. "
"Please upgrade pg_probackup binary.",
PROGRAM_VERSION, base36enc(backup->start_time), backup->program_version);
/* Revalidation is attempted for DONE, ORPHAN and CORRUPT backups */
if (backup->status != BACKUP_STATUS_OK &&
backup->status != BACKUP_STATUS_DONE &&
@@ -156,9 +163,10 @@ pgBackupValidateFiles(void *arg)
{
int i;
validate_files_arg *arguments = (validate_files_arg *)arg;
int num_files = parray_num(arguments->files);
pg_crc32 crc;
for (i = 0; i < parray_num(arguments->files); i++)
for (i = 0; i < num_files; i++)
{
struct stat st;
pgFile *file = (pgFile *) parray_get(arguments->files, i);
@@ -186,9 +194,9 @@ pgBackupValidateFiles(void *arg)
if (file->is_cfs)
continue;
/* print progress */
elog(VERBOSE, "Validate files: (%d/%lu) %s",
i + 1, (unsigned long) parray_num(arguments->files), file->path);
if (progress)
elog(INFO, "Progress: (%d/%d). Process file \"%s\"",
i + 1, num_files, file->path);
if (stat(file->path, &st) == -1)
{
tests/archive.py
+18 -18
View File
@@ -21,7 +21,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -79,7 +79,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -228,7 +228,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -283,7 +283,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -352,7 +352,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -418,7 +418,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -432,7 +432,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
master.slow_start()
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
master.psql(
@@ -481,7 +481,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
# RESTORE FULL BACKUP TAKEN FROM replica
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node'))
node.cleanup()
self.restore_node(backup_dir, 'replica', data_dir=node.data_dir)
node.append_conf(
@@ -550,14 +550,14 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
'archive_timeout': '10s'}
)
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.init_pb(backup_dir)
@@ -640,7 +640,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -648,7 +648,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
'archive_timeout': '10s'}
)
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.init_pb(backup_dir)
@@ -723,7 +723,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -732,7 +732,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
if self.get_version(node) < 100000:
pg_receivexlog_path = self.get_bin_path('pg_receivexlog')
else:
@@ -742,7 +742,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
[
pg_receivexlog_path, '-p', str(node.port), '--synchronous',
'-D', os.path.join(backup_dir, 'wal', 'node')
], async=True)
], asynchronous=True)
if pg_receivexlog.returncode:
self.assertFalse(
@@ -797,7 +797,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -806,7 +806,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
if self.get_version(node) < self.version_to_num('10.0'):
return unittest.skip('You need PostgreSQL 10 for this test')
else:
@@ -816,7 +816,7 @@ class ArchiveTest(ProbackupTest, unittest.TestCase):
[
pg_receivexlog_path, '-p', str(node.port), '--synchronous',
'-Z', '9', '-D', os.path.join(backup_dir, 'wal', 'node')
], async=True)
], asynchronous=True)
if pg_receivexlog.returncode:
self.assertFalse(
tests/auth_test.py
+21 -19
View File
@@ -32,7 +32,7 @@ class SimpleAuthTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -43,7 +43,7 @@ class SimpleAuthTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql("postgres", "CREATE ROLE backup with LOGIN")
@@ -126,7 +126,8 @@ class SimpleAuthTest(ProbackupTest, unittest.TestCase):
"test1", "create table t1 as select generate_series(0,100)")
node.append_conf("postgresql.auto.conf", "ptrack_enable = 'on'")
node.restart()
node.stop()
node.slow_start()
try:
self.backup_node(
@@ -203,25 +204,26 @@ class AuthTest(unittest.TestCase):
cls.pb.add_instance(cls.backup_dir, cls.node.name, cls.node)
cls.pb.set_archiving(cls.backup_dir, cls.node.name, cls.node)
try:
cls.node.start()
cls.node.slow_start()
except StartNodeException:
raise unittest.skip("Node hasn't started")
cls.node.safe_psql("postgres",
"CREATE ROLE backup WITH LOGIN PASSWORD 'password'; \
GRANT USAGE ON SCHEMA pg_catalog TO backup; \
GRANT EXECUTE ON FUNCTION current_setting(text) TO backup; \
GRANT EXECUTE ON FUNCTION pg_is_in_recovery() TO backup; \
GRANT EXECUTE ON FUNCTION pg_start_backup(text, boolean, boolean) TO backup; \
GRANT EXECUTE ON FUNCTION pg_stop_backup() TO backup; \
GRANT EXECUTE ON FUNCTION pg_stop_backup(boolean) TO backup; \
GRANT EXECUTE ON FUNCTION pg_create_restore_point(text) TO backup; \
GRANT EXECUTE ON FUNCTION pg_switch_xlog() TO backup; \
GRANT EXECUTE ON FUNCTION txid_current() TO backup; \
GRANT EXECUTE ON FUNCTION txid_current_snapshot() TO backup; \
GRANT EXECUTE ON FUNCTION txid_snapshot_xmax(txid_snapshot) TO backup; \
GRANT EXECUTE ON FUNCTION pg_ptrack_clear() TO backup; \
GRANT EXECUTE ON FUNCTION pg_ptrack_get_and_clear(oid, oid) TO backup;")
cls.node.safe_psql(
"postgres",
"CREATE ROLE backup WITH LOGIN PASSWORD 'password'; "
"GRANT USAGE ON SCHEMA pg_catalog TO backup; "
"GRANT EXECUTE ON FUNCTION current_setting(text) TO backup; "
"GRANT EXECUTE ON FUNCTION pg_is_in_recovery() TO backup; "
"GRANT EXECUTE ON FUNCTION pg_start_backup(text, boolean, boolean) TO backup; "
"GRANT EXECUTE ON FUNCTION pg_stop_backup() TO backup; "
"GRANT EXECUTE ON FUNCTION pg_stop_backup(boolean) TO backup; "
"GRANT EXECUTE ON FUNCTION pg_create_restore_point(text) TO backup; "
"GRANT EXECUTE ON FUNCTION pg_switch_xlog() TO backup; "
"GRANT EXECUTE ON FUNCTION txid_current() TO backup; "
"GRANT EXECUTE ON FUNCTION txid_current_snapshot() TO backup; "
"GRANT EXECUTE ON FUNCTION txid_snapshot_xmax(txid_snapshot) TO backup; "
"GRANT EXECUTE ON FUNCTION pg_ptrack_clear() TO backup; "
"GRANT EXECUTE ON FUNCTION pg_ptrack_get_and_clear(oid, oid) TO backup;")
cls.pgpass_file = os.path.join(os.path.expanduser('~'), '.pgpass')
@classmethod
tests/backup_test.py
+29 -40
View File
@@ -17,7 +17,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""standart backup modes with ARCHIVE WAL method"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -27,7 +27,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
show_backup = self.show_pb(backup_dir, 'node')[0]
@@ -89,7 +89,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""full backup with smooth checkpoint"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -97,7 +97,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(
backup_dir, 'node', node,
@@ -113,7 +113,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""page-level backup without validated full backup"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'ptrack_enable': 'on'}
)
@@ -121,7 +121,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
try:
self.backup_node(backup_dir, 'node', node, backup_type="page")
@@ -169,7 +169,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""page-level backup with corrupted full backup"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'ptrack_enable': 'on'}
)
@@ -177,7 +177,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
file = os.path.join(
@@ -195,12 +195,13 @@ class BackupTest(ProbackupTest, unittest.TestCase):
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertTrue(
"INFO: Validate backups of the instance 'node'\n" in e.message and
"WARNING: Backup file \"{0}\" is not found\n".format(
"INFO: Validate backups of the instance 'node'" in e.message and
"WARNING: Backup file".format(
file) in e.message and
"WARNING: Backup {0} data files are corrupted\n".format(
"is not found".format(file) in e.message and
"WARNING: Backup {0} data files are corrupted".format(
backup_id) in e.message and
"WARNING: Some backups are not valid\n" in e.message,
"WARNING: Some backups are not valid" in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
@@ -233,7 +234,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""ptrack multi thread backup mode"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'ptrack_enable': 'on'}
)
@@ -241,7 +242,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(
backup_dir, 'node', node,
@@ -261,7 +262,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""ptrack multi thread backup mode and stream"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -272,7 +273,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(
backup_dir, 'node', node, backup_type="full",
@@ -292,7 +293,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""make node, corrupt some page, check that backup failed"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -301,7 +302,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(
backup_dir, 'node', node,
@@ -349,7 +350,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""make node, corrupt some page, check that backup failed"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -358,7 +359,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(
backup_dir, 'node', node, backup_type="full",
@@ -383,7 +384,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
f.write(b"bla")
f.flush()
f.close
node.start()
node.slow_start()
try:
self.backup_node(
@@ -422,7 +423,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"""PGPRO-1376 """
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -431,7 +432,7 @@ class BackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(
node, 'tblspace1',
@@ -459,22 +460,9 @@ class BackupTest(ProbackupTest, unittest.TestCase):
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,1000) i")
try:
self.backup_node(
backup_dir, 'node', node, backup_type="full",
options=["-j", "4", "--stream"])
# we should die here because exception is what we expect to happen
self.assertEqual(
1, 0,
"Expecting Error because of too many levels "
"of symbolic linking\n"
" Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertTrue(
'Too many levels of symbolic links' in e.message,
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
repr(e.message), self.cmd))
backup_id_1 = self.backup_node(
backup_dir, 'node', node, backup_type="full",
options=["-j", "4", "--stream"])
node.safe_psql(
"postgres",
@@ -495,7 +483,8 @@ class BackupTest(ProbackupTest, unittest.TestCase):
).rstrip()
list = []
for root, dirs, files in os.walk(backup_dir):
for root, dirs, files in os.walk(os.path.join(
backup_dir, 'backups', 'node', backup_id_1)):
for file in files:
if file == relfilenode:
path = os.path.join(root, file)
tests/cfs_backup.py
+5 -5
View File
@@ -34,7 +34,7 @@ class CfsBackupNoEncTest(ProbackupTest, unittest.TestCase):
self.add_instance(self.backup_dir, 'node', self.node)
self.set_archiving(self.backup_dir, 'node', self.node)
self.node.start()
self.node.slow_start()
self.create_tblspace_in_node(self.node, tblspace_name, cfs=True)
@@ -745,7 +745,7 @@ class CfsBackupNoEncTest(ProbackupTest, unittest.TestCase):
self.restore_node(
self.backup_dir, 'node', self.node,
backup_id=backup_id_full, options=["-j", "4"])
self.node.start()
self.node.slow_start()
self.assertEqual(
full_result,
self.node.safe_psql("postgres", "SELECT * FROM t_heap"),
@@ -760,7 +760,7 @@ class CfsBackupNoEncTest(ProbackupTest, unittest.TestCase):
self.restore_node(
self.backup_dir, 'node', self.node,
backup_id=backup_id_page, options=["-j", "4"])
self.node.start()
self.node.slow_start()
self.assertEqual(
page_result,
self.node.safe_psql("postgres", "SELECT * FROM t_heap"),
@@ -879,7 +879,7 @@ class CfsBackupNoEncTest(ProbackupTest, unittest.TestCase):
self.restore_node(
self.backup_dir, 'node', self.node,
backup_id=backup_id_full, options=["-j", "4"])
self.node.start()
self.node.slow_start()
self.assertEqual(
full_result_1,
self.node.safe_psql("postgres", "SELECT * FROM t_heap_1"),
@@ -905,7 +905,7 @@ class CfsBackupNoEncTest(ProbackupTest, unittest.TestCase):
self.restore_node(
self.backup_dir, 'node', self.node,
backup_id=backup_id_page, options=["-j", "4"])
self.node.start()
self.node.slow_start()
self.assertEqual(
page_result_1,
self.node.safe_psql("postgres", "SELECT * FROM t_heap_1"),
tests/cfs_restore.py
+1 -1
View File
@@ -43,7 +43,7 @@ class CfsRestoreBase(ProbackupTest, unittest.TestCase):
self.add_instance(self.backup_dir, 'node', self.node)
self.set_archiving(self.backup_dir, 'node', self.node)
self.node.start()
self.node.slow_start()
self.create_tblspace_in_node(self.node, tblspace_name, cfs=True)
self.add_data_in_cluster()
tests/compatibility.py
+8 -8
View File
@@ -16,7 +16,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -47,7 +47,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
# RESTORE old FULL with new binary
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
@@ -116,7 +116,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -147,7 +147,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
# RESTORE old FULL with new binary
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
@@ -216,7 +216,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -247,7 +247,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
# RESTORE old FULL with new binary
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
@@ -316,7 +316,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -342,7 +342,7 @@ class CompatibilityTest(ProbackupTest, unittest.TestCase):
# restore OLD FULL with new binary
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
tests/compression.py
+11 -11
View File
@@ -18,7 +18,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -30,7 +30,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -131,7 +131,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -142,7 +142,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -238,7 +238,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -250,7 +250,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -348,7 +348,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -360,7 +360,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -458,7 +458,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -471,7 +471,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
try:
self.backup_node(
@@ -503,7 +503,7 @@ class CompressionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
tests/delete_test.py
+19 -16
View File
@@ -16,15 +16,15 @@ class DeleteTest(ProbackupTest, unittest.TestCase):
"""delete full backups"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
pg_options={'wal_level': 'replica'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# full backup
self.backup_node(backup_dir, 'node', node)
@@ -64,15 +64,16 @@ class DeleteTest(ProbackupTest, unittest.TestCase):
def test_delete_increment_page(self):
"""delete increment and all after him"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
pg_options={'wal_level': 'replica'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# full backup mode
self.backup_node(backup_dir, 'node', node)
@@ -104,15 +105,16 @@ class DeleteTest(ProbackupTest, unittest.TestCase):
def test_delete_increment_ptrack(self):
"""delete increment and all after him"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'ptrack_enable': 'on'}
)
pg_options={'wal_level': 'replica', 'ptrack_enable': 'on'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# full backup mode
self.backup_node(backup_dir, 'node', node)
@@ -144,15 +146,16 @@ class DeleteTest(ProbackupTest, unittest.TestCase):
def test_delete_orphaned_wal_segments(self):
"""make archive node, make three full backups, delete second backup without --wal option, then delete orphaned wals via --wal option"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
pg_options={'wal_level': 'replica'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
tests/delta.py
+56 -58
View File
@@ -23,25 +23,23 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
'max_wal_senders': '2',
'checkpoint_timeout': '300s',
'autovacuum': 'off'
}
)
'autovacuum': 'off'})
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
)
base_dir=os.path.join(module_name, fname, 'node_restored'))
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
node.safe_psql(
"postgres",
@@ -89,7 +87,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -105,7 +103,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -116,14 +114,14 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
node.safe_psql(
@@ -180,7 +178,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -196,7 +194,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -207,14 +205,14 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
node.safe_psql(
"postgres",
@@ -252,7 +250,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -266,7 +264,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -279,7 +277,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -318,7 +316,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
"--recovery-target-action=promote"]),
'\n Unexpected Error Message: {0}\n'
' CMD: {1}'.format(repr(self.output), self.cmd))
node.start()
node.slow_start()
full_result_new = node.execute("postgres", "SELECT * FROM t_heap")
self.assertEqual(full_result, full_result_new)
node.cleanup()
@@ -334,7 +332,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
"--recovery-target-action=promote"]),
'\n Unexpected Error Message: {0}\n'
' CMD: {1}'.format(repr(self.output), self.cmd))
node.start()
node.slow_start()
delta_result_new = node.execute("postgres", "SELECT * FROM t_heap")
self.assertEqual(delta_result, delta_result_new)
node.cleanup()
@@ -352,7 +350,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -365,7 +363,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
# self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -401,7 +399,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
"--recovery-target-action=promote"]),
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(self.output), self.cmd))
node.start()
node.slow_start()
full_result_new = node.execute("postgres", "SELECT * FROM t_heap")
self.assertEqual(full_result, full_result_new)
node.cleanup()
@@ -417,7 +415,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
"--recovery-target-action=promote"]),
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(self.output), self.cmd))
node.start()
node.slow_start()
delta_result_new = node.execute("postgres", "SELECT * FROM t_heap")
self.assertEqual(delta_result, delta_result_new)
node.cleanup()
@@ -434,7 +432,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -451,7 +449,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
# self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -478,7 +476,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# RESTORE NODE
restored_node = self.make_simple_node(
base_dir="{0}/{1}/restored_node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'restored_node'))
restored_node.cleanup()
tblspc_path = self.get_tblspace_path(node, 'somedata')
tblspc_path_new = self.get_tblspace_path(
@@ -517,7 +515,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -527,13 +525,13 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
self.backup_node(backup_dir, 'node', node, options=['--stream'])
@@ -599,7 +597,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -613,7 +611,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -627,7 +625,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -659,7 +657,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -681,7 +679,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# DROP DATABASE DB1
node.safe_psql(
@@ -716,7 +714,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
try:
node_restored.safe_psql('db1', 'select 1')
@@ -746,7 +744,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -761,7 +759,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -820,7 +818,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -842,7 +840,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -856,7 +854,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -868,7 +866,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
self.create_tblspace_in_node(node, 'somedata')
@@ -901,7 +899,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -949,7 +947,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -961,7 +959,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
# FULL backup
@@ -1002,7 +1000,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -1029,7 +1027,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
'postgresql.auto.conf', 'port = {0}'.format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -1043,7 +1041,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -1056,7 +1054,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -1092,7 +1090,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -1115,7 +1113,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
'postgresql.auto.conf', 'port = {0}'.format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -1125,7 +1123,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
"""make node, corrupt some page, check that backup failed"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1134,7 +1132,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(
backup_dir, 'node', node,
@@ -1184,7 +1182,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
"""make node, corrupt some page, check that backup failed"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1193,7 +1191,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(
backup_dir, 'node', node, backup_type="full",
@@ -1218,7 +1216,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
f.write(b"bla")
f.flush()
f.close
node.start()
node.slow_start()
try:
self.backup_node(
@@ -1259,7 +1257,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1307,7 +1305,7 @@ class DeltaTest(ProbackupTest, unittest.TestCase):
# Restore DELTA backup
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
node_restored.cleanup()
tests/exclude.py
+6 -6
View File
@@ -18,7 +18,7 @@ class ExcludeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -28,7 +28,7 @@ class ExcludeTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
conn = node.connect()
with node.connect("postgres") as conn:
@@ -109,7 +109,7 @@ class ExcludeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -122,7 +122,7 @@ class ExcludeTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
conn = node.connect()
with node.connect("postgres") as conn:
@@ -149,8 +149,8 @@ class ExcludeTest(ProbackupTest, unittest.TestCase):
pgdata = self.pgdata_content(node.data_dir)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
)
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
self.restore_node(
tests/expected/option_version.out
+1 -1
View File
@@ -1 +1 @@
pg_probackup 2.0.25
pg_probackup 2.0.26
tests/false_positive.py
+17 -16
View File
@@ -18,17 +18,17 @@ class FalsePositive(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
'max_wal_senders': '2'}
)
'max_wal_senders': '2'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
@@ -58,15 +58,15 @@ class FalsePositive(ProbackupTest, unittest.TestCase):
"""page-level backup with corrupted full backup"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'ptrack_enable': 'on'}
)
pg_options={'ptrack_enable': 'on'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
file = os.path.join(
@@ -120,7 +120,7 @@ class FalsePositive(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -132,7 +132,7 @@ class FalsePositive(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
@@ -204,7 +204,7 @@ class FalsePositive(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -216,7 +216,7 @@ class FalsePositive(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
@@ -300,15 +300,16 @@ class FalsePositive(ProbackupTest, unittest.TestCase):
def test_multiple_delete(self):
"""delete multiple backups"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
pg_options={'wal_level': 'replica'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
tests/helpers/ptrack_helpers.py
+218 -169
View File
@@ -7,7 +7,7 @@ import six
import testgres
import hashlib
import re
import pwd
import getpass
import select
import psycopg2
from time import sleep
@@ -89,8 +89,14 @@ def dir_files(base_dir):
def is_enterprise():
# pg_config --help
if os.name == 'posix':
cmd = [os.environ['PG_CONFIG'], '--help']
elif os.name == 'nt':
cmd = [[os.environ['PG_CONFIG']], ['--help']]
p = subprocess.Popen(
[os.environ['PG_CONFIG'], '--help'],
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
@@ -118,19 +124,19 @@ def slow_start(self, replica=False):
# raise_operational_error=False)
self.start()
if not replica:
if replica:
self.poll_query_until(
'postgres',
'SELECT pg_is_in_recovery()')
else:
while True:
try:
self.poll_query_until(
"postgres",
"SELECT not pg_is_in_recovery()")
'postgres',
'SELECT not pg_is_in_recovery()')
break
except Exception as e:
continue
else:
self.poll_query_until(
"postgres",
"SELECT pg_is_in_recovery()")
# while True:
# try:
@@ -155,18 +161,18 @@ class ProbackupTest(object):
self.test_env = os.environ.copy()
envs_list = [
"LANGUAGE",
"LC_ALL",
"PGCONNECT_TIMEOUT",
"PGDATA",
"PGDATABASE",
"PGHOSTADDR",
"PGREQUIRESSL",
"PGSERVICE",
"PGSSLMODE",
"PGUSER",
"PGPORT",
"PGHOST"
'LANGUAGE',
'LC_ALL',
'PGCONNECT_TIMEOUT',
'PGDATA',
'PGDATABASE',
'PGHOSTADDR',
'PGREQUIRESSL',
'PGSERVICE',
'PGSSLMODE',
'PGUSER',
'PGPORT',
'PGHOST'
]
for e in envs_list:
@@ -175,8 +181,8 @@ class ProbackupTest(object):
except:
pass
self.test_env["LC_MESSAGES"] = "C"
self.test_env["LC_TIME"] = "C"
self.test_env['LC_MESSAGES'] = 'C'
self.test_env['LC_TIME'] = 'C'
self.paranoia = False
if 'PG_PROBACKUP_PARANOIA' in self.test_env:
@@ -210,7 +216,7 @@ class ProbackupTest(object):
self.user = self.get_username()
self.probackup_path = None
if "PGPROBACKUPBIN" in self.test_env:
if 'PGPROBACKUPBIN' in self.test_env:
if (
os.path.isfile(self.test_env["PGPROBACKUPBIN"]) and
os.access(self.test_env["PGPROBACKUPBIN"], os.X_OK)
@@ -222,7 +228,7 @@ class ProbackupTest(object):
if not self.probackup_path:
probackup_path_tmp = os.path.join(
testgres.get_pg_config()["BINDIR"], 'pg_probackup')
testgres.get_pg_config()['BINDIR'], 'pg_probackup')
if os.path.isfile(probackup_path_tmp):
if not os.access(probackup_path_tmp, os.X_OK):
@@ -233,7 +239,7 @@ class ProbackupTest(object):
if not self.probackup_path:
probackup_path_tmp = os.path.abspath(os.path.join(
self.dir_path, "../pg_probackup"))
self.dir_path, '../pg_probackup'))
if os.path.isfile(probackup_path_tmp):
if not os.access(probackup_path_tmp, os.X_OK):
@@ -246,17 +252,22 @@ class ProbackupTest(object):
print('pg_probackup binary is not found')
exit(1)
os.environ['PATH'] = os.path.dirname(
self.probackup_path) + ":" + os.environ['PATH']
if os.name == 'posix':
os.environ['PATH'] = os.path.dirname(
self.probackup_path) + ':' + os.environ['PATH']
elif os.name == 'nt':
os.environ['PATH'] = os.path.dirname(
self.probackup_path) + ';' + os.environ['PATH']
self.probackup_old_path = None
if "PGPROBACKUPBIN_OLD" in self.test_env:
if 'PGPROBACKUPBIN_OLD' in self.test_env:
if (
os.path.isfile(self.test_env["PGPROBACKUPBIN_OLD"]) and
os.access(self.test_env["PGPROBACKUPBIN_OLD"], os.X_OK)
os.path.isfile(self.test_env['PGPROBACKUPBIN_OLD']) and
os.access(self.test_env['PGPROBACKUPBIN_OLD'], os.X_OK)
):
self.probackup_old_path = self.test_env["PGPROBACKUPBIN_OLD"]
self.probackup_old_path = self.test_env['PGPROBACKUPBIN_OLD']
else:
if self.verbose:
print('PGPROBACKUPBIN_OLD is not an executable file')
@@ -280,40 +291,37 @@ class ProbackupTest(object):
initdb_params=initdb_params, allow_streaming=set_replication)
# Sane default parameters
node.append_conf("postgresql.auto.conf", "max_connections = 100")
node.append_conf("postgresql.auto.conf", "shared_buffers = 10MB")
node.append_conf("postgresql.auto.conf", "fsync = on")
node.append_conf("postgresql.auto.conf", "wal_level = logical")
node.append_conf("postgresql.auto.conf", "hot_standby = 'off'")
node.append_conf('postgresql.auto.conf', 'max_connections = 100')
node.append_conf('postgresql.auto.conf', 'shared_buffers = 10MB')
node.append_conf('postgresql.auto.conf', 'fsync = on')
node.append_conf('postgresql.auto.conf', 'wal_level = logical')
node.append_conf('postgresql.auto.conf', 'hot_standby = off')
node.append_conf(
"postgresql.auto.conf", "log_line_prefix = '%t [%p]: [%l-1] '")
node.append_conf("postgresql.auto.conf", "log_statement = none")
node.append_conf("postgresql.auto.conf", "log_duration = on")
'postgresql.auto.conf', "log_line_prefix = '%t [%p]: [%l-1] '")
node.append_conf('postgresql.auto.conf', 'log_statement = none')
node.append_conf('postgresql.auto.conf', 'log_duration = on')
node.append_conf(
"postgresql.auto.conf", "log_min_duration_statement = 0")
node.append_conf("postgresql.auto.conf", "log_connections = on")
node.append_conf("postgresql.auto.conf", "log_disconnections = on")
'postgresql.auto.conf', 'log_min_duration_statement = 0')
node.append_conf('postgresql.auto.conf', 'log_connections = on')
node.append_conf('postgresql.auto.conf', 'log_disconnections = on')
# Apply given parameters
for key, value in six.iteritems(pg_options):
node.append_conf("postgresql.auto.conf", "%s = %s" % (key, value))
node.append_conf('postgresql.auto.conf', '%s = %s' % (key, value))
# Allow replication in pg_hba.conf
if set_replication:
node.append_conf(
"pg_hba.conf",
"local replication all trust\n")
node.append_conf(
"postgresql.auto.conf",
"max_wal_senders = 10")
'postgresql.auto.conf',
'max_wal_senders = 10')
return node
def create_tblspace_in_node(self, node, tblspc_name, tblspc_path=None, cfs=False):
res = node.execute(
"postgres",
"select exists"
'postgres',
'select exists'
" (select 1 from pg_tablespace where spcname = '{0}')".format(
tblspc_name)
)
@@ -329,11 +337,11 @@ class ProbackupTest(object):
cmd = "CREATE TABLESPACE {0} LOCATION '{1}'".format(
tblspc_name, tblspc_path)
if cfs:
cmd += " with (compression=true)"
cmd += ' with (compression=true)'
if not os.path.exists(tblspc_path):
os.makedirs(tblspc_path)
res = node.safe_psql("postgres", cmd)
res = node.safe_psql('postgres', cmd)
# Check that tablespace was successfully created
# self.assertEqual(
# res[0], 0,
@@ -344,13 +352,13 @@ class ProbackupTest(object):
def get_fork_size(self, node, fork_name):
return node.execute(
"postgres",
'postgres',
"select pg_relation_size('{0}')/8192".format(fork_name))[0][0]
def get_fork_path(self, node, fork_name):
return os.path.join(
node.base_dir, 'data', node.execute(
"postgres",
'postgres',
"select pg_relation_filepath('{0}')".format(
fork_name))[0][0]
)
@@ -378,7 +386,7 @@ class ProbackupTest(object):
end_page = pages_per_segment[segment_number]
else:
file_desc = os.open(
file+".{0}".format(segment_number), os.O_RDONLY
file+'.{0}'.format(segment_number), os.O_RDONLY
)
start_page = max(md5_per_page)+1
end_page = end_page + pages_per_segment[segment_number]
@@ -481,8 +489,8 @@ class ProbackupTest(object):
idx_dict['ptrack'][PageNum])
)
print(
" Old checksumm: {0}\n"
" New checksumm: {1}".format(
' Old checksumm: {0}\n'
' New checksumm: {1}'.format(
idx_dict['old_pages'][PageNum],
idx_dict['new_pages'][PageNum])
)
@@ -543,9 +551,9 @@ class ProbackupTest(object):
)
)
def run_pb(self, command, async=False, gdb=False, old_binary=False):
def run_pb(self, command, asynchronous=False, gdb=False, old_binary=False):
if not self.probackup_old_path and old_binary:
print("PGPROBACKUPBIN_OLD is not set")
print('PGPROBACKUPBIN_OLD is not set')
exit(1)
if old_binary:
@@ -559,7 +567,7 @@ class ProbackupTest(object):
print(self.cmd)
if gdb:
return GDBobj([binary_path] + command, self.verbose)
if async:
if asynchronous:
return subprocess.Popen(
self.cmd,
stdout=subprocess.PIPE,
@@ -571,7 +579,7 @@ class ProbackupTest(object):
[binary_path] + command,
stderr=subprocess.STDOUT,
env=self.test_env
).decode("utf-8")
).decode('utf-8')
if command[0] == 'backup':
# return backup ID
for line in self.output.splitlines():
@@ -580,13 +588,13 @@ class ProbackupTest(object):
else:
return self.output
except subprocess.CalledProcessError as e:
raise ProbackupException(e.output.decode("utf-8"), self.cmd)
raise ProbackupException(e.output.decode('utf-8'), self.cmd)
def run_binary(self, command, async=False):
def run_binary(self, command, asynchronous=False):
if self.verbose:
print([' '.join(map(str, command))])
try:
if async:
if asynchronous:
return subprocess.Popen(
command,
stdin=subprocess.PIPE,
@@ -599,18 +607,18 @@ class ProbackupTest(object):
command,
stderr=subprocess.STDOUT,
env=self.test_env
).decode("utf-8")
).decode('utf-8')
return self.output
except subprocess.CalledProcessError as e:
raise ProbackupException(e.output.decode("utf-8"), command)
raise ProbackupException(e.output.decode('utf-8'), command)
def init_pb(self, backup_dir, old_binary=False):
shutil.rmtree(backup_dir, ignore_errors=True)
return self.run_pb([
"init",
"-B", backup_dir
'init',
'-B', backup_dir
],
old_binary=old_binary
)
@@ -618,10 +626,10 @@ class ProbackupTest(object):
def add_instance(self, backup_dir, instance, node, old_binary=False):
return self.run_pb([
"add-instance",
"--instance={0}".format(instance),
"-B", backup_dir,
"-D", node.data_dir
'add-instance',
'--instance={0}'.format(instance),
'-B', backup_dir,
'-D', node.data_dir
],
old_binary=old_binary
)
@@ -629,9 +637,9 @@ class ProbackupTest(object):
def del_instance(self, backup_dir, instance, old_binary=False):
return self.run_pb([
"del-instance",
"--instance={0}".format(instance),
"-B", backup_dir
'del-instance',
'--instance={0}'.format(instance),
'-B', backup_dir
],
old_binary=old_binary
)
@@ -641,7 +649,7 @@ class ProbackupTest(object):
def backup_node(
self, backup_dir, instance, node, data_dir=False,
backup_type="full", options=[], async=False, gdb=False,
backup_type='full', options=[], asynchronous=False, gdb=False,
old_binary=False
):
if not node and not data_dir:
@@ -655,29 +663,29 @@ class ProbackupTest(object):
pgdata = data_dir
cmd_list = [
"backup",
"-B", backup_dir,
'backup',
'-B', backup_dir,
# "-D", pgdata,
"-p", "%i" % node.port,
"-d", "postgres",
"--instance={0}".format(instance)
'-p', '%i' % node.port,
'-d', 'postgres',
'--instance={0}'.format(instance)
]
if backup_type:
cmd_list += ["-b", backup_type]
cmd_list += ['-b', backup_type]
return self.run_pb(cmd_list + options, async, gdb, old_binary)
return self.run_pb(cmd_list + options, asynchronous, gdb, old_binary)
def merge_backup(
self, backup_dir, instance, backup_id, async=False,
self, backup_dir, instance, backup_id, asynchronous=False,
gdb=False, old_binary=False, options=[]):
cmd_list = [
"merge",
"-B", backup_dir,
"--instance={0}".format(instance),
"-i", backup_id
'merge',
'-B', backup_dir,
'--instance={0}'.format(instance),
'-i', backup_id
]
return self.run_pb(cmd_list + options, async, gdb, old_binary)
return self.run_pb(cmd_list + options, asynchronous, gdb, old_binary)
def restore_node(
self, backup_dir, instance, node=False,
@@ -687,13 +695,13 @@ class ProbackupTest(object):
data_dir = node.data_dir
cmd_list = [
"restore",
"-B", backup_dir,
"-D", data_dir,
"--instance={0}".format(instance)
'restore',
'-B', backup_dir,
'-D', data_dir,
'--instance={0}'.format(instance)
]
if backup_id:
cmd_list += ["-i", backup_id]
cmd_list += ['-i', backup_id]
return self.run_pb(cmd_list + options, old_binary=old_binary)
@@ -705,17 +713,17 @@ class ProbackupTest(object):
backup_list = []
specific_record = {}
cmd_list = [
"show",
"-B", backup_dir,
'show',
'-B', backup_dir,
]
if instance:
cmd_list += ["--instance={0}".format(instance)]
cmd_list += ['--instance={0}'.format(instance)]
if backup_id:
cmd_list += ["-i", backup_id]
cmd_list += ['-i', backup_id]
if as_json:
cmd_list += ["--format=json"]
cmd_list += ['--format=json']
if as_text:
# You should print it when calling as_text=true
@@ -750,7 +758,7 @@ class ProbackupTest(object):
# inverse list so oldest record come first
body = body[::-1]
# split string in list with string for every header element
header_split = re.split(" +", header)
header_split = re.split(' +', header)
# Remove empty items
for i in header_split:
if i == '':
@@ -762,7 +770,7 @@ class ProbackupTest(object):
for backup_record in body:
backup_record = backup_record.rstrip()
# split list with str for every backup record element
backup_record_split = re.split(" +", backup_record)
backup_record_split = re.split(' +', backup_record)
# Remove empty items
for i in backup_record_split:
if i == '':
@@ -787,7 +795,7 @@ class ProbackupTest(object):
]
# print sanitized_show
for line in sanitized_show:
name, var = line.partition(" = ")[::2]
name, var = line.partition(' = ')[::2]
var = var.strip('"')
var = var.strip("'")
specific_record[name.strip()] = var
@@ -799,13 +807,13 @@ class ProbackupTest(object):
):
cmd_list = [
"validate",
"-B", backup_dir
'validate',
'-B', backup_dir
]
if instance:
cmd_list += ["--instance={0}".format(instance)]
cmd_list += ['--instance={0}'.format(instance)]
if backup_id:
cmd_list += ["-i", backup_id]
cmd_list += ['-i', backup_id]
return self.run_pb(cmd_list + options, old_binary=old_binary)
@@ -813,48 +821,48 @@ class ProbackupTest(object):
self, backup_dir, instance,
backup_id=None, options=[], old_binary=False):
cmd_list = [
"delete",
"-B", backup_dir
'delete',
'-B', backup_dir
]
cmd_list += ["--instance={0}".format(instance)]
cmd_list += ['--instance={0}'.format(instance)]
if backup_id:
cmd_list += ["-i", backup_id]
cmd_list += ['-i', backup_id]
return self.run_pb(cmd_list + options, old_binary=old_binary)
def delete_expired(
self, backup_dir, instance, options=[], old_binary=False):
cmd_list = [
"delete", "--expired", "--wal",
"-B", backup_dir,
"--instance={0}".format(instance)
'delete', '--expired', '--wal',
'-B', backup_dir,
'--instance={0}'.format(instance)
]
return self.run_pb(cmd_list + options, old_binary=old_binary)
def show_config(self, backup_dir, instance, old_binary=False):
out_dict = {}
cmd_list = [
"show-config",
"-B", backup_dir,
"--instance={0}".format(instance)
'show-config',
'-B', backup_dir,
'--instance={0}'.format(instance)
]
res = self.run_pb(cmd_list, old_binary=old_binary).splitlines()
for line in res:
if not line.startswith('#'):
name, var = line.partition(" = ")[::2]
name, var = line.partition(' = ')[::2]
out_dict[name] = var
return out_dict
def get_recovery_conf(self, node):
out_dict = {}
with open(
os.path.join(node.data_dir, "recovery.conf"), "r"
os.path.join(node.data_dir, 'recovery.conf'), 'r'
) as recovery_conf:
for line in recovery_conf:
try:
key, value = line.split("=")
key, value = line.split('=')
except:
continue
out_dict[key.strip()] = value.strip(" '").replace("'\n", "")
@@ -870,35 +878,36 @@ class ProbackupTest(object):
else:
archive_mode = 'on'
# node.append_conf(
# "postgresql.auto.conf",
# "wal_level = archive"
# )
node.append_conf(
"postgresql.auto.conf",
"archive_mode = {0}".format(archive_mode)
'postgresql.auto.conf',
'archive_mode = {0}'.format(archive_mode)
)
archive_command = "{0} archive-push -B {1} --instance={2} ".format(
self.probackup_path, backup_dir, instance)
if os.name == 'posix':
archive_command = '"{0}" archive-push -B {1} --instance={2} '.format(
self.probackup_path, backup_dir, instance)
elif os.name == 'nt':
archive_command = '"{0}" archive-push -B {1} --instance={2} '.format(
self.probackup_path.replace("\\","\\\\"),
backup_dir.replace("\\","\\\\"),
instance)
if self.archive_compress or compress:
archive_command = archive_command + '--compress '
if overwrite:
archive_command = archive_command + '--overwrite '
if os.name == 'posix':
if self.archive_compress or compress:
archive_command = archive_command + "--compress "
archive_command = archive_command + '--wal-file-path %p --wal-file-name %f'
if overwrite:
archive_command = archive_command + "--overwrite "
archive_command = archive_command + "--wal-file-path %p --wal-file-name %f"
elif os.name == 'nt':
archive_command = archive_command + '--wal-file-path "%p" --wal-file-name "%f"'
node.append_conf(
"postgresql.auto.conf",
'postgresql.auto.conf',
"archive_command = '{0}'".format(
archive_command))
# elif os.name == 'nt':
# node.append_conf(
# "postgresql.auto.conf",
# "archive_command = 'copy %p {0}\\%f'".format(archive_dir)
# )
def set_replica(
self, master, replica,
@@ -906,18 +915,18 @@ class ProbackupTest(object):
synchronous=False
):
replica.append_conf(
"postgresql.auto.conf", "port = {0}".format(replica.port))
'postgresql.auto.conf', 'port = {0}'.format(replica.port))
replica.append_conf('postgresql.auto.conf', 'hot_standby = on')
replica.append_conf('recovery.conf', "standby_mode = 'on'")
replica.append_conf('recovery.conf', 'standby_mode = on')
replica.append_conf(
"recovery.conf",
'recovery.conf',
"primary_conninfo = 'user={0} port={1} application_name={2}"
" sslmode=prefer sslcompression=1'".format(
self.user, master.port, replica_name)
)
if synchronous:
master.append_conf(
"postgresql.auto.conf",
'postgresql.auto.conf',
"synchronous_standby_names='{0}'".format(replica_name)
)
master.append_conf(
@@ -927,7 +936,7 @@ class ProbackupTest(object):
master.reload()
def wrong_wal_clean(self, node, wal_size):
wals_dir = os.path.join(self.backup_dir(node), "wal")
wals_dir = os.path.join(self.backup_dir(node), 'wal')
wals = [
f for f in os.listdir(wals_dir) if os.path.isfile(
os.path.join(wals_dir, f))
@@ -939,39 +948,39 @@ class ProbackupTest(object):
def guc_wal_segment_size(self, node):
var = node.execute(
"postgres",
'postgres',
"select setting from pg_settings where name = 'wal_segment_size'"
)
return int(var[0][0]) * self.guc_wal_block_size(node)
def guc_wal_block_size(self, node):
var = node.execute(
"postgres",
'postgres',
"select setting from pg_settings where name = 'wal_block_size'"
)
return int(var[0][0])
def get_pgpro_edition(self, node):
if node.execute(
"postgres",
'postgres',
"select exists (select 1 from"
" pg_proc where proname = 'pgpro_edition')"
)[0][0]:
var = node.execute("postgres", "select pgpro_edition()")
var = node.execute('postgres', 'select pgpro_edition()')
return str(var[0][0])
else:
return False
def get_username(self):
""" Returns current user name """
return pwd.getpwuid(os.getuid())[0]
return getpass.getuser()
def version_to_num(self, version):
if not version:
return 0
parts = version.split(".")
parts = version.split('.')
while len(parts) < 3:
parts.append("0")
parts.append('0')
num = 0
for part in parts:
num = num * 100 + int(re.sub("[^\d]", "", part))
@@ -986,25 +995,25 @@ class ProbackupTest(object):
"""
if isinstance(node, testgres.PostgresNode):
if self.version_to_num(
node.safe_psql("postgres", "show server_version")
node.safe_psql('postgres', 'show server_version')
) >= self.version_to_num('10.0'):
node.safe_psql("postgres", "select pg_switch_wal()")
node.safe_psql('postgres', 'select pg_switch_wal()')
else:
node.safe_psql("postgres", "select pg_switch_xlog()")
node.safe_psql('postgres', 'select pg_switch_xlog()')
else:
if self.version_to_num(
node.execute("show server_version")[0][0]
node.execute('show server_version')[0][0]
) >= self.version_to_num('10.0'):
node.execute("select pg_switch_wal()")
node.execute('select pg_switch_wal()')
else:
node.execute("select pg_switch_xlog()")
node.execute('select pg_switch_xlog()')
def wait_until_replica_catch_with_master(self, master, replica):
if self.version_to_num(
master.safe_psql(
"postgres",
"show server_version")) >= self.version_to_num('10.0'):
'postgres',
'show server_version')) >= self.version_to_num('10.0'):
master_function = 'pg_catalog.pg_current_wal_lsn()'
replica_function = 'pg_catalog.pg_last_wal_replay_lsn()'
else:
@@ -1022,7 +1031,7 @@ class ProbackupTest(object):
def get_version(self, node):
return self.version_to_num(
testgres.get_pg_config()["VERSION"].split(" ")[1])
testgres.get_pg_config()['VERSION'].split(" ")[1])
def get_bin_path(self, binary):
return testgres.get_bin_path(binary)
@@ -1047,7 +1056,7 @@ class ProbackupTest(object):
except:
pass
def pgdata_content(self, directory, ignore_ptrack=True):
def pgdata_content(self, pgdata, ignore_ptrack=True):
""" return dict with directory content. "
" TAKE IT AFTER CHECKPOINT or BACKUP"""
dirs_to_ignore = [
@@ -1064,9 +1073,10 @@ class ProbackupTest(object):
# '_ptrack'
# )
directory_dict = {}
directory_dict['pgdata'] = directory
directory_dict['pgdata'] = pgdata
directory_dict['files'] = {}
for root, dirs, files in os.walk(directory, followlinks=True):
directory_dict['dirs'] = []
for root, dirs, files in os.walk(pgdata, followlinks=True):
dirs[:] = [d for d in dirs if d not in dirs_to_ignore]
for file in files:
if (
@@ -1076,7 +1086,7 @@ class ProbackupTest(object):
continue
file_fullpath = os.path.join(root, file)
file_relpath = os.path.relpath(file_fullpath, directory)
file_relpath = os.path.relpath(file_fullpath, pgdata)
directory_dict['files'][file_relpath] = {'is_datafile': False}
directory_dict['files'][file_relpath]['md5'] = hashlib.md5(
open(file_fullpath, 'rb').read()).hexdigest()
@@ -1089,12 +1099,51 @@ class ProbackupTest(object):
file_fullpath, size_in_pages
)
for root, dirs, files in os.walk(pgdata, topdown=False, followlinks=True):
for directory in dirs:
directory_path = os.path.join(root, directory)
directory_relpath = os.path.relpath(directory_path, pgdata)
found = False
for d in dirs_to_ignore:
if d in directory_relpath:
found = True
break
# check if directory already here as part of larger directory
if not found:
for d in directory_dict['dirs']:
# print("OLD dir {0}".format(d))
if directory_relpath in d:
found = True
break
if not found:
directory_dict['dirs'].append(directory_relpath)
return directory_dict
def compare_pgdata(self, original_pgdata, restored_pgdata):
""" return dict with directory content. DO IT BEFORE RECOVERY"""
fail = False
error_message = 'Restored PGDATA is not equal to original!\n'
# Compare directories
for directory in restored_pgdata['dirs']:
if directory not in original_pgdata['dirs']:
fail = True
error_message += '\nDirectory was not present'
error_message += ' in original PGDATA: {0}\n'.format(
os.path.join(restored_pgdata['pgdata'], directory))
for directory in original_pgdata['dirs']:
if directory not in restored_pgdata['dirs']:
fail = True
error_message += '\nDirectory dissappeared'
error_message += ' in restored PGDATA: {0}\n'.format(
os.path.join(restored_pgdata['pgdata'], directory))
for file in restored_pgdata['files']:
# File is present in RESTORED PGDATA
# but not present in ORIGINAL
@@ -1177,10 +1226,10 @@ class ProbackupTest(object):
host = '127.0.0.1'
return psycopg2.connect(
database="postgres",
database='postgres',
host='127.0.0.1',
port=port,
async=True
async_=True
)
def wait(self, connection):
@@ -1193,7 +1242,7 @@ class ProbackupTest(object):
elif state == psycopg2.extensions.POLL_READ:
select.select([connection.fileno()], [], [])
else:
raise psycopg2.OperationalError("poll() returned %s" % state)
raise psycopg2.OperationalError('poll() returned %s' % state)
def gdb_attach(self, pid):
return GDBobj([str(pid)], self.verbose, attach=True)
@@ -1214,7 +1263,7 @@ class GDBobj(ProbackupTest):
# Check gdb presense
try:
gdb_version, _ = subprocess.Popen(
["gdb", "--version"],
['gdb', '--version'],
stdout=subprocess.PIPE
).communicate()
except OSError:
tests/init_test.py
+3 -3
View File
@@ -14,7 +14,7 @@ class InitTest(ProbackupTest, unittest.TestCase):
"""Success normal init"""
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname))
node = self.make_simple_node(base_dir=os.path.join(module_name, fname, 'node'))
self.init_pb(backup_dir)
self.assertEqual(
dir_files(backup_dir),
@@ -66,7 +66,7 @@ class InitTest(ProbackupTest, unittest.TestCase):
"""Failure with backup catalog already existed"""
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname))
node = self.make_simple_node(base_dir=os.path.join(module_name, fname, 'node'))
self.init_pb(backup_dir)
try:
self.show_pb(backup_dir, 'node')
@@ -85,7 +85,7 @@ class InitTest(ProbackupTest, unittest.TestCase):
"""failure with backup catalog should be given as absolute path"""
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname))
node = self.make_simple_node(base_dir=os.path.join(module_name, fname, 'node'))
try:
self.run_pb(["init", "-B", os.path.relpath("%s/backup" % node.base_dir, self.dir_path)])
self.assertEqual(1, 0, 'Expecting Error due to initialization with non-absolute path in --backup-path. Output: {0} \n CMD: {1}'.format(
tests/merge.py
+580 -26
View File
@@ -3,6 +3,7 @@
import unittest
import os
from .helpers.ptrack_helpers import ProbackupTest, ProbackupException
import shutil
module_name = "merge"
@@ -18,14 +19,14 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=["--data-checksums"]
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, "node", node)
self.set_archiving(backup_dir, "node", node)
node.start()
node.slow_start()
# Do full backup
self.backup_node(backup_dir, "node", node)
@@ -107,14 +108,14 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=["--data-checksums"]
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, "node", node)
self.set_archiving(backup_dir, "node", node)
node.start()
node.slow_start()
# Do full compressed backup
self.backup_node(backup_dir, "node", node, options=[
@@ -163,11 +164,476 @@ class MergeTest(ProbackupTest, unittest.TestCase):
node.cleanup()
self.del_test_dir(module_name, fname)
def test_merge_compressed_backups_1(self):
"""
Test MERGE command with compressed backups
"""
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, "backup")
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=["--data-checksums"],
pg_options={
'autovacuum': 'off'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, "node", node)
self.set_archiving(backup_dir, "node", node)
node.slow_start()
# Fill with data
node.pgbench_init(scale=5)
# Do compressed FULL backup
self.backup_node(backup_dir, "node", node, options=[
'--compress-algorithm=zlib', '--stream'])
show_backup = self.show_pb(backup_dir, "node")[0]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "FULL")
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do compressed DELTA backup
self.backup_node(
backup_dir, "node", node, backup_type="delta",
options=['--compress-algorithm=zlib', '--stream'])
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do compressed PAGE backup
self.backup_node(
backup_dir, "node", node, backup_type="page",
options=['--compress-algorithm=zlib'])
pgdata = self.pgdata_content(node.data_dir)
show_backup = self.show_pb(backup_dir, "node")[2]
page_id = show_backup["id"]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "PAGE")
# Merge all backups
self.merge_backup(backup_dir, "node", page_id)
show_backups = self.show_pb(backup_dir, "node")
self.assertEqual(len(show_backups), 1)
self.assertEqual(show_backups[0]["status"], "OK")
self.assertEqual(show_backups[0]["backup-mode"], "FULL")
# Drop node and restore it
node.cleanup()
self.restore_node(backup_dir, 'node', node)
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
node.cleanup()
self.del_test_dir(module_name, fname)
def test_merge_compressed_and_uncompressed_backups(self):
"""
Test MERGE command with compressed and uncompressed backups
"""
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, "backup")
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=["--data-checksums"],
pg_options={
'autovacuum': 'off'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, "node", node)
self.set_archiving(backup_dir, "node", node)
node.slow_start()
# Fill with data
node.pgbench_init(scale=5)
# Do compressed FULL backup
self.backup_node(backup_dir, "node", node, options=[
'--compress-algorithm=zlib', '--stream'])
show_backup = self.show_pb(backup_dir, "node")[0]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "FULL")
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do compressed DELTA backup
self.backup_node(
backup_dir, "node", node, backup_type="delta",
options=['--compress-algorithm=zlib', '--stream'])
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do uncompressed PAGE backup
self.backup_node(
backup_dir, "node", node, backup_type="page")
pgdata = self.pgdata_content(node.data_dir)
show_backup = self.show_pb(backup_dir, "node")[2]
page_id = show_backup["id"]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "PAGE")
# Merge all backups
self.merge_backup(backup_dir, "node", page_id)
show_backups = self.show_pb(backup_dir, "node")
self.assertEqual(len(show_backups), 1)
self.assertEqual(show_backups[0]["status"], "OK")
self.assertEqual(show_backups[0]["backup-mode"], "FULL")
# Drop node and restore it
node.cleanup()
self.restore_node(backup_dir, 'node', node)
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
node.cleanup()
self.del_test_dir(module_name, fname)
def test_merge_compressed_and_uncompressed_backups_1(self):
"""
Test MERGE command with compressed and uncompressed backups
"""
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, "backup")
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=["--data-checksums"],
pg_options={
'autovacuum': 'off'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, "node", node)
self.set_archiving(backup_dir, "node", node)
node.slow_start()
# Fill with data
node.pgbench_init(scale=5)
# Do compressed FULL backup
self.backup_node(backup_dir, "node", node, options=[
'--compress-algorithm=zlib', '--stream'])
show_backup = self.show_pb(backup_dir, "node")[0]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "FULL")
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do uncompressed DELTA backup
self.backup_node(
backup_dir, "node", node, backup_type="delta",
options=['--stream'])
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do compressed PAGE backup
self.backup_node(
backup_dir, "node", node, backup_type="page",
options=['--compress-algorithm=zlib'])
pgdata = self.pgdata_content(node.data_dir)
show_backup = self.show_pb(backup_dir, "node")[2]
page_id = show_backup["id"]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "PAGE")
# Merge all backups
self.merge_backup(backup_dir, "node", page_id)
show_backups = self.show_pb(backup_dir, "node")
self.assertEqual(len(show_backups), 1)
self.assertEqual(show_backups[0]["status"], "OK")
self.assertEqual(show_backups[0]["backup-mode"], "FULL")
# Drop node and restore it
node.cleanup()
self.restore_node(backup_dir, 'node', node)
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
node.cleanup()
self.del_test_dir(module_name, fname)
def test_merge_compressed_and_uncompressed_backups_2(self):
"""
Test MERGE command with compressed and uncompressed backups
"""
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, "backup")
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=["--data-checksums"],
pg_options={
'autovacuum': 'off'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, "node", node)
self.set_archiving(backup_dir, "node", node)
node.slow_start()
# Fill with data
node.pgbench_init(scale=5)
# Do uncompressed FULL backup
self.backup_node(backup_dir, "node", node)
show_backup = self.show_pb(backup_dir, "node")[0]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "FULL")
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do compressed DELTA backup
self.backup_node(
backup_dir, "node", node, backup_type="delta",
options=['--compress-algorithm=zlib', '--stream'])
# Change data
pgbench = node.pgbench(options=['-T', '20', '-c', '2', '--no-vacuum'])
pgbench.wait()
# Do uncompressed PAGE backup
self.backup_node(
backup_dir, "node", node, backup_type="page")
pgdata = self.pgdata_content(node.data_dir)
show_backup = self.show_pb(backup_dir, "node")[2]
page_id = show_backup["id"]
self.assertEqual(show_backup["status"], "OK")
self.assertEqual(show_backup["backup-mode"], "PAGE")
# Merge all backups
self.merge_backup(backup_dir, "node", page_id)
show_backups = self.show_pb(backup_dir, "node")
self.assertEqual(len(show_backups), 1)
self.assertEqual(show_backups[0]["status"], "OK")
self.assertEqual(show_backups[0]["backup-mode"], "FULL")
# Drop node and restore it
node.cleanup()
self.restore_node(backup_dir, 'node', node)
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
node.cleanup()
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_merge_tablespaces(self):
"""
Some test here
Create tablespace with table, take FULL backup,
create another tablespace with another table and drop previous
tablespace, take page backup, merge it and restore
"""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
'max_wal_senders': '2',
'autovacuum': 'off'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
node.safe_psql(
"postgres",
"create table t_heap tablespace somedata as select i as id,"
" md5(i::text) as text, md5(i::text)::tsvector as tsvector"
" from generate_series(0,100) i"
)
# FULL backup
self.backup_node(backup_dir, 'node', node)
# Create new tablespace
self.create_tblspace_in_node(node, 'somedata1')
node.safe_psql(
"postgres",
"create table t_heap1 tablespace somedata1 as select i as id,"
" md5(i::text) as text, md5(i::text)::tsvector as tsvector"
" from generate_series(0,100) i"
)
node.safe_psql(
"postgres",
"drop table t_heap"
)
# Drop old tablespace
node.safe_psql(
"postgres",
"drop tablespace somedata"
)
# PAGE backup
backup_id = self.backup_node(backup_dir, 'node', node, backup_type="page")
pgdata = self.pgdata_content(node.data_dir)
node.stop()
shutil.rmtree(
self.get_tblspace_path(node, 'somedata'),
ignore_errors=True)
shutil.rmtree(
self.get_tblspace_path(node, 'somedata1'),
ignore_errors=True)
node.cleanup()
self.merge_backup(backup_dir, 'node', backup_id)
self.restore_node(
backup_dir, 'node', node, options=["-j", "4"])
pgdata_restored = self.pgdata_content(node.data_dir)
# this compare should fall because we lost some directories
self.compare_pgdata(pgdata, pgdata_restored)
# @unittest.skip("skip")
def test_merge_tablespaces_1(self):
"""
Create tablespace with table, take FULL backup,
create another tablespace with another table, take page backup,
drop first tablespace and take delta backup,
merge it and restore
"""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
'max_wal_senders': '2',
'autovacuum': 'off'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
# FULL backup
self.backup_node(backup_dir, 'node', node)
node.safe_psql(
"postgres",
"create table t_heap tablespace somedata as select i as id,"
" md5(i::text) as text, md5(i::text)::tsvector as tsvector"
" from generate_series(0,100) i"
)
# CREATE NEW TABLESPACE
self.create_tblspace_in_node(node, 'somedata1')
node.safe_psql(
"postgres",
"create table t_heap1 tablespace somedata1 as select i as id,"
" md5(i::text) as text, md5(i::text)::tsvector as tsvector"
" from generate_series(0,100) i"
)
# PAGE backup
self.backup_node(backup_dir, 'node', node, backup_type="page")
node.safe_psql(
"postgres",
"drop table t_heap"
)
node.safe_psql(
"postgres",
"drop tablespace somedata"
)
# DELTA backup
backup_id = self.backup_node(
backup_dir, 'node', node, backup_type="delta")
pgdata = self.pgdata_content(node.data_dir)
node.stop()
shutil.rmtree(
self.get_tblspace_path(node, 'somedata'),
ignore_errors=True)
shutil.rmtree(
self.get_tblspace_path(node, 'somedata1'),
ignore_errors=True)
node.cleanup()
self.merge_backup(backup_dir, 'node', backup_id)
self.restore_node(
backup_dir, 'node', node,
options=["-j", "4"])
pgdata_restored = self.pgdata_content(node.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# Clean after yourself
self.del_test_dir(module_name, fname)
def test_merge_page_truncate(self):
"""
@@ -179,7 +645,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -190,13 +656,13 @@ class MergeTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
node.safe_psql(
@@ -274,7 +740,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -285,13 +751,13 @@ class MergeTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
node.safe_psql(
@@ -369,7 +835,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -381,13 +847,13 @@ class MergeTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
node.safe_psql(
@@ -465,7 +931,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -478,7 +944,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -517,7 +983,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -540,7 +1006,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
'postgresql.auto.conf', 'port = {0}'.format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -553,7 +1019,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica'
@@ -563,7 +1029,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL backup
self.backup_node(backup_dir, 'node', node)
@@ -631,7 +1097,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica'
@@ -641,7 +1107,7 @@ class MergeTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL backup
self.backup_node(backup_dir, 'node', node)
@@ -698,7 +1164,8 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# read block from future
# block_size + backup_header = 8200
file = os.path.join(
backup_dir, 'backups/node', backup_id_2, 'database', new_path)
backup_dir, 'backups', 'node',
backup_id_2, 'database', new_path)
with open(file, 'rb') as f:
f.seek(8200)
block_1 = f.read(8200)
@@ -706,7 +1173,8 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# write block from future
file = os.path.join(
backup_dir, 'backups/node', backup_id, 'database', old_path)
backup_dir, 'backups', 'node',
backup_id, 'database', old_path)
with open(file, 'r+b') as f:
f.seek(8200)
f.write(block_1)
@@ -722,8 +1190,6 @@ class MergeTest(ProbackupTest, unittest.TestCase):
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertTrue(
"WARNING: Backup {0} data files are corrupted".format(
backup_id) in e.message and
"ERROR: Merging of backup {0} failed".format(
backup_id) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
@@ -732,6 +1198,93 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
def test_continue_failed_merge_2(self):
"""
Check that failed MERGE on delete can`t be continued
"""
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.slow_start()
# FULL backup
self.backup_node(backup_dir, 'node', node)
node.safe_psql(
"postgres",
"create table t_heap as select i as id,"
" md5(i::text) as text, md5(i::text)::tsvector as tsvector"
" from generate_series(0,1000) i"
)
# DELTA BACKUP
self.backup_node(
backup_dir, 'node', node, backup_type='delta'
)
node.safe_psql(
"postgres",
"delete from t_heap"
)
node.safe_psql(
"postgres",
"vacuum t_heap"
)
# DELTA BACKUP
self.backup_node(
backup_dir, 'node', node, backup_type='delta'
)
if self.paranoia:
pgdata = self.pgdata_content(node.data_dir)
backup_id = self.show_pb(backup_dir, "node")[2]["id"]
gdb = self.merge_backup(backup_dir, "node", backup_id, gdb=True)
gdb.set_breakpoint('pgFileDelete')
gdb.run_until_break()
if gdb.continue_execution_until_break(20) != 'breakpoint-hit':
print('Failed to hit breakpoint')
exit(1)
gdb._execute('signal SIGKILL')
print(self.show_pb(backup_dir, as_text=True, as_json=False))
backup_id_deleted = self.show_pb(backup_dir, "node")[1]["id"]
# Try to continue failed MERGE
try:
self.merge_backup(backup_dir, "node", backup_id)
self.assertEqual(
1, 0,
"Expecting Error because of backup corruption.\n "
"Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertTrue(
"ERROR: Backup {0} has status: DELETING".format(
backup_id_deleted) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(e.message), self.cmd))
# Clean after yourself
self.del_test_dir(module_name, fname)
# 1. always use parent link when merging (intermediates may be from different chain)
# 2. page backup we are merging with may disappear after failed merge,
# it should not be possible to continue merge after that
@@ -744,3 +1297,4 @@ class MergeTest(ProbackupTest, unittest.TestCase):
# FULL MERGING
# 3. Need new test with corrupted FULL backup
# 4. different compression levels
tests/option_test.py
+2 -2
View File
@@ -52,7 +52,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node'))
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
@@ -118,7 +118,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split(".")[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
pg_options={
'wal_level': 'logical',
'max_wal_senders': '2'})
tests/page.py
+153 -36
View File
@@ -1,6 +1,7 @@
import os
import unittest
from .helpers.ptrack_helpers import ProbackupTest, ProbackupException
from testgres import QueryException
from datetime import datetime, timedelta
import subprocess
import gzip
@@ -22,7 +23,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -33,13 +34,13 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node_restored.cleanup()
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
node.safe_psql(
@@ -115,7 +116,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -127,7 +128,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -204,7 +205,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -216,7 +217,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL BACKUP
node.safe_psql(
@@ -299,7 +300,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -316,7 +317,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -340,7 +341,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# RESTORE NODE
restored_node = self.make_simple_node(
base_dir="{0}/{1}/restored_node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'restored_node'))
restored_node.cleanup()
tblspc_path = self.get_tblspace_path(node, 'somedata')
tblspc_path_new = self.get_tblspace_path(
@@ -383,7 +384,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -396,7 +397,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
# FULL backup
@@ -426,7 +427,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -448,7 +449,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
'postgresql.auto.conf', 'port = {0}'.format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -463,7 +464,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -476,7 +477,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -507,7 +508,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -529,7 +530,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# START RESTORED NODE
node_restored.append_conf(
'postgresql.auto.conf', 'port = {0}'.format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Clean after yourself
self.del_test_dir(module_name, fname)
@@ -543,21 +544,21 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={
"hot_standby": "on"
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node_restored.cleanup()
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# Do full backup
self.backup_node(backup_dir, 'node', node)
@@ -597,7 +598,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.start()
node_restored.slow_start()
# Check restored node
count2 = node_restored.execute("postgres", "select count(*) from test")
@@ -618,7 +619,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# Initialize instance and backup directory
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={}
)
@@ -626,7 +627,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# Do full backup
self.backup_node(backup_dir, 'node', node)
@@ -655,7 +656,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# Drop node and restore it
node.cleanup()
self.restore_node(backup_dir, 'node', node)
node.start()
node.slow_start()
# Clean after yourself
node.cleanup()
@@ -672,7 +673,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -680,7 +681,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
@@ -762,7 +763,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -770,7 +771,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
@@ -883,23 +884,22 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
alien_node = self.make_simple_node(
base_dir="{0}/{1}/alien_node".format(module_name, fname)
)
base_dir=os.path.join(module_name, fname, 'alien_node'))
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.add_instance(backup_dir, 'alien_node', alien_node)
self.set_archiving(backup_dir, 'alien_node', alien_node)
alien_node.start()
alien_node.slow_start()
self.backup_node(backup_dir, 'node', node)
self.backup_node(backup_dir, 'alien_node', alien_node)
@@ -1003,7 +1003,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1012,7 +1012,7 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
@@ -1030,3 +1030,120 @@ class PageBackupTest(ProbackupTest, unittest.TestCase):
# Clean after yourself
self.del_test_dir(module_name, fname)
# @unittest.skip("skip")
def test_page_create_db(self):
"""
Make node, take full backup, create database db1, take page backup,
restore database and check it presense
"""
self.maxDiff = None
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
'max_wal_size': '10GB',
'max_wal_senders': '2',
'checkpoint_timeout': '5min',
'autovacuum': 'off'
}
)
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.slow_start()
# FULL BACKUP
node.safe_psql(
"postgres",
"create table t_heap as select i as id, md5(i::text) as text, "
"md5(i::text)::tsvector as tsvector from generate_series(0,100) i")
self.backup_node(
backup_dir, 'node', node)
# CREATE DATABASE DB1
node.safe_psql("postgres", "create database db1")
node.safe_psql(
"db1",
"create table t_heap as select i as id, md5(i::text) as text, "
"md5(i::text)::tsvector as tsvector from generate_series(0,1000) i")
# PAGE BACKUP
backup_id = self.backup_node(backup_dir, 'node', node, backup_type='page')
if self.paranoia:
pgdata = self.pgdata_content(node.data_dir)
# RESTORE
node_restored = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
self.restore_node(
backup_dir, 'node', node_restored,
backup_id=backup_id, options=["-j", "4"])
# COMPARE PHYSICAL CONTENT
if self.paranoia:
pgdata_restored = self.pgdata_content(node_restored.data_dir)
self.compare_pgdata(pgdata, pgdata_restored)
# START RESTORED NODE
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.slow_start()
node_restored.safe_psql('db1', 'select 1')
node_restored.cleanup()
# DROP DATABASE DB1
node.safe_psql(
"postgres", "drop database db1")
# SECOND PTRACK BACKUP
backup_id = self.backup_node(
backup_dir, 'node', node, backup_type='page')
if self.paranoia:
pgdata = self.pgdata_content(node.data_dir)
# RESTORE SECOND PTRACK BACKUP
self.restore_node(
backup_dir, 'node', node_restored,
backup_id=backup_id, options=["-j", "4"]
)
# COMPARE PHYSICAL CONTENT
if self.paranoia:
pgdata_restored = self.pgdata_content(
node_restored.data_dir, ignore_ptrack=False)
self.compare_pgdata(pgdata, pgdata_restored)
# START RESTORED NODE
node_restored.append_conf(
"postgresql.auto.conf", "port = {0}".format(node_restored.port))
node_restored.slow_start()
try:
node_restored.safe_psql('db1', 'select 1')
# we should die here because exception is what we expect to happen
self.assertEqual(
1, 0,
"Expecting Error because we are connecting to deleted database"
"\n Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd)
)
except QueryException as e:
self.assertTrue(
'FATAL: database "db1" does not exist' in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(e.message), self.cmd)
)
# Clean after yourself
self.del_test_dir(module_name, fname)
tests/pgpro560.py
+15 -12
View File
@@ -20,15 +20,16 @@ class CheckSystemID(ProbackupTest, unittest.TestCase):
check that backup failed
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
pg_options={'wal_level': 'replica'})
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
file = os.path.join(node.base_dir,'data', 'global', 'pg_control')
os.remove(file)
@@ -55,18 +56,20 @@ class CheckSystemID(ProbackupTest, unittest.TestCase):
check that backup failed
"""
fname = self.id().split('.')[3]
node1 = self.make_simple_node(base_dir="{0}/{1}/node1".format(module_name, fname),
node1 = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node1'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
node1.start()
node2 = self.make_simple_node(base_dir="{0}/{1}/node2".format(module_name, fname),
pg_options={'wal_level': 'replica'})
node1.slow_start()
node2 = self.make_simple_node(
base_dir=os.path.join(module_name, fname, 'node2'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
node2.start()
pg_options={'wal_level': 'replica'})
node2.slow_start()
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
tests/pgpro589.py
+2 -2
View File
@@ -19,7 +19,7 @@ class ArchiveCheck(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -30,7 +30,7 @@ class ArchiveCheck(ProbackupTest, unittest.TestCase):
# make erroneus archive_command
node.append_conf("postgresql.auto.conf", "archive_command = 'exit 0'")
node.start()
node.slow_start()
node.pgbench_init(scale=5)
pgbench = node.pgbench(
tests/ptrack.py
+35 -33
View File
@@ -21,7 +21,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -70,7 +70,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -89,11 +89,13 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# DISABLE PTRACK
node.safe_psql('postgres', "alter system set ptrack_enable to off")
node.restart()
node.stop()
node.slow_start()
# ENABLE PTRACK
node.safe_psql('postgres', "alter system set ptrack_enable to on")
node.restart()
node.stop()
node.slow_start()
# PTRACK BACKUP
try:
@@ -128,7 +130,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -139,7 +141,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
@@ -190,7 +192,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -201,7 +203,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
@@ -286,7 +288,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -298,7 +300,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
@@ -372,7 +374,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -383,7 +385,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
}
)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
)
self.init_pb(backup_dir)
@@ -448,7 +450,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -520,7 +522,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -616,7 +618,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -722,7 +724,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -806,7 +808,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -878,7 +880,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -955,7 +957,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -1003,7 +1005,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -1082,7 +1084,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -1106,7 +1108,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
"md5(i::text)::tsvector as tsvector from generate_series(0,100) i")
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.backup_node(
@@ -1159,7 +1161,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -1183,7 +1185,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True, initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -1232,7 +1234,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname)
base_dir=os.path.join(module_name, fname, 'node_restored')
)
node_restored.cleanup()
@@ -1280,7 +1282,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -1317,7 +1319,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# RESTORE
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
self.restore_node(
backup_dir, 'node',
@@ -1351,7 +1353,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -1435,7 +1437,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -1478,7 +1480,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# Restore ptrack backup
restored_node = self.make_simple_node(
base_dir="{0}/{1}/restored_node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'restored_node'))
restored_node.cleanup()
tblspc_path_new = self.get_tblspace_path(
restored_node, 'somedata_restored')
@@ -1548,7 +1550,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -1613,7 +1615,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
# RESTORE NODE
restored_node = self.make_simple_node(
base_dir="{0}/{1}/restored_node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'restored_node'))
restored_node.cleanup()
tblspc_path = self.get_tblspace_path(node, 'somedata')
tblspc_path_new = self.get_tblspace_path(
@@ -1655,7 +1657,7 @@ class PtrackTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
tests/ptrack_clean.py
+6 -6
View File
@@ -15,7 +15,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
"""Take backups of every available types and check that PTRACK is clean"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -26,7 +26,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -124,7 +124,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
"""Take backups of every available types from master and check that PTRACK on replica is clean"""
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -136,12 +136,12 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'master', master)
master.start()
master.slow_start()
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
@@ -149,7 +149,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.add_instance(backup_dir, 'replica', replica)
self.set_replica(master, replica, synchronous=True)
self.set_archiving(backup_dir, 'replica', replica, replica=True)
replica.start()
replica.slow_start(replica=True)
# Create table and indexes
master.safe_psql(
tests/ptrack_cluster.py
+12 -12
View File
@@ -15,7 +15,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_cluster_on_btree(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -26,7 +26,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -96,7 +96,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_cluster_on_gist(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -107,7 +107,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
# Create table and indexes
node.safe_psql(
@@ -174,7 +174,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_cluster_on_btree_replica(self):
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -185,12 +185,12 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'master', master)
master.start()
master.slow_start()
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
@@ -198,7 +198,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.add_instance(backup_dir, 'replica', replica)
self.set_replica(master, replica, synchronous=True)
self.set_archiving(backup_dir, 'replica', replica, replica=True)
replica.start()
replica.slow_start(replica=True)
# Create table and indexes
master.safe_psql(
@@ -274,7 +274,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_cluster_on_gist_replica(self):
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -285,12 +285,12 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'master', master)
master.start()
master.slow_start()
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
@@ -298,7 +298,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.add_instance(backup_dir, 'replica', replica)
self.set_replica(master, replica, 'replica', synchronous=True)
self.set_archiving(backup_dir, 'replica', replica, replica=True)
replica.start()
replica.slow_start(replica=True)
# Create table and indexes
master.safe_psql(
tests/ptrack_empty.py
+8 -8
View File
@@ -15,7 +15,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
"""Take backups of every available types and check that PTRACK is clean"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -27,7 +27,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -58,7 +58,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
node.safe_psql('postgres', 'checkpoint')
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
tblspace1 = self.get_tblspace_path(node, 'somedata')
@@ -93,7 +93,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
"""Take backups of every available types from master and check that PTRACK on replica is clean"""
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -103,12 +103,12 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'master', master)
master.start()
master.slow_start()
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
@@ -116,7 +116,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.add_instance(backup_dir, 'replica', replica)
self.set_replica(master, replica, synchronous=True)
self.set_archiving(backup_dir, 'replica', replica, replica=True)
replica.start()
replica.slow_start(replica=True)
# Create table
master.safe_psql(
@@ -151,7 +151,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.wait_until_replica_catch_with_master(master, replica)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node_restored'))
node_restored.cleanup()
# Take PTRACK backup
tests/ptrack_move_to_tablespace.py
+2 -2
View File
@@ -13,7 +13,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_recovery(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -24,7 +24,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
tests/ptrack_recovery.py
+3 -3
View File
@@ -14,7 +14,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_recovery(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -25,7 +25,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
self.create_tblspace_in_node(node, 'somedata')
@@ -56,7 +56,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
print('Killing postmaster. Losing Ptrack changes')
node.stop(['-m', 'immediate', '-D', node.data_dir])
if not node.status():
node.start()
node.slow_start()
else:
print("Die! Die! Why won't you die?... Why won't you die?")
exit(1)
tests/ptrack_truncate.py
+3 -3
View File
@@ -13,7 +13,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_truncate(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -74,7 +74,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_truncate_replica(self):
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -91,7 +91,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
tests/ptrack_vacuum.py
+3 -3
View File
@@ -13,7 +13,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -100,7 +100,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_replica(self):
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -117,7 +117,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
tests/ptrack_vacuum_bits_frozen.py
+3 -3
View File
@@ -13,7 +13,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_bits_frozen(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -93,7 +93,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_bits_frozen_replica(self):
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -109,7 +109,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
tests/ptrack_vacuum_bits_visibility.py
+1 -1
View File
@@ -13,7 +13,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_bits_visibility(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
tests/ptrack_vacuum_full.py
+3 -3
View File
@@ -14,7 +14,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_full(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -95,7 +95,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_full_replica(self):
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -112,7 +112,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
tests/ptrack_vacuum_truncate.py
+3 -3
View File
@@ -13,7 +13,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_truncate(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -94,7 +94,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
def test_ptrack_vacuum_truncate_replica(self):
fname = self.id().split('.')[3]
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -111,7 +111,7 @@ class SimpleTest(ProbackupTest, unittest.TestCase):
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
tests/replica.py
+13 -13
View File
@@ -3,7 +3,6 @@ import unittest
from .helpers.ptrack_helpers import ProbackupTest, ProbackupException, idx_ptrack
from datetime import datetime, timedelta
import subprocess
from sys import exit
import time
@@ -22,7 +21,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -30,7 +29,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
'max_wal_senders': '2',
'ptrack_enable': 'on'}
)
master.start()
master.slow_start()
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'master', master)
@@ -45,7 +44,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
# take full backup and restore it
self.backup_node(backup_dir, 'master', master, options=['--stream'])
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.restore_node(backup_dir, 'master', replica)
self.set_replica(master, replica)
@@ -79,7 +78,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
# RESTORE FULL BACKUP TAKEN FROM PREVIOUS STEP
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node'))
node.cleanup()
self.restore_node(backup_dir, 'replica', data_dir=node.data_dir)
@@ -138,7 +137,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -152,7 +151,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
master.slow_start()
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.backup_node(backup_dir, 'master', master)
@@ -211,7 +210,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
# RESTORE FULL BACKUP TAKEN FROM replica
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node'))
node.cleanup()
self.restore_node(backup_dir, 'replica', data_dir=node.data_dir)
@@ -289,7 +288,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -305,7 +304,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
master.slow_start()
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.backup_node(backup_dir, 'master', master)
@@ -350,7 +349,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
master = self.make_simple_node(
base_dir="{0}/{1}/master".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'master'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -364,7 +363,7 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
master.slow_start()
replica = self.make_simple_node(
base_dir="{0}/{1}/replica".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'replica'))
replica.cleanup()
self.backup_node(backup_dir, 'master', master)
@@ -402,7 +401,8 @@ class ReplicaTest(ProbackupTest, unittest.TestCase):
replica.append_conf(
'recovery.conf', "recovery_min_apply_delay = '300s'")
replica.restart()
replica.stop()
replica.slow_start(replica=True)
master.pgbench_init(scale=10)
tests/restore_test.py
+53 -52
View File
@@ -18,7 +18,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to latest from full backup"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -27,7 +27,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
pgbench = node.pgbench(
@@ -66,7 +66,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to latest from full + page backups"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -74,7 +74,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
@@ -114,7 +114,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to target timeline"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -122,7 +122,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
@@ -183,7 +183,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to target time"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -192,7 +192,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.append_conf("postgresql.auto.conf", "TimeZone = Europe/Moscow")
node.start()
node.slow_start()
node.pgbench_init(scale=2)
before = node.execute("postgres", "SELECT * FROM pgbench_branches")
@@ -232,7 +232,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to target xid"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -240,7 +240,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
with node.connect("postgres") as con:
@@ -293,7 +293,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery with target inclusive false"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -304,7 +304,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
with node.connect("postgres") as con:
@@ -358,7 +358,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to target lsn"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -371,7 +371,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
with node.connect("postgres") as con:
@@ -431,7 +431,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to target lsn"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -444,7 +444,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
with node.connect("postgres") as con:
@@ -505,7 +505,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to latest from archive full+ptrack backups"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'ptrack_enable': 'on'}
)
@@ -513,7 +513,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
@@ -553,7 +553,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery to latest from archive full+ptrack+ptrack backups"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'ptrack_enable': 'on'}
)
@@ -561,7 +561,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
@@ -608,7 +608,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery in stream mode to latest from full + ptrack backups"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -620,7 +620,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
@@ -663,7 +663,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -675,7 +675,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=2)
@@ -729,7 +729,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -741,7 +741,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# wal_segment_size = self.guc_wal_segment_size(node)
node.pgbench_init(scale=2)
@@ -794,7 +794,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery using tablespace-mapping option"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -805,7 +805,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# Create tablespace
tblspc_path = os.path.join(node.base_dir, "tblspc")
@@ -832,15 +832,17 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertEqual(
self.assertIn(
'ERROR: restore destination is not empty: "{0}"'.format(node.data_dir),
e.message,
'ERROR: restore destination is not empty: "{0}"\n'.format(
node.data_dir),
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(e.message), self.cmd))
# 2 - Try to restore to existing tablespace directory
tblspc_path_tmp = os.path.join(node.base_dir, "tblspc_tmp")
os.rename(tblspc_path, tblspc_path_tmp)
node.cleanup()
os.rename(tblspc_path_tmp, tblspc_path)
try:
self.restore_node(backup_dir, 'node', node)
# we should die here because exception is what we expect to happen
@@ -850,10 +852,9 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"not empty.\n Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertEqual(
self.assertIn(
'ERROR: restore tablespace destination is not empty:',
e.message,
'ERROR: restore tablespace destination '
'is not empty: "{0}"\n'.format(tblspc_path),
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(e.message), self.cmd))
@@ -913,7 +914,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""recovery using tablespace-mapping option and page backup"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -921,7 +922,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# Full backup
self.backup_node(backup_dir, 'node', node)
@@ -995,7 +996,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1004,7 +1005,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(
backup_dir, 'node', node, options=["--stream"])
@@ -1045,7 +1046,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1054,7 +1055,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(
backup_dir, 'node', node, options=["--stream"])
@@ -1094,7 +1095,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1103,7 +1104,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(
backup_dir, 'node', node, options=["--stream"])
@@ -1143,7 +1144,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1151,7 +1152,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
if self.paranoia:
@@ -1198,7 +1199,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1206,7 +1207,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
@@ -1247,7 +1248,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
def test_zags_block_corrupt(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1255,7 +1256,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
@@ -1303,7 +1304,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1325,7 +1326,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
def test_zags_block_corrupt_1(self):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={
'wal_level': 'replica',
@@ -1336,7 +1337,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
@@ -1395,7 +1396,7 @@ class RestoreTest(ProbackupTest, unittest.TestCase):
self.switch_wal_segment(node)
node_restored = self.make_simple_node(
base_dir="{0}/{1}/node_restored".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node_restored'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
tests/retention_test.py
+6 -6
View File
@@ -15,7 +15,7 @@ class RetentionTest(ProbackupTest, unittest.TestCase):
"""purge backups using redundancy-based retention policy"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -23,7 +23,7 @@ class RetentionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
with open(os.path.join(
backup_dir, 'backups', 'node',
@@ -72,7 +72,7 @@ class RetentionTest(ProbackupTest, unittest.TestCase):
"""purge backups using window-based retention policy"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -80,7 +80,7 @@ class RetentionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
with open(
os.path.join(
@@ -126,7 +126,7 @@ class RetentionTest(ProbackupTest, unittest.TestCase):
"""purge backups using window-based retention policy"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -134,7 +134,7 @@ class RetentionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
tests/show_test.py
+12 -12
View File
@@ -15,7 +15,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -23,7 +23,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.assertEqual(
self.backup_node(
@@ -43,7 +43,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -51,7 +51,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.assertEqual(
self.backup_node(
@@ -71,7 +71,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -79,7 +79,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
@@ -117,7 +117,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -125,7 +125,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
@@ -146,7 +146,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -154,7 +154,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
@@ -177,7 +177,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -185,7 +185,7 @@ class OptionTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
tests/validate_test.py
+121 -92
View File
@@ -21,7 +21,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -56,7 +56,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
with open(log_file_path) as f:
self.assertTrue(
'LOG: File: {0} blknum 1, empty page'.format(file) in f.read(),
'{0} blknum 1, empty page'.format(file_path) in f.read(),
'Failed to detect nullified block')
self.validate_pb(backup_dir)
@@ -73,7 +73,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -117,9 +117,9 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertEqual(
self.assertIn(
'ERROR: Backup satisfying target options is not found',
e.message,
'ERROR: Backup satisfying target options is not found.\n',
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
repr(e.message), self.cmd))
@@ -213,7 +213,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -221,7 +221,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL
backup_id_1 = self.backup_node(backup_dir, 'node', node)
@@ -249,7 +249,8 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# Corrupt some file
file = os.path.join(
backup_dir, 'backups/node', backup_id_2, 'database', file_path)
backup_dir, 'backups', 'node',
backup_id_2, 'database', file_path)
with open(file, "r+b", 0) as f:
f.seek(42)
f.write(b"blah")
@@ -298,7 +299,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -306,7 +307,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
@@ -342,7 +343,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# Corrupt some file in FULL backup
file_full = os.path.join(
backup_dir, 'backups/node',
backup_dir, 'backups', 'node',
backup_id_1, 'database', file_path_t_heap)
with open(file_full, "rb+", 0) as f:
f.seek(84)
@@ -352,7 +353,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# Corrupt some file in PAGE1 backup
file_page1 = os.path.join(
backup_dir, 'backups/node',
backup_dir, 'backups', 'node',
backup_id_2, 'database', file_path_t_heap_1)
with open(file_page1, "rb+", 0) as f:
f.seek(42)
@@ -379,8 +380,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.assertTrue(
'INFO: Validating backup {0}'.format(
backup_id_1) in e.message and
'WARNING: Invalid CRC of backup file "{0}"'.format(
file_full) in e.message and
'WARNING: Invalid CRC of backup file' in e.message and
'WARNING: Backup {0} data files are corrupted'.format(
backup_id_1) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
@@ -421,7 +421,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -429,7 +429,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL1
backup_id_1 = self.backup_node(backup_dir, 'node', node)
@@ -499,7 +499,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# Corrupt some file in PAGE2 and PAGE5 backups
file_page1 = os.path.join(
backup_dir, 'backups/node', backup_id_3, 'database', file_page_2)
backup_dir, 'backups', 'node', backup_id_3, 'database', file_page_2)
with open(file_page1, "rb+", 0) as f:
f.seek(84)
f.write(b"blah")
@@ -507,7 +507,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
f.close
file_page4 = os.path.join(
backup_dir, 'backups/node', backup_id_6, 'database', file_page_5)
backup_dir, 'backups', 'node', backup_id_6, 'database', file_page_5)
with open(file_page4, "rb+", 0) as f:
f.seek(42)
f.write(b"blah")
@@ -547,8 +547,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.assertTrue(
'INFO: Validating backup {0}'.format(
backup_id_3) in e.message and
'WARNING: Invalid CRC of backup file "{0}"'.format(
file_page1) in e.message and
'WARNING: Invalid CRC of backup file' in e.message and
'WARNING: Backup {0} data files are corrupted'.format(
backup_id_3) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
@@ -620,7 +619,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -628,7 +627,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# FULL1
backup_id_1 = self.backup_node(backup_dir, 'node', node)
@@ -698,7 +697,8 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# Corrupt some file in PAGE2 and PAGE5 backups
file_page1 = os.path.join(
backup_dir, 'backups/node', backup_id_3, 'database', file_page_2)
backup_dir, 'backups', 'node',
backup_id_3, 'database', file_page_2)
with open(file_page1, "rb+", 0) as f:
f.seek(84)
f.write(b"blah")
@@ -706,7 +706,8 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
f.close
file_page4 = os.path.join(
backup_dir, 'backups/node', backup_id_6, 'database', file_page_5)
backup_dir, 'backups', 'node',
backup_id_6, 'database', file_page_5)
with open(file_page4, "rb+", 0) as f:
f.seek(42)
f.write(b"blah")
@@ -747,8 +748,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.assertTrue(
'INFO: Validating backup {0}'.format(
backup_id_3) in e.message and
'WARNING: Invalid CRC of backup file "{0}"'.format(
file_page1) in e.message and
'WARNING: Invalid CRC of backup file' in e.message and
'WARNING: Backup {0} data files are corrupted'.format(
backup_id_3) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
@@ -804,7 +804,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -812,7 +812,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
@@ -851,7 +851,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# Corrupt some file in FULL backup
file_full = os.path.join(
backup_dir, 'backups/node', backup_id_2,
backup_dir, 'backups', 'node', backup_id_2,
'database', file_path_t_heap1)
with open(file_full, "rb+", 0) as f:
f.seek(84)
@@ -906,8 +906,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.assertTrue(
'INFO: Validating backup {0}'.format(
backup_id_2) in e.message and
'WARNING: Invalid CRC of backup file "{0}"'.format(
file_full) in e.message and
'WARNING: Invalid CRC of backup file' in e.message and
'WARNING: Backup {0} data files are corrupted'.format(
backup_id_2) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(
@@ -952,7 +951,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
expect FULL to gain status CORRUPT, PAGE1 and PAGE2 to gain status ORPHAN,
try to restore backup with --no-validation option"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -960,11 +959,13 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
"create table t_heap as select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(0,10000) i")
"create table t_heap as select i as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,10000) i")
file_path_t_heap = node.safe_psql(
"postgres",
"select pg_relation_filepath('t_heap')").rstrip()
@@ -973,14 +974,18 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
node.safe_psql(
"postgres",
"insert into t_heap select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(0,10000) i")
"insert into t_heap select i as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,10000) i")
# PAGE1
backup_id_2 = self.backup_node(backup_dir, 'node', node, backup_type='page')
# PAGE2
node.safe_psql(
"postgres",
"insert into t_heap select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(20000,30000) i")
"insert into t_heap select i as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(20000,30000) i")
backup_id_3 = self.backup_node(backup_dir, 'node', node, backup_type='page')
# FULL1
@@ -989,11 +994,15 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# PAGE3
node.safe_psql(
"postgres",
"insert into t_heap select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(30000,40000) i")
"insert into t_heap select i as id, "
"md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(30000,40000) i")
backup_id_5 = self.backup_node(backup_dir, 'node', node, backup_type='page')
# Corrupt some file in FULL backup
file_full = os.path.join(backup_dir, 'backups/node', backup_id_1, 'database', file_path_t_heap)
file_full = os.path.join(
backup_dir, 'backups', 'node',
backup_id_1, 'database', file_path_t_heap)
with open(file_full, "rb+", 0) as f:
f.seek(84)
f.write(b"blah")
@@ -1009,7 +1018,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.assertTrue(
'INFO: Validating backup {0}'.format(backup_id_1) in e.message
and "INFO: Validate backups of the instance 'node'" in e.message
and 'WARNING: Invalid CRC of backup file "{0}"'.format(file_full) in e.message
and 'WARNING: Invalid CRC of backup file' in e.message
and 'WARNING: Backup {0} data files are corrupted'.format(backup_id_1) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(repr(e.message), self.cmd))
@@ -1038,7 +1047,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
corrupt file in FULL backup and run validate on instance,
expect FULL to gain status CORRUPT, PAGE1 and PAGE2 to gain status ORPHAN"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1046,11 +1055,14 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
"create table t_heap as select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(0,10000) i")
"create table t_heap as select i as id, "
"md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,10000) i")
file_path_t_heap = node.safe_psql(
"postgres",
"select pg_relation_filepath('t_heap')").rstrip()
@@ -1059,27 +1071,40 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
node.safe_psql(
"postgres",
"insert into t_heap select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(0,10000) i")
"insert into t_heap select i as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(0,10000) i")
# PAGE1
backup_id_2 = self.backup_node(backup_dir, 'node', node, backup_type='page')
backup_id_2 = self.backup_node(
backup_dir, 'node', node, backup_type='page')
# PAGE2
node.safe_psql(
"postgres",
"insert into t_heap select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(20000,30000) i")
backup_id_3 = self.backup_node(backup_dir, 'node', node, backup_type='page')
"insert into t_heap select i as id, md5(i::text) as text, "
"md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(20000,30000) i")
backup_id_3 = self.backup_node(
backup_dir, 'node', node, backup_type='page')
# FULL1
backup_id_4 = self.backup_node(backup_dir, 'node', node)
backup_id_4 = self.backup_node(
backup_dir, 'node', node)
# PAGE3
node.safe_psql(
"postgres",
"insert into t_heap select i as id, md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector from generate_series(30000,40000) i")
"insert into t_heap select i as id, "
"md5(i::text) as text, md5(repeat(i::text,10))::tsvector as tsvector "
"from generate_series(30000,40000) i")
backup_id_5 = self.backup_node(backup_dir, 'node', node, backup_type='page')
# Corrupt some file in FULL backup
file_full = os.path.join(backup_dir, 'backups/node', backup_id_1, 'database', file_path_t_heap)
file_full = os.path.join(
backup_dir, 'backups', 'node',
backup_id_1, 'database', file_path_t_heap)
with open(file_full, "rb+", 0) as f:
f.seek(84)
f.write(b"blah")
@@ -1089,13 +1114,16 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
# Validate Instance
try:
self.validate_pb(backup_dir, 'node')
self.assertEqual(1, 0, "Expecting Error because of data files corruption.\n Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
self.assertEqual(
1, 0,
"Expecting Error because of data files corruption.\n "
"Output: {0} \n CMD: {1}".format(
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertTrue(
'INFO: Validating backup {0}'.format(backup_id_1) in e.message
and "INFO: Validate backups of the instance 'node'" in e.message
and 'WARNING: Invalid CRC of backup file "{0}"'.format(file_full) in e.message
and 'WARNING: Invalid CRC of backup file' in e.message
and 'WARNING: Backup {0} data files are corrupted'.format(backup_id_1) in e.message,
'\n Unexpected Error Message: {0}\n CMD: {1}'.format(repr(e.message), self.cmd))
@@ -1113,7 +1141,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""make archive node, take FULL1, PAGE1,PAGE2,FULL2,PAGE3,PAGE4 backups, corrupt all wal files, run validate, expect errors"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1121,7 +1149,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id_1 = self.backup_node(backup_dir, 'node', node)
@@ -1175,7 +1203,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
def test_validate_corrupt_wal_2(self):
"""make archive node, make full backup, corrupt all wal files, run validate to real xid, expect errors"""
fname = self.id().split('.')[3]
node = self.make_simple_node(base_dir="{0}/{1}/node".format(module_name, fname),
node = self.make_simple_node(base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1183,7 +1211,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
with node.connect("postgres") as con:
con.execute("CREATE TABLE tbl0005 (a text)")
@@ -1247,7 +1275,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1255,7 +1283,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.pgbench_init(scale=3)
@@ -1281,8 +1309,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
repr(self.output), self.cmd))
except ProbackupException as e:
self.assertTrue(
"WAL segment \"{0}\" is absent".format(
file) in e.message and
"is absent" in e.message and
"WARNING: There are not enough WAL records to consistenly "
"restore backup {0}".format(backup_id) in e.message and
"WARNING: Backup {0} WAL segments are corrupted".format(
@@ -1325,7 +1352,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -1333,7 +1360,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
@@ -1419,7 +1446,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1427,7 +1454,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(
backup_dir, 'node', node, options=["--stream"])
@@ -1461,7 +1488,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1470,7 +1497,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
recovery_time = self.show_pb(
@@ -1491,7 +1518,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node1 = self.make_simple_node(
base_dir="{0}/{1}/node1".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node1'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1500,13 +1527,13 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node1', node1)
self.set_archiving(backup_dir, 'node1', node1)
node1.start()
node1.slow_start()
backup_id = self.backup_node(
backup_dir, 'node1', node1, options=["--stream"])
node2 = self.make_simple_node(
base_dir="{0}/{1}/node2".format(module_name, fname))
base_dir=os.path.join(module_name, fname, 'node2'))
node2.cleanup()
node1.psql(
@@ -1528,7 +1555,8 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
node2.append_conf(
'postgresql.auto.conf', 'archive_mode = on')
node2.restart()
node2.stop()
node2.slow_start()
timeline_node1 = node1.get_control_data()["Latest checkpoint's TimeLineID"]
timeline_node2 = node2.get_control_data()["Latest checkpoint's TimeLineID"]
@@ -1582,7 +1610,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={
@@ -1594,7 +1622,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
self.backup_node(backup_dir, 'node', node, backup_type='page')
@@ -1708,7 +1736,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1717,7 +1745,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
self.backup_node(backup_dir, 'node', node, backup_type='page')
@@ -1820,7 +1848,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -1829,7 +1857,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
self.backup_node(backup_dir, 'node', node, backup_type='page')
@@ -2170,7 +2198,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -2179,7 +2207,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
self.backup_node(backup_dir, 'node', node)
self.backup_node(backup_dir, 'node', node, backup_type='page')
@@ -2394,7 +2422,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
# initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica'}
)
@@ -2405,7 +2433,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
node.safe_psql(
"postgres",
@@ -2425,7 +2453,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
backup_id = self.backup_node(
backup_dir, 'node', node, backup_type="full",
options=["-j", "4"], async=False, gdb=False)
options=["-j", "4"], asynchronous=False, gdb=False)
node.stop()
node.cleanup()
@@ -2472,7 +2500,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -2481,7 +2509,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# CHAIN1
self.backup_node(backup_dir, 'node', node)
@@ -2614,7 +2642,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -2623,7 +2651,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# CHAIN1
self.backup_node(backup_dir, 'node', node)
@@ -2735,7 +2763,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -2744,7 +2772,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# CHAIN1
self.backup_node(backup_dir, 'node', node)
@@ -2923,7 +2951,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
"""
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -2932,7 +2960,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
# CHAIN1
self.backup_node(backup_dir, 'node', node)
@@ -3071,7 +3099,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
""" PGPRO-2096 """
fname = self.id().split('.')[3]
node = self.make_simple_node(
base_dir="{0}/{1}/node".format(module_name, fname),
base_dir=os.path.join(module_name, fname, 'node'),
set_replication=True,
initdb_params=['--data-checksums'],
pg_options={'wal_level': 'replica', 'max_wal_senders': '2'}
@@ -3080,7 +3108,7 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.init_pb(backup_dir)
self.add_instance(backup_dir, 'node', node)
self.set_archiving(backup_dir, 'node', node)
node.start()
node.slow_start()
backup_id = self.backup_node(backup_dir, 'node', node)
@@ -3105,11 +3133,12 @@ class ValidateTest(ProbackupTest, unittest.TestCase):
self.run_binary(
[
pg_resetxlog_path,
'-D',
os.path.join(backup_dir, 'backups', 'node', backup_id, 'database'),
'-o 42',
'-f'
],
async=False)
asynchronous=False)
md5_after = hashlib.md5(
open(pg_control_path, 'rb').read()).hexdigest()