db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Code Issues Releases Activity
86f71349ef
pgbackrest/test/lib/pgBackRestTest/Module/Real/RealAllTest.pm

1025 lines
52 KiB
Perl
Raw Normal View History

Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
Split full module into mock and real to allow better test combinations and save time in CI.
2017-06-20 00:55:00 +02:00
# Test All Commands On PostgreSQL Clusters
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
Split full module into mock and real to allow better test combinations and save time in CI.
2017-06-20 00:55:00 +02:00
package pgBackRestTest::Module::Real::RealAllTest;
Moved test and env modules to new directories to avoid namespace conflicts with common tests.
2017-05-12 22:43:04 +02:00
use parent 'pgBackRestTest::Env::HostEnvTest';
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
# Perl includes
####################################################################################################################################
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
use File::Basename qw(dirname);
Rename BackRestDoc Perl module to pgBackRestDoc. This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523. More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 21:41:56 +02:00
use pgBackRestDoc::Common::Exception;
use pgBackRestDoc::Common::Ini;
use pgBackRestDoc::Common::Log;
Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo. The primary source for project info is now src/version.h. The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 23:57:02 +02:00
use pgBackRestDoc::ProjectInfo;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
use pgBackRestTest::Common::ContainerTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Common::DbVersion;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
use pgBackRestTest::Common::ExecuteTest;
use pgBackRestTest::Common::FileTest;
use pgBackRestTest::Common::RunTest;
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
use pgBackRestTest::Common::VmTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Common::Storage;
use pgBackRestTest::Common::StoragePosix;
use pgBackRestTest::Common::StorageRepo;
use pgBackRestTest::Common::Wait;
use pgBackRestTest::Env::ArchiveInfo;
use pgBackRestTest::Env::BackupInfo;
use pgBackRestTest::Env::InfoCommon;
Moved test and env modules to new directories to avoid namespace conflicts with common tests.
2017-05-12 22:43:04 +02:00
use pgBackRestTest::Env::Host::HostBaseTest;
use pgBackRestTest::Env::Host::HostBackupTest;
use pgBackRestTest::Env::Host::HostDbTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Env::Host::HostDbTest;
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
use pgBackRestTest::Env::HostEnvTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Env::Manifest;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Remove Perl Db module and LibC dependencies. This was mostly dead code except the DB_BACKUP_ADVISORY_LOCK constant, moved to the real/all test module, and the function that pulls info from pg_control, moved to ExpireEnvTest.pm.
2020-03-06 14:21:17 +02:00
####################################################################################################################################
# Backup advisory lock
####################################################################################################################################
use constant DB_BACKUP_ADVISORY_LOCK => '12340078987004321';
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
# run
####################################################################################################################################
sub run
{
my $self = shift;
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
# Should the test use lz4 compression?
my $bLz4Compress = true;
S3 repository support.
2017-06-12 16:52:32 +02:00
foreach my $bS3 (false, true)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
S3 repository support.
2017-06-12 16:52:32 +02:00
foreach my $bHostBackup ($bS3 ? (true) : (false, true))
{
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# Standby should only be tested for pg versions that support it
S3 repository support.
2017-06-12 16:52:32 +02:00
foreach my $bHostStandby ($bS3 ? (false) : (false, true))
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# Master and standby backup destinations on need to be tested on one db version since it is not version specific
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
foreach my $strBackupDestination (
S3 repository support.
2017-06-12 16:52:32 +02:00
$bS3 || $bHostBackup ? (HOST_BACKUP) : $bHostStandby ? (HOST_DB_MASTER, HOST_DB_STANDBY) : (HOST_DB_MASTER))
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
my $strCompressType = $bHostBackup && !$bHostStandby ? (vmWithLz4($self->vm()) && $bLz4Compress ? LZ4 : GZ) : NONE;
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
my $bRepoEncrypt = ($strCompressType ne NONE && !$bS3) ? true : false;
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
Add lz4 compression support. LZ4 compresses data faster than gzip but at a lower ratio. This can be a good tradeoff in certain scenarios. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest.
2020-03-10 20:45:27 +02:00
# If compression was used then switch it for the next test that uses compression
if ($strCompressType ne NONE)
{
$bLz4Compress = !$bLz4Compress;
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Increment the run, log, and decide whether this unit test should be run
Only check expect logs on CentOS 7. Variations in distros cause false negatives in tests but don't add much value.
2017-11-15 00:16:39 +02:00
my $hyVm = vmGet();
my $strDbVersionMostRecent = ${$hyVm->{$self->vm()}{&VM_DB_TEST}}[-1];
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
next if (!$self->begin(
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
"bkp ${bHostBackup}, sby ${bHostStandby}, dst ${strBackupDestination}, cmp ${strCompressType}, s3 ${bS3}, " .
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
"enc ${bRepoEncrypt}",
Only check expect logs on CentOS 7. Variations in distros cause false negatives in tests but don't add much value.
2017-11-15 00:16:39 +02:00
# Use the most recent db version on the expect vm for expect testing
$self->vm() eq VM_EXPECT && $self->pgVersion() eq $strDbVersionMostRecent));
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# Skip when s3 and host backup tests when there is more than one version of pg being tested and this is not the last one
Only check expect logs on CentOS 7. Variations in distros cause false negatives in tests but don't add much value.
2017-11-15 00:16:39 +02:00
if (($bS3 || $bHostBackup) && (@{$hyVm->{$self->vm()}{&VM_DB_TEST}} > 1 && $strDbVersionMostRecent ne $self->pgVersion()))
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
{
Only check expect logs on CentOS 7. Variations in distros cause false negatives in tests but don't add much value.
2017-11-15 00:16:39 +02:00
&log(INFO, "skipped - this test is run this OS using PG ${strDbVersionMostRecent}");
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
next;
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# Skip hot standby tests if the system does not support hot standby
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
if ($bHostStandby && $self->pgVersion() < PG_VERSION_HOT_STANDBY)
{
&log(INFO, 'skipped - this version of PostgreSQL does not support hot standby');
next;
}
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# Skip backup destinations other than backup host when standby except for one arbitrary db version
if ($bHostStandby && $strBackupDestination ne HOST_BACKUP && $self->pgVersion() ne PG_VERSION_96)
{
Automatically check that all supported PostgreSQL versions are being tested on a single default VM.
2017-06-27 21:48:34 +02:00
&log(INFO, 'skipped - standby with backup destination other than backup host is tested on PG ' . PG_VERSION_96);
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
next;
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create hosts, file object, and config
S3 repository support.
2017-06-12 16:52:32 +02:00
my ($oHostDbMaster, $oHostDbStandby, $oHostBackup, $oHostS3) = $self->setup(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
false, $self->expect(),
{bHostBackup => $bHostBackup, bStandby => $bHostStandby, strBackupDestination => $strBackupDestination,
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
strCompressType => $strCompressType, bArchiveAsync => false, bS3 => $bS3, bRepoEncrypt => $bRepoEncrypt});
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# Only perform extra tests on certain runs to save time
my $bTestLocal = $self->runCurrent() == 1;
my $bTestExtra =
$bTestLocal || $self->runCurrent() == 4 || ($self->runCurrent() == 6 && $self->pgVersion() eq PG_VERSION_96);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# If S3 set process max to 2. This seems like the best place for parallel testing since it will help speed S3 processing
# without slowing down the other tests too much.
if ($bS3)
{
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({&CFGDEF_SECTION_GLOBAL => {'process-max' => 2}});
$oHostDbMaster->configUpdate({&CFGDEF_SECTION_GLOBAL => {'process-max' => 2}});
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterCreate();
# Create the stanza
$oHostBackup->stanzaCreate('main create stanza info files');
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
# Get passphrase to access the Manifest file from backup.info - returns undefined if repo not encrypted
my $strCipherPass =
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
(new pgBackRestTest::Env::BackupInfo($oHostBackup->repoBackupPath()))->cipherPassSub();
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
# Create a manifest with the pg version to get version-specific paths
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
my $oManifest = new pgBackRestTest::Env::Manifest(BOGUS, {bLoad => false, strDbVersion => $self->pgVersion(),
Require PostgreSQL catalog version when instantiating a Manifest object (and not loading it from disk). Contributed by Cynthia Shang.
2018-07-16 23:25:15 +02:00
iDbCatalogVersion => $self->dbCatalogVersion($self->pgVersion()),
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
strCipherPass => $strCipherPass, strCipherPassSub => $bRepoEncrypt ? ENCRYPTION_KEY_BACKUPSET : undef});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Static backup parameters
my $fTestDelay = 1;
# Restore test string
my $strDefaultMessage = 'default';
my $strFullMessage = 'full';
my $strStandbyMessage = 'standby';
my $strIncrMessage = 'incr';
my $strTimeMessage = 'time';
my $strXidMessage = 'xid';
my $strNameMessage = 'name';
Clarify which timeline should be used for timeline integration test.
2019-09-27 19:37:59 +02:00
my $strTimelineMessage = 'timeline';
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create two new databases
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
{
$oHostDbMaster->sqlExecute('create database test1', {bAutoCommit => true});
$oHostDbMaster->sqlExecute('create database test2', {bAutoCommit => true});
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Test check command and stanza create
#---------------------------------------------------------------------------------------------------------------------------
if ($bTestExtra)
{
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
# In this section the same comment can be used multiple times so make it a variable that can be set once and reused
my $strComment = undef;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
# Archive and backup info file names
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
my $strArchiveInfoFile = $oHostBackup->repoArchivePath(ARCHIVE_INFO_FILE);
my $strArchiveInfoCopyFile = $oHostBackup->repoArchivePath(ARCHIVE_INFO_FILE . INI_COPY_EXT);
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
my $strArchiveInfoOldFile = "${strArchiveInfoFile}.old";
my $strArchiveInfoCopyOldFile = "${strArchiveInfoCopyFile}.old";
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
my $strBackupInfoFile = $oHostBackup->repoBackupPath(FILE_BACKUP_INFO);
my $strBackupInfoCopyFile = $oHostBackup->repoBackupPath(FILE_BACKUP_INFO . INI_COPY_EXT);
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
my $strBackupInfoOldFile = "${strBackupInfoFile}.old";
my $strBackupInfoCopyOldFile = "${strBackupInfoCopyFile}.old";
# Move the archive.info files to simulate missing file
forceStorageMove(storageRepo(), $strArchiveInfoFile, $strArchiveInfoOldFile, {bRecurse => false});
forceStorageMove(storageRepo(), $strArchiveInfoCopyFile, $strArchiveInfoCopyOldFile, {bRecurse => false});
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->check(
'fail on missing archive.info file',
{iTimeout => 0.1, iExpectedExitStatus => ERROR_FILE_MISSING});
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
# Backup.info was created earlier so restore archive info files
forceStorageMove(storageRepo(), $strArchiveInfoOldFile, $strArchiveInfoFile, {bRecurse => false});
forceStorageMove(storageRepo(), $strArchiveInfoCopyOldFile, $strArchiveInfoCopyFile, {bRecurse => false});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Check ERROR_ARCHIVE_DISABLED error
$strComment = 'fail on archive_mode=off';
$oHostDbMaster->clusterRestart({bIgnoreLogError => true, bArchiveEnabled => false});
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
$oHostBackup->backup(CFGOPTVAL_BACKUP_TYPE_FULL, $strComment, {iExpectedExitStatus => ERROR_ARCHIVE_DISABLED});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_ARCHIVE_DISABLED});
# If running the remote tests then also need to run check locally
if ($bHostBackup)
{
$oHostBackup->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_ARCHIVE_DISABLED});
}
# Check ERROR_ARCHIVE_COMMAND_INVALID error
$strComment = 'fail on invalid archive_command';
$oHostDbMaster->clusterRestart({bIgnoreLogError => true, bArchive => false});
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
$oHostBackup->backup(CFGOPTVAL_BACKUP_TYPE_FULL, $strComment, {iExpectedExitStatus => ERROR_ARCHIVE_COMMAND_INVALID});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_ARCHIVE_COMMAND_INVALID});
# If running the remote tests then also need to run check locally
if ($bHostBackup)
{
$oHostBackup->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_ARCHIVE_COMMAND_INVALID});
}
# When archive-check=n then ERROR_ARCHIVE_TIMEOUT will be raised instead of ERROR_ARCHIVE_COMMAND_INVALID
# ??? But maybe we should error with the fact that that option is not valid
$strComment = 'fail on archive timeout when archive-check=n';
$oHostDbMaster->check(
$strComment,
{iTimeout => 0.1, iExpectedExitStatus => ERROR_ARCHIVE_TIMEOUT, strOptionalParam => '--no-archive-check'});
# Stop the cluster ignoring any errors in the postgresql log
$oHostDbMaster->clusterStop({bIgnoreLogError => true});
# Providing a sufficient archive-timeout, verify that the check command runs successfully.
$strComment = 'verify success';
$oHostDbMaster->clusterStart();
$oHostDbMaster->check($strComment, {iTimeout => 5});
# If running the remote tests then also need to run check locally
if ($bHostBackup)
{
$oHostBackup->check($strComment, {iTimeout => 5});
}
# Check archive mismatch due to upgrade error
$strComment = 'fail on archive mismatch after upgrade';
# load the archive info file and munge it for testing by breaking the database version
$oHostBackup->infoMunge(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->repoArchivePath(ARCHIVE_INFO_FILE),
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
{&INFO_ARCHIVE_SECTION_DB => {&INFO_ARCHIVE_KEY_DB_VERSION => '8.0'},
&INFO_ARCHIVE_SECTION_DB_HISTORY => {1 => {&INFO_ARCHIVE_KEY_DB_VERSION => '8.0'}}});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
$oHostDbMaster->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_FILE_INVALID});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# If running the remote tests then also need to run check locally
if ($bHostBackup)
{
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
$oHostBackup->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_FILE_INVALID});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Restore the file to its original condition
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->infoRestore($oHostBackup->repoArchivePath(ARCHIVE_INFO_FILE));
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Check archive_timeout error when WAL segment is not found
$strComment = 'fail on archive timeout';
$oHostDbMaster->clusterRestart({bIgnoreLogError => true, bArchiveInvalid => true});
$oHostDbMaster->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_ARCHIVE_TIMEOUT});
# If running the remote tests then also need to run check locally
if ($bHostBackup)
{
$oHostBackup->check($strComment, {iTimeout => 0.1, iExpectedExitStatus => ERROR_ARCHIVE_TIMEOUT});
}
# Restart the cluster ignoring any errors in the postgresql log
$oHostDbMaster->clusterRestart({bIgnoreLogError => true});
# With a valid archive info, create the backup.info file by running a backup then munge the backup.info file.
# Check backup mismatch error
$strComment = 'fail on backup info mismatch';
# Load the backup.info file and munge it for testing by breaking the database version and system id
$oHostBackup->infoMunge(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->repoBackupPath(FILE_BACKUP_INFO),
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{&INFO_BACKUP_SECTION_DB =>
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
{&INFO_BACKUP_KEY_DB_VERSION => '8.0', &INFO_BACKUP_KEY_SYSTEM_ID => 6999999999999999999},
&INFO_BACKUP_SECTION_DB_HISTORY =>
{1 => {&INFO_BACKUP_KEY_DB_VERSION => '8.0', &INFO_BACKUP_KEY_SYSTEM_ID => 6999999999999999999}}});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Run the test
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
$oHostDbMaster->check($strComment, {iTimeout => 5, iExpectedExitStatus => ERROR_FILE_INVALID});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# If running the remote tests then also need to run check locally
if ($bHostBackup)
{
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
$oHostBackup->check($strComment, {iTimeout => 5, iExpectedExitStatus => ERROR_FILE_INVALID});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Restore the file to its original condition
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->infoRestore($oHostBackup->repoBackupPath(FILE_BACKUP_INFO));
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
# ??? Removed temporarily until manifest build can be brought back into the check command
Improve check command to verify that the backup manifest can be built. Contributed by Cynthia Shang.
2017-12-13 18:16:27 +02:00
# Create a directory in pg_data location that is only readable by root to ensure manifest->build is called by check
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
# my $strDir = $oHostDbMaster->dbBasePath() . '/rootreaddir';
# executeTest('sudo mkdir ' . $strDir);
# executeTest("sudo chown root:root ${strDir}");
# executeTest("sudo chmod 400 ${strDir}");
#
# $strComment = 'confirm master manifest->build executed';
# $oHostDbMaster->check($strComment, {iTimeout => 5, iExpectedExitStatus => ERROR_PATH_OPEN});
# executeTest("sudo rmdir ${strDir}");
Improve check command to verify that the backup manifest can be built. Contributed by Cynthia Shang.
2017-12-13 18:16:27 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Providing a sufficient archive-timeout, verify that the check command runs successfully now with valid
# archive.info and backup.info files
$strComment = 'verify success after backup';
$oHostDbMaster->check($strComment, {iTimeout => 5});
# If running the remote tests then also need to run check locally
if ($bHostBackup)
{
$oHostBackup->check($strComment, {iTimeout => 5});
}
# Restart the cluster ignoring any errors in the postgresql log
$oHostDbMaster->clusterRestart({bIgnoreLogError => true});
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
# Stanza Create
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
#-----------------------------------------------------------------------------------------------------------------------
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
# With data existing in the archive and backup directory, move info files and confirm failure
forceStorageMove(storageRepo(), $strArchiveInfoFile, $strArchiveInfoOldFile, {bRecurse => false});
forceStorageMove(storageRepo(), $strArchiveInfoCopyFile, $strArchiveInfoCopyOldFile, {bRecurse => false});
forceStorageMove(storageRepo(), $strBackupInfoFile, $strBackupInfoOldFile, {bRecurse => false});
forceStorageMove(storageRepo(), $strBackupInfoCopyFile, $strBackupInfoCopyOldFile, {bRecurse => false});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
$oHostBackup->stanzaCreate(
'fail on backup info file missing from non-empty dir', {iExpectedExitStatus => ERROR_PATH_NOT_EMPTY});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
# Change the database version by copying a new pg_control file to a new pg-path to use for db mismatch test
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
storageTest()->pathCreate(
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
$oHostDbMaster->dbPath() . '/testbase/' . DB_PATH_GLOBAL,
{strMode => '0700', bIgnoreExists => true, bCreateParent => true});
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
$self->controlGenerate(
$oHostDbMaster->dbPath() . '/testbase', $self->pgVersion() eq PG_VERSION_94 ? PG_VERSION_95 : PG_VERSION_94);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
# Run stanza-create online to confirm proper handling of configValidation error against new pg-path
$oHostBackup->stanzaCreate('fail on database mismatch with directory',
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{strOptionalParam => ' --pg1-path=' . $oHostDbMaster->dbPath() . '/testbase/',
iExpectedExitStatus => ERROR_DB_MISMATCH});
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
# Remove the directories to be able to create the stanza
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
forceStorageRemove(storageRepo(), $oHostBackup->repoBackupPath(), {bRecurse => true});
forceStorageRemove(storageRepo(), $oHostBackup->repoArchivePath(), {bRecurse => true});
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
# Stanza Upgrade - tests configValidate code - all other tests in synthetic integration tests
#-----------------------------------------------------------------------------------------------------------------------
The stanza-create/update/delete commands are implemented entirely in C. Contributed by Cynthia Shang.
2019-08-21 22:26:28 +02:00
# Run stanza-create offline to create files needing to be upgraded (using new pg-path)
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
$oHostBackup->stanzaCreate('successfully create stanza files to be upgraded',
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{strOptionalParam => ' --pg1-path=' . $oHostDbMaster->dbPath() . '/testbase/ --no-online --force'});
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
my $oArchiveInfo = new pgBackRestTest::Env::ArchiveInfo($oHostBackup->repoArchivePath());
my $oBackupInfo = new pgBackRestTest::Env::BackupInfo($oHostBackup->repoBackupPath());
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
# Read info files to confirm the files were created with a different database version
if ($self->pgVersion() eq PG_VERSION_94)
{
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
$self->testResult(sub {$oArchiveInfo->test(INFO_ARCHIVE_SECTION_DB, INFO_ARCHIVE_KEY_DB_VERSION, undef,
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
PG_VERSION_95)}, true, 'archive upgrade forced with pg mismatch');
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
$self->testResult(sub {$oBackupInfo->test(INFO_BACKUP_SECTION_DB, INFO_BACKUP_KEY_DB_VERSION, undef,
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
PG_VERSION_95)}, true, 'backup upgrade forced with pg mismatch');
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
}
else
{
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
$self->testResult(sub {$oArchiveInfo->test(INFO_ARCHIVE_SECTION_DB, INFO_ARCHIVE_KEY_DB_VERSION, undef,
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
PG_VERSION_94)}, true, 'archive create forced with pg mismatch in prep for stanza-upgrade');
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
$self->testResult(sub {$oBackupInfo->test(INFO_BACKUP_SECTION_DB, INFO_BACKUP_KEY_DB_VERSION, undef,
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
PG_VERSION_94)}, true, 'backup create forced with pg mismatch in prep for stanza-upgrade');
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
# Run stanza-upgrade online with the default pg-path to correct the info files
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
$oHostBackup->stanzaUpgrade('upgrade stanza files online');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
# Reread the info files and confirm the result
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
$oArchiveInfo = new pgBackRestTest::Env::ArchiveInfo($oHostBackup->repoArchivePath());
$oBackupInfo = new pgBackRestTest::Env::BackupInfo($oHostBackup->repoBackupPath());
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
$self->testResult(sub {$oArchiveInfo->test(INFO_ARCHIVE_SECTION_DB, INFO_ARCHIVE_KEY_DB_VERSION, undef,
Added stanza-upgrade command. The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. Contributed by Cynthia Shang.
2017-04-03 16:42:55 +02:00
$self->pgVersion())}, true, 'archive upgrade online corrects db');
$self->testResult(sub {$oBackupInfo->test(INFO_BACKUP_SECTION_DB, INFO_BACKUP_KEY_DB_VERSION, undef,
$self->pgVersion())}, true, 'backup upgrade online corrects db');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Full backup
#---------------------------------------------------------------------------------------------------------------------------
# Create the table where test messages will be stored
$oHostDbMaster->sqlExecute("create table test (message text not null)");
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->sqlExecute("insert into test values ('$strDefaultMessage')");
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
# Acquire the backup advisory lock so it looks like a backup is running
if (!$oHostDbMaster->sqlSelectOne('select pg_try_advisory_lock(' . DB_BACKUP_ADVISORY_LOCK . ')'))
{
confess 'unable to acquire advisory lock for testing';
}
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
$oHostBackup->backup(
CFGOPTVAL_BACKUP_TYPE_FULL, 'fail on backup lock exists', {iExpectedExitStatus => ERROR_LOCK_ACQUIRE});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Release the backup advisory lock so the next backup will succeed
if (!$oHostDbMaster->sqlSelectOne('select pg_advisory_unlock(' . DB_BACKUP_ADVISORY_LOCK . ')'))
{
confess 'unable to release advisory lock';
}
}
$oHostDbMaster->sqlExecute("update test set message = '$strFullMessage'");
# Required to set hint bits to be sent to the standby to make the heap match on both sides
$oHostDbMaster->sqlSelectOneTest('select message from test', $strFullMessage);
Update integration tests in real/all that use test points. Test points are not supported by the new C code so these will be replaced with unit tests. The fact that the tests still pass even when the changes aren't made mid-backup (except application_name) shows how weak they were in the first place. Even so, this does represent a regression in (soon to be be removed) Perl coverage.
2019-11-26 18:32:12 +02:00
my $strFullBackup = $oHostBackup->backup(
CFGOPTVAL_BACKUP_TYPE_FULL, 'update during backup',
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{strOptionalParam => ' --buffer-size=16384'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
# Enabled async archiving
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({&CFGDEF_SECTION_GLOBAL => {'archive-async' => 'y'}});
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
# Kick out a bunch of archive logs to exercise async archiving. Only do this when compressed and remote to slow it
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# down enough to make it evident that the async process is working.
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
if ($bTestExtra && $strCompressType ne NONE && $strBackupDestination eq HOST_BACKUP)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
&log(INFO, ' multiple wal switches to exercise async archiving');
$oHostDbMaster->sqlExecute("create table wal_activity (id int)");
$oHostDbMaster->sqlWalRotate();
$oHostDbMaster->sqlExecute("insert into wal_activity values (1)");
$oHostDbMaster->sqlWalRotate();
$oHostDbMaster->sqlExecute("insert into wal_activity values (2)");
$oHostDbMaster->sqlWalRotate();
$oHostDbMaster->sqlExecute("insert into wal_activity values (3)");
$oHostDbMaster->sqlWalRotate();
$oHostDbMaster->sqlExecute("insert into wal_activity values (4)");
$oHostDbMaster->sqlWalRotate();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Setup replica
#---------------------------------------------------------------------------------------------------------------------------
if ($bHostStandby)
{
my %oRemapHash;
$oRemapHash{&MANIFEST_TARGET_PGDATA} = $oHostDbStandby->dbBasePath();
if ($oHostDbStandby->pgVersion() >= PG_VERSION_92)
{
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbStandby->linkRemap($oManifest->walPath(), $oHostDbStandby->dbPath() . '/' . $oManifest->walPath());
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
$oHostDbStandby->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
'restore backup on replica', 'latest',
Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf. This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal. The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 23:39:45 +02:00
{rhRemapHash => \%oRemapHash, strType => CFGOPTVAL_RESTORE_TYPE_STANDBY,
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
strOptionalParam =>
Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf. This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal. The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 23:39:45 +02:00
' --recovery-option="primary_conninfo=host=' . HOST_DB_MASTER .
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
' port=' . $oHostDbMaster->pgPort() . ' user=replicator"'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbStandby->clusterStart({bHotStandby => true});
# Make sure streaming replication is on
$oHostDbMaster->sqlSelectOneTest(
"select client_addr || '-' || state from pg_stat_replication", $oHostDbStandby->ipGet() . '/32-streaming');
# Check that the cluster was restored properly
$oHostDbStandby->sqlSelectOneTest('select message from test', $strFullMessage);
# Update message for standby
$oHostDbMaster->sqlExecute("update test set message = '$strStandbyMessage'");
Up to seven standbys can be configured for backup from standby.` Contributed by Cynthia Shang.
2017-09-01 01:15:44 +02:00
if ($oHostDbStandby->pgVersion() >= PG_VERSION_BACKUP_STANDBY)
{
Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. * repo-* options are now indexed although only one is allowed. * List deprecated option names in documentation and command-line help.
2018-02-04 01:27:38 +02:00
# If there is only a master and a replica and the replica is the backup destination, then if pg2-host and pg3-host
Up to seven standbys can be configured for backup from standby.` Contributed by Cynthia Shang.
2017-09-01 01:15:44 +02:00
# are BOGUS, confirm failure to reach the master
if (!$bHostBackup && $bHostStandby && $strBackupDestination eq HOST_DB_STANDBY)
{
my $strStandbyBackup = $oHostBackup->backup(
CFGOPTVAL_BACKUP_TYPE_FULL, 'backup from standby, failure to reach master',
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{bStandby => true, iExpectedExitStatus => ERROR_DB_CONNECT, strOptionalParam => '--pg8-host=' . BOGUS});
Up to seven standbys can be configured for backup from standby.` Contributed by Cynthia Shang.
2017-09-01 01:15:44 +02:00
}
else
{
my $strStandbyBackup = $oHostBackup->backup(
CFGOPTVAL_BACKUP_TYPE_FULL, 'backup from standby, failure to access at least one standby',
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{bStandby => true, iExpectedExitStatus => ERROR_DB_CONNECT, strOptionalParam => '--pg8-host=' . BOGUS});
Up to seven standbys can be configured for backup from standby.` Contributed by Cynthia Shang.
2017-09-01 01:15:44 +02:00
}
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
my $strStandbyBackup = $oHostBackup->backup(
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
CFGOPTVAL_BACKUP_TYPE_FULL, 'backup from standby',
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{bStandby => true,
iExpectedExitStatus => $oHostDbStandby->pgVersion() >= PG_VERSION_BACKUP_STANDBY ? undef : ERROR_CONFIG,
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
strOptionalParam => '--repo1-retention-full=1'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
if ($oHostDbStandby->pgVersion() >= PG_VERSION_BACKUP_STANDBY)
{
$strFullBackup = $strStandbyBackup;
}
The check command is implemented entirely in C. Note that building the manifest on each host has been temporarily removed. This feature will likely be brought back as a non-default option (after the manifest code has been fully migrated to C) since it can be fairly expensive.
2019-10-09 00:04:09 +02:00
# ??? Removed temporarily until manifest build can be brought back into the check command
# # Create a directory in pg_data location that is only readable by root to ensure manifest->build is called by check
# my $strDir = $oHostDbStandby->dbBasePath() . '/rootreaddir';
# executeTest('sudo mkdir ' . $strDir);
# executeTest("sudo chown root:root ${strDir}");
# executeTest("sudo chmod 400 ${strDir}");
#
# my $strComment = 'confirm standby manifest->build executed';
#
# # If there is an invalid host, the final error returned from check will be the inability to resolve the name which is
# # an open error instead of a read error
# if (!$oHostDbStandby->bogusHost())
# {
# $oHostDbStandby->check($strComment, {iTimeout => 5, iExpectedExitStatus => ERROR_PATH_OPEN});
# }
# else
# {
# $oHostDbStandby->check($strComment, {iTimeout => 5, iExpectedExitStatus => ERROR_FILE_READ});
# }
#
# # Remove the directory in pg_data location that is only readable by root
# executeTest("sudo rmdir ${strDir}");
Improve check command to verify that the backup manifest can be built. Contributed by Cynthia Shang.
2017-12-13 18:16:27 +02:00
# Confirm the check command runs without error on a standby (when a bogus host is not configured)
Fix manifest test in the check command when tablespaces are present. Fixed by Cynthia Shang. Reported by Thomas Flatley.
2018-03-20 01:54:03 +02:00
if (!$oHostDbStandby->bogusHost())
Improve check command to verify that the backup manifest can be built. Contributed by Cynthia Shang.
2017-12-13 18:16:27 +02:00
{
$oHostDbStandby->check('verify check command on standby');
}
Include archive_status directory in online backups. The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log. Reported by Stephen Frost.
2017-07-24 13:57:47 +02:00
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
# Shutdown the standby before creating tablespaces (this will error since paths are different)
Include archive_status directory in online backups. The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log. Reported by Stephen Frost.
2017-07-24 13:57:47 +02:00
$oHostDbStandby->clusterStop({bIgnoreLogError => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Execute stop and make sure the backup fails
#---------------------------------------------------------------------------------------------------------------------------
# Restart the cluster to check for any errors before continuing since the stop tests will definitely create errors and
# the logs will to be deleted to avoid causing issues further down the line.
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestExtra && !$bS3)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
$oHostDbMaster->clusterRestart();
$oHostDbMaster->stop();
The backup command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C except as noted below. A backup can now be initiated from a linked directory. The link will not be stored in the manifest or recreated on restore. If a link or directory does not already exist in the restore location then a directory will be created. The logic for creating backup labels has been improved and it should no longer be possible to get a backup label earlier than the latest backup even with timezone changes or clock skew. This has never been an issue in the field that we know of, but we found it in testing. For online backups all times are fetched from the PostgreSQL primary host (before only copy start was). This doesn't affect backup integrity but it does prevent clock skew between hosts affecting backup duration reporting. Archive copy now works as expected when the archive and backup have different compression settings, i.e. when one is compressed and the other is not. This was a long-standing bug in the Perl code. Resume will now work even if hardlink settings have been changed. Reviewed by Cynthia Shang.
2019-12-14 00:14:26 +02:00
$oHostBackup->backup(
CFGOPTVAL_BACKUP_TYPE_INCR, 'attempt backup when stopped',
{iExpectedExitStatus => $oHostBackup == $oHostDbMaster ? ERROR_STOP : ERROR_DB_CONNECT});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->start();
}
Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 23:38:05 +02:00
# Setup the time targets
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$oHostDbMaster->sqlExecute("update test set message = '$strTimeMessage'");
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 23:38:05 +02:00
my $strTimeTarget = $oHostDbMaster->sqlSelectOne("select current_timestamp");
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
&log(INFO, " time target is ${strTimeTarget}");
# Incr backup - fail on archive_mode=always when version >= 9.5
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal && $oHostDbMaster->pgVersion() >= PG_VERSION_95)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
# Set archive_mode=always
$oHostDbMaster->clusterRestart({bArchiveAlways => true});
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
$oHostBackup->backup(
CFGOPTVAL_BACKUP_TYPE_INCR, 'fail on archive_mode=always', {iExpectedExitStatus => ERROR_FEATURE_NOT_SUPPORTED});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Reset the cluster to a normal state so the next test will work
$oHostDbMaster->clusterRestart();
}
# Incr backup
#---------------------------------------------------------------------------------------------------------------------------
# Create a tablespace directory
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
storageTest()->pathCreate($oHostDbMaster->tablespacePath(1), {strMode => '0700', bCreateParent => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Also create it on the standby so replay won't fail
if (defined($oHostDbStandby))
{
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
storageTest()->pathCreate($oHostDbStandby->tablespacePath(1), {strMode => '0700', bCreateParent => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
$oHostDbMaster->sqlExecute(
"create tablespace ts1 location '" . $oHostDbMaster->tablespacePath(1) . "'", {bAutoCommit => true});
Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. The integration tests that were supposed to prevent this regression did not work as intended. They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace. All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid. However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions. Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore. The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again. Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 22:47:40 +02:00
$oHostDbMaster->sqlExecute("alter table test set tablespace ts1");
# Create a table in the tablespace that will not be modified again to be sure it does get full page writes in the WAL later
$oHostDbMaster->sqlExecute("create table test_exists (id int) tablespace ts1", {bCommit => true, bCheckPoint => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create a table in the tablespace
$oHostDbMaster->sqlExecute("create table test_remove (id int)");
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->sqlExecute("update test set message = '$strDefaultMessage'");
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
# Create a database in the tablespace and a table to check
$oHostDbMaster->sqlExecute("create database test3 with tablespace ts1", {bAutoCommit => true});
$oHostDbMaster->sqlExecute(
'create table test3_exists (id int);' .
'insert into test3_exists values (1);',
{strDb => 'test3', bAutoCommit => true});
if ($bTestLocal)
{
# Create a table in test1 to check - test1 will not be restored
$oHostDbMaster->sqlExecute(
'create table test1_zeroed (id int);' .
'insert into test1_zeroed values (1);',
{strDb => 'test1', bAutoCommit => true});
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Start a backup so the next backup has to restart it. This test is not required for PostgreSQL >= 9.6 since backups
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
# are run in non-exclusive mode.
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal && $oHostDbMaster->pgVersion() >= PG_VERSION_93 && $oHostDbMaster->pgVersion() < PG_VERSION_96)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
$oHostDbMaster->sqlSelectOne("select pg_start_backup('test backup that will cause an error', true)");
# Verify that an error is returned if the backup is already running
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
$oHostBackup->backup(
CFGOPTVAL_BACKUP_TYPE_INCR, 'fail on backup already running', {iExpectedExitStatus => ERROR_DB_QUERY});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Restart the cluster ignoring any errors in the postgresql log
$oHostDbMaster->clusterRestart({bIgnoreLogError => true});
# Start a new backup to make the next test restart it
$oHostDbMaster->sqlSelectOne("select pg_start_backup('test backup that will be restarted', true)");
}
# Drop a table
$oHostDbMaster->sqlExecute('drop table test_remove');
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->sqlExecute("update test set message = '$strIncrMessage'", {bCommit => true});
Update integration tests in real/all that use test points. Test points are not supported by the new C code so these will be replaced with unit tests. The fact that the tests still pass even when the changes aren't made mid-backup (except application_name) shows how weak they were in the first place. Even so, this does represent a regression in (soon to be be removed) Perl coverage.
2019-11-26 18:32:12 +02:00
# Exercise --delta checksum option
my $strIncrBackup = $oHostBackup->backup(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
CFGOPTVAL_BACKUP_TYPE_INCR, 'update during backup', {strOptionalParam => '--stop-auto --buffer-size=32768 --delta'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fix manifest test in the check command when tablespaces are present. Fixed by Cynthia Shang. Reported by Thomas Flatley.
2018-03-20 01:54:03 +02:00
# Ensure the check command runs properly with a tablespace unless there is a bogus host
if (!$oHostBackup->bogusHost())
{
$oHostBackup->check( 'check command with tablespace', {iTimeout => 5});
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Setup the xid target
#---------------------------------------------------------------------------------------------------------------------------
my $strXidTarget = undef;
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
$oHostDbMaster->sqlExecute("update test set message = '$strXidMessage'", {bCommit => false});
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strXidTarget = $oHostDbMaster->sqlSelectOne("select txid_current()");
$oHostDbMaster->sqlCommit();
&log(INFO, " xid target is ${strXidTarget}");
}
# Setup the name target
#---------------------------------------------------------------------------------------------------------------------------
my $strNameTarget = 'backrest';
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
$oHostDbMaster->sqlExecute("update test set message = '$strNameMessage'", {bCommit => true});
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
if ($oHostDbMaster->pgVersion() >= PG_VERSION_91)
{
$oHostDbMaster->sqlExecute("select pg_create_restore_point('${strNameTarget}')");
}
&log(INFO, " name target is ${strNameTarget}");
}
# Create a table and data in database test2
#---------------------------------------------------------------------------------------------------------------------------
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
# Initialize variables for SHA1 and path of the pg_filenode.map for the database that will not be restored
my $strDb1TablePath;
my $strDb1TableSha1;
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
{
$oHostDbMaster->sqlExecute(
'create table test (id int);' .
'insert into test values (1);' .
'create table test_ts1 (id int) tablespace ts1;' .
'insert into test_ts1 values (2);',
{strDb => 'test2', bAutoCommit => true});
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
$oHostDbMaster->sqlWalRotate();
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
# Get the SHA1 and path of the table for the database that will not be restored
$strDb1TablePath = $oHostDbMaster->dbBasePath(). "/base/" .
$oHostDbMaster->sqlSelectOne("select oid from pg_database where datname='test1'") . "/" .
$oHostDbMaster->sqlSelectOne("select relfilenode from pg_class where relname='test1_zeroed'", {strDb => 'test1'});
$strDb1TableSha1 = storageTest()->hashSize($strDb1TablePath);
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Restore (type = default)
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
# Expect failure because postmaster.pid exists
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostDbMaster->restore('postmaster running', 'latest', {iExpectedExitStatus => ERROR_POSTMASTER_RUNNING});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
$oHostDbMaster->clusterStop();
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
# Expect failure because db path is not empty
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostDbMaster->restore('path not empty', 'latest', {iExpectedExitStatus => ERROR_PATH_NOT_EMPTY});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Drop and recreate db path
testPathRemove($oHostDbMaster->dbBasePath());
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
storageTest()->pathCreate($oHostDbMaster->dbBasePath(), {strMode => '0700'});
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
testPathRemove($oHostDbMaster->dbPath() . qw{/} . $oManifest->walPath());
storageTest()->pathCreate($oHostDbMaster->dbPath() . qw{/} . $oManifest->walPath(), {strMode => '0700'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
testPathRemove($oHostDbMaster->tablespacePath(1));
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
storageTest()->pathCreate($oHostDbMaster->tablespacePath(1), {strMode => '0700'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Now the restore should work
$oHostDbMaster->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
undef, 'latest',
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
{strOptionalParam => ($bTestLocal ? ' --db-include=test2 --db-include=test3' : '') . ' --buffer-size=16384'});
# Test that the first database has not been restored since --db-include did not include test1
if ($bTestLocal)
{
my ($strSHA1, $lSize) = storageTest()->hashSize($strDb1TablePath);
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
# Create a zeroed sparse file in the test directory that is the same size as the filenode.map. We need to use the
# posix driver directly to do this because handles cannot be passed back from the C code.
my $oStorageTrunc = new pgBackRestTest::Common::Storage($self->testPath(), new pgBackRestTest::Common::StoragePosix());
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
my $strTestTable = $self->testPath() . "/testtable";
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
my $oDestinationFileIo = $oStorageTrunc->openWrite($strTestTable);
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
$oDestinationFileIo->open();
# Truncate to the original size which will create a sparse file.
Add Perl interface to C storage layer. Maintaining the storage layer/drivers in two languages is burdensome. Since the integration tests require the Perl storage layer/drivers we'll need them even after the core code is migrated to C. Create an interface layer so the Perl code can be removed and new storage drivers/features introduced without adding Perl equivalents. The goal is to move the integration tests to C so this interface will eventually be removed. That being the case, the interface was designed for maximum compatibility to ease the transition. The result looks a bit hacky but we'll improve it as needed until it can be retired.
2019-06-26 14:24:58 +02:00
if (!truncate($oDestinationFileIo->handle(), $lSize))
{
confess "unable to truncate '$strTestTable' with handle " . $oDestinationFileIo->handle();
}
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
$oDestinationFileIo->close();
# Confirm the test filenode.map and the database test1 filenode.map are zeroed
my ($strSHA1Test, $lSizeTest) = storageTest()->hashSize($strTestTable);
$self->testResult(sub {($strSHA1Test eq $strSHA1) && ($lSizeTest == $lSize) && ($strSHA1 ne $strDb1TableSha1)},
true, 'database test1 not restored');
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart();
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
$oHostDbMaster->sqlSelectOneTest('select message from test', $bTestLocal ? $strNameMessage : $strIncrMessage);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
# Once the cluster is back online, make sure the database & table in the tablespace exists properly
if ($bTestLocal)
{
$oHostDbMaster->sqlSelectOneTest('select id from test_ts1', 2, {strDb => 'test2'});
$oHostDbMaster->sqlDisconnect({strDb => 'test2'});
$oHostDbMaster->sqlSelectOneTest('select id from test3_exists', 1, {strDb => 'test3'});
$oHostDbMaster->sqlDisconnect({strDb => 'test3'});
}
Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. The integration tests that were supposed to prevent this regression did not work as intended. They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace. All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid. However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions. Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore. The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again. Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 22:47:40 +02:00
# The tablespace path should exist and have files in it
my $strTablespacePath = $oHostDbMaster->tablespacePath(1);
# Version <= 8.4 always places a PG_VERSION file in the tablespace
if ($oHostDbMaster->pgVersion() <= PG_VERSION_84)
{
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
if (!storageTest()->exists("${strTablespacePath}/" . DB_FILE_PGVERSION))
Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. The integration tests that were supposed to prevent this regression did not work as intended. They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace. All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid. However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions. Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore. The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again. Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 22:47:40 +02:00
{
confess &log(ASSERT, "unable to find '" . DB_FILE_PGVERSION . "' in tablespace path '${strTablespacePath}'");
}
}
# Version >= 9.0 creates a special path using the version and catalog number
else
{
# Backup info will have the catalog number
Rename BackRestDoc Perl module to pgBackRestDoc. This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523. More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 21:41:56 +02:00
my $oBackupInfo = new pgBackRestDoc::Common::Ini(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
storageRepo(), $oHostBackup->repoBackupPath(FILE_BACKUP_INFO),
{bLoad => false, strContent => ${storageRepo()->get($oHostBackup->repoBackupPath(FILE_BACKUP_INFO))}});
Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. The integration tests that were supposed to prevent this regression did not work as intended. They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace. All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid. However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions. Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore. The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again. Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 22:47:40 +02:00
# Construct the special path
$strTablespacePath .=
'/PG_' . $oHostDbMaster->pgVersion() . qw{_} . $oBackupInfo->get(INFO_BACKUP_SECTION_DB, INFO_BACKUP_KEY_CATALOG);
# Check that path exists
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
if (!storageTest()->pathExists($strTablespacePath))
Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. The integration tests that were supposed to prevent this regression did not work as intended. They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace. All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid. However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions. Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore. The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again. Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 22:47:40 +02:00
{
confess &log(ASSERT, "unable to find tablespace path '${strTablespacePath}'");
}
}
# Make sure there are some files in the tablespace path (exclude PG_VERSION if <= 8.4 since that was tested above)
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
if (grep(!/^PG\_VERSION$/i, storageTest()->list($strTablespacePath)) == 0)
Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4. The integration tests that were supposed to prevent this regression did not work as intended. They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace. All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid. However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions. Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore. The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again. Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 22:47:40 +02:00
{
confess &log(ASSERT, "no files found in tablespace path '${strTablespacePath}'");
}
# This table should exist to prove that the tablespace was restored. It has not been updated since it was created so it
# should not be created by any full page writes. Once it is verified to exist it can be dropped.
$oHostDbMaster->sqlSelectOneTest("select count(*) from test_exists", 0);
$oHostDbMaster->sqlExecute('drop table test_exists');
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
# Now it should be OK to drop database test2 and test3
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
{
$oHostDbMaster->sqlExecute('drop database test2', {bAutoCommit => true});
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# The test table lives in ts1 so it needs to be moved or dropped
if ($oHostDbMaster->pgVersion() >= PG_VERSION_90)
{
$oHostDbMaster->sqlExecute('alter table test set tablespace pg_default');
}
# Drop for older versions
else
{
$oHostDbMaster->sqlExecute('drop table test');
}
# And drop the tablespace
Fixed inability to restore a single database contained in a tablespace using --db-include. Fixed by Cynthia Shang.
2018-01-30 23:13:54 +02:00
$oHostDbMaster->sqlExecute('drop database test3', {bAutoCommit => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->sqlExecute("drop tablespace ts1", {bAutoCommit => true});
# Restore (restore type = immediate, inclusive)
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if (($bTestLocal || $bHostStandby) && $oHostDbMaster->pgVersion() >= PG_VERSION_94)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
&log(INFO, ' testing recovery type = ' . CFGOPTVAL_RESTORE_TYPE_IMMEDIATE);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStop();
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
$oHostDbMaster->restore(
undef, $strFullBackup, {bForce => true, strType => CFGOPTVAL_RESTORE_TYPE_IMMEDIATE, strTargetAction => 'promote'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart();
$oHostDbMaster->sqlSelectOneTest(
'select message from test', ($bHostStandby ? $strStandbyMessage : $strFullMessage));
}
# Restore (restore type = xid, inclusive)
#---------------------------------------------------------------------------------------------------------------------------
Store recovery file name in integration when testing preserve recovery. This makes the test a little more maintainable and is friendly with the changes needed for PostgreSQL 12.
2019-09-27 16:27:07 +02:00
my $strRecoveryFile = undef;
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
&log(INFO, ' testing recovery type = ' . CFGOPTVAL_RESTORE_TYPE_XID);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStop();
executeTest('rm -rf ' . $oHostDbMaster->dbBasePath() . "/*");
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
executeTest('rm -rf ' . $oHostDbMaster->dbPath() . qw{/} . $oManifest->walPath() . '/*');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->restore(
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
undef, $strIncrBackup,
{bForce => true, strType => CFGOPTVAL_RESTORE_TYPE_XID, strTarget => $strXidTarget,
strTargetAction => $oHostDbMaster->pgVersion() >= PG_VERSION_91 ? 'promote' : undef,
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
strTargetTimeline => $oHostDbMaster->pgVersion() >= PG_VERSION_12 ? 'current' : undef,
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
strOptionalParam => '--tablespace-map-all=../../tablespace', bTablespace => false});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Save recovery file to test so we can use it in the next test
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
$strRecoveryFile = $oHostDbMaster->pgVersion() >= PG_VERSION_12 ? 'postgresql.auto.conf' : DB_FILE_RECOVERYCONF;
Store recovery file name in integration when testing preserve recovery. This makes the test a little more maintainable and is friendly with the changes needed for PostgreSQL 12.
2019-09-27 16:27:07 +02:00
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
storageTest()->copy(
Store recovery file name in integration when testing preserve recovery. This makes the test a little more maintainable and is friendly with the changes needed for PostgreSQL 12.
2019-09-27 16:27:07 +02:00
$oHostDbMaster->dbBasePath() . qw{/} . $strRecoveryFile, $self->testPath() . qw{/} . $strRecoveryFile);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart();
$oHostDbMaster->sqlSelectOneTest('select message from test', $strXidMessage);
$oHostDbMaster->sqlExecute("update test set message = '$strTimelineMessage'");
}
# Restore (restore type = preserve, inclusive)
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
&log(INFO, ' testing recovery type = ' . CFGOPTVAL_RESTORE_TYPE_PRESERVE);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStop();
executeTest('rm -rf ' . $oHostDbMaster->dbBasePath() . "/*");
PostgreSQL 10 support.
2017-09-01 18:29:34 +02:00
executeTest('rm -rf ' . $oHostDbMaster->dbPath() . qw{/} . $oManifest->walPath() . '/*');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
executeTest('rm -rf ' . $oHostDbMaster->tablespacePath(1) . "/*");
# Restore recovery file that was saved in last test
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
storageTest()->move($self->testPath . "/${strRecoveryFile}", $oHostDbMaster->dbBasePath() . "/${strRecoveryFile}");
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fix recovery test failing in PostgreSQL 12.0. This test was not creating recovery.signal when testing with --type=preserve. The preserve recovery type only keeps existing files and does not create any. RC1 was just ignoring recovery.signal and going right into recovery. Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.
2019-10-12 15:26:19 +02:00
# Also touch recovery.signal when required
if ($oHostDbMaster->pgVersion() >= PG_VERSION_12)
{
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
storageTest()->put($oHostDbMaster->dbBasePath() . "/" . DB_FILE_RECOVERYSIGNAL);
Fix recovery test failing in PostgreSQL 12.0. This test was not creating recovery.signal when testing with --type=preserve. The preserve recovery type only keeps existing files and does not create any. RC1 was just ignoring recovery.signal and going right into recovery. Weirdly, 12.0 used restore_command to do crash recovery which made the problem harder to diagnose, but this has now been fixed in PostgreSQL and should be released in 12.1.
2019-10-12 15:26:19 +02:00
}
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostDbMaster->restore(undef, 'latest', {strType => CFGOPTVAL_RESTORE_TYPE_PRESERVE});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart();
$oHostDbMaster->sqlSelectOneTest('select message from test', $strXidMessage);
$oHostDbMaster->sqlExecute("update test set message = '$strTimelineMessage'");
}
Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 23:38:05 +02:00
# Restore (restore type = time, inclusive, automatically select backup) - there is no exclusive time test because I can't
# find a way to find the exact commit time of a transaction.
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
&log(INFO, ' testing recovery type = ' . CFGOPTVAL_RESTORE_TYPE_TIME);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStop();
$oHostDbMaster->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
undef, 'latest',
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
{bDelta => true, strType => CFGOPTVAL_RESTORE_TYPE_TIME, strTarget => $strTimeTarget,
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
strTargetAction => $oHostDbMaster->pgVersion() >= PG_VERSION_91 ? 'promote' : undef,
Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. Currently a limited number of date formats are recognized and timezone names are not allowed, only timezone offsets.
2020-01-30 23:38:05 +02:00
strTargetTimeline => $oHostDbMaster->pgVersion() >= PG_VERSION_12 ? 'current' : undef,
strBackupExpected => $strFullBackup});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart();
$oHostDbMaster->sqlSelectOneTest('select message from test', $strTimeMessage);
# Restore (restore type = xid, exclusive)
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
&log(INFO, ' testing recovery type = ' . CFGOPTVAL_RESTORE_TYPE_XID);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStop();
$oHostDbMaster->restore(
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
undef, $strIncrBackup,
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
{bDelta => true, strType => CFGOPTVAL_RESTORE_TYPE_XID, strTarget => $strXidTarget, bTargetExclusive => true,
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
strTargetAction => $oHostDbMaster->pgVersion() >= PG_VERSION_91 ? 'promote' : undef,
strTargetTimeline => $oHostDbMaster->pgVersion() >= PG_VERSION_12 ? 'current' : undef});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart();
$oHostDbMaster->sqlSelectOneTest('select message from test', $strIncrMessage);
}
# Restore (restore type = name)
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal && $oHostDbMaster->pgVersion() >= PG_VERSION_91)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
&log(INFO, ' testing recovery type = ' . CFGOPTVAL_RESTORE_TYPE_NAME);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStop();
$oHostDbMaster->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
undef, 'latest',
Add explicit promotes to recovery integration tests. PostgreSQL 12 will shutdown in these cases which seems to be the correct action (according to the documentation) when hot_standby = off, but older versions are promoting instead. Set target_action explicitly so all versions will behave the same way. This does beg the question of whether the PostgreSQL 12 behavior is wrong (though it matches the docs) or the previous versions are.
2019-09-27 19:04:36 +02:00
{bDelta => true, bForce => true, strType => CFGOPTVAL_RESTORE_TYPE_NAME, strTarget => $strNameTarget,
PostgreSQL 12 support. Recovery settings are now written into postgresql.auto.conf instead of recovery.conf. Existing recovery_target* settings will be commented out to help avoid conflicts. A comment is added before recovery settings to identify them as written by pgBackRest since it is unclear how, in general, old settings will be removed. recovery.signal and standby.signal are automatically created based on the recovery settings.
2019-10-01 19:20:43 +02:00
strTargetAction => 'promote',
strTargetTimeline => $oHostDbMaster->pgVersion() >= PG_VERSION_12 ? 'current' : undef});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart();
$oHostDbMaster->sqlSelectOneTest('select message from test', $strNameMessage);
}
Clarify which timeline should be used for timeline integration test.
2019-09-27 19:37:59 +02:00
# Restore (restore type = default, timeline = created by type = xid, inclusive recovery)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestLocal && $oHostDbMaster->pgVersion() >= PG_VERSION_84)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
&log(INFO, ' testing recovery type = ' . CFGOPTVAL_RESTORE_TYPE_DEFAULT);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStop();
Clarify which timeline should be used for timeline integration test.
2019-09-27 19:37:59 +02:00
# The timeline to use for this test is subject to change based on tests being added or removed above. The best thing
# would be to automatically grab the timeline after the restore, but since this test has been stable for a long time
# it does not seem worth the effort to automate.
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->restore(
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
undef, $strIncrBackup,
Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf. This could be accomplished previously by setting --recovery-option=standby_mode=on but PostgreSQL 12 requires standby mode to be enabled by a special file named standby.signal. The new restore type allows us to maintain a common interface between PostgreSQL versions.
2019-09-26 23:39:45 +02:00
{bDelta => true,
strType => $oHostDbMaster->pgVersion() >= PG_VERSION_90 ?
CFGOPTVAL_RESTORE_TYPE_STANDBY : CFGOPTVAL_RESTORE_TYPE_DEFAULT,
strTargetTimeline => 4});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostDbMaster->clusterStart({bHotStandby => true});
$oHostDbMaster->sqlSelectOneTest('select message from test', $strTimelineMessage, {iTimeout => 120});
}
Fixed --no-online tests to suppress expected errors.
2017-02-03 03:44:42 +02:00
# Stop clusters to catch any errors in the postgres log
#---------------------------------------------------------------------------------------------------------------------------
Include archive_status directory in online backups. The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log. Reported by Stephen Frost.
2017-07-24 13:57:47 +02:00
$oHostDbMaster->clusterStop();
Fixed --no-online tests to suppress expected errors.
2017-02-03 03:44:42 +02:00
# Test no-online backups
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Eliminate redundancy in real tests.
2017-06-21 22:07:13 +02:00
if ($bTestExtra & !$bS3)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Fixed --no-online tests to suppress expected errors.
2017-02-03 03:44:42 +02:00
# Create a postmaster.pid file so it appears that the server is running
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
storageTest()->put($oHostDbMaster->dbBasePath() . '/postmaster.pid', '99999');
Fixed --no-online tests to suppress expected errors.
2017-02-03 03:44:42 +02:00
# Incr backup - make sure a --no-online backup fails
#-----------------------------------------------------------------------------------------------------------------------
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostBackup->backup(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
CFGOPTVAL_BACKUP_TYPE_INCR, 'fail on --no-online',
{iExpectedExitStatus => ERROR_POSTMASTER_RUNNING, strOptionalParam => '--no-online'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fixed --no-online tests to suppress expected errors.
2017-02-03 03:44:42 +02:00
# Incr backup - allow --no-online backup to succeed with --force
#-----------------------------------------------------------------------------------------------------------------------
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostBackup->backup(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
CFGOPTVAL_BACKUP_TYPE_INCR, 'succeed on --no-online with --force', {strOptionalParam => '--no-online --force'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
Bypass database checks when stanza-delete issued with force. Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted. Contributed by Cynthia Shang. Suggested by Roman.
2019-05-15 19:14:58 +02:00
# Stanza-delete --force without access to pgbackrest on database host
#---------------------------------------------------------------------------------------------------------------------------
if ($bTestExtra && !$bS3 && $bHostBackup)
{
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
# With stanza-delete --force, allow stanza to be deleted regardless of accessibility of database host
Bypass database checks when stanza-delete issued with force. Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted. Contributed by Cynthia Shang. Suggested by Roman.
2019-05-15 19:14:58 +02:00
if ($bHostBackup)
{
$oHostDbMaster->stop();
$oHostBackup->stop({strStanza => $self->stanza});
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->stanzaDelete(
"delete stanza with --force when pgbackrest on pg host not accessible", {strOptionalParam => ' --force'});
Bypass database checks when stanza-delete issued with force. Previously it was not possible to delete a stanza if the PostgreSQL server could not be contacted. Contributed by Cynthia Shang. Suggested by Roman.
2019-05-15 19:14:58 +02:00
$oHostDbMaster->start();
$oHostBackup->start();
}
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
}
}
S3 repository support.
2017-06-12 16:52:32 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
1;
Reference in New Issue Copy Permalink