db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-16 10:20:02 +02:00
Code Issues Releases Activity
2c96327e65
pgbackrest/test/lib/pgBackRestTest/Module/Mock/MockAllTest.pm

1123 lines
58 KiB
Perl
Raw Normal View History

Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
Split full module into mock and real to allow better test combinations and save time in CI.
2017-06-20 00:55:00 +02:00
# Test All Commands on Mock Data
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
Split full module into mock and real to allow better test combinations and save time in CI.
2017-06-20 00:55:00 +02:00
package pgBackRestTest::Module::Mock::MockAllTest;
Moved test and env modules to new directories to avoid namespace conflicts with common tests.
2017-05-12 22:43:04 +02:00
use parent 'pgBackRestTest::Env::HostEnvTest';
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
# Perl includes
####################################################################################################################################
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
Improvements to Ini.pm. * Refactor Ini.pm to facilitate testing. * Complete statement/branch coverage for Ini.pm. * Improved functions used to test/munge manifest and info files.
2017-04-10 19:24:45 +02:00
use File::Basename qw(basename dirname);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename BackRestDoc Perl module to pgBackRestDoc. This is consistent with the way BackRest and BackRest test were renamed way back in 18fd2523. More modules will be moving to pgBackRestDoc soon so renaming now reduces churn later.
2020-03-10 21:41:56 +02:00
use pgBackRestDoc::Common::Exception;
use pgBackRestDoc::Common::Ini;
use pgBackRestDoc::Common::Log;
use pgBackRestDoc::Common::String;
Move pgBackRest::Version module to pgBackRestDoc::ProjectInfo. The primary source for project info is now src/version.h. The pgBackRestDoc::ProjectInfo module loads the project info from src/version.h at runtime so there is no need to update it.
2020-03-10 23:57:02 +02:00
use pgBackRestDoc::ProjectInfo;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
use pgBackRestTest::Common::ContainerTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Common::DbVersion;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
use pgBackRestTest::Common::ExecuteTest;
use pgBackRestTest::Common::FileTest;
use pgBackRestTest::Common::RunTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Common::StorageRepo;
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
use pgBackRestTest::Common::VmTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Common::Wait;
use pgBackRestTest::Env::ArchiveInfo;
use pgBackRestTest::Env::BackupInfo;
Moved test and env modules to new directories to avoid namespace conflicts with common tests.
2017-05-12 22:43:04 +02:00
use pgBackRestTest::Env::Host::HostBackupTest;
S3 repository support.
2017-06-12 16:52:32 +02:00
use pgBackRestTest::Env::Host::HostS3Test;
use pgBackRestTest::Env::HostEnvTest;
Move Perl modules out of lib directory. This directory was once the home of the production Perl code but since f0ef73db this is no longer true. Move the modules to test in most cases, except where the module is expected to be useful for the doc engine beyond the expected lifetime of the Perl test code (about a year if all goes well). The exception is pgBackRest::Version which requires more work to migrate since it is used to track pgBackRest versions.
2020-03-10 21:12:44 +02:00
use pgBackRestTest::Env::InfoCommon;
use pgBackRestTest::Env::Manifest;
####################################################################################################################################
# backupRegExpGet
#
# Generate a regexp depending on the backups that need to be found.
####################################################################################################################################
sub backupRegExpGet
{
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$bFull,
$bDifferential,
$bIncremental,
$bAnchor
) =
logDebugParam
(
__PACKAGE__ . '::backupRegExpGet', \@_,
{name => 'bFull', default => false},
{name => 'bDifferential', default => false},
{name => 'bIncremental', default => false},
{name => 'bAnchor', default => true}
);
# One of the types must be selected
if (!($bFull || $bDifferential || $bIncremental))
{
confess &log(ASSERT, 'at least one backup type must be selected');
}
# Standard regexp to match date and time formatting
my $strDateTimeRegExp = "[0-9]{8}\\-[0-9]{6}";
# Start the expression with the anchor if requested, date/time regexp and full backup indicator
my $strRegExp = ($bAnchor ? '^' : '') . $strDateTimeRegExp . 'F';
# Add the diff and/or incr expressions if requested
if ($bDifferential || $bIncremental)
{
# If full requested then diff/incr is optional
if ($bFull)
{
$strRegExp .= "(\\_";
}
# Else diff/incr is required
else
{
$strRegExp .= "\\_";
}
# Append date/time regexp for diff/incr
$strRegExp .= $strDateTimeRegExp;
# Filter on both diff/incr
if ($bDifferential && $bIncremental)
{
$strRegExp .= '(D|I)';
}
# Else just diff
elsif ($bDifferential)
{
$strRegExp .= 'D';
}
# Else just incr
else
{
$strRegExp .= 'I';
}
# If full requested then diff/incr is optional
if ($bFull)
{
$strRegExp .= '){0,1}';
}
}
# Append the end anchor if requested
$strRegExp .= $bAnchor ? "\$" : '';
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'strRegExp', value => $strRegExp}
);
}
####################################################################################################################################
# backupLabel
#
# Get unique backup label.
####################################################################################################################################
sub backupLabel
{
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$oStorageRepo,
$strRepoBackupPath,
$strType,
$strBackupLabelLast,
$lTimestampStart
) =
logDebugParam
(
__PACKAGE__ . '::backupLabelFormat', \@_,
{name => 'oStorageRepo', trace => true},
{name => 'strRepoBackupPath', trace => true},
{name => 'strType', trace => true},
{name => 'strBackupLabelLast', required => false, trace => true},
{name => 'lTimestampStart', trace => true}
);
# Create backup label
my $strBackupLabel = backupLabelFormat($strType, $strBackupLabelLast, $lTimestampStart);
# Make sure that the timestamp has not already been used by a prior backup. This is unlikely for online backups since there is
# already a wait after the manifest is built but it's still possible if the remote and local systems don't have synchronized
# clocks. In practice this is most useful for making offline testing faster since it allows the wait after manifest build to
# be skipped by dealing with any backup label collisions here.
if ($oStorageRepo->list(
$strRepoBackupPath,
{strExpression =>
($strType eq CFGOPTVAL_BACKUP_TYPE_FULL ? '^' : '_') . timestampFileFormat(undef, $lTimestampStart) .
($strType eq CFGOPTVAL_BACKUP_TYPE_FULL ? 'F' : '(D|I)$')}) ||
$oStorageRepo->list(
"${strRepoBackupPath}/" . PATH_BACKUP_HISTORY . '/' . timestampFormat('%4d', $lTimestampStart),
{strExpression =>
($strType eq CFGOPTVAL_BACKUP_TYPE_FULL ? '^' : '_') . timestampFileFormat(undef, $lTimestampStart) .
($strType eq CFGOPTVAL_BACKUP_TYPE_FULL ? 'F' : '(D|I)\.manifest\.gz$'),
bIgnoreMissing => true}))
{
waitRemainder();
$strBackupLabel = backupLabelFormat($strType, $strBackupLabelLast, time());
}
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'strBackupLabel', value => $strBackupLabel, trace => true}
);
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
2017-01-30 20:59:00 +02:00
####################################################################################################################################
# Build PostgreSQL pages for testing
####################################################################################################################################
sub pageBuild
{
my $tPageSource = shift;
my $iBlockNo = shift;
Use static checksums in mock/all integration tests. Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests. We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 20:56:20 +02:00
my $iChecksum = shift;
Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
2017-01-30 20:59:00 +02:00
my $iWalId = shift;
my $iWalOffset = shift;
Use static checksums in mock/all integration tests. Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests. We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 20:56:20 +02:00
return
(defined($iWalId) ? pack('I', $iWalId) . pack('I', $iWalOffset) : substr($tPageSource, 0, 8)) . pack('S', $iChecksum) .
substr($tPageSource, 10);
Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
2017-01-30 20:59:00 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
####################################################################################################################################
# run
####################################################################################################################################
sub run
{
my $self = shift;
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
foreach my $rhRun
(
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
{vm => VM2, remote => false, tls => false, storage => GCS, encrypt => false, delta => true, compress => BZ2},
{vm => VM2, remote => true, tls => false, storage => AZURE, encrypt => true, delta => false, compress => GZ},
{vm => VM3, remote => false, tls => false, storage => POSIX, encrypt => false, delta => true, compress => ZST},
{vm => VM3, remote => true, tls => true, storage => S3, encrypt => true, delta => false, compress => LZ4},
{vm => VM4, remote => false, tls => false, storage => GCS, encrypt => false, delta => false, compress => GZ},
{vm => VM4, remote => true, tls => true, storage => S3, encrypt => true, delta => true, compress => ZST},
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
)
S3 repository support.
2017-06-12 16:52:32 +02:00
{
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
# Only run tests for this vm
Allow mock integration tests for all VM types. Previously the mock integration tests would be skipped for VMs other than the standard four used in CI. Now VMs outside the standard four will run the same tests as VM4 (currently U18).
2019-11-02 11:35:48 +02:00
next if ($rhRun->{vm} ne vmTest($self->vm()));
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
# Increment the run, log, and decide whether this unit test should be run
my $bRemote = $rhRun->{remote};
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
my $bTls = $rhRun->{tls};
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
my $strStorage = $rhRun->{storage};
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
my $bEncrypt = $rhRun->{encrypt};
my $bDeltaBackup = $rhRun->{delta};
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
my $strCompressType = $rhRun->{compress};
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Increment the run, log, and decide whether this unit test should be run
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
if (!$self->begin("rmt ${bRemote}, tls ${bTls}, storage ${strStorage}, enc ${bEncrypt}, delta ${bDeltaBackup}")) {next}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create hosts, file object, and config
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
my ($oHostDbPrimary, $oHostDbStandby, $oHostBackup) = $self->setup(
Add TLS Server. The TLS server is an alternative to using SSH for protocol connections to remote hosts. This command is currently experimental and intended only for trial and testing. As such, the new commands and options will not show up in the command-line help unless directly requested.
2021-10-18 20:32:41 +02:00
true, $self->expect(),
{bHostBackup => $bRemote, bTls => $bTls, strStorage => $strStorage, bRepoEncrypt => $bEncrypt,
strCompressType => NONE});
Reduce process redundancy in mock tests.
2017-06-22 00:47:25 +02:00
# If S3 set process max to 2. This seems like the best place for parallel testing since it will help speed S3 processing
# without slowing down the other tests too much.
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
if ($strStorage eq S3)
Reduce process redundancy in mock tests.
2017-06-22 00:47:25 +02:00
{
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({&CFGDEF_SECTION_GLOBAL => {'process-max' => 2}});
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->configUpdate({&CFGDEF_SECTION_GLOBAL => {'process-max' => 2}});
Reduce process redundancy in mock tests.
2017-06-22 00:47:25 +02:00
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
# Reduce log level to warn because parallel tests do not create deterministic logs
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({&CFGDEF_SECTION_GLOBAL => {'log-level-console' => lc(WARN)}});
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->configUpdate({&CFGDEF_SECTION_GLOBAL => {'log-level-console' => lc(WARN)}});
Reduce process redundancy in mock tests.
2017-06-22 00:47:25 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Get base time
my $lTime = time() - 10000;
# Build the manifest
my %oManifest;
Rename PGBACKREST/BACKREST constants to PROJECT. This brings consistency between the C and Perl constants and allows for easier code reuse.
2018-11-25 02:05:03 +02:00
$oManifest{&INI_SECTION_BACKREST}{&INI_KEY_VERSION} = PROJECT_VERSION;
$oManifest{&INI_SECTION_BACKREST}{&INI_KEY_FORMAT} = REPOSITORY_FORMAT;
Set archive-check option in manifest correctly when offline. Archive check does not run when in offline backup mode but the option was set to true in the manifest. It's harmless since these options are informational only but it could cause confusion when debugging.
2019-11-28 15:27:21 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_ARCHIVE_CHECK} = JSON::PP::false;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_ARCHIVE_COPY} = JSON::PP::true;
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_BACKUP_STANDBY} = JSON::PP::false;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_BUFFER_SIZE} = 16384;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_CHECKSUM_PAGE} = JSON::PP::true;
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_COMPRESS} = JSON::PP::false;
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_COMPRESS_TYPE} = NONE;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_COMPRESS_LEVEL} = 3;
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_COMPRESS_LEVEL_NETWORK} = $bRemote ? 1 : 3;
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_HARDLINK} = JSON::PP::false;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_ONLINE} = JSON::PP::false;
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = $strStorage eq S3 ? 2 : 1;
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_DELTA} = JSON::PP::false;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
if ($bEncrypt)
Repository encryption support. Contributed by Cynthia Shang.
2017-11-06 19:51:12 +02:00
{
$oManifest{&INI_SECTION_CIPHER}{&INI_KEY_CIPHER_PASS} = 'REPLACEME';
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_DB}{&MANIFEST_KEY_CATALOG} = 201409291;
$oManifest{&MANIFEST_SECTION_BACKUP_DB}{&MANIFEST_KEY_CONTROL} = 942;
Automate generation of WAL and pg_control test files. The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-19 03:02:54 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_DB}{&MANIFEST_KEY_SYSTEM_ID} = 1000000000000000094;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_DB}{&MANIFEST_KEY_DB_VERSION} = PG_VERSION_94;
$oManifest{&MANIFEST_SECTION_BACKUP_DB}{&MANIFEST_KEY_DB_ID} = 1;
$oManifest{&MANIFEST_SECTION_BACKUP_TARGET}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_PATH} =
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbBasePath();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_TARGET}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_TYPE} = MANIFEST_VALUE_PATH;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_FILE_PGVERSION, PG_VERSION_94,
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
'184473f470864e067ee3a22e64b47b0a1c356f29', $lTime, undef, true);
# Load sample page
Remove obsolete methods in pgBackRest::Storage::Storage module. All the methods in this module will need to be implemented via the command-line in order to get rid of LibC, so the first step is to reduce the code in the module as much as possible. First remove storageDb() and use storageTest() instead. Then create storageTest() using pgBackRestTest::Common::Storage which has no dependencies on LibC. Now the only storage using the LibC interface is storageRepo(). Remove all link functions since those operations cannot be performed on a repo unless it is Posix, in which case the LibC interface is not needed. Same for owner(). Remove pathSync() because syncs are not required in the tests. No test data is reused after a crash. Path create/exists functions should never be explicitly performed on a repo so remove those. File exists can be implemented by calling info() instead. Remove encryption detection functions which were only used by Backup/Archive::Info reconstruct() which are now obsolete. Remove all filters except pgBackRest::Storage::Filter::CipherBlock since they are not being used. That also means there are no filters returning results so remove all the result code. Move hashSize() and pathAbsolute() into pgBackRest::Storage::Base where they can be shared between pgBackRest::Storage::Storage and pgBackRestTest::Common::Storage.
2020-03-06 21:10:09 +02:00
my $tBasePage = ${storageTest()->get($self->dataPath() . '/page.bin')};
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
my $iBasePageChecksum = 0x1B99;
# Create base path
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base');
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/1');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/1/12000', $tBasePage,
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
'22c98d248ff548311eda88559e4a8405ed77c003', $lTime);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/1/' . DB_FILE_PGVERSION,
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
PG_VERSION_94, '184473f470864e067ee3a22e64b47b0a1c356f29', $lTime, '660');
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!$bRemote)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->executeSimple(
'chown 7777 ' . $oHostDbPrimary->dbBasePath() . '/base/1/' . DB_FILE_PGVERSION, undef, 'root');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{MANIFEST_TARGET_PGDATA . '/base/1/' . DB_FILE_PGVERSION}
{&MANIFEST_SUBKEY_USER} = INI_FALSE;
}
my $tPageInvalid17000 = $tBasePage . $tBasePage;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/16384');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/16384/17000', $tPageInvalid17000,
'e0101dd8ffb910c9c202ca35b5f828bcb9697bed', $lTime, undef, undef, '1');
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/16384/' . DB_FILE_PGVERSION,
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
PG_VERSION_94, '184473f470864e067ee3a22e64b47b0a1c356f29', $lTime);
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!$bRemote)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->executeSimple(
'chown :7777 ' . $oHostDbPrimary->dbBasePath() . '/base/16384/' . DB_FILE_PGVERSION, undef, 'root');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{MANIFEST_TARGET_PGDATA . '/base/16384/' . DB_FILE_PGVERSION}
{&MANIFEST_SUBKEY_GROUP} = INI_FALSE;
}
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
my $tPageValid =
Use static checksums in mock/all integration tests. Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests. We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 20:56:20 +02:00
pageBuild($tBasePage, 0, 0x1b99) .
pageBuild($tBasePage, 1, 0x1b9a) .
pageBuild($tBasePage, 2, 0x1b97) .
Retry on page checksum validation failure during backup. Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL. Also remove tests made redundant by the test merge in b4897077.
2022-02-23 20:05:53 +02:00
("\0" x 8192);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Retry on page checksum validation failure during backup. Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL. Also remove tests made redundant by the test merge in b4897077.
2022-02-23 20:05:53 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768/33000', $tPageValid, '1d73a0052828531770e7c155aeb22338e017e196',
Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
2017-01-30 20:59:00 +02:00
$lTime);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fixed an issue where tables over 1GB would report page checksum errors after the first segment. Reported by Stephen Frost.
2017-03-01 21:15:46 +02:00
my $iBlockOffset = 32767 * 131072;
my $tPageValidSeg32767 =
Use static checksums in mock/all integration tests. Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests. We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 20:56:20 +02:00
pageBuild($tBasePage, $iBlockOffset + 0, 0xf7de) .
pageBuild($tBasePage, $iBlockOffset + 1, 0xf7df) .
Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. Reported by Stephen Frost.
2017-03-13 21:50:13 +02:00
("\0" x 8192) .
Retry on page checksum validation failure during backup. Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL. Also remove tests made redundant by the test merge in b4897077.
2022-02-23 20:05:53 +02:00
("\0" x 8192);
Fixed an issue where tables over 1GB would report page checksum errors after the first segment. Reported by Stephen Frost.
2017-03-01 21:15:46 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Fixed an issue where tables over 1GB would report page checksum errors after the first segment. Reported by Stephen Frost.
2017-03-01 21:15:46 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768/33000.32767', $tPageValidSeg32767,
Retry on page checksum validation failure during backup. Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL. Also remove tests made redundant by the test merge in b4897077.
2022-02-23 20:05:53 +02:00
'1d11c42e6080e805a7b12bf9f83f4def548d92ac', $lTime);
Fixed an issue where tables over 1GB would report page checksum errors after the first segment. Reported by Stephen Frost.
2017-03-01 21:15:46 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
my $tPageInvalid33001 =
Use static checksums in mock/all integration tests. Using static values serves as a better cross-check against the page checksum code. The downside is that these checksums may not work with some big endian systems but in that case neither will the unit tests. We can also remove the page checksum interface from LibC which brings us one step closer to eliminating it.
2020-03-05 20:56:20 +02:00
pageBuild($tBasePage, 1, 0x1b9a) .
pageBuild($tBasePage, 1, 0x1b9a) .
pageBuild($tBasePage, 2, 0x1b97) .
pageBuild($tBasePage, 0, 0x1b99) .
pageBuild($tBasePage, 0, 0x1b99) .
pageBuild($tBasePage, 0, 0x1b99) .
pageBuild($tBasePage, 6, 0x1b9b) .
pageBuild($tBasePage, 0, 0x1b99);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768/33001', $tPageInvalid33001,
'6bf316f11d28c28914ea9be92c00de9bea6d9a6b', $lTime, undef, undef, '0, [3, 5], 7');
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768/' . DB_FILE_PGVERSION,
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
PG_VERSION_94, '184473f470864e067ee3a22e64b47b0a1c356f29', $lTime);
# Create global path
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'global');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Support configurable WAL segment size. PostgreSQL 11 introduces configurable WAL segment sizes, from 1MB to 1GB. There are two areas that needed to be updated to support this: building the archive-get queue and checking that WAL has been archived after a backup. Both operations require the WAL segment size to properly build a list. Checking the archive after a backup is still implemented in Perl and has an active database connection, so just get the WAL segment size from the database. The archive-get command does not have a connection to the database, so get the WAL segment size from pg_control instead. This requires a deeper inspection of pg_control than has been done in the past, so it seemed best to copy the relevant data structures from each version of PostgreSQL and build a generic interface layer to address them. While this approach is a bit verbose, it has the advantage of being relatively simple, and can easily be updated for new versions of PostgreSQL. Since the integration tests generate pg_control files for testing, teach Perl how to generate files with the correct offsets for both 32-bit and 64-bit architectures.
2018-09-25 11:24:42 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, DB_FILE_PGCONTROL, '[replaceme]',
$self->archBits() == 32 ? '8107e546c59c72a8c1818fc3610d7cc1e5623660' : '4c77c900f7af0d9ab13fa9982051a42e0b637f6c',
$lTime - 100, undef, true);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Copy pg_control
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$self->controlGenerate($oHostDbPrimary->dbBasePath(), PG_VERSION_94);
utime($lTime - 100, $lTime - 100, $oHostDbPrimary->dbBasePath() . '/' . DB_FILE_PGCONTROL)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
or confess &log(ERROR, "unable to set time");
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{MANIFEST_TARGET_PGDATA . '/' . DB_FILE_PGCONTROL}
{&MANIFEST_SUBKEY_SIZE} = 8192;
# Create tablespace path
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGTBLSPC);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create paths/files to ignore
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!$bRemote)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
# Create temp dir and file that will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/' . DB_FILE_PREFIX_TMP);
$oHostDbPrimary->dbFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/' . DB_FILE_PREFIX_TMP . '/' . DB_FILE_PREFIX_TMP . '.1', 'IGNORE');
# Create pg_dynshmem dir and file - only file will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGDYNSHMEM);
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGDYNSHMEM . '/anything.tmp', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create pg_notify dir and file - only file will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGNOTIFY);
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGNOTIFY . '/anything.tmp', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create pg_replslot dir and file - only file will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGREPLSLOT);
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGREPLSLOT . '/anything.tmp', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create pg_serial dir and file - only file will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSERIAL);
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSERIAL . '/anything.tmp', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fix comment typos. This includes some variable names in tests which don't seem important enough for their own commits. Contributed by Josh Soref.
2019-08-26 18:05:36 +02:00
# Create pg_snapshots dir and file - only file will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSNAPSHOTS);
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSNAPSHOTS . '/anything.tmp', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create pg_stat_tmp dir and file - only file will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSTATTMP);
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSTATTMP . '/anything.tmp', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create pg_subtrans dir and file - only file will be ignored
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSUBTRANS);
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_PATH_PGSUBTRANS . '/anything.tmp', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# More files to ignore
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_FILE_POSTGRESQLAUTOCONFTMP, 'IGNORE');
Rename PostgreSQL pid file constants and tests.
2021-10-14 01:36:59 +02:00
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_FILE_POSTMTROPTS, 'IGNORE');
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_FILE_RECOVERYCONF, 'IGNORE');
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, DB_FILE_RECOVERYDONE, 'IGNORE');
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'global/' . DB_FILE_PGINTERNALINIT, 'IGNORE');
Exclude temporary and unlogged relation (table/index) files from backup. Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced. Contributed by Cynthia Shang.
2018-07-31 00:53:34 +02:00
# Unlog and temp files to ignore (unlog _init will NOT be ignored)
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768/44000_init', $tPageValid,
Retry on page checksum validation failure during backup. Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL. Also remove tests made redundant by the test merge in b4897077.
2022-02-23 20:05:53 +02:00
'1d73a0052828531770e7c155aeb22338e017e196', $lTime);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768/44000', 'IGNORE');
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/32768/t333_44000', 'IGNORE');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
Move help/version integration tests to mock/all. Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
2018-04-08 18:16:32 +02:00
# Help and Version. These have complete unit tests, so here just make sure there is output from the command line.
#---------------------------------------------------------------------------------------------------------------------------
if ($self->runCurrent() == 1)
{
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->executeSimple($self->backrestExe() . " version", {oLogTest => $self->expect()});
$oHostDbPrimary->executeSimple($self->backrestExe() . " help version", {oLogTest => $self->expect()});
Move help/version integration tests to mock/all. Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
2018-04-08 18:16:32 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Full backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
my $strType = CFGOPTVAL_BACKUP_TYPE_FULL;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
my $strOptionalParam = '--manifest-save-threshold=3';
my $strTestPoint;
# Create the archive info file
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->stanzaCreate('create required data for stanza', {strOptionalParam => '--no-online'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
# Create a link to postgresql.conf
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
storageTest()->pathCreate($oHostDbPrimary->dbPath() . '/pg_config', {strMode => '0700', bCreateParent => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
testFileCreate(
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbPath() . '/pg_config/postgresql.conf', "listen_addresses = *\n", $lTime - 100);
testLinkCreate($oHostDbPrimary->dbPath() . '/pg_config/postgresql.conf.link', './postgresql.conf');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestLinkCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'postgresql.conf',
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
'../pg_config/postgresql.conf', true);
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
# Create a link to pg_hba.conf
testFileCreate(
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbPath() . '/pg_config/pg_hba.conf', "CONTENTS\n", $lTime - 100);
testLinkCreate($oHostDbPrimary->dbPath() . '/pg_config/pg_hba.conf.link', './pg_hba.conf');
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestLinkCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_hba.conf',
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
'../pg_config/pg_hba.conf', true);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create stat directory link and file
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
storageTest()->pathCreate($oHostDbPrimary->dbPath() . '/pg_stat', {strMode => '0700', bCreateParent => true});
$oHostDbPrimary->manifestLinkCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_stat', '../pg_stat');
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA . '/pg_stat', 'global.stat', 'stats',
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
'e350d5ce0153f3e22d5db21cf2a4eff00f3ee877', $lTime - 100, undef, true);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_clog');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Improve support for special characters in filenames. % characters caused issues in backup/restore due to filenames being appended directly into a format string. Reserved XML characters (<>&') caused issues in the S3 driver due to improper escaping. Add a file with all common special characters to regression testing.
2018-09-10 16:54:34 +02:00
# Create file with special characters
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Use minio for integration testing. ScalityS3 has not received any maintenance in years and is slow to start which is bad for testing. Replace it with minio which starts quickly and ships as a single executable or a tiny container. Minio has stricter limits on allowable characters but should still provide enough coverage to show that our encoding is working correctly. This commit also includes the upgrade to openssl 1.1.1 in the Ubuntu 18.04 container.
2019-07-03 04:20:35 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'special-!_.*\'()&!@;:+,?', undef, undef, $lTime, undef, true);
Improve support for special characters in filenames. % characters caused issues in backup/restore due to filenames being appended directly into a format string. Reserved XML characters (<>&') caused issues in the S3 driver due to improper escaping. Add a file with all common special characters to regression testing.
2018-09-10 16:54:34 +02:00
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = 1;
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
my $strFullBackup = $oHostBackup->backup(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strType, 'create pg_stat link, pg_clog dir',
{oExpectedManifest => \%oManifest,
Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes.
2017-09-19 16:14:18 +02:00
strOptionalParam => $strOptionalParam .
# Pass ssh path to make sure it is used
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
($bRemote ? ' --cmd-ssh=/usr/bin/ssh' : '') .
Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes.
2017-09-19 16:14:18 +02:00
# Pass bogus ssh port to make sure it is passed through the protocol layer (it won't be used)
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
($bRemote ? ' --pg1-port=9999' : '') .
Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes.
2017-09-19 16:14:18 +02:00
# Pass bogus socket path to make sure it is passed through the protocol layer (it won't be used)
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
($bRemote ? ' --pg1-socket-path=/test_socket_path' : '') .
' --buffer-size=16384 --checksum-page --process-max=1',
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 22:24:34 +02:00
strRepoType => $strStorage eq POSIX ? CIFS : undef, strTest => $strTestPoint, fTestDelay => 0});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = $strStorage eq S3 ? 2 : 1;
Integration test improvements for disk and memory efficiency. Set log-level-file=off when more that one test will run. In this case is it impossible to see the logs anyway since they will be automatically cleaned up after the test. This improves performance pretty dramatically since trace-level logging is expensive. If a singe integration test is run then log-level-file is trace by default but can be changed with the --log-level-test-file option. Reduce buffer-size to 64k to save memory during testing and allow more processes to run in parallel. Update log replacement rules so that these options can change without affecting expect logs.
2019-12-17 22:23:07 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_BUFFER_SIZE} = 65536;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Stop operations and make sure the correct error occurs
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
if ($strStorage eq POSIX)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Remove start/stop --force integration tests in mock/all. These tests require test points which are not being implemented in the C code. This functionality is fully tested in the command/control unit tests so integration tests are no longer required.
2019-11-25 14:45:58 +02:00
# Test global stop
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->stop({bForce => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostBackup->backup(
$strType, 'global stop',
{oExpectedManifest => \%oManifest, iExpectedExitStatus => ERROR_STOP});
# Test stanza stop
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->stop({strStanza => $oHostDbPrimary->stanza()});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# This time a warning should be generated
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->stop({strStanza => $oHostDbPrimary->stanza()});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oHostBackup->backup(
$strType, 'stanza stop',
{oExpectedManifest => \%oManifest, iExpectedExitStatus => ERROR_STOP});
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->start({strStanza => $self->stanza()});
$oHostDbPrimary->start();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# This time a warning should be generated
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->start();
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Resume Full Backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_FULL;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# These files should never be backed up (this requires the next backup to do --force)
Rename PostgreSQL pid file constants and tests.
2021-10-14 01:36:59 +02:00
testFileCreate($oHostDbPrimary->dbBasePath() . '/' . DB_FILE_POSTMTRPID, 'JUNK');
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
testFileCreate($oHostDbPrimary->dbBasePath() . '/' . DB_FILE_BACKUPLABELOLD, 'JUNK');
testFileCreate($oHostDbPrimary->dbBasePath() . '/' . DB_FILE_RECOVERYCONF, 'JUNK');
testFileCreate($oHostDbPrimary->dbBasePath() . '/' . DB_FILE_RECOVERYDONE, 'JUNK');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create files in root tblspc paths that should not be copied or deleted.
# This will be checked later after a --force restore.
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
my $strDoNotDeleteFile = $oHostDbPrimary->tablespacePath(1, 2) . '/donotdelete.txt';
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
storageTest()->pathCreate(dirname($strDoNotDeleteFile), {strMode => '0700', bCreateParent => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
testFileCreate($strDoNotDeleteFile, 'DONOTDELETE-1-2');
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
storageTest()->pathCreate($oHostDbPrimary->tablespacePath(1), {strMode => '0700', bCreateParent => true});
testFileCreate($oHostDbPrimary->tablespacePath(1) . '/donotdelete.txt', 'DONOTDELETE-1');
storageTest()->pathCreate($oHostDbPrimary->tablespacePath(2), {strMode => '0700', bCreateParent => true});
testFileCreate($oHostDbPrimary->tablespacePath(2) . '/donotdelete.txt', 'DONOTDELETE-2');
storageTest()->pathCreate($oHostDbPrimary->tablespacePath(2, 2), {strMode => '0700', bCreateParent => true});
testFileCreate($oHostDbPrimary->tablespacePath(2, 2) . '/donotdelete.txt', 'DONOTDELETE-2-2');
storageTest()->pathCreate($oHostDbPrimary->tablespacePath(11), {strMode => '0700', bCreateParent => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
# Resume by copying the valid full backup over the last aborted full backup if it exists, or by creating a new path
my $strResumeBackup = (storageRepo()->list(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->repoBackupPath(), {strExpression => backupRegExpGet(true, true, true), strSortOrder => 'reverse'}))[0];
Fix backup labels in mock/all resume integration tests. These were not getting updated to match the directory name when the manifests were copied. The Perl code didn't care but the C code expects labels to be set correctly.
2019-12-06 18:48:41 +02:00
my $strResumeLabel = $strResumeBackup ne $strFullBackup ?
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$strResumeBackup : backupLabel(storageRepo(), $oHostBackup->repoBackupPath(), $strType, undef, time());
my $strResumePath = $oHostBackup->repoBackupPath($strResumeLabel);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
forceStorageRemove(storageRepo(), $strResumePath, {bRecurse => true});
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
forceStorageMove(storageRepo(), $oHostBackup->repoBackupPath($strFullBackup), $strResumePath);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
# Set ownership on base directory to bogus values
if (!$bRemote)
{
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->executeSimple('chown 7777:7777 ' . $oHostDbPrimary->dbBasePath(), undef, 'root');
$oHostDbPrimary->executeSimple('chmod 777 ' . $oHostDbPrimary->dbBasePath(), undef, 'root');
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_USER} = INI_FALSE;
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_GROUP} = INI_FALSE;
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_MODE} = '0777';
}
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
$oHostBackup->manifestMunge(
basename($strResumePath),
Fix backup labels in mock/all resume integration tests. These were not getting updated to match the directory name when the manifests were copied. The Perl code didn't care but the C code expects labels to be set correctly.
2019-12-06 18:48:41 +02:00
{&MANIFEST_SECTION_BACKUP => {&MANIFEST_KEY_LABEL => $strResumeLabel},
&MANIFEST_SECTION_TARGET_FILE =>
{(&MANIFEST_TARGET_PGDATA . '/' . &DB_FILE_PGVERSION) => {&MANIFEST_SUBKEY_CHECKSUM => undef}}},
Improvements to Ini.pm. * Refactor Ini.pm to facilitate testing. * Complete statement/branch coverage for Ini.pm. * Improved functions used to test/munge manifest and info files.
2017-04-10 19:24:45 +02:00
false);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
# Remove the main manifest so the backup appears aborted
forceStorageRemove(storageRepo(), "${strResumePath}/" . FILE_MANIFEST);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create a temp file in backup temp root to be sure it's deleted correctly
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
my $strTempFile = "${strResumePath}/file.tmp";
S3 repository support.
2017-06-12 16:52:32 +02:00
Azure support for repository storage. Azure and Azure-compatible object stores can now be used for repository storage. Currently only shared key authentication is supported but SAS will be added soon.
2020-07-02 22:24:34 +02:00
if ($strStorage ne POSIX)
S3 repository support.
2017-06-12 16:52:32 +02:00
{
Remove dependency on aws cli for testing. This tool was only being used it a few places but was a pretty large dependency. Rework the forceStorageMove() code using our storage layer and replace one aws cli cp with a storage put. Also, remove the Dockerfile that was once used to build the Scality S3 test container.
2019-10-09 20:38:24 +02:00
storageRepo()->put($strTempFile, "TEMP");
S3 repository support.
2017-06-12 16:52:32 +02:00
}
else
{
Remove pgbackrest test user. This user was created before we tested in containers to ensure isolation between the pg and repo hosts which were then just directories. The downside is that this resulted in a lot of sudos to set the pgbackrest user and to remove files which did not belong to the main test user. Containers provide isolation without needing separate users so we can now safely remove the pgbackrest user. This allows us to remove most sudos, except where they are explicitly needed in tests. While we're at it, remove the code that installed the Perl C library (which also required sudo) and simply add the build path to @INC instead.
2019-10-12 15:45:18 +02:00
executeTest("touch ${strTempFile}", {bRemote => $bRemote});
S3 repository support.
2017-06-12 16:52:32 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. Contributed by Cynthia Shang.
2018-08-24 22:50:33 +02:00
# Add zero-sized file
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'zero_from_start', undef,
Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. Contributed by Cynthia Shang.
2018-08-24 22:50:33 +02:00
undef, $lTime, undef, true);
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
# Add files for testing backups when time changes but content doesn't, and when content changes but time and size don't
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'changetime.txt', 'SIZE', '88087292ed82e26f3eb824d0bffc05ccf7a30f8d', $lTime,
undef, true);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'changecontent.txt', 'CONTENT', '238a131a3e8eb98d1fc5b27d882ca40b7618fd2a', $lTime,
undef, true);
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
# Create files to be excluded with the --exclude option
$oHostBackup->configUpdate(
{(CFGDEF_SECTION_GLOBAL . ':backup') =>
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{'exclude' => ['postgresql.auto.conf', 'pg_log/', 'pg_log2', 'apipe']}});
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbLinkCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'postgresql.auto.conf',
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
'../pg_config/postgresql.conf', true);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_log');
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_log/logfile', 'IGNORE');
$oHostDbPrimary->dbPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_log2');
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_log2/logfile', 'IGNORE');
executeTest('mkfifo ' . $oHostDbPrimary->dbBasePath() . '/apipe');
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strFullBackup = $oHostBackup->backup(
$strType, 'resume',
Remove obsolete integration tests from mock/all. The protocol timeout tests have been superceded by unit tests. The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto. It didn't hurt anything but it did add 200ms to each test where it appeared. Catalog and control version tests were redundant. The database version and system id tests covered the important code paths and the C code gets these values from a lookup table. Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 23:06:27 +02:00
{oExpectedManifest => \%oManifest,
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
strOptionalParam => '--force --checksum-page' . ($bDeltaBackup ? ' --delta' : '')});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Use the PG_FILE_POSTMTRPID constant where appropriate. Do the same in Perl with the MANIFEST_FILE_POSTMTRPID constant.
2022-01-20 15:41:05 +02:00
# Remove pid so restore will succeed (the rest will be cleaned up by the delta)
Rename PostgreSQL pid file constants and tests.
2021-10-14 01:36:59 +02:00
storageTest->remove($oHostDbPrimary->dbBasePath() . '/' . DB_FILE_POSTMTRPID);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Restore - tests various mode, extra files/paths, missing files/paths
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Munge permissions/modes on files that will be fixed by the restore
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!$bRemote)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->executeSimple(
"chown :7777 " . $oHostDbPrimary->dbBasePath() . '/base/1/' . DB_FILE_PGVERSION, undef, 'root');
$oHostDbPrimary->executeSimple(
"chmod 600 " . $oHostDbPrimary->dbBasePath() . '/base/1/' . DB_FILE_PGVERSION, undef, 'root');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Create a path and file that are not in the manifest
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbPathCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'deleteme');
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'deleteme/deleteme.txt', 'DELETEME');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Change path mode
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbPathMode(\%oManifest, MANIFEST_TARGET_PGDATA, 'base', '0777');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Remove a path
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbPathRemove(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_clog');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Remove a file
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbFileRemove(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/16384/17000');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
# Restore will set invalid user and group to root since the base path user/group are also invalid
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!$bRemote)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 13:52:02 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_USER} = 'root';
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_GROUP} = 'root';
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{MANIFEST_TARGET_PGDATA . '/base/1/' . DB_FILE_PGVERSION}
{&MANIFEST_SUBKEY_USER} = 'root';
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{MANIFEST_TARGET_PGDATA . '/base/16384/' . DB_FILE_PGVERSION}
{&MANIFEST_SUBKEY_GROUP} = 'root';
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
'add and delete files', $strFullBackup,
{rhExpectedManifest => \%oManifest, bDelta => true, strUser => !$bRemote ? 'root' : undef,
strOptionalParam => ' --link-all' . ($bRemote ? ' --cmd-ssh=/usr/bin/ssh' : '')});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
# Remove excludes now that they just create noise in the log
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({(CFGDEF_SECTION_GLOBAL . ':backup') => {'exclude' => []}});
Reduce the scope of mock/all exclusion tests. Run exclusions only on the tests where they will have an effect to reduce churn in the expect logs when they change.
2019-12-02 00:47:47 +02:00
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
# Run again to fix permissions
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!$bRemote)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
# Reset the base path user and group for the next restore so files will be reset to the base path user/group
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->executeSimple(
'chown ' . TEST_USER . ':' . TEST_GROUP . ' ' . $oHostDbPrimary->dbBasePath(), undef, 'root');
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
$oHostBackup->manifestMunge(
$strFullBackup,
{&MANIFEST_SECTION_TARGET_PATH =>
{&MANIFEST_TARGET_PGDATA =>
Fix test writing "null" into manifest files. "null" is not allowed in the manifest format (null values should be missing instead) but Perl was treating the invalid values written by this test as if they were missing. Update the test code to remove the values rather than setting them to "null".
2019-08-18 21:29:18 +02:00
{&MANIFEST_SUBKEY_USER => undef, &MANIFEST_SUBKEY_GROUP => undef}}},
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
false);
Simplify base path mode in mock/all integration tests. Change the mode back to 0700 earlier to reduce churn in the expect logs. This will be especially important in a future commit that gets the defaults exclusively from the base path.
2022-01-21 15:52:51 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_USER} = TEST_USER;
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_GROUP} = TEST_GROUP;
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
Simplify base path mode in mock/all integration tests. Change the mode back to 0700 earlier to reduce churn in the expect logs. This will be especially important in a future commit that gets the defaults exclusively from the base path.
2022-01-21 15:52:51 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{MANIFEST_TARGET_PGDATA . '/base/1/' . DB_FILE_PGVERSION}
{&MANIFEST_SUBKEY_USER} = TEST_USER;
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{MANIFEST_TARGET_PGDATA . '/base/16384/' . DB_FILE_PGVERSION}
{&MANIFEST_SUBKEY_GROUP} = TEST_GROUP;
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
'fix permissions', $strFullBackup,
{rhExpectedManifest => \%oManifest, bDelta => true, strUser => 'root',
strOptionalParam => ' --link-all --log-level-console=detail'});
Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 19:46:35 +02:00
# Fix and remove files that are now owned by root
Update last tests that required sudo. All tests should now run in a sudo-less environment.
2019-10-16 17:05:24 +02:00
$oHostBackup->executeSimple('chown -R ' . TEST_USER . ':' . TEST_GROUP . ' ' . $oHostBackup->logPath(), undef, 'root');
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->executeSimple('rm -rf ' . $oHostDbPrimary->lockPath() . '/*', undef, 'root');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
# Change an existing link to the wrong directory
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbFileRemove(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_stat');
$oHostDbPrimary->dbLinkCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_stat', '../wrong');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
'fix broken symlink', $strFullBackup,
{rhExpectedManifest => \%oManifest, bDelta => true,
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
strOptionalParam => ' --link-all' . ($bRemote ? ' --compress-level-network=0' : '')});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
# Restore links as directories
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Munge the user to make sure it gets reset on the next run
$oHostBackup->manifestMunge(
Improvements to Ini.pm. * Refactor Ini.pm to facilitate testing. * Complete statement/branch coverage for Ini.pm. * Improved functions used to test/munge manifest and info files.
2017-04-10 19:24:45 +02:00
$strFullBackup,
{&MANIFEST_SECTION_TARGET_FILE =>
{&MANIFEST_FILE_PGCONTROL => {&MANIFEST_SUBKEY_USER => 'bogus', &MANIFEST_SUBKEY_GROUP => 'bogus'}}},
false);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Restore succeeds
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestLinkMap(\%oManifest, MANIFEST_TARGET_PGDATA . '/pg_stat');
$oHostDbPrimary->manifestLinkMap(\%oManifest, MANIFEST_TARGET_PGDATA . '/postgresql.conf');
$oHostDbPrimary->manifestLinkMap(\%oManifest, MANIFEST_TARGET_PGDATA . '/pg_hba.conf');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
'restore links as directories', $strFullBackup,
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
{rhExpectedManifest => \%oManifest, bDelta => true, bForce => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
# No longer need pg_hba.conf since it is no longer a link and doesn't provide additional coverage
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileRemove(\%oManifest, MANIFEST_TARGET_PGDATA, 'pg_hba.conf');
Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. Reported by Vitaliy Kukharik.
2018-05-02 18:19:54 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Incr backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_INCR;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestReference(\%oManifest, $strFullBackup);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Simplify base path mode in mock/all integration tests. Change the mode back to 0700 earlier to reduce churn in the expect logs. This will be especially important in a future commit that gets the defaults exclusively from the base path.
2022-01-21 15:52:51 +02:00
# Fix mode on base path so defaults are simpler in backups
if (!$bRemote)
{
$oHostDbPrimary->executeSimple('chmod 700 ' . $oHostDbPrimary->dbBasePath(), undef, 'root');
$oManifest{&MANIFEST_SECTION_TARGET_PATH}{&MANIFEST_TARGET_PGDATA}{&MANIFEST_SUBKEY_MODE} = '0700';
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Add tablespace 1
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestTablespaceCreate(\%oManifest, 1);
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/1', '16384');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/1', '16384/tablespace1.txt', 'TBLSPCB',
'14c44cef6287269b08d41de489fd492bb9fc795d', $lTime - 100, undef, undef, false);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(\%oManifest, MANIFEST_TARGET_PGDATA, 'badchecksum.txt', 'BADCHECKSUM',
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
'f927212cd08d11a42a666b2f04235398e9ceeb51', $lTime, undef, true);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'changesize.txt', 'SIZE', '88087292ed82e26f3eb824d0bffc05ccf7a30f8d', $lTime,
undef, true);
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Add zero-length file to mock/all test.
2018-07-05 21:40:50 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'zerosize.txt', '', undef, $lTime - 100, undef, true);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Create temp dir and file that will be ignored
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!$bRemote)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbPathCreate(\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/1', DB_FILE_PREFIX_TMP);
$oHostDbPrimary->dbFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/1', DB_FILE_PREFIX_TMP . '/' . DB_FILE_PREFIX_TMP . '.1', 'IGNORE');
}
Remove test point infrastructure. 82df7e6f and 9856fef5 updated tests that used test points in preparation for the feature not being available in the C code. Since tests points are no longer used remove the infrastructure. Also remove one stray --test option in mock/all that was essentially a noop but no longer works now that the option has been removed.
2019-12-10 20:16:47 +02:00
my $strBackup = $oHostBackup->backup($strType, 'add tablespace 1', {oExpectedManifest => \%oManifest});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Resume Incr Backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_INCR;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
# Create resumable backup from last backup
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$strResumeLabel = backupLabel(storageRepo(), $oHostBackup->repoBackupPath(), $strType, substr($strBackup, 0, 16), time());
$strResumePath = $oHostBackup->repoBackupPath($strResumeLabel);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
forceStorageRemove(storageRepo(), $strResumePath);
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
forceStorageMove(storageRepo(), $oHostBackup->repoBackupPath($strBackup), $strResumePath);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
# Munge manifest so the resumed file in the repo appears to be bad
Create test matrix for mock/all to increase coverage and reduce tests. The same test configurations are run on all four test VMs, which seems a real waste of resources. Vary the tests per VM to increase coverage while reducing the total number of tests. Be sure to include each major feature (remote, s3, encryption) in each VM at least once.
2019-03-02 15:01:02 +02:00
if ($bEncrypt || $bRemote)
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
{
$oHostBackup->manifestMunge(
basename($strResumePath),
{&MANIFEST_SECTION_TARGET_FILE =>
{(&MANIFEST_TARGET_PGDATA . '/badchecksum.txt') => {&MANIFEST_SUBKEY_CHECKSUM => BOGUS}}},
false);
}
# Change contents of resumed file without changing size so it will throw a nasty error about the repo having been corrupted
else
{
storageRepo()->put("${strResumePath}/pg_data/badchecksum.txt", 'BDDCHECKSUM');
}
Fix backup labels in mock/all resume integration tests. These were not getting updated to match the directory name when the manifests were copied. The Perl code didn't care but the C code expects labels to be set correctly.
2019-12-06 18:48:41 +02:00
# Write correct label into resumable manifest
$oHostBackup->manifestMunge(
basename($strResumePath), {&MANIFEST_SECTION_BACKUP => {&MANIFEST_KEY_LABEL => $strResumeLabel}},false);
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
# Change contents/size of a db file to make sure it recopies (and does not resume)
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'changesize.txt', 'SIZE+MORE', '3905d5be2ec8d67f41435dab5e0dcda3ae47455d', $lTime,
undef, true);
# Change contents/time of a db file to make sure it recopies (and does not resume)
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/1', '16384/tablespace1.txt', 'TBLSPC1',
'd85de07d6421d90aa9191c11c889bfde43680f0f', $lTime, undef, undef, false);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
# Remove the main manifest so the backup appears aborted
forceStorageRemove(storageRepo(), "${strResumePath}/" . FILE_MANIFEST);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Add tablespace 2
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestTablespaceCreate(\%oManifest, 2);
$oHostDbPrimary->manifestPathCreate(\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/2', '32768');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/2', '32768/tablespace2.txt', 'TBLSPC2',
'dc7f76e43c46101b47acc55ae4d593a9e6983578', $lTime, undef, undef, false);
Additional backup exclusions. * Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup. * Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-04 14:26:57 +02:00
# Make sure pg_internal.init is ignored in tablespaces
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->dbFileCreate(\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/2', '32768/' . DB_FILE_PGINTERNALINIT, 'IGNORE');
Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. Reported by blogh.
2017-03-02 20:50:29 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Also create tablespace 11 to be sure it does not conflict with path of tablespace 1
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestTablespaceCreate(\%oManifest, 11);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
# Change only the time to be in the past on a valid file and update the timestamp in the expected manifest
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
utime($lTime - 100, $lTime - 100, $oHostDbPrimary->dbBasePath() . '/changetime.txt')
or confess &log(ERROR, "unable to set time for file ".$oHostDbPrimary->dbBasePath() . '/changetime.txt');
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/changetime.txt'}{&MANIFEST_SUBKEY_TIMESTAMP} = $lTime - 100;
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
# Change the content of the changecontent file to be the same size but leave the timestamp the same on the file
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
storageTest()->put($oHostDbPrimary->dbBasePath() . '/changecontent.txt', 'CHGCONT');
utime($lTime, $lTime, $oHostDbPrimary->dbBasePath() . '/changecontent.txt')
or confess &log(ERROR, "unable to set time for file ".$oHostDbPrimary->dbBasePath() . '/changecontent.txt');
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. There are a number of cases where a checksum delta is more appropriate than the default time-based delta: * Timeline has switched since the prior backup * File timestamp is older than recorded in the prior backup * File size changed but timestamp did not * File timestamp is in the future compared to the start of the backup * Online option has changed since the prior backup A practical example is that checksum delta will be enabled after a failover to standby due to the timeline switch. In this case, timestamps can't be trusted and our recommendation has been to run a full backup, which can impact the retention schedule and requires manual intervention. Now, a checksum delta will be performed if the backup type is incr/diff. This means more CPU will be used during the backup but the backup size will be smaller and the retention schedule will not be impacted. Contributed by Cynthia Shang.
2018-11-01 17:31:25 +02:00
# The changecontent & changetime files have conditions that will force the delta option to be turned on which should result
# in the reference of changecontent to be removed but the reference to changetime to stay since the checksum wouldn't change
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/changecontent.txt'}{&MANIFEST_SUBKEY_CHECKSUM} =
"a094d94583e209556d03c3c5da33131a065f1689";
delete($oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/changecontent.txt'}{&MANIFEST_SUBKEY_REFERENCE});
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/changetime.txt'}{&MANIFEST_SUBKEY_TIMESTAMP} = $lTime - 100;
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = 1;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strBackup = $oHostBackup->backup(
Reduce process redundancy in mock tests.
2017-06-22 00:47:25 +02:00
$strType, 'resume and add tablespace 2',
Remove obsolete integration tests from mock/all. The protocol timeout tests have been superceded by unit tests. The TEST_BACKUP_RESUME test point was incorrectly included into a number of tests, probably a copy pasto. It didn't hurt anything but it did add 200ms to each test where it appeared. Catalog and control version tests were redundant. The database version and system id tests covered the important code paths and the C code gets these values from a lookup table. Finally, fix an incomplete update to the backup.info file while munging for tests.
2019-11-21 23:06:27 +02:00
{oExpectedManifest => \%oManifest,
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
strOptionalParam => '--process-max=1' . ($bDeltaBackup ? ' --delta' : '')});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = $strStorage eq S3 ? 2 : 1;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
if (!$bRemote)
{
# Remove the size-changed test file to avoid expect log churn
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileRemove(\%oManifest, MANIFEST_TARGET_PGDATA, 'changesize.txt');
Add checksum delta for incremental backups. Use checksums rather than timestamps to determine if files have changed. This is useful in cases where the timestamps may not be trustworthy, e.g. when performing an incremental after failing over to a standby. If checksum delta is enabled then checksums will be used for verification of resumed backups, even if they are full. Resumes have always used checksums to verify the files in the repository, enabling delta performs checksums on the database files as well. Note that the user must manually enable this feature in cases were it would be useful or just keep in enabled all the time. A future commit will address automatically enabling the feature in cases where it seems likely to be useful. Contributed by Cynthia Shang.
2018-09-19 17:12:45 +02:00
}
Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. All resumed backups should be considered inconsistent. A resumed backup can be identified by checking the log for the message "aborted backup of same type exists, will be cleaned to remove invalid files and resumed". Reported by David Youatt, Yogesh Sharma, Stephen Frost.
2018-07-03 20:01:57 +02:00
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
# Drop tablespace 11
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_DIFF;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Drop tablespace 11
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestTablespaceDrop(\%oManifest, 11);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = 1;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strBackup = $oHostBackup->backup(
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
$strType, 'drop tablespace 11',
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{oExpectedManifest => \%oManifest, strOptionalParam => '--process-max=1' . ($bDeltaBackup ? ' --delta' : '')});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = $strStorage eq S3 ? 2 : 1;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Restore
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Remap the base and tablespace paths
my %oRemapHash;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oRemapHash{&MANIFEST_TARGET_PGDATA} = $oHostDbPrimary->dbBasePath(2);
storageTest()->pathCreate($oHostDbPrimary->dbBasePath(2), {strMode => '0700', bCreateParent => true});
$oRemapHash{&MANIFEST_TARGET_PGTBLSPC . '/1'} = $oHostDbPrimary->tablespacePath(1, 2);
$oRemapHash{&MANIFEST_TARGET_PGTBLSPC . '/2'} = $oHostDbPrimary->tablespacePath(2, 2);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
'remap all paths', $strBackup, {rhExpectedManifest => \%oManifest, rhRemapHash => \%oRemapHash});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Restore (make sure file in root tablespace path is not deleted by --delta)
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
'ensure file in tblspc root remains after --delta', $strBackup,
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
{rhExpectedManifest => \%oManifest, rhRemapHash => \%oRemapHash, bDelta => true});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
if (!-e $strDoNotDeleteFile)
{
confess "${strDoNotDeleteFile} was deleted by --delta";
}
# Incr Backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_INCR;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestReference(\%oManifest, $strBackup);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Update Perl page checksum expression. This expression determines which files contain page checksums but it was also including the directory above the relation directories. In a real PostgreSQL installation this not a problem because these directories don't contain any files. However, our tests place a file in `base` which the Perl code thought should have page checksums while the new C code says no. Update the expression to document the change and avoid churn in the expect logs later.
2019-11-25 14:37:09 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/base2.txt', 'BASE2', '09b5e31766be1dba1ec27de82f975c1b6eea2a92', $lTime);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestTablespaceDrop(\%oManifest, 1, 2);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/2', '32768/tablespace2b.txt', 'TBLSPC2B',
'e324463005236d83e6e54795dbddd20a74533bf3', $lTime, undef, undef, false);
# Munge the version to make sure it gets corrected on the next run
Improvements to Ini.pm. * Refactor Ini.pm to facilitate testing. * Complete statement/branch coverage for Ini.pm. * Improved functions used to test/munge manifest and info files.
2017-04-10 19:24:45 +02:00
$oHostBackup->manifestMunge($strBackup, {&INI_SECTION_BACKREST => {&INI_KEY_VERSION => '0.00'}}, false);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = 1;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strBackup = $oHostBackup->backup(
$strType, 'add files and remove tablespace 2',
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
{oExpectedManifest => \%oManifest, strOptionalParam => '--process-max=1'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = $strStorage eq S3 ? 2 : 1;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Incr Backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_INCR;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestReference(\%oManifest, $strBackup);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/16384/17000', 'BASEUPDT', '9a53d532e27785e681766c98516a5e93f096a501',
$lTime, undef, undef, false);
# Perform the backup
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
$strBackup =$oHostBackup->backup($strType, 'update files', {oExpectedManifest => \%oManifest});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Diff Backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_DIFF;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestReference(\%oManifest, $strFullBackup, true);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = 1;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strBackup = $oHostBackup->backup(
Reduce process redundancy in mock tests.
2017-06-22 00:47:25 +02:00
$strType, 'updates since last full', {oExpectedManifest => \%oManifest,
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
strOptionalParam => '--process-max=1' . ($bDeltaBackup ? ' --delta' : '')});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = $strStorage eq S3 ? 2 : 1;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
# Diff Backup with files removed
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestReference(\%oManifest, $strFullBackup, true);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
$strType = CFGOPTVAL_BACKUP_TYPE_DIFF;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
# Enable compression to ensure a warning is raised (reset when gz to avoid log churn since it is the default)
if ($strCompressType eq GZ)
{
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({&CFGDEF_SECTION_GLOBAL => {'compress-type' => undef}});
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
}
else
{
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({&CFGDEF_SECTION_GLOBAL => {'compress-type' => $strCompressType}});
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
}
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
# Enable hardlinks (except for s3) to ensure a warning is raised
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
if ($strStorage eq POSIX)
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
{
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$oHostBackup->configUpdate({&CFGDEF_SECTION_GLOBAL => {'repo1-hardlink' => 'y'}});
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
}
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = 1;
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileRemove(\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/2', '32768/tablespace2b.txt', true);
$oHostDbPrimary->manifestFileRemove(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/base2.txt', true);
$oHostDbPrimary->manifestFileRemove(\%oManifest, MANIFEST_TARGET_PGDATA, 'base/16384/17000');
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGTBLSPC . '/2', '32768/tablespace2c.txt', 'TBLSPCBIGGER',
'dfcb8679956b734706cf87259d50c88f83e80e66', $lTime, undef, undef, false);
Remove integration tests that are now covered in the unit tests. Most of these tests are just checking that errors are thrown when required. These are well covered in various unit tests. The "cannot resume" tests are also well covered in the backup unit tests. Finally, config warnings are well covered in the config unit tests. There is more to be done here, but this accounts for the low-hanging fruit.
2019-12-18 03:14:45 +02:00
$oHostBackup->backup(
$strType, 'remove files',
Update integration tests in mock/all that use test points. Test points will not be available in the C code so update these tests as best as possible without using them. This represents a loss of coverage for the Perl code (soon to be removed) which will be made up in the C code with unit tests.
2019-11-25 14:48:52 +02:00
{oExpectedManifest => \%oManifest,
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
strOptionalParam => '--process-max=1' . ($bDeltaBackup ? ' --delta' : '')});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_PROCESS_MAX} = $strStorage eq S3 ? 2 : 1;
Add additional options to backup.manifest for debugging purposes. Add the buffer-size, compress-level, compress-level-network, and process-max options to the backup:option section in backup.manifest to aid in debugging. It may also make sense to propagate these options up to backup.info so they can be displayed in the info command, but for now this is deemed sufficient. Contributed by blogh.
2019-03-10 11:03:52 +02:00
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Full Backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_FULL;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
# Now the compression and hardlink changes will take effect
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_COMPRESS} = JSON::PP::true;
Add infrastructure for multiple compression type support. Add compress-type option and deprecate compress option. Since the compress option is boolean it won't work with multiple compression types. Add logic to cfgLoadUpdateOption() to update compress-type if it is not set directly. The compress option should no longer be referenced outside the cfgLoadUpdateOption() function. Add common/compress/helper module to contain interface functions that work with multiple compression types. Code outside this module should no longer call specific compression drivers, though it may be OK to reference a specific compression type using the new interface (e.g., saving backup history files in gz format). Unit tests only test compression using the gz format because other formats may not be available in all builds. It is the job of integration tests to exercise all compression types. Additional compression types will be added in future commits.
2020-03-06 21:41:03 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_COMPRESS_TYPE} = $strCompressType;
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
if ($strStorage eq POSIX)
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
{
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_HARDLINK} = JSON::PP::true;
$oHostBackup->{bHardLink} = true;
}
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestReference(\%oManifest);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/16384/17000', 'BASEUPDT2', '7579ada0808d7f98087a0a586d0df9de009cdc33',
$lTime, undef, undef, false);
$oManifest{&MANIFEST_SECTION_BACKUP_OPTION}{&MANIFEST_KEY_CHECKSUM_PAGE} = JSON::PP::false;
$strFullBackup = $oHostBackup->backup(
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
$strType, 'update file', {oExpectedManifest => \%oManifest});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Call expire
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
$oHostBackup->expire({iRetentionFull => 1});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Diff Backup
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
$strType = CFGOPTVAL_BACKUP_TYPE_DIFF;
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestReference(\%oManifest, $strFullBackup);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->manifestFileCreate(
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
\%oManifest, MANIFEST_TARGET_PGDATA, 'base/base2.txt', 'BASE2UPDT', 'cafac3c59553f2cfde41ce2e62e7662295f108c0',
$lTime, undef, undef, false);
# Munge the prior manifest so that option-checksum-page is missing to be sure the logic works for backups before page
# checksums were introduced
Improvements to Ini.pm. * Refactor Ini.pm to facilitate testing. * Complete statement/branch coverage for Ini.pm. * Improved functions used to test/munge manifest and info files.
2017-04-10 19:24:45 +02:00
$oHostBackup->manifestMunge(
$strFullBackup, {&MANIFEST_SECTION_BACKUP_OPTION => {&MANIFEST_KEY_CHECKSUM_PAGE => undef}}, false);
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$strBackup = $oHostBackup->backup(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
$strType, 'add file', {oExpectedManifest => \%oManifest, strOptionalParam => '--checksum-page'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Selective Restore
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Remove mapping for tablespace 1
delete($oRemapHash{&MANIFEST_TARGET_PGTBLSPC . '/1'});
# Remove checksum to match zeroed files
delete($oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/base/32768/33000'}{&MANIFEST_SUBKEY_CHECKSUM});
delete($oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/base/32768/33001'}{&MANIFEST_SUBKEY_CHECKSUM});
delete($oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_tblspc/2/PG_9.4_201409291/32768/tablespace2.txt'}
{&MANIFEST_SUBKEY_CHECKSUM});
delete($oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_tblspc/2/PG_9.4_201409291/32768/tablespace2c.txt'}
{&MANIFEST_SUBKEY_CHECKSUM});
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
'selective restore 16384', 'latest',
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
{rhExpectedManifest => \%oManifest, rhRemapHash => \%oRemapHash, bDelta => true,
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
strOptionalParam => '--db-include=16384'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Restore checksum values for next test
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/base/32768/33000'}{&MANIFEST_SUBKEY_CHECKSUM} =
Retry on page checksum validation failure during backup. Rather than attempting to filter page checksum failures by LSN, just retry when there is a page checksum failure. If the page has not changed since the last read report it as an error. If the page has changed, then PostgreSQL must be modifying the page so we can ignore the error because a full page write (and possibly updates) will be in the WAL. Also remove tests made redundant by the test merge in b4897077.
2022-02-23 20:05:53 +02:00
'1d73a0052828531770e7c155aeb22338e017e196';
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/base/32768/33001'}{&MANIFEST_SUBKEY_CHECKSUM} =
'6bf316f11d28c28914ea9be92c00de9bea6d9a6b';
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_tblspc/2/PG_9.4_201409291/32768/tablespace2.txt'}
{&MANIFEST_SUBKEY_CHECKSUM} = 'dc7f76e43c46101b47acc55ae4d593a9e6983578';
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_tblspc/2/PG_9.4_201409291/32768/tablespace2c.txt'}
{&MANIFEST_SUBKEY_CHECKSUM} = 'dfcb8679956b734706cf87259d50c88f83e80e66';
# Remove checksum to match zeroed file
delete($oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/base/16384/17000'}{&MANIFEST_SUBKEY_CHECKSUM});
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
'selective restore 32768', 'latest',
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
{rhExpectedManifest => \%oManifest, rhRemapHash => \%oRemapHash, bDelta => true,
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
strOptionalParam => '--db-include=32768'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_TARGET_FILE}{'pg_data/base/16384/17000'}{&MANIFEST_SUBKEY_CHECKSUM} =
'7579ada0808d7f98087a0a586d0df9de009cdc33';
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
'error on invalid id', 'latest',
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
{rhExpectedManifest => \%oManifest, rhRemapHash => \%oRemapHash, bDelta => true,
iExpectedExitStatus => ERROR_DB_MISSING, strOptionalParam => '--log-level-console=warn --db-include=7777'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
'error on system id', 'latest',
Move restore test infrastructure to HostBackup.pm. Required to test restores on the backup server, a fairly common scenario. Improve the restore function to accept optional parameters rather than a long list of parameters. In passing, clean up extraneous use of strType and strComment variables.
2017-12-04 00:08:49 +02:00
{rhExpectedManifest => \%oManifest, rhRemapHash => \%oRemapHash, bDelta => true,
iExpectedExitStatus => ERROR_DB_INVALID, strOptionalParam => '--log-level-console=warn --db-include=1'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
# Compact Restore
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
executeTest('rm -rf ' . $oHostDbPrimary->dbBasePath(2) . "/*");
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
my $strDbPath = $oHostDbPrimary->dbBasePath(2) . '/base';
Storage and IO layer refactor: Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.) Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 23:51:41 +02:00
storageTest()->pathCreate($strDbPath, {strMode => '0700'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oRemapHash{&MANIFEST_TARGET_PGDATA} = $strDbPath;
delete($oRemapHash{&MANIFEST_TARGET_PGTBLSPC . '/2'});
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$oHostDbPrimary->restore(
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
'no tablespace remap', 'latest',
The restore command is implemented entirely in C. For the most part this is a direct migration of the Perl code into C. There is one important behavioral change with regard to how file permissions are handled. The Perl code tried to set ownership as it was in the manifest even when running as an unprivileged user. This usually just led to errors and frustration. The C code works like this: If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried. If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name. Reviewed by Cynthia Shang.
2019-09-26 13:52:02 +02:00
{rhExpectedManifest => \%oManifest, rhRemapHash => \%oRemapHash, bTablespace => false,
Reduce log level for all expect tests to detail. The C code is designed to be efficient rather than deterministic at the debug log level. As we move more testing from integration to unit tests it makes less sense to try and maintain the expect logs at this log level. Most of the expect logs have already been moved to detail level but mock/all still had tests at debug level. Change the logging defaults in the config file and remove as many references to log-level-console as possible.
2019-05-23 00:23:44 +02:00
strOptionalParam => '--tablespace-map-all=../../tablespace'});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
$oManifest{&MANIFEST_SECTION_BACKUP_TARGET}{'pg_tblspc/2'}{&MANIFEST_SUBKEY_PATH} = '../../tablespace/ts2';
$oManifest{&MANIFEST_SECTION_TARGET_LINK}{'pg_data/pg_tblspc/2'}{&MANIFEST_SUBKEY_DESTINATION} = '../../tablespace/ts2';
# Dump out history path at the end to verify all history files are being recorded. This test is only performed locally
# because for some reason sort order is different when this command is executed via ssh (even though the content of the
# directory is identical).
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Make storage type more generic in the integration tests. Rather than bS3 use strStorage which can indicate more than two storage types. For the moment there are still only two storage types but this change is required before more can be added.
2020-05-13 00:55:20 +02:00
if (!$bRemote && $strStorage eq POSIX)
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
{
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
executeTest(
'ls -1Rtr ' . $oHostBackup->repoBackupPath(PATH_BACKUP_HISTORY),
{oLogTest => $self->expect(), bRemote => $bRemote});
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable. Fixed by Cynthia Shang.
2017-05-12 22:05:44 +02:00
# Test backup from standby warning that standby not configured so option reset
Configuration rules are now pulled from the C library when present.
2017-08-25 22:47:47 +02:00
#---------------------------------------------------------------------------------------------------------------------------
Full/Synthetic test refactor. * Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage. * Change log level of hardlink logging to detail. * Cast size in S3 manifest to integer.
2017-06-15 21:32:10 +02:00
if (!defined($oHostDbStandby))
Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable. Fixed by Cynthia Shang.
2017-05-12 22:05:44 +02:00
{
$strBackup = $oHostBackup->backup(
Rename most instances of master to primary in tests. This aligns better with general PostgreSQL usage and our own documentation (updated in 4bcef702). Usage in the backup.manifest tests has not been updated since it might break the file format.
2020-06-16 20:06:38 +02:00
$strType, 'option backup-standby reset - backup performed from primary', {oExpectedManifest => \%oManifest,
Remove LibC. This was the interface between Perl and C introduced in 36a5349b but since f0ef73db has only been used by the Perl integration tests. This is expensive code to maintain just for testing. The main dependency was the interface to storage, no matter where it was located, e.g. S3. Replace this with the new-introduced repo commands (d3c83453) that allow access to repo storage via the command line. The other dependency was on various cfgOption* functions and CFGOPT_ constants that were convenient but not necessary. Replace these with hard-coded strings in most places and create new constants for commonly used values. Remove all auto-generated Perl code. This means that the error list will no longer be maintained automatically so copy used errors to Common::Exception.pm. This file will need to be maintained manually going forward but there is not likely to be much churn as the Perl integration tests are being retired. Update test.pl and related code to remove LibC builds. Ding, dong, LibC is dead.
2020-03-09 23:41:59 +02:00
strOptionalParam => '--log-level-console=info --backup-standby'});
Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable. Fixed by Cynthia Shang.
2017-05-12 22:05:44 +02:00
}
Split test modules into separate files. Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 15:22:59 +02:00
}
}
1;
Reference in New Issue Copy Permalink