db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-01-18 04:58:51 +02:00
Code Issues Releases Activity

Added check command.

Browse Source 
The check command validates that pgBackRest is configured correctly for archiving and backups.

Contributed by Cynthia Shang.
This commit is contained in:
Cynthia Shang 2016-06-12 09:13:46 -04:00 committed by David Steele
parent 7c9eaf7210
commit 7e45ed8366
16 changed files with 546 additions and 58 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
bin/pgbackrest
View File

@ -109,6 +109,14 @@ eval
exitSafe(new pgBackRest::Archive()->process());
}
################################################################################################################################
# Process check command
################################################################################################################################
if (commandTest(CMD_CHECK))
{
exitSafe(new pgBackRest::Archive()->check());
}
################################################################################################################################
# Process start/stop commands
################################################################################################################################

1
doc/lib/BackRestDoc/Common/DocRender.pm
View File

@ -66,6 +66,7 @@ my $oRenderTag =
'path' => ['', ''],
'cmd' => ['', ''],
'br-option' => ['', ''],
'pg-setting' => ['', ''],
'param' => ['', ''],
'setting' => ['', ''],
'code' => ['', ''],

8
doc/xml/backlog.xml
View File

@ -32,14 +32,6 @@
<p>If <postgres/> crashes during a backup it may not be able to recover if the backup label is present. Copy and delete right after start_backup(). Stop backup will want to delete it so it might be necessary to copy it back or at least touch a file that <postgres/> can delete. Check after the backup is complete to make sure it's really gone.</p>
</section>
<section id="wal-archive-working">
<title>Ability to test that WAL archiving is working</title>
<p><link url="{[github-url-issues]}/149">Github Issue</link></p>
<p>Add a new command, archive-test, that will execute pg_switch_xlog and check that the xlog makes it to the archive.</p>
</section>
<section id="processes-vs-threads">
<title>Abandon threads and go to processes</title>

32
doc/xml/reference.xml
View File

@ -72,8 +72,16 @@
<config-section id="general" name="General">
<text>The <setting>general</setting> section defines settings that are shared between multiple operations.</text>
<!-- CONFIG - GENERAL SECTION - BUFFER-SIZE KEY -->
<!-- CONFIG - GENERAL SECTION - ARCHIVE-TIMEOUT -->
<config-key-list>
<config-key id="archive-timeout" name="Archive Timeout">
<summary>Archive timeout.</summary>
<text>Set maximum time, in seconds, to wait for WAL segments to reach the archive. The timeout applies to the <cmd>check</cmd> command and to the <cmd>backup</cmd> command when waiting for WAL segments required to make the backup consistent to be archived.</text>
<example>30</example>
</config-key>
<config-key id="buffer-size" name="Buffer Size">
<summary>Buffer size for file operations.</summary>
@ -125,7 +133,7 @@
<config-key id="db-timeout" name="Database Timeout">
<summary>Database query timeout.</summary>
<text>Sets the timeout for queries against the database. This includes the <code>pg_start_backup()</code> and <code>pg_stop_backup()</code> functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set <setting>startfast=y</setting> and you know that the database cluster will not generate many WAL segments during the backup).</text>
<text>Sets the timeout, in seconds, for queries against the database. This includes the <code>pg_start_backup()</code> and <code>pg_stop_backup()</code> functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set <setting>startfast=y</setting> and you know that the database cluster will not generate many WAL segments during the backup).</text>
<example>600</example>
</config-key>
@ -203,7 +211,6 @@
<config-section id="backup" name="Backup">
<text>The <setting>backup</setting> section defines settings related to backup.</text>
<!-- CONFIG - BACKUP SECTION - BACKUP-HOST KEY -->
<config-key-list>
<!-- CONFIG - BACKUP SECTION - ARCHIVE-CHECK -->
<config-key id="archive-check" name="Check Archive">
@ -225,6 +232,7 @@
<example>y</example>
</config-key>
<!-- CONFIG - BACKUP SECTION - BACKUP-HOST KEY -->
<config-key id="backup-host" name="Backup Host">
<summary>Backup host when operating remotely via SSH.</summary>
@ -620,6 +628,24 @@
</command-example-list>
</command>
<!-- OPERATION - CHECK COMMAND -->
<command id="check" name="Check">
<summary>Check the configuration.</summary>
<text>The <cmd>check</cmd> command validates that <backrest/> and the <pg-setting>archive_command</pg-setting> setting are configured correctly for archiving and backups. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the database or the backup host.
Note that <code>pg_create_restore_point('pgBackRest Archive Check')</code> and <code>pg_switch_xlog()</code> are called to force <postgres/> to archive a WAL segment. Restore points are only supported in <postgres/> &gt;= 9.1 so for older versions the <cmd>check</cmd> command may fail if there has been no write activity since the last log rotation.</text>
<command-example-list>
<command-example>
<text><code-block title="">
{[backrest-exe]} --stanza=db check
</code-block>
</text>
</command-example>
</command-example-list>
</command>
<!-- OPERATION - EXPIRE COMMAND -->
<command id="expire" name="Expire">
<summary>Expire backups that exceed retention.</summary>

14
doc/xml/release.xml
View File

@ -85,6 +85,20 @@
<release-list>
<release date="XXXX-XX-XX" version="1.03dev" title="UNDER DEVELOPMENT">
<release-core-list>
<release-feature-list>
<release-item>
<release-item-contributor-list>
<release-item-ideator id="steele.david"/>
<release-item-contributor id="shang.cynthia"/>
<release-item-reviewer id="steele.david"/>
</release-item-contributor-list>
<p>Added <cmd>check</cmd> command to validate that <backrest/> is configured correctly for archiving and backups.</p>
</release-item>
</release-feature-list>
</release-core-list>
<release-doc-list>
<release-bug-list>
<release-item>

46
doc/xml/user-guide.xml
View File

@ -412,6 +412,33 @@
<exe-retry/>
</execute>
</execute-list> -->
</section>
<!-- SECTION => QUICKSTART - CHECK CONFIGURATION -->
<section id="check-configuration" depend="configure-archiving">
<title>Check the Configuration</title>
<cmd-description key="check"/>
<execute-list host="{[host-db-master]}">
<title>Check the configuration</title>
<execute output="y">
<exe-cmd>{[project-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-log-level-console=info check</exe-cmd>
<exe-highlight> successfully stored in the archive at </exe-highlight>
</execute>
</execute-list>
<!-- Decided not to show the error in this part of the user guide but added as a debug statement for reference. -->
<execute-list keyword="debug" host="{[host-db-master]}">
<title>Example of an invalid configuration</title>
<execute output="y" err-expect="157">
<exe-cmd>{[project-exe]} {[dash]}-stanza={[postgres-cluster-demo]} --archive-timeout=.1 check</exe-cmd>
<exe-highlight>could not find WAL segment|did not reach the archive</exe-highlight>
</execute>
</execute-list>
</section>
<!-- SECTION => QUICKSTART - PERFORM BACKUP -->
@ -1251,6 +1278,25 @@
</execute-list>
<p>Commands are run the same as on a single host configuration except that the <cmd>backup</cmd> and <cmd>expire</cmd> command are run from the <host>backup</host> host and all other commands are run from the <host>database</host> host.</p>
<p>Check that the configuration is correct on both the <host>database</host> and <host>backup</host> hosts. More information about the <cmd>check</cmd> command can be found in <link section="/quickstart/check-configuration">Check the Configuration</link>.</p>
<execute-list host="{[host-db-master]}">
<title>Check the configuration</title>
<execute output="y" filter="n" >
<exe-cmd>{[project-exe]} {[dash]}-stanza={[postgres-cluster-demo]} check</exe-cmd>
</execute>
</execute-list>
<execute-list host="{[host-backup]}">
<title>Check the configuration</title>
<execute user="backrest" output="y" filter="n" >
<exe-cmd>{[project-exe]} {[dash]}-stanza={[postgres-cluster-demo]} check</exe-cmd>
</execute>
</execute-list>
</section>
<!-- SECTION => BACKUP HOST - PERFORM BACKUP -->

135
lib/pgBackRest/Archive.pm
View File

@ -11,6 +11,7 @@ use Exporter qw(import);
our @EXPORT = qw();
use Fcntl qw(SEEK_CUR O_RDONLY O_WRONLY O_CREAT);
use File::Basename qw(dirname basename);
use Scalar::Util qw(blessed);
use lib dirname($0);
use pgBackRest::Common::Exception;
@ -235,7 +236,7 @@ sub walFileName
# If waiting and no WAL segment was found then throw an error
if (@stryWalFileName == 0 && defined($iWaitSeconds))
{
confess &log(ERROR, "could not find WAL segment ${strWalSegment} after " . waitInterval($oWait) . ' second(s)');
confess &log(ERROR, "could not find WAL segment ${strWalSegment} after ${iWaitSeconds} second(s)", ERROR_ARCHIVE_TIMEOUT);
}
# Return from function and log return values if any
@ -511,7 +512,7 @@ sub pushProcess
$self->push($ARGV[1], $bArchiveAsync);
# Fork is async archiving is enabled
# Fork if async archiving is enabled
if ($bArchiveAsync)
{
# Fork and disable the async archive flag if this is the parent process
@ -1035,4 +1036,134 @@ sub range
);
}
####################################################################################################################################
# check
#
# Validates the database configuration and checks that the archive logs can be read by backup. This will alert the user to any
# misconfiguration, particularly of archiving, that would result in the inability of a backup to complete (e.g waiting at the end
# until it times out because it could not find the WAL file).
####################################################################################################################################
sub check
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my $strOperation = logDebugParam(__PACKAGE__ . '->check');
# Initialize default file object
$self->{oFile} = new pgBackRest::File
(
optionGet(OPTION_STANZA),
optionGet(OPTION_REPO_PATH),
optionRemoteType(),
protocolGet()
);
# Initialize the database object
$self->{oDb} = new pgBackRest::Db();
# Validate the database configuration
$self->{oDb}->configValidate($self->{oFile}, optionGet(OPTION_DB_PATH));
# Force archiving
my $strWalSegment = $self->{oDb}->xlogSwitch();
# Get the timeout and error message to display - if it is 0 we are testing
my $iArchiveTimeout = optionGet(OPTION_ARCHIVE_TIMEOUT);
# Initialize the result variables
my $iResult = 0;
my $strResultMessage = undef;
# Record the start time to wait for the archive.info file to be written
my $oWait = waitInit($iArchiveTimeout);
my $strArchiveId = undef;
my $strArchiveFile = undef;
# Turn off console logging to control when to display the error
logLevelSet(undef, OFF);
# Wait for the archive.info to be written. If it does not get written within the timout period then report the last error.
do
{
eval
{
$strArchiveId = $self->getCheck($self->{oFile});
# Clear any previous errors if we've found the archive.info
$iResult = 0;
};
if ($@)
{
my $oMessage = $@;
# If this is a backrest error then capture the last code and message else confess
if (blessed($oMessage) && $oMessage->isa('pgBackRest::Common::Exception'))
{
$iResult = $oMessage->code();
$strResultMessage = $oMessage->message();
}
else
{
confess $oMessage;
}
}
} while (!defined($strArchiveId) && waitMore($oWait));
# If able to get the archive id then check the archived WAL file with the time remaining
if ($iResult == 0)
{
eval
{
$strArchiveFile = $self->walFileName($self->{oFile}, $strArchiveId, $strWalSegment, false, $iArchiveTimeout);
};
# If this is a backrest error then capture the last code and message else confess
if ($@)
{
my $oMessage = $@;
# If a backrest exception then return the code else confess
if (blessed($oMessage) && $oMessage->isa('pgBackRest::Common::Exception'))
{
$iResult = $oMessage->code();
$strResultMessage = $oMessage->message();
}
else
{
confess $oMessage;
}
}
}
# Reset the console logging
logLevelSet(undef, optionGet(OPTION_LOG_LEVEL_CONSOLE));
# Display results
if ($iResult == 0)
{
&log(INFO,
"WAL segment ${strWalSegment} successfully stored in the archive at '" .
$self->{oFile}->pathGet(PATH_BACKUP_ARCHIVE, "$strArchiveId/${strArchiveFile}") . "'");
}
else
{
&log(ERROR, $strResultMessage, $iResult);
&log(WARN,
"WAL segment ${strWalSegment} did not reach the archive:\n" .
"HINT: Check the archive_command to ensure that all options are correct (especialy --stanza).\n" .
"HINT: Check the PostreSQL server log for errors.");
}
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'iResult', value => $iResult, trace => true}
);
}
1;

4
lib/pgBackRest/Backup.pm
View File

@ -855,7 +855,7 @@ sub process
# Create the modification time for the archive logs
my $lModificationTime = time();
# After the backup has been stopped, need to make a copy of the archive logs need to make the db consistent
# After the backup has been stopped, need to make a copy of the archive logs to make the db consistent
logDebugMisc($strOperation, "retrieve archive logs ${strArchiveStart}:${strArchiveStop}");
my $oArchive = new pgBackRest::Archive();
my $strArchiveId = $oArchive->getCheck($self->{oFile});
@ -863,7 +863,7 @@ sub process
foreach my $strArchive (@stryArchive)
{
my $strArchiveFile = $oArchive->walFileName($self->{oFile}, $strArchiveId, $strArchive, false, 600);
my $strArchiveFile = $oArchive->walFileName($self->{oFile}, $strArchiveId, $strArchive, false, optionGet(OPTION_ARCHIVE_TIMEOUT));
if (optionGet(OPTION_BACKUP_ARCHIVE_COPY))
{

2
lib/pgBackRest/Common/Exception.pm
View File

@ -132,6 +132,8 @@ use constant ERROR_DB_MISSING => ERROR_MIN
push @EXPORT, qw(ERROR_DB_MISSING);
use constant ERROR_DB_INVALID => ERROR_MINIMUM + 56;
push @EXPORT, qw(ERROR_DB_INVALID);
use constant ERROR_ARCHIVE_TIMEOUT => ERROR_MINIMUM + 57;
push @EXPORT, qw(ERROR_ARCHIVE_TIMEOUT);
use constant ERROR_INVALID_VALUE => ERROR_MAXIMUM - 1;
push @EXPORT, qw(ERROR_INVALID_VALUE);

6
lib/pgBackRest/Common/Wait.pm
View File

@ -17,6 +17,12 @@ use Time::HiRes qw(gettimeofday usleep);
use lib dirname($0) . '/../lib';
use pgBackRest::Common::Log;
####################################################################################################################################
# Wait constants
####################################################################################################################################
use constant WAIT_TIME_MINIMUM => .1;
push @EXPORT, qw(WAIT_TIME_MINIMUM);
####################################################################################################################################
# waitRemainder
####################################################################################################################################

69
lib/pgBackRest/Config/Config.pm
View File

@ -17,6 +17,7 @@ use lib dirname($0) . '/../lib';
use pgBackRest::Common::Exception;
use pgBackRest::Common::Ini;
use pgBackRest::Common::Log;
use pgBackRest::Common::Wait;
use pgBackRest::Protocol::Common;
use pgBackRest::Protocol::RemoteMaster;
use pgBackRest::Version;
@ -45,6 +46,9 @@ use constant CMD_ARCHIVE_PUSH => 'archive-
use constant CMD_BACKUP => 'backup';
push @EXPORT, qw(CMD_BACKUP);
$oCommandHash{&CMD_BACKUP} = true;
use constant CMD_CHECK => 'check';
push @EXPORT, qw(CMD_CHECK);
$oCommandHash{&CMD_CHECK} = true;
use constant CMD_EXPIRE => 'expire';
push @EXPORT, qw(CMD_EXPIRE);
$oCommandHash{&CMD_EXPIRE} = true;
@ -239,6 +243,8 @@ use constant OPTION_TEST_POINT => 'test-poi
# GENERAL Section
#-----------------------------------------------------------------------------------------------------------------------------------
use constant OPTION_ARCHIVE_TIMEOUT => 'archive-timeout';
push @EXPORT, qw(OPTION_ARCHIVE_TIMEOUT);
use constant OPTION_BUFFER_SIZE => 'buffer-size';
push @EXPORT, qw(OPTION_BUFFER_SIZE);
use constant OPTION_CONFIG_REMOTE => 'config-remote';
@ -380,6 +386,13 @@ use constant OPTION_DEFAULT_TEST_NO_FORK => false;
# GENERAL Section
#-----------------------------------------------------------------------------------------------------------------------------------
use constant OPTION_DEFAULT_ARCHIVE_TIMEOUT => 60;
push @EXPORT, qw(OPTION_DEFAULT_ARCHIVE_TIMEOUT);
use constant OPTION_DEFAULT_ARCHIVE_TIMEOUT_MIN => WAIT_TIME_MINIMUM;
push @EXPORT, qw(OPTION_DEFAULT_ARCHIVE_TIMEOUT_MIN);
use constant OPTION_DEFAULT_ARCHIVE_TIMEOUT_MAX => 86400;
push @EXPORT, qw(OPTION_DEFAULT_ARCHIVE_TIMEOUT_MAX);
use constant OPTION_DEFAULT_BUFFER_SIZE => 4194304;
push @EXPORT, qw(OPTION_DEFAULT_BUFFER_SIZE);
use constant OPTION_DEFAULT_BUFFER_SIZE_MIN => 16384;
@ -406,6 +419,11 @@ use constant OPTION_DEFAULT_COMPRESS_LEVEL_NETWORK_MAX => 9;
use constant OPTION_DEFAULT_DB_TIMEOUT => 1800;
push @EXPORT, qw(OPTION_DEFAULT_DB_TIMEOUT);
use constant OPTION_DEFAULT_DB_TIMEOUT_MIN => WAIT_TIME_MINIMUM;
push @EXPORT, qw(OPTION_DEFAULT_DB_TIMEOUT_MIN);
use constant OPTION_DEFAULT_DB_TIMEOUT_MAX => 86400 * 7;
push @EXPORT, qw(OPTION_DEFAULT_DB_TIMEOUT_MAX);
use constant OPTION_DEFAULT_CONFIG => '/etc/' . BACKREST_EXE . '.conf';
push @EXPORT, qw(OPTION_DEFAULT_CONFIG);
use constant OPTION_DEFAULT_LOCK_PATH => '/tmp/' . BACKREST_EXE;
@ -493,6 +511,12 @@ use constant OPTION_DEFAULT_DB_USER => 'postgres
####################################################################################################################################
# Option Rule Hash
#
# pgbackrest will throw an error if:
# 1) an option is provided when executing the command that is not listed in the OPTION_RULE_COMMAND section of the Option Rule Hash
# 2) or an option is not provided when executing the command and it is listed in the OPTION_RULE_COMMAND section as "true"
# If an OPTION_RULE_COMMAND is set to "false" then pgbackrest will not throw an error if the option is missing and also will not throw an
# error if it exists.
####################################################################################################################################
my %oOptionRule =
(
@ -508,6 +532,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_EXPIRE => true,
&CMD_INFO => true,
&CMD_REMOTE => true,
@ -609,6 +634,10 @@ my %oOptionRule =
{
&OPTION_RULE_REQUIRED => true
},
&CMD_CHECK =>
{
&OPTION_RULE_REQUIRED => true
},
&CMD_EXPIRE =>
{
&OPTION_RULE_REQUIRED => true
@ -852,6 +881,19 @@ my %oOptionRule =
# GENERAL Section
#-------------------------------------------------------------------------------------------------------------------------------
&OPTION_ARCHIVE_TIMEOUT =>
{
&OPTION_RULE_SECTION => CONFIG_SECTION_GLOBAL,
&OPTION_RULE_TYPE => OPTION_TYPE_FLOAT,
&OPTION_RULE_DEFAULT => OPTION_DEFAULT_ARCHIVE_TIMEOUT,
&OPTION_RULE_ALLOW_RANGE => [OPTION_DEFAULT_ARCHIVE_TIMEOUT_MIN, OPTION_DEFAULT_ARCHIVE_TIMEOUT_MAX],
&OPTION_RULE_COMMAND =>
{
&CMD_BACKUP => true,
&CMD_CHECK => true,
},
},
&OPTION_BUFFER_SIZE =>
{
&OPTION_RULE_SECTION => CONFIG_SECTION_GLOBAL,
@ -863,6 +905,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_EXPIRE => false,
&CMD_INFO => true,
&CMD_REMOTE => true,
@ -873,13 +916,15 @@ my %oOptionRule =
&OPTION_DB_TIMEOUT =>
{
&OPTION_RULE_SECTION => CONFIG_SECTION_GLOBAL,
&OPTION_RULE_TYPE => OPTION_TYPE_INTEGER,
&OPTION_RULE_TYPE => OPTION_TYPE_FLOAT,
&OPTION_RULE_DEFAULT => OPTION_DEFAULT_DB_TIMEOUT,
&OPTION_RULE_ALLOW_RANGE => [OPTION_DEFAULT_DB_TIMEOUT_MIN, OPTION_DEFAULT_DB_TIMEOUT_MAX],
&OPTION_RULE_COMMAND =>
{
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_EXPIRE => false,
&CMD_INFO => true,
&CMD_REMOTE => true,
@ -913,6 +958,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_EXPIRE => false,
&CMD_INFO => true,
&CMD_REMOTE => true,
@ -931,6 +977,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_EXPIRE => false,
&CMD_INFO => true,
&CMD_REMOTE => true,
@ -948,6 +995,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_INFO => true,
&CMD_RESTORE => true
},
@ -963,6 +1011,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_INFO => false,
&CMD_EXPIRE => false,
&CMD_REMOTE => true,
@ -1001,6 +1050,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_INFO => true,
&CMD_REMOTE => true,
&CMD_RESTORE => true,
@ -1020,6 +1070,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_INFO => true,
&CMD_REMOTE => true,
&CMD_RESTORE => true,
@ -1082,6 +1133,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_INFO => true,
&CMD_RESTORE => true
}
@ -1109,6 +1161,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_EXPIRE => true,
&CMD_INFO => true,
&CMD_RESTORE => true,
@ -1137,6 +1190,7 @@ my %oOptionRule =
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_EXPIRE => true,
&CMD_INFO => true,
&CMD_RESTORE => true,
@ -1216,6 +1270,7 @@ my %oOptionRule =
{
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_CHECK => true,
&CMD_INFO => true,
&CMD_RESTORE => true
},
@ -1230,6 +1285,7 @@ my %oOptionRule =
{
&CMD_ARCHIVE_GET => true,
&CMD_ARCHIVE_PUSH => true,
&CMD_CHECK => true,
&CMD_INFO => true,
&CMD_RESTORE => true
},
@ -1467,7 +1523,11 @@ my %oOptionRule =
{
&OPTION_RULE_REQUIRED => false
},
&CMD_BACKUP => true
&CMD_BACKUP => true,
&CMD_CHECK =>
{
&OPTION_RULE_REQUIRED => false
}
},
},
@ -1479,6 +1539,7 @@ my %oOptionRule =
&OPTION_RULE_COMMAND =>
{
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_REMOTE => true
}
},
@ -1491,6 +1552,7 @@ my %oOptionRule =
&OPTION_RULE_COMMAND =>
{
&CMD_BACKUP => true,
&CMD_CHECK => true,
&CMD_REMOTE => true
}
},
@ -1502,7 +1564,8 @@ my %oOptionRule =
&OPTION_RULE_DEFAULT => OPTION_DEFAULT_DB_USER,
&OPTION_RULE_COMMAND =>
{
&CMD_BACKUP => true
&CMD_BACKUP => true,
&CMD_CHECK => true,
},
&OPTION_RULE_REQUIRED => false,
&OPTION_RULE_DEPEND =>

63
lib/pgBackRest/Config/ConfigHelpData.pm
View File

@ -90,6 +90,19 @@ my $oConfigHelpData =
"\${stanza} is the backup stanza."
},
# ARCHIVE-TIMEOUT Option Help
#---------------------------------------------------------------------------------------------------------------------------
'archive-timeout' =>
{
section => 'general',
summary =>
"Archive timeout.",
description =>
"Set maximum time, in seconds, to wait for WAL segments to reach the archive. The timeout applies to the check " .
"command and to the backup command when waiting for WAL segments required to make the backup consistent to " .
"be archived."
},
# BACKUP-HOST Option Help
#---------------------------------------------------------------------------------------------------------------------------
'backup-host' =>
@ -277,10 +290,10 @@ my $oConfigHelpData =
summary =>
"Database query timeout.",
description =>
"Sets the timeout for queries against the database. This includes the pg_start_backup() and pg_stop_backup() " .
"functions which can each take a substantial amount of time. Because of this the timeout should be kept " .
"high unless you know that these functions will return quickly (i.e. if you have set startfast=y and you " .
"know that the database cluster will not generate many WAL segments during the backup)."
"Sets the timeout, in seconds, for queries against the database. This includes the pg_start_backup() and " .
"pg_stop_backup() functions which can each take a substantial amount of time. Because of this the timeout " .
"should be kept high unless you know that these functions will return quickly (i.e. if you have set " .
"startfast=y and you know that the database cluster will not generate many WAL segments during the backup)."
},
# DB-USER Option Help
@ -734,6 +747,7 @@ my $oConfigHelpData =
{
'archive-check' => 'section',
'archive-copy' => 'section',
'archive-timeout' => 'section',
'buffer-size' => 'section',
'cmd-remote' => 'section',
'compress' => 'section',
@ -817,6 +831,47 @@ my $oConfigHelpData =
}
},
# CHECK Command Help
#---------------------------------------------------------------------------------------------------------------------------
'check' =>
{
summary =>
"Check the configuration.",
description =>
"The check command validates that pgBackRest and the archive_command setting are configured correctly for " .
"archiving and backups. It detects misconfigurations, particularly in archiving, that result in incomplete " .
"backups because required WAL segments did not reach the archive. The command can be run on the database or " .
"the backup host.\n" .
"\n" .
"Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog() are called to force " .
"PostgreSQL to archive a WAL segment. Restore points are only supported in PostgreSQL >= 9.1 so for older " .
"versions the check command may fail if there has been no write activity since the last log rotation.",
option =>
{
'archive-timeout' => 'section',
'backup-host' => 'section',
'backup-user' => 'section',
'buffer-size' => 'section',
'cmd-remote' => 'section',
'compress-level' => 'section',
'compress-level-network' => 'section',
'config' => 'default',
'config-remote' => 'section',
'db-path' => 'section',
'db-port' => 'section',
'db-socket-path' => 'section',
'db-timeout' => 'section',
'db-user' => 'section',
'log-level-console' => 'section',
'log-level-file' => 'section',
'log-path' => 'section',
'neutral-umask' => 'section',
'repo-path' => 'section',
'stanza' => 'default'
}
},
# EXPIRE Command Help
#---------------------------------------------------------------------------------------------------------------------------
'expire' =>

119
lib/pgBackRest/Db.pm
View File

@ -627,20 +627,8 @@ sub backupStart
{name => 'bStartFast'}
);
# Get the version from the control file
my ($strDbVersion) = $self->info($oFile, $strDbPath);
# Get version and db path from the database
my ($fCompareDbVersion, $strCompareDbPath) = $self->versionGet();
# Error if the version from the control file and the configured db-path do not match the values obtained from the database
if (!($strDbVersion == $fCompareDbVersion && $strDbPath eq $strCompareDbPath))
{
confess &log(ERROR,
"version '${fCompareDbVersion}' and db-path '${strCompareDbPath}' queried from cluster does not match" .
" version '${strDbVersion}' and db-path '${strDbPath}' read from '${strDbPath}/" . DB_FILE_PGCONTROL . "'\n" .
"HINT: the db-path and db-port settings likely reference different clusters", ERROR_DB_MISMATCH);
}
# Validate the database configuration
$self->configValidate($oFile, $strDbPath);
# Only allow start-fast option for version >= 8.4
if ($self->{strDbVersion} < PG_VERSION_84 && $bStartFast)
@ -649,20 +637,6 @@ sub backupStart
$bStartFast = false;
}
# Error if archive_mode = always (support has not been added yet)
if ($self->executeSql('show archive_mode') eq 'always')
{
confess &log(ERROR, "archive_mode=always not supported", ERROR_FEATURE_NOT_SUPPORTED);
}
# Check if archive_command is set
my $strArchiveCommand = $self->executeSql('show archive_command');
if (index($strArchiveCommand, BACKREST_EXE) == -1)
{
confess &log(ERROR, 'archive_command must contain \'' . BACKREST_EXE . '\'', ERROR_ARCHIVE_COMMAND_INVALID);
}
# Acquire the backup advisory lock to make sure that backups are not running from multiple backup servers against the same
# database cluster. This lock helps make the stop-auto option safe.
if (!$self->executeSqlOne('select pg_try_advisory_lock(' . DB_BACKUP_ADVISORY_LOCK . ')'))
@ -760,4 +734,93 @@ sub backupStop
);
}
####################################################################################################################################
# configValidate
#
# Validate the database configuration and archiving.
####################################################################################################################################
sub configValidate
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$oFile,
$strDbPath
) =
logDebugParam
(
__PACKAGE__ . '->configValidate', \@_,
{name => 'oFile'},
{name => 'strDbPath'}
);
# Get the version from the control file
my ($strDbVersion) = $self->info($oFile, $strDbPath);
# Get version and db path from the database
my ($fCompareDbVersion, $strCompareDbPath) = $self->versionGet();
# Error if the version from the control file and the configured db-path do not match the values obtained from the database
if (!($strDbVersion == $fCompareDbVersion && $strDbPath eq $strCompareDbPath))
{
confess &log(ERROR,
"version '${fCompareDbVersion}' and db-path '${strCompareDbPath}' queried from cluster does not match" .
" version '${strDbVersion}' and db-path '${strDbPath}' read from '${strDbPath}/" . DB_FILE_PGCONTROL . "'\n" .
"HINT: the db-path and db-port settings likely reference different clusters", ERROR_DB_MISMATCH);
}
# Error if archive_mode = always (support has not been added yet)
if ($self->executeSql('show archive_mode') eq 'always')
{
confess &log(ERROR, "archive_mode=always not supported", ERROR_FEATURE_NOT_SUPPORTED);
}
# Check if archive_command is set
my $strArchiveCommand = $self->executeSql('show archive_command');
if (index($strArchiveCommand, BACKREST_EXE) == -1)
{
confess &log(ERROR, 'archive_command must contain \'' . BACKREST_EXE . '\'', ERROR_ARCHIVE_COMMAND_INVALID);
}
return logDebugReturn
(
$strOperation
);
}
####################################################################################################################################
# xlogSwitch
#
# Forces a switch to the next transaction log in order to archive the current log.
####################################################################################################################################
sub xlogSwitch
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my $strOperation = logDebugParam(__PACKAGE__ . '->xlogSwitch');
# Create a restore point to ensure current xlog will be archived. For versions <= 9.0 activity will need to be generated by
# the user if there have been no writes since the last xlog switch.
if ($self->{strDbVersion} >= PG_VERSION_91)
{
$self->executeSql("select pg_create_restore_point('pgBackRest Archive Check');");
}
my $strWalFileName = $self->executeSqlRow('select pg_xlogfile_name from pg_xlogfile_name(pg_switch_xlog());');
&log(INFO, "switch xlog ${strWalFileName}");
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'strXlogFileName', value => $strWalFileName}
);
}
1;

2
lib/pgBackRest/Version.pm
View File

@ -26,7 +26,7 @@ use constant BACKREST_EXE => lc(BACKRE
# repositories or manifests can be read - that's the job of the format number.
#-----------------------------------------------------------------------------------------------------------------------------------
our # 'our' keyword is on a separate line to make the ExtUtils::MakeMaker parser happy.
$VERSION = '1.02';
$VERSION = '1.03dev';
push @EXPORT, qw($VERSION);

32
test/lib/pgBackRestTest/BackupCommonTest.pm
View File

@ -292,6 +292,7 @@ sub BackRestTestBackup_ClusterStart
my $bHotStandby = shift;
my $bArchive = shift;
my $bArchiveAlways = shift;
my $bArchiveInvalid = shift;
# Set default
$iPort = defined($iPort) ? $iPort : BackRestTestCommon_DbPortGet();
@ -307,7 +308,8 @@ sub BackRestTestBackup_ClusterStart
}
# Create the archive command
my $strArchive = BackRestTestCommon_CommandMainAbsGet() . ' --stanza=' . BackRestTestCommon_StanzaGet() .
my $strArchive = BackRestTestCommon_CommandMainAbsGet() . ' --stanza=' .
(defined($bArchiveInvalid) ? 'bogus' : BackRestTestCommon_StanzaGet()) .
' --config=' . BackRestTestCommon_DbPathGet() . '/pgbackrest.conf archive-push %p';
# Start the cluster
@ -389,6 +391,7 @@ sub BackRestTestBackup_ClusterCreate
my $iPort = shift;
my $bArchive = shift;
my $strXlogPath = shift;
my $bArchiveInvalid = shift;
# Defaults
$strPath = defined($strPath) ? $strPath : BackRestTestCommon_DbCommonPathGet();
@ -399,7 +402,7 @@ sub BackRestTestBackup_ClusterCreate
'/initdb' . (BackRestTestCommon_DbVersion() >= PG_VERSION_92 ? ' --xlogdir=${strXlogPath}' : '') .
" --pgdata=${strPath} --auth=trust");
BackRestTestBackup_ClusterStart($strPath, $iPort, undef, $bArchive);
BackRestTestBackup_ClusterStart($strPath, $iPort, undef, $bArchive, undef, $bArchiveInvalid);
# Connect user session
BackRestTestBackup_PgConnect();
@ -1306,6 +1309,31 @@ sub BackRestTestBackup_Backup
return BackRestTestBackup_BackupEnd();
}
####################################################################################################################################
# BackRestTestBackup_Check
####################################################################################################################################
push @EXPORT, qw(BackRestTestBackup_Check);
sub BackRestTestBackup_Check
{
my $strStanza = shift;
my $bRemote = shift;
my $iArchiveTimeout = shift;
my $strComment = shift;
my $iExpectedExitStatus = shift;
$strComment = "check" . (defined($strStanza) ? " ${strStanza}" : '') . " (" . $strComment . ")";
&log(INFO, " $strComment");
my $strCommand = ($bRemote ? BackRestTestCommon_CommandMainAbsGet() : BackRestTestCommon_CommandMainGet()) .
' --config=' . ($bRemote ? BackRestTestCommon_RepoPathGet() : BackRestTestCommon_DbPathGet()) .
"/pgbackrest.conf --archive-timeout=${iArchiveTimeout} --stanza=${strStanza} check --log-level-console=detail";
executeTest($strCommand,
{bRemote => $bRemote, strComment => $strComment, iExpectedExitStatus => $iExpectedExitStatus,
oLogTest => $oBackupLogTest});
}
####################################################################################################################################
# BackRestTestBackup_Info
####################################################################################################################################

63
test/lib/pgBackRestTest/BackupTest.pm
View File

@ -1717,7 +1717,8 @@ sub BackRestTestBackup_Test
# Create the cluster
if ($bCreate)
{
BackRestTestBackup_ClusterCreate();
# For the 'fail on missing archive.info file' test, the archive.info file must not be found so set archive invalid
BackRestTestBackup_ClusterCreate(undef, undef, undef, undef, true);
$bCreate = false;
}
@ -1754,16 +1755,68 @@ sub BackRestTestBackup_Test
# Test invalid archive command
#-----------------------------------------------------------------------------------------------------------------------
$strType = BACKUP_TYPE_FULL;
$strComment = 'fail on invalid archive_command';
# NOTE: This must run before the success test since that will create the archive.info file
$strComment = 'fail on missing archive.info file';
BackRestTestBackup_Check($strStanza, $bRemote, 0.1, $strComment, ERROR_FILE_MISSING);
# Clean up the archive_timeout error from the postgresql log by stopping the cluster and removing the log file before
# running the next test
BackRestTestBackup_ClusterStop(undef, undef, true);
BackRestTestCommon_FileRemove(BackRestTestCommon_DbCommonPathGet() . '/postgresql.log');
# Check archive_command_not_set error
BackRestTestBackup_ClusterStop();
$strComment = 'fail on invalid archive_command';
BackRestTestBackup_ClusterStart(undef, undef, undef, false);
BackRestTestBackup_Backup($strType, $strStanza, $strComment, {iExpectedExitStatus => ERROR_ARCHIVE_COMMAND_INVALID});
BackRestTestBackup_Backup(
$strType, $strStanza, $strComment,
{iExpectedExitStatus => ERROR_ARCHIVE_COMMAND_INVALID});
BackRestTestBackup_Check($strStanza, $bRemote, 0.1, $strComment, ERROR_ARCHIVE_COMMAND_INVALID);
# If running the remote tests then also need to run check locally
if ($bRemote)
{
BackRestTestBackup_Check($strStanza, false, 0.1, "${strComment} - remote", ERROR_ARCHIVE_COMMAND_INVALID);
}
# Clean up the archive_command error from the postgresql log by stopping the cluster and removing the log file before
# running the next test
BackRestTestBackup_ClusterStop(undef, undef, true);
BackRestTestCommon_FileRemove(BackRestTestCommon_DbCommonPathGet() . '/postgresql.log');
# Providing a sufficient archive-timeout, verify that the check command runs successfully.
$strComment = 'verify success';
BackRestTestBackup_ClusterStart();
BackRestTestBackup_Check($strStanza, $bRemote, 5, $strComment, 0);
# If running the remote tests then also need to run check locally
if ($bRemote)
{
BackRestTestBackup_Check($strStanza, false, 5, "${strComment} - remote", 0);
}
# Check archive_timeout error
$strComment = 'fail on archive timeout';
BackRestTestBackup_ClusterStop();
BackRestTestBackup_ClusterStart(undef, undef, undef, undef, undef, true);
BackRestTestBackup_Check($strStanza, $bRemote, 0.1, $strComment, ERROR_ARCHIVE_TIMEOUT);
# If running the remote tests then also need to run check locally
if ($bRemote)
{
BackRestTestBackup_Check($strStanza, false, 0.1, "${strComment} - remote", ERROR_ARCHIVE_TIMEOUT);
}
# Clean up the archive_timeout error from the postgresql log by stopping the cluster and removing the log file
# before running the next test
BackRestTestBackup_ClusterStop(undef, undef, true);
BackRestTestCommon_FileRemove(BackRestTestCommon_DbCommonPathGet() . '/postgresql.log');
# Reset the cluster to a normal state so the next test will work
BackRestTestBackup_ClusterStop();
BackRestTestBackup_ClusterStart();
# Full backup