1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-03-05 15:05:48 +02:00

v2.02: Parallel Asynchronous Archive Get and Configuration Includes

Bug Fixes:

* Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
* Fix archive-copy throwing "path not found" error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
* Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
* Fix delta restore failing when a linked file is missing.
* Fix rendering of key/value and list options in help. (Reported by Clinton Adams.)

Features:

* Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
* Add support for additional pgBackRest configuration files in the directory specified by the --config-include-path option. Add --config-path option for overriding the default base path of the --config and --config-include-path option. (Contributed by Cynthia Shang.)
* Add repo-s3-token option to allow temporary credentials tokens to be configured. pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire. (Contributed by Yogesh Sharma.)

Improvements:

* Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. (Contributed by Cynthia Shang.)
* Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path.
* Show command parameters as well as command options in initial info log message.
* Rename archive-queue-max option to archive-push-queue-max to avoid confusion with the new archive-get-queue-max option. The old option name will continue to be accepted.
This commit is contained in:
David Steele 2018-05-06 19:53:42 -04:00
parent 4d6a51ac47
commit ee8aafb3ca
8 changed files with 2323 additions and 844 deletions

View File

@ -6,9 +6,9 @@ pgBackRest aims to be a simple, reliable backup and restore system that can seam
Instead of relying on traditional backup tools like tar and rsync, pgBackRest implements all backup features internally and uses a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows for better solutions to database-specific backup challenges. The custom remote protocol allows for more flexibility and limits the types of connections that are required to perform a backup which increases security.
pgBackRest [v2.01](https://github.com/pgbackrest/pgbackrest/releases/tag/release/2.01) is the current stable release. Release notes are on the [Releases](http://www.pgbackrest.org/release.html) page.
pgBackRest [v2.02](https://github.com/pgbackrest/pgbackrest/releases/tag/release/2.02) is the current stable release. Release notes are on the [Releases](http://www.pgbackrest.org/release.html) page.
pgBackRest v1 will receive bugs fixes only until EOL. Documentation for v1 can be found [here](http://www.pgbackrest.org/1).
Documentation for v1 can be found [here](http://www.pgbackrest.org/1). No further releases are planned for v1 because v2 is backward-compatible with v1 options and repositories.
## Features
@ -60,13 +60,15 @@ If the repository is on a backup server, compression is performed on the databas
The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are taken for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Parallel processing can lead to a dramatic reduction in restore times.
### Parallel WAL Archiving
### Parallel, Asynchronous WAL Push & Get
Dedicated commands are included for both pushing WAL to the archive and retrieving WAL from the archive.
Dedicated commands are included for pushing WAL to the archive and getting WAL from the archive. Both commands support parallelism to accelerate processing and run asynchronously to provide the fastest possible response time to PostgreSQL.
The push command automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. The push and get commands both ensure that the database and repository match by comparing PostgreSQL versions and system identifiers. This precludes the possibility of misconfiguring the WAL archive location.
WAL push automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. Asynchronous WAL push allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.
Asynchronous archiving allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.
Asynchronous WAL get maintains a local queue of WAL segments that are decompressed and ready for replay. This reduces the time needed to provide WAL to PostgreSQL which maximizes replay speed. Higher-latency connections and storage (such as S3) benefit the most.
The push and get commands both ensure that the database and repository match by comparing PostgreSQL versions and system identifiers. This virtually eliminates the possibility of misconfiguring the WAL archive location.
### Tablespace & Link Support

File diff suppressed because it is too large Load Diff

View File

@ -36,7 +36,7 @@
<p><backrest/> <link url="{[github-url-base]}/releases/tag/release/{[version-stable]}">v{[version-stable]}</link> is the current stable release. Release notes are on the <link page="{[backrest-page-release]}">Releases</link> page.</p>
<p><backrest/> <proper>v1</proper> will receive bugs fixes only until EOL. Documentation for <proper>v1</proper> can be found <link url="{[backrest-url-base]}/1">here</link>.</p>
<p>Documentation for <proper>v1</proper> can be found <link url="{[backrest-url-base]}/1">here</link>. No further releases are planned for <proper>v1</proper> because <proper>v2</proper> is backward-compatible with <proper>v1</proper> options and repositories.</p>
</section>
<section id="features">
@ -109,13 +109,15 @@
</section>
<section id="parallel-archiving">
<title>Parallel WAL Archiving</title>
<title>Parallel, Asynchronous WAL Push &amp; Get</title>
<p>Dedicated commands are included for both pushing WAL to the archive and retrieving WAL from the archive.</p>
<p>Dedicated commands are included for pushing WAL to the archive and getting WAL from the archive. Both commands support parallelism to accelerate processing and run asynchronously to provide the fastest possible response time to <postgres/>.</p>
<p>The push command automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. The push and get commands both ensure that the database and repository match by comparing <postgres/> versions and system identifiers. This precludes the possibility of misconfiguring the WAL archive location.</p>
<p>WAL push automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. Asynchronous WAL push allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.</p>
<p>Asynchronous archiving allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.</p>
<p>Asynchronous WAL get maintains a local queue of WAL segments that are decompressed and ready for replay. This reduces the time needed to provide WAL to <postgres/> which maximizes replay speed. Higher-latency connections and storage (such as <proper>S3</proper>) benefit the most.</p>
<p>The push and get commands both ensure that the database and repository match by comparing <postgres/> versions and system identifiers. This virtually eliminates the possibility of misconfiguring the WAL archive location.</p>
</section>
<section id="tablespace-link-support">

View File

@ -12,7 +12,7 @@
</intro>
<release-list>
<release date="XXXX-XX-XX" version="2.02dev" title="UNDER DEVELOPMENT">
<release date="2018-05-06" version="2.02" title="Parallel Asynchronous Archive Get and Configuration Includes">
<release-core-list>
<release-bug-list>
<release-item>
@ -41,7 +41,7 @@
</release-item>
<release-item>
<p>Fix delta restore failing when a linked file was missing.</p>
<p>Fix delta restore failing when a linked file is missing.</p>
</release-item>
<release-item>
@ -76,10 +76,6 @@
</release-feature-list>
<release-improvement-list>
<release-item>
<p>Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path. Remove recursive path sync functionality since it is no longer used.</p>
</release-item>
<release-item>
<release-item-contributor-list>
<release-item-contributor id="shang.cynthia"/>
@ -89,11 +85,15 @@
</release-item>
<release-item>
<p>Show command parameters as well as command options in initial info message.</p>
<p>Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path.</p>
</release-item>
<release-item>
<p>Rename <br-option>archive-queue-max</br-option> option to <br-option>archive-push-queue-max</br-option> to avoid confusion with the <br-option>archive-get-queue-max</br-option> option. The old option name will continue to be accepted.</p>
<p>Show command parameters as well as command options in initial info log message.</p>
</release-item>
<release-item>
<p>Rename <br-option>archive-queue-max</br-option> option to <br-option>archive-push-queue-max</br-option> to avoid confusion with the new <br-option>archive-get-queue-max</br-option> option. The old option name will continue to be accepted.</p>
</release-item>
</release-improvement-list>
@ -111,7 +111,7 @@
</release-item>
<release-item>
<p>Storage object improvements. Convert all functions to variadic functions. Enforce read-only storage. Add <code>storageLocalWrite()</code> helper function. Add <code>storageCopy()</code>, <code>storageExists()</code>, <code>storageMove()</code>, <code>storageNewRead()</code>/<code>storageNewWrite()</code>, <code>storagePathCreate()</code>, <code>storagePathRemove()</code>, <code>storagePathSync()</code>, and <code>storageRemove()</code>. Add <code>StorageFileRead</code> and <code>StorageFileWrite</code> objects. Abstract Posix driver code into a separate module. Call <code>storagePathCreate()</code> from the Perl Posix driver.</p>
<p>Storage object improvements. Convert all functions to variadic functions. Enforce read-only storage. Add <code>storageLocalWrite()</code> helper function. Add <code>storageCopy()</code>, <code>storageExists()</code>, <code>storageMove()</code>, <code>storageNewRead()</code>/<code>storageNewWrite()</code>, <code>storagePathCreate()</code>, <code>storagePathRemove()</code>, <code>storagePathSync()</code>, and <code>storageRemove()</code>. Add <code>StorageFileRead</code> and <code>StorageFileWrite</code> objects. Abstract Posix driver code into a separate module. Call <code>storagePathRemove()</code> from the Perl Posix driver.</p>
</release-item>
<release-item>
@ -168,7 +168,7 @@
</release-item>
<release-item>
<p>Split debug and assert code into separate headers. Assert can be used earlier because it only depends on the error-handler and not logging. Add <code>ASSERT()</code> that is preserved in production builds.</p>
<p>Split debug and assert code into separate headers. Assert can be used earlier because it only depends on the error-handler and not logging. Add <code>ASSERT()</code> macro which is preserved in production builds.</p>
</release-item>
<release-item>
@ -255,7 +255,7 @@
</release-item>
<release-item>
<p>Add <code>HARNESS_FORK</code> for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.</p>
<p>Add <code>HARNESS_FORK</code> macros for tests that require fork(). A standard pattern for tests makes fork() easier to use and should help prevent some common mistakes.</p>
</release-item>
<release-item>

View File

@ -39,7 +39,7 @@ push @EXPORT, qw(backrestBin backrestBinSet);
# Defines the current version of the BackRest executable. The version number is used to track features but does not affect what
# repositories or manifests can be read - that's the job of the format number.
#-----------------------------------------------------------------------------------------------------------------------------------
use constant BACKREST_VERSION => '2.02dev';
use constant BACKREST_VERSION => '2.02';
push @EXPORT, qw(BACKREST_VERSION);
# Format Format Number

View File

@ -6,7 +6,7 @@ package pgBackRest::LibCAuto;
# Library version (.999 indicates development version)
sub libcAutoVersion
{
return '2.02.999';
return '2.02';
}
# Configuration option value constants

View File

@ -17,6 +17,6 @@ Standard binary name
/***********************************************************************************************************************************
Version of the software. Currently this value is maintained in Version.pm and updated by test.pl.
***********************************************************************************************************************************/
#define PGBACKREST_VERSION "2.02dev"
#define PGBACKREST_VERSION "2.02"
#endif

View File

@ -151,6 +151,14 @@ lib/pgBackRest/Archive/Common.pm:
class: core
type: perl
lib/pgBackRest/Archive/Get/Async.pm:
class: core
type: perl
lib/pgBackRest/Archive/Get/File.pm:
class: core
type: perl
lib/pgBackRest/Archive/Get/Get.pm:
class: core
type: perl
@ -539,6 +547,22 @@ src/cipher/random.h:
class: core
type: c/h
src/command/archive/common.c:
class: core
type: c
src/command/archive/common.h:
class: core
type: c/h
src/command/archive/get/get.c:
class: core
type: c
src/command/archive/get/get.h:
class: core
type: c/h
src/command/archive/push/push.c:
class: core
type: c
@ -811,6 +835,14 @@ src/perl/exec.h:
class: core
type: c/h
src/postgres/info.c:
class: core
type: c
src/postgres/info.h:
class: core
type: c/h
src/postgres/pageChecksum.c:
class: core
type: c
@ -819,19 +851,59 @@ src/postgres/pageChecksum.h:
class: core
type: c/h
src/storage/driver/posix.c:
class: core
type: c
src/storage/driver/posix.h:
src/postgres/type.h:
class: core
type: c/h
src/storage/file.c:
src/postgres/version.h:
class: core
type: c/h
src/storage/driver/posix/driver.c:
class: core
type: c
src/storage/file.h:
src/storage/driver/posix/driver.h:
class: core
type: c/h
src/storage/driver/posix/driverFile.c:
class: core
type: c
src/storage/driver/posix/driverFile.h:
class: core
type: c/h
src/storage/driver/posix/driverRead.c:
class: core
type: c
src/storage/driver/posix/driverRead.h:
class: core
type: c/h
src/storage/driver/posix/driverWrite.c:
class: core
type: c
src/storage/driver/posix/driverWrite.h:
class: core
type: c/h
src/storage/fileRead.c:
class: core
type: c
src/storage/fileRead.h:
class: core
type: c/h
src/storage/fileWrite.c:
class: core
type: c
src/storage/fileWrite.h:
class: core
type: c/h
@ -843,6 +915,10 @@ src/storage/helper.h:
class: core
type: c/h
src/storage/info.h:
class: core
type: c/h
src/storage/storage.c:
class: core
type: c
@ -955,15 +1031,15 @@ test/lib/pgBackRestTest/Env/S3EnvTest.pm:
class: test/harness
type: perl
test/lib/pgBackRestTest/Module/Archive/ArchiveCommonTest.pm:
test/lib/pgBackRestTest/Module/Archive/ArchiveCommonPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Archive/ArchiveGetTest.pm:
test/lib/pgBackRestTest/Module/Archive/ArchiveGetPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Archive/ArchiveInfoUnitTest.pm:
test/lib/pgBackRestTest/Module/Archive/ArchiveInfoUnitPerlTest.pm:
class: test/module
type: perl
@ -971,11 +1047,11 @@ test/lib/pgBackRestTest/Module/Archive/ArchivePushPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Backup/BackupInfoUnitTest.pm:
test/lib/pgBackRestTest/Module/Backup/BackupInfoUnitPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Backup/BackupUnitTest.pm:
test/lib/pgBackRestTest/Module/Backup/BackupUnitPerlTest.pm:
class: test/module
type: perl
@ -983,7 +1059,7 @@ test/lib/pgBackRestTest/Module/Common/CommonEncodePerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Common/CommonHttpClientTest.pm:
test/lib/pgBackRestTest/Module/Common/CommonHttpClientPerlTest.pm:
class: test/module
type: perl
@ -991,7 +1067,7 @@ test/lib/pgBackRestTest/Module/Common/CommonIniPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Common/CommonIoBufferedTest.pm:
test/lib/pgBackRestTest/Module/Common/CommonIoBufferedPerlTest.pm:
class: test/module
type: perl
@ -999,7 +1075,7 @@ test/lib/pgBackRestTest/Module/Common/CommonIoHandlePerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Common/CommonIoProcessTest.pm:
test/lib/pgBackRestTest/Module/Common/CommonIoProcessPerlTest.pm:
class: test/module
type: perl
@ -1007,11 +1083,11 @@ test/lib/pgBackRestTest/Module/Common/CommonLogPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Info/InfoUnitTest.pm:
test/lib/pgBackRestTest/Module/Info/InfoUnitPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Manifest/ManifestAllTest.pm:
test/lib/pgBackRestTest/Module/Manifest/ManifestAllPerlTest.pm:
class: test/module
type: perl
@ -1043,11 +1119,11 @@ test/lib/pgBackRestTest/Module/Performance/PerformanceIoTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Protocol/ProtocolCommonMinionTest.pm:
test/lib/pgBackRestTest/Module/Protocol/ProtocolCommonMinionPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperTest.pm:
test/lib/pgBackRestTest/Module/Protocol/ProtocolHelperPerlTest.pm:
class: test/module
type: perl
@ -1055,19 +1131,19 @@ test/lib/pgBackRestTest/Module/Real/RealAllTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Stanza/StanzaAllTest.pm:
test/lib/pgBackRestTest/Module/Stanza/StanzaAllPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageFilterCipherBlockTest.pm:
test/lib/pgBackRestTest/Module/Storage/StorageFilterCipherBlockPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageFilterGzipTest.pm:
test/lib/pgBackRestTest/Module/Storage/StorageFilterGzipPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageFilterShaTest.pm:
test/lib/pgBackRestTest/Module/Storage/StorageFilterShaPerlTest.pm:
class: test/module
type: perl
@ -1075,27 +1151,27 @@ test/lib/pgBackRestTest/Module/Storage/StorageHelperPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageLocalTest.pm:
test/lib/pgBackRestTest/Module/Storage/StorageLocalPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StoragePosixTest.pm:
test/lib/pgBackRestTest/Module/Storage/StoragePosixPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageS3AuthTest.pm:
test/lib/pgBackRestTest/Module/Storage/StorageS3AuthPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageS3CertTest.pm:
test/lib/pgBackRestTest/Module/Storage/StorageS3CertPerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageS3RequestTest.pm:
test/lib/pgBackRestTest/Module/Storage/StorageS3PerlTest.pm:
class: test/module
type: perl
test/lib/pgBackRestTest/Module/Storage/StorageS3Test.pm:
test/lib/pgBackRestTest/Module/Storage/StorageS3RequestPerlTest.pm:
class: test/module
type: perl
@ -1107,6 +1183,10 @@ test/src/common/harnessConfig.h:
class: test/harness
type: c/h
test/src/common/harnessFork.h:
class: test/harness
type: c/h
test/src/common/harnessTest.c:
class: test/harness
type: c
@ -1123,6 +1203,14 @@ test/src/common/logTest.h:
class: test/harness
type: c/h
test/src/module/archive/commonTest.c:
class: test/module
type: c
test/src/module/archive/getTest.c:
class: test/module
type: c
test/src/module/archive/pushTest.c:
class: test/module
type: c
@ -1263,6 +1351,10 @@ test/src/module/perl/execTest.c:
class: test/module
type: c
test/src/module/postgres/infoTest.c:
class: test/module
type: c
test/src/module/postgres/pageChecksumTest.c:
class: test/module
type: c