db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2025-09-16 09:06:18 +02:00
Code Issues Releases Activity

Added documentation for parallel archive-push.

Browse Source
This commit is contained in:
David Steele
2017-02-05 10:35:53 -05:00
parent 8b5f37c2a8
commit 2237c3fc1b
4 changed files with 41 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@@ -10,9 +10,9 @@ pgBackRest [v1.12](https://github.com/pgbackrest/pgbackrest/releases/tag/release
## Features
### Multi-process Backup & Restore
### Parallel Backup & Restore
Compression is usually the bottleneck during backup operations but, even with now ubiquitous multi-core servers, most database backup solutions are still single-process. pgBackRest solves the compression bottleneck with multi-processing.
Compression is usually the bottleneck during backup operations but, even with now ubiquitous multi-core servers, most database backup solutions are still single-process. pgBackRest solves the compression bottleneck with parallel processing.
Utilizing multiple cores for compression makes it possible to achieve 1TB/hr raw throughput even on a 1Gb/s link. More cores and a larger pipe lead to even higher throughput.
@@ -48,15 +48,15 @@ If the repository is on a backup server, compression is performed on the databas
### Delta Restore
The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are taken for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Multi-processing can lead to a dramatic reduction in restore times.
The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are taken for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Parallel processing can lead to a dramatic reduction in restore times.
### Advanced Archiving
### Parallel WAL Archiving
Dedicated commands are included for both pushing WAL to the archive and retrieving WAL from the archive.
The push command automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. The push and get commands both ensure that the database and repository match by comparing PostgreSQL versions and system identifiers. This precludes the possibility of misconfiguring the WAL archive location.
Asynchronous archiving allows compression and transfer to be offloaded to another process which maintains a continuous connection to the remote server, improving throughput significantly. This can be a critical feature for databases with extremely high write volume.
Asynchronous archiving allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.
### Tablespace & Link Support

16
doc/xml/index.xml
View File

@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE doc SYSTEM "doc.dtd">
<doc title="{[project]}" subtitle="Reliable {[postgres]} Backup &amp; Restore" toc="n">
<description>{[project]} provides fast, reliable backup and restore for {[postgres]} and seamlessly scales to terabyte scale databases by implementing stream compression and multi-processing.</description>
<description>{[project]} provides fast, reliable backup and restore for {[postgres]} and seamlessly scales to terabyte scale databases by implementing stream compression and parallel processing.</description>
<variable-list>
<!-- Variables used by the rest of the script -->
@@ -40,10 +40,10 @@
<section id="features">
<title>Features</title>
<section id="multi-process">
<title>Multi-process Backup &amp; Restore</title>
<section id="parallel-backup-restore">
<title>Parallel Backup &amp; Restore</title>
<p>Compression is usually the bottleneck during backup operations but, even with now ubiquitous multi-core servers, most database backup solutions are still single-process. <backrest/> solves the compression bottleneck with multi-processing.</p>
<p>Compression is usually the bottleneck during backup operations but, even with now ubiquitous multi-core servers, most database backup solutions are still single-process. <backrest/> solves the compression bottleneck with parallel processing.</p>
<p>Utilizing multiple cores for compression makes it possible to achieve 1TB/hr raw throughput even on a 1Gb/s link. More cores and a larger pipe lead to even higher throughput.</p>
</section>
@@ -93,17 +93,17 @@
<section id="delta-restore">
<title>Delta Restore</title>
<p>The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are taken for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Multi-processing can lead to a dramatic reduction in restore times.</p>
<p>The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are taken for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Parallel processing can lead to a dramatic reduction in restore times.</p>
</section>
<section id="advanced-archiving">
<title>Advanced Archiving</title>
<section id="parallel-archiving">
<title>Parallel WAL Archiving</title>
<p>Dedicated commands are included for both pushing WAL to the archive and retrieving WAL from the archive.</p>
<p>The push command automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. The push and get commands both ensure that the database and repository match by comparing <postgres/> versions and system identifiers. This precludes the possibility of misconfiguring the WAL archive location.</p>
<p>Asynchronous archiving allows compression and transfer to be offloaded to another process which maintains a continuous connection to the remote server, improving throughput significantly. This can be a critical feature for databases with extremely high write volume.</p>
<p>Asynchronous archiving allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.</p>
</section>
<section id="tablespace-link-support">

55
doc/xml/user-guide.xml
View File

@@ -1364,6 +1364,8 @@
<backrest-config-option section="global" key="backup-host">{[host-backup]}</backrest-config-option>
<backrest-config-option section="global" key="backup-user">backrest</backrest-config-option>
<backrest-config-option section="global" key="log-level-file">detail</backrest-config-option>
<backrest-config-option section="global" key="log-level-stderr">off</backrest-config-option>
</backrest-config>
@@ -1482,19 +1484,35 @@
</execute>
</execute-list>
<p>The spool path must be configured and asynchronous archiving enabled.</p>
<p>The spool path must be configured and asynchronous archiving enabled. Asynchronous archiving automatically confers some benefit by reducing the number of ssh connections made to the backup server, but setting <br-option>process-max</br-option> can drastically improve performance. Be sure not to set <br-option>process-max</br-option> so high that it affects normal database operations.</p>
<backrest-config host="{[host-db-master]}" file="{[backrest-config-demo]}">
<title>Configure the spool path and asynchronous archiving</title>
<backrest-config-option section="global" key="spool-path">{[spool-path]}</backrest-config-option>
<backrest-config-option section="global" key="archive-async">y</backrest-config-option>
<backrest-config-option section="global:archive-push" key="process-max">2</backrest-config-option>
</backrest-config>
<p>The <cmd>check</cmd> command ensures that asynchronous archiving is working.</p>
<p>The <file>archive-async.log</file> file can be used to monitor the activity of the asynchronous process. A good way to test this is to quickly push a number of WAL segments.</p>
<execute-list host="{[host-db-master]}">
<title>Check asynchronous archiving</title>
<title>Test parallel asynchronous archiving</title>
<execute output="n" show="n">
<exe-cmd>rm -f /var/log/pgbackrest/demo-archive-async.log</exe-cmd>
</execute>
<execute output="n">
<exe-cmd>
psql -c "
select pg_create_restore_point('test async push'); select pg_switch_xlog();
select pg_create_restore_point('test async push'); select pg_switch_xlog();
select pg_create_restore_point('test async push'); select pg_switch_xlog();
select pg_create_restore_point('test async push'); select pg_switch_xlog();
select pg_create_restore_point('test async push'); select pg_switch_xlog();"
</exe-cmd>
</execute>
<execute>
<exe-cmd>{[project-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-log-level-console=info check</exe-cmd>
@@ -1502,37 +1520,14 @@
</execute>
</execute-list>
<!-- DEBUG test to ensure async archiving is working in batch until better tests can be written -->
<execute-list host="{[host-db-master]}" keyword="debug">
<title>DEBUG asynchronous archiving</title>
<p>Now the log file will contain parallel, asynchronous activity.</p>
<execute output="n">
<exe-cmd>rm /var/log/pgbackrest/demo-archive-async.log</exe-cmd>
</execute>
<execute output="n">
<exe-cmd>
psql -c "
select pg_create_restore_point('test asynchronous archiving');
select pg_switch_xlog();
select pg_create_restore_point('test asynchronous archiving');
select pg_switch_xlog();
select pg_create_restore_point('test asynchronous archiving');
select pg_switch_xlog();
select pg_create_restore_point('test asynchronous archiving');
select pg_switch_xlog();
select pg_create_restore_point('test asynchronous archiving');
select pg_switch_xlog();"
</exe-cmd>
</execute>
<execute output="n">
<exe-cmd>sleep 5</exe-cmd>
</execute>
<execute-list host="{[host-db-master]}">
<title>Check results in the log</title>
<execute output="y">
<exe-cmd>cat /var/log/pgbackrest/demo-archive-async.log</exe-cmd>
<exe-highlight>WAL segments to archive</exe-highlight>
<exe-highlight> WAL file\(s\) to archive|pushed WAL file 0000000</exe-highlight>
</execute>
</execute-list>
</section>

6
lib/pgBackRest/Archive/ArchivePushAsync.pm
View File

@@ -215,8 +215,8 @@ sub processQueue
if ($self->{oArchiveProcess}->jobTotal() > 0)
{
&log(INFO,
@{$stryWalFile} . " new WAL file(s) to archive: " .
${$stryWalFile}[0] . (@{$stryWalFile} > 1 ? " ... ${$stryWalFile}[-1]" : ''));
'push ' . @{$stryWalFile} . ' WAL file(s) to archive: ' .
${$stryWalFile}[0] . (@{$stryWalFile} > 1 ? "...${$stryWalFile}[-1]" : ''));
eval
{
@@ -248,7 +248,7 @@ sub processQueue
$iOkTotal++;
&log(DETAIL, "pushed WAL file ${strWalFile} to archive");
&log(DETAIL, "pushed WAL file ${strWalFile} to archive", undef, undef, undef, $hJob->{iProcessId});
}
}