db/pgbackrest
1
0
Fork 0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Code Issues Releases Activity

Implemented issue #148: Website with basic user guide.

Browse Source
This commit is contained in:
David Steele 2015-10-28 10:10:36 +01:00
parent 3dc0142244
commit 57a06ba3b1
39 changed files with 4815 additions and 2923 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -5,3 +5,4 @@ test/test
test/vm/.vagrant
test/nytprof.out
test/nytprof/*
doc/html/*

124
CHANGELOG.md
View File

@ -1,22 +1,26 @@
# pgBackRest - Change Log
# pgBackRest<br/>Change Log
## v0.87: VIENNA MILESTONE - UNDER DEVELOPMENT
__No Release Date Set__
*
* Added a new user guide that covers pgBackRest basics and some advanced topics including PITR. Much more to come, but it's a start.
* The website, markdown, and command-line help are now all generated from the same XML source.
* The `backup_label.old` and `recovery.done` files are now excluded from backups.
## v0.85: Start/Stop Commands and Minor Bug Fixes
__Released October 8, 2015__
* Added new feature to allow all pgBackRest operations to be stopped or started using the `stop` and `start` commands. This prevents any pgBackRest processes from running on a system where PostgreSQL is shutdown or the system needs to be quiesced for some reason.
* Added new feature to allow all pgBackRest operations to be stopped or started using the `stop` and `start` commands. This prevents any pgBackRest processes from running on a system where PostgreSQL is shutdown or the system needs to be quiesced for some reason.
* Removed dependency on `IO::String` module.
* Fixed an issue where an error could be returned after a backup or restore completely successfully.
* Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed. This scenario was likely if the backup process got terminated during the copy phase.
* Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed. This scenario was likely if the backup process got terminated during the copy phase.
* Experimental support for PostgreSQL 9.5 beta1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for `--target-resume` tests (this functionality has changed in 9.5) and there is no testing yet for `.partial` WAL segments.
* Experimental support for PostgreSQL 9.5 beta1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for `--target-resume` tests (this functionality has changed in 9.5) and there is no testing yet for `.partial` WAL segments.
## v0.82: Refactoring, Command-line Help, and Minor Bug Fixes
__Released September 14, 2015__
@ -31,9 +35,9 @@ __Released September 14, 2015__
* Fixed an issue where the pgBackRest version number was not being updated in `backup.info` and `archive.info` after an upgrade/downgrade.
* Fixed an issue where the `info` command was throwing an exception when the repository contained no stanzas. _Reported by Stephen Frost_.
* Fixed an issue where the `info` command was throwing an exception when the repository contained no stanzas. _Reported by Stephen Frost_.
* Fixed an issue where the PostgreSQL `pg_stop_backup()` NOTICEs were being output to stderr. _Reported by Stephen Frost_.
* Fixed an issue where the PostgreSQL `pg_stop_backup()` NOTICEs were being output to stderr. _Reported by Stephen Frost_.
* Renamed `recovery-setting` option and section to `recovery-option` to be more consistent with pgBackRest naming conventions.
@ -43,45 +47,45 @@ __Released September 14, 2015__
* Added dynamic module loading to speed up commands, especially asynchronous archiving.
* Expiration tests are now synthetic rather than based on actual backups. This will allow development of more advanced expiration features.
* Expiration tests are now synthetic rather than based on actual backups. This will allow development of more advanced expiration features.
* Experimental support for PostgreSQL 9.5 alpha2. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for `--target-resume` tests (this functionality has changed in 9.5) and there is no testing yet for `.partial` WAL segments.
* Experimental support for PostgreSQL 9.5 alpha2. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for `--target-resume` tests (this functionality has changed in 9.5) and there is no testing yet for `.partial` WAL segments.
## v0.80: DBI Support, Stability, and Convenience Features
__Released August 9, 2015__
* Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the `info` command. Only `text` output was affected -- `json` output reported the correct epoch values. _Reported by Michael Renner_.
* Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the `info` command. Only `text` output was affected -- `json` output reported the correct epoch values. _Reported by Michael Renner_.
* Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.
* Now using Perl `DBI` and `DBD::Pg` for connections to PostgreSQL rather than `psql`. The `cmd-psql` and `cmd-psql-option` settings have been removed and replaced with `db-port` and `db-socket-path`. Follow the instructions in [Installation](USERGUIDE.md#installation) to install `DBD::Pg` on your operating system.
* Now using Perl `DBI` and `DBD::Pg` for connections to PostgreSQL rather than `psql`. The `cmd-psql` and `cmd-psql-option` settings have been removed and replaced with `db-port` and `db-socket-path`. Follow the instructions in [Installation](USERGUIDE.md#installation) to install `DBD::Pg` on your operating system.
* Add [stop-auto](USERGUIDE.md#stop-auto-key) option to allow failed backups to automatically be stopped when a new backup starts.
* Add [db-timeout](USERGUIDE.md#db-timeout-key) option to limit the amount of time pgBackRest will wait for pg_start_backup() and pg_stop_backup() to return.
* Remove `pg_control` file at the beginning of the restore and copy it back at the very end. This prevents the possibility that a partial restore can be started by PostgreSQL.
* Remove `pg_control` file at the beginning of the restore and copy it back at the very end. This prevents the possibility that a partial restore can be started by PostgreSQL.
* The repository is now created and updated with consistent directory and file modes. By default `umask` is set to `0000` but this can be disabled with the `neutral-umask` setting.
* The repository is now created and updated with consistent directory and file modes. By default `umask` is set to `0000` but this can be disabled with the `neutral-umask` setting. _Reported by Cynthia Shang_
* Added checks to be sure the `db-path` setting is consistent with `db-port` by comparing the `data_directory` as reported by the cluster against the `db-path` setting and the version as reported by the cluster against the value read from `pg_control`. The `db-socket-path` setting is checked to be sure it is an absolute path.
* Added checks to be sure the `db-path` setting is consistent with `db-port` by comparing the `data_directory` as reported by the cluster against the `db-path` setting and the version as reported by the cluster against the value read from `pg_control`. The `db-socket-path` setting is checked to be sure it is an absolute path.
* Experimental support for PostgreSQL 9.5 alpha1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for `--target-resume` tests (this functionality has changed in 9.5) and there is no testing yet for `.partial` WAL segments.
* Experimental support for PostgreSQL 9.5 alpha1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for `--target-resume` tests (this functionality has changed in 9.5) and there is no testing yet for `.partial` WAL segments.
* Major refactoring of the protocol layer to support future development.
* Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.
* Split most of `README.md` out into `USERGUIDE.md` and `CHANGELOG.md` because it was becoming unwieldy. Changed most references to "database" in the user guide to "database cluster" for clarity.
* Split most of `README.md` out into `USERGUIDE.md` and `CHANGELOG.md` because it was becoming unwieldy. Changed most references to "database" in the user guide to "database cluster" for clarity.
## v0.78: Remove CPAN Dependencies, Stability Improvements
__Released July 13, 2015__
* Removed dependency on CPAN packages for multi-threaded operation. While it might not be a bad idea to update the `threads` and `Thread::Queue` packages, it is no longer necessary.
* Removed dependency on CPAN packages for multi-threaded operation. While it might not be a bad idea to update the `threads` and `Thread::Queue` packages, it is no longer necessary.
* Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.
* Modified wait backoff to use a Fibonacci rather than geometric sequence. This will make wait time grow less aggressively while still giving reasonable values.
* Modified wait backoff to use a Fibonacci rather than geometric sequence. This will make wait time grow less aggressively while still giving reasonable values.
* More options for regression tests and improved code to run in a variety of environments.
@ -90,72 +94,72 @@ __Released June 30, 2015__
* Removed `pg_backrest_remote` and added the functionality to `pg_backrest` as the `remote` command.
* Added file and directory syncs to the `File` object for additional safety during backup/restore and archiving. _Suggested by Andres Freund_.
* Added file and directory syncs to the `File` object for additional safety during backup/restore and archiving. _Suggested by Andres Freund_.
* Support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6. _Reported by Eric Radman._
* Support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6. _Reported by Eric Radman._
* Improved error message when backup is run without `archive_command` set and without `--no-archive-check` specified. _Reported by Eric Radman_.
* Improved error message when backup is run without `archive_command` set and without `--no-archive-check` specified. _Reported by Eric Radman_.
* Moved version number out of the `VERSION` file to `Version.pm` to better support packaging. _Suggested by Michael Renner_.
* Moved version number out of the `VERSION` file to `Version.pm` to better support packaging. _Suggested by Michael Renner_.
* Replaced `IPC::System::Simple` and `Net::OpenSSH` with `IPC::Open3` to eliminate CPAN dependency for multiple operating systems.
## v0.75: New Repository Format, Info Command and Experimental 9.5 Support
__Released June 14, 2015__
* **IMPORTANT NOTE**: This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, and the binary names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. The `pg_backrest.conf` file has not changed but you'll need to change any references to `pg_backrest.pl` in cron (or elsewhere) to `pg_backrest` (without the `.pl` extension).
* **IMPORTANT NOTE**: This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, and the binary names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. The `pg_backrest.conf` file has not changed but you'll need to change any references to `pg_backrest.pl` in cron (or elsewhere) to `pg_backrest` (without the `.pl` extension).
* Add `info` command.
* More efficient file ordering for `backup`. Files are copied in descending size order so a single thread does not end up copying a large file at the end. This had already been implemented for `restore`.
* More efficient file ordering for `backup`. Files are copied in descending size order so a single thread does not end up copying a large file at the end. This had already been implemented for `restore`.
* Logging now uses unbuffered output. This should make log files that are being written by multiple threads less chaotic. _Suggested by Michael Renner_.
* Logging now uses unbuffered output. This should make log files that are being written by multiple threads less chaotic. _Suggested by Michael Renner_.
* Experimental support for PostgreSQL 9.5. This may break when the control version or WAL magic changes but will be updated in each release.
* Experimental support for PostgreSQL 9.5. This may break when the control version or WAL magic changes but will be updated in each release.
## v0.70: Stability Improvements for Archiving, Improved Logging and Help
__Released June 1, 2015__
* Fixed an issue where `archive-copy` would fail on an incr/diff backup when `hardlink=n`. In this case the `pg_xlog` path does not already exist and must be created. _Reported by Michael Renner_
* Fixed an issue where `archive-copy` would fail on an incr/diff backup when `hardlink=n`. In this case the `pg_xlog` path does not already exist and must be created. _Reported by Michael Renner_
* Allow duplicate WAL segments to be archived when the checksum matches. This is necessary for some recovery scenarios.
* Allow duplicate WAL segments to be archived when the checksum matches. This is necessary for some recovery scenarios.
* Allow comments/disabling in `pg_backrest.conf` using the `#` character. Only `#` characters in the forst character of the line are honored. _Suggested by Michael Renner_.
* Allow comments/disabling in `pg_backrest.conf` using the `#` character. Only `#` characters in the forst character of the line are honored. _Suggested by Michael Renner_.
* Better logging before `pg_start_backup()` to make it clear when the backup is waiting on a checkpoint. _Suggested by Michael Renner_.
* Better logging before `pg_start_backup()` to make it clear when the backup is waiting on a checkpoint. _Suggested by Michael Renner_.
* Various command behavior, help and logging fixes. _Reported by Michael Renner_.
* Various command behavior, help and logging fixes. _Reported by Michael Renner_.
* Fixed an issue in async archiving where `archive-push` was not properly returning 0 when `archive-max-mb` was reached and moved the async check after transfer to avoid having to remove the stop file twice. Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong. _Reported by Michael Renner_.
* Fixed an issue in async archiving where `archive-push` was not properly returning 0 when `archive-max-mb` was reached and moved the async check after transfer to avoid having to remove the stop file twice. Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong. _Reported by Michael Renner_.
* Fixed a locking issue that could allow multiple operations of the same type against a single stanza. This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. _Reported by Michael Renner_.
* Fixed a locking issue that could allow multiple operations of the same type against a single stanza. This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. _Reported by Michael Renner_.
* Replaced `JSON` module with `JSON::PP` which ships with core Perl.
## v0.65: Improved Resume and Restore Logging, Compact Restores
__Released May 11, 2015__
* Better resume support. Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses. More unit tests to verify each resume case.
* Better resume support. Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses. More unit tests to verify each resume case.
* Resume is now optional. Use the `resume` setting or `--no-resume` from the command line to disable.
* Resume is now optional. Use the `resume` setting or `--no-resume` from the command line to disable.
* More info messages during restore. Previously, most of the restore messages were debug level so not a lot was output in the log.
* More info messages during restore. Previously, most of the restore messages were debug level so not a lot was output in the log.
* Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.
* Added `tablespace` setting to allow tablespaces to be restored into the `pg_tblspc` path. This produces compact restores that are convenient for development, staging, etc. Currently these restores cannot be backed up as pgBackRest expects only links in the `pg_tblspc` path.
* Added `tablespace` setting to allow tablespaces to be restored into the `pg_tblspc` path. This produces compact restores that are convenient for development, staging, etc. Currently these restores cannot be backed up as pgBackRest expects only links in the `pg_tblspc` path.
## v0.61: Bug Fix for Uncompressed Remote Destination
__Released April 21, 2015__
* Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination. The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
* Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination. The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
## v0.60: Better Version Support and WAL Improvements
__Released April 19, 2015__
* Pushing duplicate WAL now generates an error. This worked before only if checksums were disabled.
* Pushing duplicate WAL now generates an error. This worked before only if checksums were disabled.
* Database System IDs are used to make sure that all WAL in an archive matches up. This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.
* Database System IDs are used to make sure that all WAL in an archive matches up. This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.
* Regression tests working back to PostgreSQL 8.3.
@ -168,28 +172,28 @@ __Released March 25, 2015__
* All options can now be set on the command-line making `pg_backrest.conf` optional.
* De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.
* De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.
* Added option `--no-start-stop` to allow backups when Postgres is shut down. If `postmaster.pid` is present then `--force` is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.
* Added option `--no-start-stop` to allow backups when Postgres is shut down. If `postmaster.pid` is present then `--force` is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.
* Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplified a number of things. Issue #28 has been created for checksum deltas.
* Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplified a number of things. Issue #28 has been created for checksum deltas.
* Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.
* Removed dependency on `Moose`. It wasn't being used extensively and makes for longer startup times.
* Removed dependency on `Moose`. It wasn't being used extensively and makes for longer startup times.
* Checksum for `backup.manifest` to detect a corrupted/modified manifest.
* Link `latest` always points to the last backup. This has been added for convenience and to make restores simpler.
* Link `latest` always points to the last backup. This has been added for convenience and to make restores simpler.
* More comprehensive unit tests in all areas.
## v0.30: Core Restructuring and Unit Tests
__Released October 5, 2014__
* Complete rewrite of `BackRest::File` module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.
* Complete rewrite of `BackRest::File` module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.
* Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.
* Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.
* Removed dependency on `Storable` and replaced with a custom ini file implementation.
@ -200,24 +204,24 @@ __Released October 5, 2014__
## v0.19: Improved Error Reporting/Handling
__Released May 13, 2014__
* Working on improving error handling in the `File` object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).
* Working on improving error handling in the `File` object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).
* Found and squashed a nasty bug where `file_copy()` was defaulted to ignore errors. There was also an issue in `file_exists()` that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.
* Found and squashed a nasty bug where `file_copy()` was defaulted to ignore errors. There was also an issue in `file_exists()` that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.
## v0.18: Return Soft Error When Archive Missing
__Released April 13, 2014__
* The `archive-get` command returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets PostgreSQL know that that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream.
* The `archive-get` command returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets PostgreSQL know that that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream.
## v0.17: Warn When Archive Directories Cannot Be Deleted
__Released April 3, 2014__
* If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).
* If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).
## v0.16: RequestTTY=yes for SSH Sessions
__Released April 1, 2014__
* Added `RequestTTY=yes` to ssh sessions. Hoping this will prevent random lockups.
* Added `RequestTTY=yes` to ssh sessions. Hoping this will prevent random lockups.
## v0.15: Added archive-get
__Released March 29, 2014__
@ -229,21 +233,21 @@ __Released March 29, 2014__
## v0.11: Minor Fixes
__Released March 26, 2014__
* Removed `master_stderr_discard` option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code.
* Removed `master_stderr_discard` option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code.
* Changed lock file conflicts on `backup` and `expire` commands to `ERROR`. They were set to `DEBUG` due to a copy-and-paste from the archive locks.
* Changed lock file conflicts on `backup` and `expire` commands to `ERROR`. They were set to `DEBUG` due to a copy-and-paste from the archive locks.
## v0.10: Backup and Archiving are Functional
__Released March 5, 2014__
* No restore functionality, but the backup directories are consistent PostgreSQL data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.
* No restore functionality, but the backup directories are consistent PostgreSQL data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.
* Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.
* Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.
* Backups are multi-threaded, but the `Net::OpenSSH` library does not appear to be 100% thread-safe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.
* Backups are multi-threaded, but the `Net::OpenSSH` library does not appear to be 100% thread-safe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.
* Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.
* Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.
* The `backup.manifest` is being written as `Storable` because `Config::IniFile` does not seem to handle large files well. Would definitely like to save these as human-readable text.
* The `backup.manifest` is being written as `Storable` because `Config::IniFile` does not seem to handle large files well. Would definitely like to save these as human-readable text.
* Absolutely no documentation (outside the code). Well, excepting these release notes.
* Absolutely no documentation (outside the code). Well, excepting these release notes.

3
LICENSE
View File

@ -1,6 +1,7 @@
The MIT License (MIT)
Copyright (c) 2013-2015 David Steele
Portions Copyright (c) 2015, The PostgreSQL Global Development Group
Portions Copyright (c) 2013-2015, David Steele
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in

30
README.md
View File

@ -1,6 +1,8 @@
# pgBackRest - Simple Postgres Backup & Restore
# pgBackRest<br/>Reliable PostgreSQL Backup & Restore
pgBackRest aims to be a simple backup and restore system that can seamlessly scale up to the largest databases and workloads.
## Introduction
pgBackRest aims to be a simple, reliable backup and restore system that can seamlessly scale up to the largest databases and workloads.
Primary pgBackRest features:
@ -20,41 +22,39 @@ Primary pgBackRest features:
- Restore remapping base/tablespaces
- Support for PostgreSQL >= 8.3
Instead of relying on traditional backup tools like tar and rsync, pgBackRest implements all backup features internally and uses a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows for better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security.
Instead of relying on traditional backup tools like tar and rsync, pgBackRest implements all backup features internally and uses a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows for better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security.
pgBackRest uses the gitflow model of development. This means that the master branch contains only the release history, i.e. each commit represents a single release and release tags are always from the master branch. The dev branch contains a single commit for each feature or fix and more accurately depicts the development history. Actual development is done on feature (dev_*) branches and squashed into dev after regression tests have passed. In this model dev is considered stable and can be released at any time. As such, the dev branch does not have any special version modifiers.
pgBackRest is hosted at [GitHub](https://github.com/pgmasters/backrest) and follows the gitflow model of development. This means that the master branch contains only the release history, i.e. each commit represents a single release and release tags are always from the master branch. The dev branch contains a single commit for each feature or fix and more accurately depicts the development history. Actual development is done on feature (dev_*) branches and squashed into dev after regression tests have passed. In this model dev is considered stable and can be released at any time. As such, the dev branch does not have any special version modifiers.
## Getting Started
pgBackRest strives to be easy to configure and operate:
* [Installation instructions](USERGUIDE.md#installation) for major operating systems.
- [User guide](http://www.pgbackrest.org/backrest/user-guide.html) for Ubuntu 12.04 & 14.04 / PostgreSQL 9.4.
- [Command reference](http://www.pgbackrest.org/backrest/command.html) for command-line operations.
- [Configuration reference](http://www.pgbackrest.org/backrest/configuration.html) for creating rich pgBackRest configurations.
* [Sample configurations](USERGUIDE.md#examples) that cover most basic use cases.
* [Command guide](USERGUIDE.md#commands) for command-line operations.
* [Settings documentation](USERGUIDE.md#setttings) for creating complex configurations and more detail on options.
## Contributing
Contributions to pgBackRest are always welcome!
Code fixes or new features can be submitted via pull requests. Ideas for new features and improvements to existing functionality or documentation can be [submitted as issues](http://github.com/pgmasters/backrest/issues).
Code fixes or new features can be submitted via pull requests. Ideas for new features and improvements to existing functionality or documentation can be [submitted as issues](https://github.com/pgmasters/backrest/issues).
Bug reports should be [submitted as issues](http://github.com/pgmasters/backrest/issues). Please provide as much information as possible to aid in determining the cause of the problem.
Bug reports should be [submitted as issues](https://github.com/pgmasters/backrest/issues). Please provide as much information as possible to aid in determining the cause of the problem.
You will always receive credit in the [change log](https://github.com/pgmasters/backrest/blob/master/CHANGELOG.md) for your contributions.
## Support
pgBackRest is completely free and open source under the [MIT](https://github.com/pgmasters/backrest/blob/master/LICENSE) license. You may use it for personal or commercial purposes without any restrictions whatsoever. Bug reports are taken very seriously and will be addressed as quickly as possible.
pgBackRest is completely free and open source under the [MIT](https://github.com/pgmasters/backrest/blob/master/LICENSE) license. You may use it for personal or commercial purposes without any restrictions whatsoever. Bug reports are taken very seriously and will be addressed as quickly as possible.
Creating a robust disaster recovery policy with proper replication and backup strategies can be a very complex and daunting task. You may find that you need help during the architecture phase and ongoing support to ensure that your enterprise continues running smoothly.
Creating a robust disaster recovery policy with proper replication and backup strategies can be a very complex and daunting task. You may find that you need help during the architecture phase and ongoing support to ensure that your enterprise continues running smoothly.
[Crunchy Data](http://www.crunchydatasolutions.com) provides packaged versions of pgBackRest for major operating systems and expert full life-cycle commercial support for pgBackRest and all things PostgreSQL. [Crunchy Data](http://www.crunchydatasolutions.com) is committed to providing open source solutions with no vendor lock-in so cross-compatibility with the community version of pgBackRest is always strictly maintained.
[Crunchy Data](http://www.crunchydatasolutions.com) provides packaged versions of pgBackRest for major operating systems and expert full life-cycle commercial support for pgBackRest and all things PostgreSQL. [Crunchy Data](http://www.crunchydatasolutions.com) is committed to providing open source solutions with no vendor lock-in so cross-compatibility with the community version of pgBackRest is always strictly maintained.
Please visit [Crunchy Backup Manager](http://crunchydatasolutions.com/crunchy-backup-manager) for more information.
Please visit [Crunchy Backup Manager](http://www.crunchydatasolutions.com/crunchy-backup-manager) for more information.
## Recognition

941
USERGUIDE.md
View File

@ -1,941 +0,0 @@
# pgBackRest - User Guide
## Installation
pgBackRest is written entirely in Perl. Some additional modules will need to be installed depending on the OS.
### Ubuntu 12.04/14.04 Setup
* Install required Perl modules:
```
apt-get install libdbd-pg-perl
```
### CentOS 6 Setup
* Install Perl and required modules:
```
yum install perl perl-Time-HiRes perl-parent perl-JSON perl-Digest-SHA perl-DBD-Pg
```
### CentOS 7 Setup
* Install Perl and required modules:
```
yum install perl perl-Thread-Queue perl-JSON-PP perl-Digest-SHA perl-DBD-Pg
```
### Software Installation
pgBackRest can be installed by downloading the most recent release:
https://github.com/pgmasters/backrest/releases
pgBackRest can be installed anywhere but it's best (though not required) to install it in the same location on all systems.
### Regression Test Setup
* Create the backrest user
The backrest user must be created on the same system and in the same group as the user you will use for testing (which can be any user you prefer). For example:
```
adduser -g <test-user-group> backrest
```
* Setup password-less SSH login between the test user and the backrest user
The test user should be able to `ssh backrest@127.0.0.1` and the backrest user should be able to `ssh <testuser>@127.0.0.1` without requiring any passwords. This article (http://archive.oreilly.com/pub/h/66) has details on how to accomplish this. Do the logons both ways at the command line before running regression tests.
* Give group read and execute permissions to `~/backrest/test`:
Usually this can be accomplished by running the following as the test user:
```
chmod 750 ~
```
* Running regression:
Running the full regression suite is generally not necessary. Run the following first:
```
./test.pl --module=backup --test=full --db-version=all --thread-max=<# threads>
```
This will run full backup/restore regression with a variety of options on all installed versions of PostgreSQL. If you are only interested in one version then modify the `db-version` setting to X.X (e.g. 9.4). `--thread-max` can be omitted if you are running single-threaded.
If there are errors in this test then run full regression to help isolate problems:
```
./test.pl --db-version=all --thread-max=<# threads>
```
Report regression test failures at https://github.com/pgmasters/backrest/issues.
## Configuration
pgBackRest can be used entirely with command-line parameters but a configuration file is more practical for installations that are complex or set a lot of options. The default location for the configuration file is `/etc/pg_backrest.conf`.
Each system where pgBackRest is installed should have a repository owned by the user that will be running pgBackRest on that system. Normally this will be the `postgres` user on a database server and the `backrest` user on a backup server. See [repo-path](USERGUIDE.md#repo-path-key) for more information on how repositories are used.
### Examples
#### Confguring Postgres for Archiving
Modify the following settings in `postgresql.conf`:
```
wal_level = archive
archive_mode = on
archive_command = '/path/to/backrest/bin/pg_backrest --stanza=db archive-push %p'
```
Replace the path with the actual location where pgBackRest was installed. The stanza parameter should be changed to the actual stanza name for your database cluster.
#### Minimal Configuration
The absolute minimum required to run pgBackRest (if all defaults are accepted) is the database cluster path.
`/etc/pg_backrest.conf`:
```
[main]
db-path=/data/db
```
The `db-path` option could also be provided on the command line, but it's best to use a configuration file as options tend to pile up quickly.
#### Simple Single Host Configuration
This configuration is appropriate for a small installation where backups are being made locally or to a remote file system that is mounted locally. A number of additional options are set:
- `db-port` - Custom port for PostgreSQL.
- `compress` - Disable compression (handy if the file system is already compressed).
- `repo-path` - Path to the pgBackRest repository where backups and WAL archive are stored.
- `log-level-file` - Set the file log level to debug (Lots of extra info if something is not working as expected).
- `hardlink` - Create hardlinks between backups (but never between full backups).
- `thread-max` - Use 2 threads for backup/restore operations.
`/etc/pg_backrest.conf`:
```
[global:general]
compress=n
repo-path=/path/to/db/repo
[global:log]
log-level-file=debug
[global:backup]
hardlink=y
thread-max=2
[main]
db-path=/data/db
db-port=5555
```
#### Simple Multiple Host Configuration
This configuration is appropriate for a small installation where backups are being made remotely. Make sure that postgres@db-host has trusted ssh to backrest@backup-host and vice versa. This configuration assumes that you have pg_backrest in the same path on both servers.
`/etc/pg_backrest.conf` on the db host:
```
[global:general]
repo-path=/path/to/db/repo
repo-remote-path=/path/to/backup/repo
[global:backup]
backup-host=backup.mydomain.com
backup-user=backrest
[global:archive]
archive-async=y
[main]
db-path=/data/db
```
`/etc/pg_backrest.conf` on the backup host:
```
[global:general]
repo-path=/path/to/backup/repo
[main]
db-host=db.mydomain.com
db-path=/data/db
db-user=postgres
```
### Setttings
#### `command` section
The `command` section defines the location of external commands that are used by pgBackRest.
##### `cmd-remote` key
pgBackRest exe path on the remote host.
Required only if the path to pg_backrest is different on the local and remote systems. If not defined, the remote exe path will be set the same as the local exe path.
```
default: same as local
example: cmd-remote=/usr/lib/backrest/bin/pg_backrest_remote.pl
```
#### `log` section
The `log` section defines logging-related settings.
##### `log-level-file` key
Level for file logging.
The following log levels are supported:
- `off` - No logging at all (not recommended)
- `error` - Log only errors
- `warn` - Log warnings and errors
- `info` - Log info, warnings, and errors
- `debug` - Log debug, info, warnings, and errors
- `trace` - Log trace (very verbose debugging), debug, info, warnings, and errors
```
default: info
example: log-level-file=debug
```
##### `log-level-console` key
Level for console logging.
The following log levels are supported:
- `off` - No logging at all (not recommended)
- `error` - Log only errors
- `warn` - Log warnings and errors
- `info` - Log info, warnings, and errors
- `debug` - Log debug, info, warnings, and errors
- `trace` - Log trace (very verbose debugging), debug, info, warnings, and errors
```
default: warn
example: log-level-console=error
```
#### `general` section
The `general` section defines settings that are shared between multiple operations.
##### `buffer-size` key
Buffer size for file operations.
Set the buffer size used for copy, compress, and uncompress functions. A maximum of 3 buffers will be in use at a time per thread. An additional maximum of 256K per thread may be used for zlib buffers.
```
default: 4194304
allow: 16384 - 8388608
example: buffer-size=32768
```
##### `compress` key
Use file compression.
Enable gzip compression. Backup files are compatible with command-line gzip tools.
```
default: y
example: compress=n
```
##### `compress-level` key
Compression level for stored files.
Sets the zlib level to be used for file compression when `compress=y`.
```
default: 6
allow: 0-9
example: compress-level=9
```
##### `compress-level-network` key
Compression level for network transfer when `compress=n`.
Sets the zlib level to be used for protocol compression when `compress=n` and the database cluster is not on the same host as the backup. Protocol compression is used to reduce network traffic but can be disabled by setting `compress-level-network=0`. When `compress=y` the `compress-level-network` setting is ignored and `compress-level` is used instead so that the file is only compressed once. SSH compression is always disabled.
```
default: 3
allow: 0-9
example: compress-level-network=1
```
##### `config-remote` key
pgBackRest remote configuration file.
Sets the location of the remote configuration file. This is only required if the remote configuration file is in a different location than the local configuration file.
```
default: /etc/doc.pl.conf
example: config-remote=/etc/pg_backrest_remote.conf
```
##### `db-timeout` key
Database query timeout.
Sets the timeout for queries against the database. This includes the `pg_start_backup()` and `pg_stop_backup()` functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set `startfast=y` and you know that the database cluster will not generate many WAL segments during the backup).
```
default: 1800
example: db-timeout=600
```
##### `neutral-umask` key
Use a neutral umask.
Sets the umask to 0000 so modes in the repository as created in a sensible way. The default directory mode is 0750 and default file mode is 0640. The lock and log directories set the directory and file mode to 0770 and 0660 respectively.
To use the executing user's umask instead specify `neutral-umask=n` in the config file or `--no-neutral-umask` on the command line.
```
default: y
example: neutral-umask=n
```
##### `repo-path` key
Repository path where WAL segments, backups, logs, etc are stored.
The repository serves as both storage and working area for pgBackRest. In a simple installation where the backups are stored locally to the database server there will be only one repository which will contain everything: backups, archives, logs, locks, etc.
If the backups are being done remotely then the backup server's repository will contain backups, archives, locks and logs while the database server's repository will contain only locks and logs. However, if asynchronous archiving is enabled then the database server's repository will also contain a spool directory for archive logs that have not yet been pushed to the remote repository.
Each system where pgBackRest is installed should have a repository directory configured. Storage requirements vary based on usage. The main backup repository will need the most space as it contains both backups and WAL segments for whatever retention you have specified. The database repository only needs significant space if asynchronous archiving is enabled and then it will act as an overflow for WAL segments and might need to be large depending on your database activity.
If you are new to backup then it will be difficult to estimate in advance how much space you'll need. The best thing to do it take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will change over time as your database evolves.
```
default: /var/lib/backup
example: repo-path=/data/db/backrest
```
##### `repo-remote-path` key
Remote repository path where WAL segments, backups, logs, etc are stored.
The remote repository is relative to the current installation of pgBackRest. On a database server the backup server will be remote and visa versa for the backup server where the database server will be remote. This option is only required if the remote repository has a different path than the local repository.
```
example: repo-remote-path=/backup/backrest
```
##### `thread-max` key
Max threads to use in process.
Each thread will perform compression and transfer to make the command run faster, but don't set `thread-max` so high that it impacts database performance.
```
default: 1
example: thread-max=4
```
##### `thread-timeout` key
Max time a thread can run.
This limits the amount of time (in seconds) that a thread might be stuck due to unforeseen issues executing the command. Has no affect when `thread-max=1`.
```
example: thread-timeout=3600
```
#### `backup` section
The `backup` section defines settings related to backup.
##### `archive-check` key
Check that WAL segments are present in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
```
default: y
example: archive-check=n
```
##### `archive-copy` key
Copy WAL segments needed for consistency to the backup.
This slightly paranoid option protects against corruption or premature expiration in the WAL segment archive by storing the WAL segments directly in the backup. PITR won't be possible without the WAL segment archive and this option also consumes more space.
Even though WAL segments will be restored with the backup, PostgreSQL will ignore them if a `recovery.conf` file exists and instead use `archive_command` to fetch WAL segments. Specifying `type=none` when restoring will not create `recovery.conf` and force PostgreSQL to use the WAL segments in pg_xlog. This will get the database cluster to a consistent state.
```
default: n
example: archive-copy=y
```
##### `backup-host` key
Backup host when operating remotely via SSH.
Make sure that trusted SSH authentication is configured between the db host and the backup host.
When backing up to a locally mounted network filesystem this setting is not required.
```
example: backup-host=backup.domain.com
```
##### `backup-user` key
Backup host user when `backup-host` is set.
Defines the user that will be used for operations on the backup server. Preferably this is not the `postgres` user but rather some other user like `backrest`. If PostgreSQL runs on the backup server the `postgres` user can be placed in the `backrest` group so it has read permissions on the repository without being able to damage the contents accidentally.
```
example: backup-user=backrest
```
##### `hardlink` key
Hardlink files between backups.
Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each backup is a full backup. Be careful, though, because modifying files that are hard-linked can affect all the backups in the set.
```
default: n
example: hardlink=y
```
##### `manifest-save-threshold` key
Manifest save threshold during backup.
Defines how often the manifest will be saved during a backup (in bytes). Saving the manifest is important because it stores the checksums and allows the resume function to work efficiently. The actual threshold used is 1% of the backup size or `manifest-save-threshold`, whichever is greater.
```
default: 1073741824
example: manifest-save-threshold=5368709120
```
##### `resume` key
Allow resume of failed backup.
Defines whether the resume feature is enabled. Resume can greatly reduce the amount of time required to run a backup after a previous backup of the same type has failed. It adds complexity, however, so it may be desirable to disable in environments that do not require the feature.
```
default: y
example: resume=false
```
##### `start-fast` key
Force a checkpoint to start backup quickly.
Forces a checkpoint (by passing `true` to the `fast` parameter of `pg_start_backup()`) so the backup begins immediately. Otherwise the backup will start after the next regular checkpoint.
This feature only works in PostgreSQL <= `8.3`.
```
default: n
example: start-fast=y
```
##### `stop-auto` key
Stop prior failed backup on new backup.
This will only be done if an exclusive advisory lock can be acquired to demonstrate that the prior failed backup process has really stopped.
This feature relies on pg_is_in_backup() so only works on PostgreSQL >= `9.3`.
The setting is disabled by default because it assumes that pgBackRest is the only process doing exclusive online backups. It depends on an advisory lock that only pgBackRest sets so it may abort other processes that do exclusive online backups. Note that `base_backup` and `pg_dump` are safe to use with this setting because they do not call `pg_start_backup()` so are not exclusive.
```
default: n
example: stop-auto=y
```
#### `archive` section
The `archive` section defines parameters when doing async archiving. This means that the archive files will be stored locally, then a background process will pick them and move them to the backup.
##### `archive-async` key
Archive WAL segments asynchronously.
WAL segments will be copied to the local repo, then a process will be forked to compress the segment and transfer it to the remote repo if configured. Control will be returned to PostgreSQL as soon as the WAL segment is copied locally.
```
default: n
example: archive-async=y
```
##### `archive-max-mb` key
Limit size of the local asynchronous archive queue when `archive-async=y`.
After the limit is reached, the following will happen:
- pgBackRest will notify Postgres that the archive was successfully backed up, then DROP IT.
- An error will be logged to the console and also to the Postgres log.
- A stop file will be written in the lock directory and no more archive files will be backed up until it is removed.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
The purpose of this feature is to prevent the log volume from filling up at which point Postgres will stop completely. Better to lose the backup than have PostgreSQL go down.
To start normal archiving again you'll need to remove the stop file which will be located at `${repo-path}/lock/${stanza}-archive.stop` where `${repo-path}` is the path set in the `general` section, and `${stanza}` is the backup stanza.
```
example: archive-max-mb=1024
```
#### `restore` section
The `restore` section defines settings used for restoring backups.
##### `tablespace` key
Restore tablespaces into original or remapped paths.
Defines whether tablespaces will be be restored into their original (or remapped) paths or stored directly under the `pg_tblspc` path. Disabling this setting produces compact restores that are convenient for development, staging, etc. Currently these restores cannot be backed up as pgBackRest expects only links in the `pg_tblspc` path. If no tablespaces are present this this setting has no effect.
```
default: y
example: tablespace=n
```
#### `expire` section
The `expire` section defines how long backups will be retained. Expiration only occurs when the number of complete backups exceeds the allowed retention. In other words, if full-retention is set to 2, then there must be 3 complete backups before the oldest will be expired. Make sure you always have enough space for retention + 1 backups.
##### `retention-full` key
Number of full backups to retain.
When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When not defined then all full backups will be kept.
```
example: retention-full=2
```
##### `retention-diff` key
Number of differential backups to retain.
When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept.
```
example: retention-diff=3
```
##### `retention-archive-type` key
Backup type for WAL retention.
If set to full, then pgBackRest will keep archive logs for the number of full backups defined by `retention-archive`. If set to diff (differential), then pgBackRest will keep archive logs for the number of differential backups defined by `retention-archive`.
If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs that are older than the oldest backup but there may occasionally be reasons for doing so.
```
default: full
example: retention-archive-type=diff
```
##### `retention-archive` key
Number of backups worth of WAL to retain.
Number of backups worth of archive log to keep. If this is set less than your backup retention then be sure you set `archive-copy=y` or you won't be able to restore some older backups.
For example, if `retention-archive=2` and `retention-full=4`, then any backups older than the most recent two full backups will not have WAL segments in the archive to make them consistent. To solve this, set `archive-copy=y` and use `type=none` when restoring. This issue will be addressed in a future release but for now be careful with this setting.
```
example: retention-archive=2
```
#### `stanza` section
A stanza defines the backup configuration for a specific PostgreSQL database cluster. The stanza section must define the database cluster path and host/user if the database cluster is remote. Also, any global configuration sections can be overridden to define stanza-specific settings.
##### `db-host` key
Cluster host for operating remotely via SSH.
Used for backups where the database cluster host is different from the backup host.
```
example: db-host=db.domain.com
```
##### `db-user` key
Cluster host logon user when `db-host` is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally `postgres`, the default.
```
default: postgres
example: db-user=db_owner
```
##### `db-path` key
Cluster data directory.
This should be the same as the `data_directory` setting in `postgresql.conf`. Even though this value can be read from `postgresql.conf` or the database cluster it is prudent to set it in case those resources are not available during a restore or cold backup scenario.
The `db-path` option is tested against the value reported by PostgreSQL on every hot backup so it should always be current.
```
required: y
example: db-path=/data/db
```
##### `db-port` key
Cluster port.
Port that PostgreSQL is running on. This usually does not need to be specified as most database clusters run on the default port.
```
default: 5432
example: db-port=6543
```
##### `db-socket-path` key
cluster unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there usually no need to specify this setting unless the socket directory was explicitly modified with the `unix_socket_directory` setting in `postgressql.conf`.
```
example: db-socket-path=/var/run/postgresql
```
## Commands
### General Options
These options are either global or used by all commands.
#### `config` option
pgBackRest configuration file.
Use this option to specify a different configuration file than the default.
```
default: /etc/pg_backrest.conf
example: config=/var/lib/backrest/pg_backrest.conf
```
#### `stanza` option
Command stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one Postgres database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
Examples of how to configure a stanza can be found in the `configuration examples` section.
```
required: y
example: stanza=main
```
### Commands
#### `backup` command
Backup a database cluster.
pgBackRest does not have a built-in scheduler so it's best to run it from cron or some other scheduling mechanism.
##### `type` option
Backup type.
The following backup types are supported:
- `full` - all database cluster files will be copied and there will be no dependencies on previous backups.
- `incr` - incremental from the last successful backup.
- `diff` - like an incremental backup but always based on the last full backup.
```
default: incr
example: --type=full
```
##### `no-start-stop` option
Perform cold backup.
This option prevents pgBackRest from running `pg_start_backup()` and `pg_stop_backup()` on the database cluster. In order for this to work PostgreSQL should be shut down and pgBackRest will generate an error if it is not.
The purpose of this option is to allow cold backups. The `pg_xlog` directory is copied as-is and `archive-check` is automatically disabled for the backup.
```
default: n
```
##### `force` option
Force a cold backup.
When used with `--no-start-stop` a backup will be run even if pgBackRest thinks that PostgreSQL is running. **This option should be used with extreme care as it will likely result in a bad backup.**
There are some scenarios where a backup might still be desirable under these conditions. For example, if a server crashes and the database cluster volume can only be mounted read-only, it would be a good idea to take a backup even if `postmaster.pid` is present. In this case it would be better to revert to the prior backup and replay WAL, but possibly there is a very important transaction in a WAL segment that did not get archived.
```
default: n
```
##### Example: Full Backup
```
pg_backrest --stanza=db --type=full backup
```
Run a `full` backup on the `db` stanza. `--type` can also be set to `incr` or `diff` for incremental or differential backups. However, if no `full` backup exists then a `full` backup will be forced even if `incr` or `diff` is requested.
#### `archive-push` command
Push a WAL segment to the archive.
The WAL segment may be pushed immediately to the archive or stored locally depending on the value of `archive-async`
##### Example
```
pg_backrest --stanza=db archive-push %p
```
Accepts a WAL segment from PostgreSQL and archives it in the repository defined by `repo-path`. `%p` is how PostgreSQL specifies the location of the WAL segment to be archived.
#### `archive-get` command
Get a WAL segment from the archive.
WAL segments are required for restoring a PostgreSQL cluster or maintaining a replica.
##### Example
```
pg_backrest --stanza=db archive-get %f %p
```
Retrieves a WAL segment from the repository. This command is used in `recovery.conf` to restore a backup, perform PITR, or as an alternative to streaming for keeping a replica up to date. `%f` is how PostgreSQL specifies the WAL segment it needs and `%p` is the location where it should be copied.
#### `expire` command
Expire backups that exceed retention.
pgBackRest does backup rotation but is not concerned with when the backups were created. If two full backups are configured for retention, pgBackRest will keep two full backups no matter whether they occur two hours or two weeks apart.
##### Example
```
pg_backrest --stanza=db expire
```
Expire (rotate) any backups that exceed the defined retention. Expiration is run automatically after every successful backup, so there is no need to run this command separately unless you have reduced retention, usually to free up some space.
#### `restore` command
Restore a database cluster.
This command is generally run manually, but there are instances where it might be automated.
##### `set` option
Backup set to restore.
The backup set to be restored. `latest` will restore the latest backup, otherwise provide the name of the backup to restore.
```
default: latest
example: --set=20150131-153358F_20150131-153401I
```
##### `delta` option
Restore using delta.
By default the PostgreSQL data and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
```
default: n
```
##### `force` option
Force a restore.
By itself this option forces the PostgreSQL data and tablespace paths to be completely overwritten. In combination with `--delta` a timestamp/size delta will be performed instead of using checksums.
```
default: n
```
##### `type` option
Recovery type.
The following recovery types are supported:
- `default` - recover to the end of the archive stream.
- `name` - recover the restore point specified in `--target`.
- `xid` - recover to the transaction id specified in `--target`.
- `time` - recover to the time specified in `--target`.
- `preserve` - preserve the existing `recovery.conf` file.
- `none` - no `recovery.conf` file is written so PostgreSQL will attempt to achieve consistency using WAL segments present in `pg_xlog`. Provide the required WAL segments or use the `archive-copy` setting to include them with the backup.
```
default: default
example: --type=xid
```
##### `target` option
Recovery target.
Defines the recovery target when `--type` is `name`, `xid`, or `time`.
```
required: y
example: "--target=2015-01-30 14:15:11 EST"
```
##### `target-exclusive` option
Stop just before the recovery target is reached.
Defines whether recovery to the target would be exclusive (the default is inclusive) and is only valid when `--type` is `time` or `xid`. For example, using `--target-exclusive` would exclude the contents of transaction `1007` when `--type=xid` and `--target=1007`. See the `recovery_target_inclusive` option in the PostgreSQL docs for more information.
```
default: n
```
##### `target-resume` option
Resume when recovery target is reached.
Specifies whether recovery should resume when the recovery target is reached. See `pause_at_recovery_target` in the PostgreSQL docs for more information.
```
default: n
```
##### `target-timeline` option
Recover along a timeline.
See `recovery_target_timeline` in the PostgreSQL docs for more information.
```
example: --target-timeline=3
```
##### `recovery-option` option
Set an option in `recovery.conf`.
See http://www.postgresql.org/docs/X.X/static/recovery-config.html for details on recovery.conf options (replace X.X with your PostgreSQL version). This option can be used multiple times.
Note: The `restore_command` option will be automatically generated but can be overridden with this option. Be careful about specifying your own `restore_command` as pgBackRest is designed to handle this for you. Target Recovery options (recovery_target_name, recovery_target_time, etc.) are generated automatically by pgBackRest and should not be set with this option.
Recovery settings can also be set in the `restore:recovery-option` section of pg_backrest.conf. For example:
```
[restore:recovery-option]
primary_conn_info=db.mydomain.com
standby_mode=on
```
Since pgBackRest does not start PostgreSQL after writing the `recovery.conf` file, it is always possible to edit/check `recovery.conf` before manually restarting.
```
example: --recovery-option primary_conninfo=db.mydomain.com
```
##### `tablespace-map` option
Modify a tablespace path.
Moves a tablespace to a new location during the restore. This is useful when tablespace locations are not the same on a replica, or an upgraded system has different mount points.
Since PostgreSQL 9.2 tablespace locations are not stored in pg_tablespace so moving tablespaces can be done with impunity. However, moving a tablespace to the `data_directory` is not recommended and may cause problems. For more information on moving tablespaces http://www.databasesoup.com/2013/11/moving-tablespaces.html is a good resource.
```
example: --tablespace-map ts_01=/db/ts_01
```
##### Example: Restore Latest
```
pg_backrest --stanza=db --type=name --target=release restore
```
Restores the latest database cluster backup and then recovers to the `release` restore point.
#### `info` command
Retrieve information about backups.
The `info` command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.
For machine-readable output use `--output=json`. The JSON output contains far more information than the text output, however **this feature is currently experimental so the format may change between versions**.
##### `output` option
Output format.
The following output types are supported:
- `text` - Human-readable summary of backup information.
- `json` - Exhaustive machine-readable backup information in JSON format.
```
default: text
example: --output=json
```
##### Example: Information for a single stanza
```
pg_backrest --stanza=db --output=json info
```
Get information about backups in the `db` stanza.
##### Example: Information for all stanzas
```
pg_backrest --output=json info
```
Get information about backups for all stanzas in the repository.
#### `help` command
Get help.
Three levels of help are provided. If no command is specified then general help will be displayed. If a command is specified then a full description of the command will be displayed along with a list of valid options. If an option is specified in addition to a command then the a full description of the option as it applies to the command will be displayed.
##### Example: Help for the backup command
```
pg_backrest help backup
```
Get help for the backup command.
##### Example: Help for backup command, --force option
```
pg_backrest help backup force
```
Get help for the force option of the backup command.
#### `start` command
Allow pgBackRest processes to run.
If the pgBackRest processes were previously stopped using the `stop` command then they can be started again using the `start` command. Note that this will not immediately start up any pgBackRest processes but they are allowed to run.
##### Example: Start processes for stanza main
```
pg_backrest --stanza=main start
```
Allows pgBackRest processes to run for the `main` stanza.
#### `stop` command
Stop pgBackRest processes from running.
Does not allow any new pgBackRest processes to run. By default running processes will be allowed to complete successfully. Use the `--force` option to terminate running processes.
pgBackRest processes will return an error if they are run after the stop command completes.
##### `force` option
Force all pgBackRest processes to stop.
This option will send TERM signals to all running pgBackRest processes to effect a graceful but immediate shutdown. Note that this will also shutdown processes that were initiated on another system but have remotes running on the current system. For instance, if a backup was started on the backup server then running `stop --force` on the database server will shutdown the backup process on the backup server.
```
default: n
```
##### Example: Stop processes for all stanzas
```
pg_backrest stop
```
Stop new pgBackRest processes for all stanzas but allow any current process to complete.
#### `version` command
Get version.
Displays installed pgBackRest version.
##### Example: Get version
```
pg_backrest version
```
Get pgBackRest version.

349
doc/css/default.css Normal file
View File

@ -0,0 +1,349 @@
/*******************************************************************************
Html and body
*******************************************************************************/
html
{
background-color: #f8f8f8;
font-family: Avenir, Corbel, sans-serif;
font-size: 12pt;
margin-top: 8px;
margin-left: 1%;
margin-right: 1%;
width: 98%;
}
body
{
margin: 0px auto;
padding: 0px;
width: 100%;
}
@media (min-width: 1000px)
{
body
{
width: 1000px;
}
}
/*******************************************************************************
Link default styling
*******************************************************************************/
a:link, a:visited, a:hover, a:active
{
text-decoration: underline;
color: black;
}
/*******************************************************************************
Header
*******************************************************************************/
.page-header
{
width:100%;
text-align:center;
/*float:left;*/
}
.page-header-title
{
font-size: xx-large;
font-weight: bolder;
}
.page-header-subtitle
{
position: relative;
top: -.25em;
font-size: large;
font-weight: bold;
}
/*******************************************************************************
Menu
*******************************************************************************/
.page-menu
{
/*margin-top: .25em;*/
/*font-size: 14pt;*/
}
.menu-body
{
text-align: center;
/*font-weight: 600;*/
/*border-bottom: 2px #dddddd solid;*/
}
.menu
{
white-space: nowrap;
display: inline;
margin-left: 6px;
}
.menu:first-of-type
{
margin-left: 0px;
}
.menu:before
{
content: "[";
/*margin-right: .5em;*/
}
.menu:after
{
content: "]";
/*margin-left: .5em;*/
}
.menu-link
{
/*margin-left: 2px;
margin-right: 2px;*/
}
a.menu-link:link, a.menu-link:visited, a.menu-link:active
{
text-decoration: none;
}
a.menu-link:hover
{
text-decoration: underline;
}
/*******************************************************************************
Table of Contents
*******************************************************************************/
.page-toc-body
{
margin-top: .25em;
}
.section1-toc:first-of-type
{
margin-top: 0;
}
.section1-toc
{
margin-top: .5em;
}
.section1-toc-title
{
/*margin-top: .25em;*/
font-size: 14pt;
}
.section2-toc-title
{
font-size: 12pt;
margin-left: 2em;
}
.section3-toc-title
{
margin-left: 4em;
}
/*******************************************************************************
Section
*******************************************************************************/
.section1
{
/*display:block;*/
margin-top: 1em;
}
.section2, .section3
{
/*display:block;*/
margin-top: .5em;
}
.section1-title, .section2-title, .section3-title, .page-toc-title
{
/*display:block;*/
margin-top: .5em;
font-weight: 500;
}
.section1-title, .page-toc-title
{
background-color: #dddddd;
font-size: 16pt;
padding-left: .5em;
}
.section2
{
/*margin-left: 2em;*/
}
.section2-title
{
border-bottom: 2px #cccccc solid;
font-size: 14pt;
}
.section3
{
margin-left: 1em;
margin-top: 1em;
/*border-top: 1px #cccccc solid;*/
}
.section3:first-of-type
{
border-top: none;
}
.section3-title
{
display: inline;
font-size: 13pt;
/*text-decoration: underline;*/
border-bottom: 1px #bbbbbb solid;
}
.section-intro
{
margin-top: .75em;
}
.section-body
{
}
/*******************************************************************************
Config, Execute, and Code Block Elements
*******************************************************************************/
.section-body-text
{
margin-top: 1em;
}
.section-body-text:first-of-type
{
margin-top: .75em;
}
.execute,
.config,
.code-block
{
/*white-space: nowrap;*/
margin-left: 1em;
margin-right: 1em;
margin-top: 1em;
margin-bottom: 1em;
}
.execute-title,
.config-title
{
font-size: 11pt;
margin-bottom: .125em;
}
.execute-body,
.config-body,
.code-block
{
background-color: #444444;
/*overflow-x:auto;*/
font-family: monospace;
unicode-bidi: embed;
color: white;
font-size: 10pt;
}
.execute-body-cmd-first, .execute-body-cmd,
.execute-body-output, .execute-body-output-highlight,
.config-body-title,
.code-block
{
padding-left: .5em;
padding-right: .5em;
}
.execute-body-cmd-first, .execute-body-output,
.config-body-title, .config-body-output,
.code-block
{
padding-top: .25em;
}
.execute-body-cmd-first, .execute-body-cmd, .execute-body-output,
.config-body-title, .config-body-output,
.code-block
{
padding-bottom: .25em;
}
.execute-body-output,
.config-body-output,
.code-block
{
background-color: #606060;
}
.execute-body-output-highlight
{
background-color: green;
}
.execute-body-output-highlight-error
{
background-color: firebrick;
}
.execute-body-output, .execute-body-output-highlight, .execute-body-output-highlight-error,
.config-body-output
{
padding-left: 2em;
}
.code-block
{
}
.section-body-execute-output
{
}
/*******************************************************************************
Keywords
*******************************************************************************/
.backrest, .cmd, .postgres
{
font-weight: 500;
}
.path, .br-option, .br-setting, .pg-option, .pg-setting, .id, .user, .file, .path
{
/*font-weight: 600;*/
/*background-color: #e8e8e8;*/
unicode-bidi: embed;
font-family: monospace;
white-space: nowrap;
/*font-size: 11pt;*/
}
/*******************************************************************************
Footer
*******************************************************************************/
.page-footer
{
border-top: 2px #cccccc solid;
margin-top: 1em;
padding-top: .25em;
font-size: x-small;
white-space: nowrap;
text-align: center;
}

175
doc/doc.pl
View File

@ -19,9 +19,10 @@ use Pod::Usage qw(pod2usage);
use XML::Checker::Parser;
use lib dirname($0) . '/lib';
use BackRestDoc::Doc;
use BackRestDoc::DocConfig;
use BackRestDoc::DocRender;
use BackRestDoc::Common::Doc;
use BackRestDoc::Common::DocConfig;
use BackRestDoc::Html::DocHtmlSite;
use BackRestDoc::Common::DocRender;
use lib dirname($0) . '/../lib';
use BackRest::Common::Log;
@ -51,72 +52,6 @@ doc.pl [options] [operation]
use constant OP_MAIN => 'Main';
use constant OP_MAIN_DOC_PROCESS => OP_MAIN . '::docProcess';
use constant OP_MAIN_OPTION_LIST_PROCESS => OP_MAIN . '::optionListProcess';
####################################################################################################################################
# optionListProcess
####################################################################################################################################
sub optionListProcess
{
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$oOptionListOut,
$strCommand,
$oOptionRuleRef
) =
logDebugParam
(
OP_MAIN_OPTION_LIST_PROCESS, \@_,
{name => 'oOptionListOut'},
{name => 'strCommand', required => false},
{name => 'oOptionRuleRef'}
);
foreach my $oOptionOut ($oOptionListOut->nodeList())
{
my $strOption = $oOptionOut->paramGet('id');
if ($strOption eq 'help' || $strOption eq 'version')
{
next;
}
if (!defined($$oOptionRuleRef{$strOption}{&OPTION_RULE_TYPE}))
{
confess "unable to find option $strOption";
}
if (defined(optionDefault($strOption, $strCommand)))
{
$oOptionOut->fieldSet('required', false);
if ($$oOptionRuleRef{$strOption}{&OPTION_RULE_TYPE} eq &OPTION_TYPE_BOOLEAN)
{
$oOptionOut->fieldSet('default', optionDefault($strOption, $strCommand) ? 'y' : 'n');
}
else
{
$oOptionOut->fieldSet('default', optionDefault($strOption, $strCommand));
}
}
else
{
$oOptionOut->fieldSet('required', optionRequired($strOption, $strCommand));
}
if (defined($strCommand))
{
$oOptionOut->fieldSet('cmd', true);
}
if ($strOption eq 'cmd-remote')
{
$oOptionOut->fieldSet('default', 'same as local');
}
}
}
####################################################################################################################################
# Load command line parameters and config
@ -127,11 +62,17 @@ my $bQuiet = false; # Sets log level to ERROR
my $strLogLevel = 'info'; # Log level for tests
my $strProjectName = 'pgBackRest'; # Project name to use in docs
my $strExeName = 'pg_backrest'; # Exe name to use in docs
my $bHtml = false; # Generate full html documentation
my $strHtmlRoot = '/'; # Root html page
my $bNoExe = false; # Should commands be executed when buildng help? (for testing only)
GetOptions ('help' => \$bHelp,
'version' => \$bVersion,
'quiet' => \$bQuiet,
'log-level=s' => \$strLogLevel)
'log-level=s' => \$strLogLevel,
'html' => \$bHtml,
'html-root=s' => \$strHtmlRoot,
'no-exe', \$bNoExe)
or pod2usage(2);
# Display version and exit if requested
@ -167,83 +108,47 @@ sub docProcess
$strOperation,
$strXmlIn,
$strMdOut,
$bManual
$oHtmlSite
) =
logDebugParam
(
OP_MAIN_DOC_PROCESS, \@_,
{name => 'strXmlIn'},
{name => 'strMdOut'},
{name => 'bManual', default => false}
{name => 'oHtmlSite'}
);
# Build the document from xml
my $oDoc = new BackRestDoc::Doc($strXmlIn);
# Build commands pulled from the code
if ($bManual)
{
# Get the option rules
my $oOptionRule = optionRuleGet();
# Ouput general options
my $oOperationGeneralOptionListOut =
$oDoc->nodeGet('operation')->nodeGet('operation-general')->nodeGet('option-list');
optionListProcess($oOperationGeneralOptionListOut, undef, $oOptionRule);
# Ouput commands
my $oCommandListOut = $oDoc->nodeGet('operation')->nodeGet('command-list');
foreach my $oCommandOut ($oCommandListOut->nodeList())
{
my $strCommand = $oCommandOut->paramGet('id');
my $oOptionListOut = $oCommandOut->nodeGet('option-list', false);
if (defined($oOptionListOut))
{
optionListProcess($oOptionListOut, $strCommand, $oOptionRule);
}
my $oExampleListOut = $oCommandOut->nodeGet('command-example-list');
foreach my $oExampleOut ($oExampleListOut->nodeList())
{
if (defined($oExampleOut->paramGet('title', false)))
{
$oExampleOut->paramSet('title', 'Example: ' . $oExampleOut->paramGet('title'));
}
else
{
$oExampleOut->paramSet('title', 'Example');
}
}
}
# Ouput config section
my $oConfigSectionListOut = $oDoc->nodeGet('config')->nodeGet('config-section-list');
foreach my $oConfigSectionOut ($oConfigSectionListOut->nodeList())
{
my $oOptionListOut = $oConfigSectionOut->nodeGet('config-key-list', false);
if (defined($oOptionListOut))
{
optionListProcess($oOptionListOut, undef, $oOptionRule);
}
}
}
my $oDoc = new BackRestDoc::Common::Doc($strXmlIn);
# Write markdown
my $oRender = new BackRestDoc::DocRender('markdown', $strProjectName, $strExeName);
$oRender->save($strMdOut, $oRender->process($oDoc));
my $oRender = new BackRestDoc::Common::DocRender('markdown', $strProjectName, $strExeName);
$oRender->save($strMdOut, $oHtmlSite->variableReplace($oRender->process($oDoc)));
}
my $oRender = new BackRestDoc::DocRender('text', $strProjectName, $strExeName);
my $oDocConfig = new BackRestDoc::DocConfig(new BackRestDoc::Doc("${strBasePath}/xml/reference.xml"), $oRender);
my $oRender = new BackRestDoc::Common::DocRender('text', $strProjectName, $strExeName);
my $oDocConfig = new BackRestDoc::Common::DocConfig(new BackRestDoc::Common::Doc("${strBasePath}/xml/reference.xml"), $oRender);
$oDocConfig->helpDataWrite();
docProcess("${strBasePath}/xml/readme.xml", "${strBasePath}/../README.md");
docProcess("${strBasePath}/xml/reference.xml", "${strBasePath}/../USERGUIDE.md", true);
docProcess("${strBasePath}/xml/changelog.xml", "${strBasePath}/../CHANGELOG.md");
# !!! Create Html Site Object to perform variable replacements on markdown and test
# !!! This should be replaced with a more generic site object in the future
my $oHtmlSite =
new BackRestDoc::Html::DocHtmlSite
(
new BackRestDoc::Common::DocRender('html', $strProjectName, $strExeName),
$oDocConfig,
"${strBasePath}/xml",
"${strBasePath}/html",
"${strBasePath}/css/default.css",
$strHtmlRoot,
!$bNoExe
);
docProcess("${strBasePath}/xml/index.xml", "${strBasePath}/../README.md", $oHtmlSite);
docProcess("${strBasePath}/xml/change-log.xml", "${strBasePath}/../CHANGELOG.md", $oHtmlSite);
# Only generate the HTML site when requested
if ($bHtml)
{
$oHtmlSite->process();
}

204
doc/html/default.css
View File

@ -1,204 +0,0 @@
/*******************************************************************************
Html and body
*******************************************************************************/
html
{
background-color: #F8F8F8;
font-family: Avenir, Corbel, sans-serif;
font-size: medium;
margin-top: 8px;
margin-left: 1%;
margin-right: 1%;
width: 98%;
}
body
{
margin: 0px auto;
padding: 0px;
width: 100%;
}
@media (min-width: 1000px)
{
body
{
width: 1000px;
}
}
/*******************************************************************************
Link default styling
*******************************************************************************/
a:link
{
text-decoration: none;
color: black;
}
a:visited
{
text-decoration: none;
color: black;
}
a:hover
{
text-decoration: underline;
color: black;
}
a:active
{
text-decoration: none;
color: black;
}
/*******************************************************************************
Header
*******************************************************************************/
.header
{
width:100%;
text-align:center;
float:left;
}
.header-title
{
font-size: 28pt;
font-weight: bolder;
}
.header-subtitle
{
position: relative;
top: -.25em;
font-size: larger;
font-weight: bold;
}
/*******************************************************************************
Menu
*******************************************************************************/
.menu-set
{
text-align: center;
font-weight: 600;
border-bottom: 2px #dddddd solid;
}
.menu-first, .menu
{
white-space: nowrap;
display: inline;
}
.menu
{
margin-left: 6px;
}
.menu-link
{
margin-left: 2px;
margin-right: 2px;
}
/*******************************************************************************
Section
*******************************************************************************/
.section1
{
display:block;
margin-top: 8px;
}
section1-header
{
display:block;
background-color: #dddddd;
font-size: 14pt;
padding-left: 4px;
margin-bottom: 4px;
}
/*******************************************************************************
SubSection
*******************************************************************************/
doc-configure-section
{
display:block;
margin-top: 8px;
}
doc-configure-section-header
{
display:block;
border-bottom: 2px #cccccc solid;
font-size: large;
font-weight: 500;
margin-bottom: 4px;
}
/*******************************************************************************
SubSection2
*******************************************************************************/
doc-configure-key
{
display:block;
margin-top: 8px;
margin-left: 2em;
margin-right: 2em;
}
doc-configure-key-header
{
display:block;
font-size: medium;
font-weight: 500;
border-bottom: 1px #dddddd solid;
margin-bottom: 4px;
}
/*******************************************************************************
Code & Detail
*******************************************************************************/
doc-code, doc-code-block, doc-id, doc-file, doc-function, doc-detail,
doc-setting
{
font-family: "Lucida Console", Monaco, monospace;
font-size: smaller;
}
doc-id, doc-file, doc-function
{
white-space: pre;
}
doc-code, doc-code-block, doc-detail-block
{
background-color: #eeeeee;
}
doc-setting, doc-file
{
background-color: #e0e0e0;
}
doc-code-block, doc-detail-block
{
margin: 8px;
padding: 8px;
display:block;
}
doc-detail
{
display: block;
}
doc-detail-value
{
margin-left: .5em;
}

135
doc/lib/BackRestDoc/Doc.pm → doc/lib/BackRestDoc/Common/Doc.pm
View File

@ -1,15 +1,16 @@
####################################################################################################################################
# DOC MODULE
####################################################################################################################################
package BackRestDoc::Doc;
package BackRestDoc::Common::Doc;
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
use Exporter qw(import);
our @EXPORT = qw();
# use Exporter qw(import);
# our @EXPORT = qw();
use File::Basename qw(dirname);
use Scalar::Util qw(blessed);
use lib dirname($0) . '/../lib';
use BackRest::Common::Log;
@ -30,6 +31,7 @@ use constant OP_DOC_PARAM_GET => OP_DOC .
use constant OP_DOC_PARAM_SET => OP_DOC . '->paramSet';
use constant OP_DOC_PARSE => OP_DOC . '->parse';
use constant OP_DOC_VALUE_GET => OP_DOC . '->valueGet';
use constant OP_DOC_VALUE_SET => OP_DOC . '->valueSet';
####################################################################################################################################
# CONSTRUCTOR
@ -52,35 +54,46 @@ sub new
logDebugParam
(
OP_DOC_NEW, \@_,
{name => 'strFileName'}
{name => 'strFileName', required => false}
);
my $oParser = XML::Checker::Parser->new(ErrorContext => 2, Style => 'Tree');
$oParser->set_sgml_search_path(dirname($self->{strFileName}) . '/dtd');
my $oTree;
eval
# Load the doc from a file if one has been defined
if (defined($self->{strFileName}))
{
local $XML::Checker::FAIL = sub
{
my $iCode = shift;
my $oParser = XML::Checker::Parser->new(ErrorContext => 2, Style => 'Tree');
$oParser->set_sgml_search_path(dirname($self->{strFileName}) . '/dtd');
die XML::Checker::error_string($iCode, @_);
my $oTree;
eval
{
local $XML::Checker::FAIL = sub
{
my $iCode = shift;
die XML::Checker::error_string($iCode, @_);
};
$oTree = $oParser->parsefile($self->{strFileName});
};
$oTree = $oParser->parsefile($self->{strFileName});
};
# Report any error that stopped parsing
if ($@)
{
$@ =~ s/at \/.*?$//s; # remove module line number
die "malformed xml in '$self->{strFileName}':\n" . trim($@);
}
# Report any error that stopped parsing
if ($@)
# Parse and build the doc
$self->{oDoc} = $self->build($self->parse(${$oTree}[0], ${$oTree}[1]));
}
# Else create a blank doc
else
{
$@ =~ s/at \/.*?$//s; # remove module line number
die "malformed xml in '$self->{strFileName}':\n" . trim($@);
$self->{oDoc} = {name => 'doc', children => []};
}
# Parse and build the doc
$self->{oDoc} = $self->build($self->parse(${$oTree}[0], ${$oTree}[1]));
$self->{strName} = 'root';
# Return from function and log return values if any
@ -116,7 +129,8 @@ sub parse
my %oOut;
my $iIndex = 0;
my $bText = $strName eq 'text' || $strName eq 'li' || $strName eq 'code-block';
my $bText = $strName eq 'text' || $strName eq 'li' || $strName eq 'code-block' || $strName eq 'p' || $strName eq 'title' ||
$strName eq 'summary';
# Store the node name
$oOut{name} = $strName;
@ -218,7 +232,7 @@ sub build
);
# Initialize the node object
my $oOut = {name => $$oDoc{name}, children => []};
my $oOut = {name => $$oDoc{name}, children => [], value => $$oDoc{value}};
my $strError = "in node $$oDoc{name}";
# Get all params
@ -230,7 +244,11 @@ sub build
}
}
if (defined($$oDoc{children}))
if ($$oDoc{name} eq 'p' || $$oDoc{name} eq 'title' || $$oDoc{name} eq 'summary')
{
$$oOut{field}{text} = $oDoc;
}
elsif (defined($$oDoc{children}))
{
for (my $iIndex = 0; $iIndex < @{$$oDoc{children}}; $iIndex++)
{
@ -241,11 +259,11 @@ sub build
{
$$oOut{field}{text} = $oSub;
}
elsif (defined($$oSub{value}))
elsif (defined($$oSub{value}) && !defined($$oSub{param}))
{
$$oOut{field}{$strName} = $$oSub{value};
}
elsif (!defined($$oSub{children}))
elsif (!defined($$oSub{children}) && !defined($$oSub{value}) && !defined($$oSub{param}))
{
$$oOut{field}{$strName} = true;
}
@ -310,7 +328,7 @@ sub nodeGetById
if (!defined($oNode) && $bRequired)
{
confess "unable to find child ${strName} (${strId}) in node $$oDoc{name}";
confess "unable to find child ${strName}" . (defined($strId) ? " (${strId})" : '') . " in node $$oDoc{name}";
}
# Return from function and log return values if any
@ -333,6 +351,27 @@ sub nodeGet
return $self->nodeGetById(shift, undef, shift);
}
####################################################################################################################################
# nodeAdd
#
# Add a node to to the current doc's child list
####################################################################################################################################
sub nodeAdd
{
my $self = shift;
my $strName = shift;
my $strValue = shift;
my $oParam = shift;
my $oField = shift;
my $oDoc = $self->{oDoc};
my $oNode = {name => $strName, value => $strValue, param => $oParam, field => $oField};
push(@{$$oDoc{children}}, $oNode);
return $self->nodeBless($oNode);
}
####################################################################################################################################
# nodeBless
#
@ -467,6 +506,24 @@ sub valueGet
);
}
####################################################################################################################################
# valueSet
####################################################################################################################################
sub valueSet
{
my $self = shift;
my $strValue = shift;
# Assign function parameters, defaults, and log debug info
my $strOperation = logDebugParam(OP_DOC_VALUE_SET);
# Set the value
${$self->{oDoc}}{value} = $strValue;
# Return from function and log return values if any
return logDebugReturn($strOperation);
}
####################################################################################################################################
# paramGet
#
@ -562,6 +619,28 @@ sub textGet
return $self->nodeBless($self->paramGet('text', shift, 'field'));
}
####################################################################################################################################
# textSet
#
# Get a field from a node.
####################################################################################################################################
sub textSet
{
my $self = shift;
my $oText = shift;
if (blessed($oText) && $oText->isa('BackRestDoc::Common::Doc'))
{
$oText = $oText->{oDoc};
}
elsif (ref($oText) ne 'HASH')
{
$oText = {name => 'text', children => [$oText]};
}
return $self->paramSet('text', $oText, 'field');
}
####################################################################################################################################
# fieldSet
#

331
doc/lib/BackRestDoc/DocConfig.pm → doc/lib/BackRestDoc/Common/DocConfig.pm
View File

@ -1,7 +1,7 @@
####################################################################################################################################
# DOC CONFIG MODULE
####################################################################################################################################
package BackRestDoc::DocConfig;
package BackRestDoc::Common::DocConfig;
use strict;
use warnings FATAL => qw(all);
@ -26,6 +26,13 @@ use constant OP_DOC_CONFIG => 'DocConfi
use constant OP_DOC_CONFIG_NEW => OP_DOC_CONFIG . '->new';
use constant OP_DOC_CONFIG_PROCESS => OP_DOC_CONFIG . '->process';
use constant OP_DOC_CONFIG_HELP_DATA_WRITE => OP_DOC_CONFIG . '->helpDataWrite';
use constant OP_DOC_CONFIG_HELP_CONFIG_DOC_GET => OP_DOC_CONFIG . '->helpConfigDocGet';
####################################################################################################################################
# Help types
####################################################################################################################################
use constant CONFIG_HELP_NAME => 'name';
use constant CONFIG_HELP_EXAMPLE => 'example';
####################################################################################################################################
# CONSTRUCTOR
@ -187,6 +194,9 @@ sub process
$$oCommandOption{&CONFIG_HELP_SUMMARY} = $oOptionDoc->nodeGet('summary')->textGet();
$$oCommandOption{&CONFIG_HELP_DESCRIPTION} = $oOptionDoc->textGet();
$$oCommandOption{&CONFIG_HELP_EXAMPLE} = $oOptionDoc->fieldGet('example');
$$oCommandOption{&CONFIG_HELP_NAME} = $oOptionDoc->paramGet('name');
# If the option did not come from the command also store in global option list. This prevents duplication of commonly
# used options.
@ -201,7 +211,9 @@ sub process
$$oOption{&CONFIG_HELP_SECTION} = $strSection;
}
$$oOption{&CONFIG_HELP_NAME} = $oOptionDoc->paramGet('name');
$$oOption{&CONFIG_HELP_DESCRIPTION} = $$oCommandOption{&CONFIG_HELP_DESCRIPTION};
$$oOption{&CONFIG_HELP_EXAMPLE} = $oOptionDoc->fieldGet('example');
}
}
}
@ -458,4 +470,321 @@ sub helpDataWrite
logDebugReturn($strOperation);
}
####################################################################################################################################
# helpConfigDocGet
#
# Get the xml for configuration help.
####################################################################################################################################
sub helpConfigDocGet
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my $strOperation = logDebugParam(OP_DOC_CONFIG_HELP_CONFIG_DOC_GET);
# Build a hash of the sections
my $oConfigHash = $self->{oConfigHash};
my $oConfigDoc = $self->{oDoc}->nodeGet('config');
my $oSectionHash = {};
foreach my $strOption (sort(keys($$oConfigHash{&CONFIG_HELP_OPTION})))
{
my $oOption = $$oConfigHash{&CONFIG_HELP_OPTION}{$strOption};
if (defined($$oOption{&CONFIG_HELP_SECTION}))
{
$$oSectionHash{$$oOption{&CONFIG_HELP_SECTION}}{$strOption} = true;
}
}
my $oDoc = new BackRestDoc::Common::Doc();
$oDoc->paramSet('title', $oConfigDoc->paramGet('title'));
# Output the introduction
my $oIntroSectionDoc = $oDoc->nodeAdd('section', undef, {id => 'introduction'});
$oIntroSectionDoc->nodeAdd('title')->textSet('Introduction');
$oIntroSectionDoc->textSet($oConfigDoc->textGet());
foreach my $strSection (sort(keys($oSectionHash)))
{
my $oSectionElement = $oDoc->nodeAdd('section', undef, {id => "section-${strSection}"});
my $oSectionDoc = $oConfigDoc->nodeGet('config-section-list')->nodeGetById('config-section', $strSection);
$oSectionElement->
nodeAdd('title')->textSet(
{name => 'text',
children=> [$oSectionDoc->paramGet('name') . ' Section (', {name => 'id', value => $strSection}, ')']});
foreach my $strOption (sort(keys($$oSectionHash{$strSection})))
{
$self->helpOptionGet(undef, $strOption, $oSectionElement, $$oConfigHash{&CONFIG_HELP_OPTION}{$strOption});
}
}
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'oDoc', value => $oDoc}
);
}
####################################################################################################################################
# helpCommandDocGet
#
# Get the xml for command help.
####################################################################################################################################
sub helpCommandDocGet
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my $strOperation = logDebugParam(OP_DOC_CONFIG_HELP_CONFIG_DOC_GET);
# Internal option find function
#
# The option may be stored with the command or in the option list depending on whether it's generic or command-specific
sub optionFind
{
my $oConfigHelpData = shift;
my $oOptionRule = shift;
my $strCommand = shift;
my $strOption = shift;
my $strSection = CONFIG_HELP_COMMAND;
my $oOption = $$oConfigHelpData{&CONFIG_HELP_COMMAND}{$strCommand}{&CONFIG_HELP_OPTION}{$strOption};
if ($$oOption{&CONFIG_HELP_SOURCE} eq CONFIG_HELP_SOURCE_DEFAULT)
{
$strSection = CONFIG_SECTION_GENERAL;
}
elsif ($$oOption{&CONFIG_HELP_SOURCE} eq CONFIG_HELP_SOURCE_SECTION)
{
$oOption = $$oConfigHelpData{&CONFIG_HELP_OPTION}{$strOption};
if (defined($$oOption{&CONFIG_HELP_SECTION}))
{
$strSection = $$oOption{&CONFIG_HELP_SECTION};
if ($strSection eq CONFIG_SECTION_COMMAND)
{
$strSection = CONFIG_SECTION_GENERAL;
}
}
else
{
$strSection = CONFIG_SECTION_GENERAL;
}
if (($strSection ne CONFIG_SECTION_GENERAL && $strSection ne CONFIG_SECTION_LOG &&
$strSection ne CONFIG_SECTION_STANZA && $strSection ne CONFIG_SECTION_EXPIRE) ||
$strSection eq $strCommand)
{
$strSection = CONFIG_HELP_COMMAND;
}
}
# if (defined(optionDefault($strOption, $strCommand)))
# {
# if ($$oOptionRule{$strOption}{&OPTION_RULE_TYPE} eq &OPTION_TYPE_BOOLEAN)
# {
# $$oOption{&CONFIG_HELP_DEFAULT} = optionDefault($strOption, $strCommand) ? 'y' : 'n';
# }
# else
# {
# $$oOption{&CONFIG_HELP_DEFAULT} = optionDefault($strOption, $strCommand);
# }
# }
#
# if (optionTest($strOption) && optionSource($strOption) ne CONFIG_HELP_SOURCE_DEFAULT)
# {
# if ($$oOptionRule{$strOption}{&OPTION_RULE_TYPE} eq &OPTION_TYPE_BOOLEAN)
# {
# $$oOption{&CONFIG_HELP_CURRENT} = optionGet($strOption) ? 'y' : 'n';
# }
# else
# {
# $$oOption{&CONFIG_HELP_CURRENT} = optionGet($strOption);
# }
# }
return $oOption, $strSection;
}
# Working variables
my $oConfigHash = $self->{oConfigHash};
my $oOperationDoc = $self->{oDoc}->nodeGet('operation');
my $oOptionRule = optionRuleGet();
my $oDoc = new BackRestDoc::Common::Doc();
$oDoc->paramSet('title', $oOperationDoc->paramGet('title'));
# Output the introduction
my $oIntroSectionDoc = $oDoc->nodeAdd('section', undef, {id => 'introduction'});
$oIntroSectionDoc->nodeAdd('title')->textSet('Introduction');
$oIntroSectionDoc->textSet($oOperationDoc->textGet());
foreach my $strCommand (sort(keys($$oConfigHash{&CONFIG_HELP_COMMAND})))
{
my $oCommandHash = $$oConfigHash{&CONFIG_HELP_COMMAND}{$strCommand};
my $oSectionElement = $oDoc->nodeAdd('section', undef, {id => "command-${strCommand}"});
my $oCommandDoc = $oOperationDoc->nodeGet('command-list')->nodeGetById('command', $strCommand);
$oSectionElement->
nodeAdd('title')->textSet(
{name => 'text',
children=> [$oCommandDoc->paramGet('name') . ' Command (', {name => 'id', value => $strCommand}, ')']});
$oSectionElement->textSet($$oCommandHash{&CONFIG_HELP_DESCRIPTION});
# use Data::Dumper;
# confess Dumper($oDoc->{oDoc});
if (defined($$oCommandHash{&CONFIG_HELP_OPTION}))
{
my $oCategory = {};
foreach my $strOption (sort(keys(%{$$oCommandHash{&CONFIG_HELP_OPTION}})))
{
my ($oOption, $strCategory) = optionFind($oConfigHash, $oOptionRule, $strCommand, $strOption);
$$oCategory{$strCategory}{$strOption} = $oOption;
}
# Iterate sections
foreach my $strCategory (sort(keys(%{$oCategory})))
{
my $oOptionListElement = $oSectionElement->nodeAdd('section', undef, {id => "category-${strCategory}"});
$oOptionListElement->
nodeAdd('title')->textSet(ucfirst($strCategory) . ' Options');
# Iterate options
foreach my $strOption (sort(keys(%{$$oCategory{$strCategory}})))
{
$self->helpOptionGet($strCommand, $strOption, $oOptionListElement,
$$oCommandHash{&CONFIG_HELP_OPTION}{$strOption});
}
}
}
}
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'oDoc', value => $oDoc}
);
}
####################################################################################################################################
# helpOptionGet
#
# Get the xml for an option.
####################################################################################################################################
sub helpOptionGet
{
my $self = shift;
my $strCommand = shift;
my $strOption = shift;
my $oParentElement = shift;
my $oOptionHash = shift;
# Create the option section
my $oOptionElement = $oParentElement->nodeAdd('section', undef, {id => "option-${strOption}"});
# Set the option section title
$oOptionElement->
nodeAdd('title')->textSet(
{name => 'text',
children=> [$$oOptionHash{&CONFIG_HELP_NAME} . ' Option (', {name => 'id', value => "--${strOption}"}, ')']});
# Add the option summary and description
$oOptionElement->
nodeAdd('p')->textSet($$oOptionHash{&CONFIG_HELP_SUMMARY});
$oOptionElement->
nodeAdd('p')->textSet($$oOptionHash{&CONFIG_HELP_DESCRIPTION});
# Get the default value (or required=n if there is no default)
my $strCodeBlock;
if (defined(optionDefault($strOption, $strCommand)) || $strOption eq OPTION_REPO_REMOTE_PATH)
{
my $strDefault;
if ($strOption eq OPTION_CONFIG || $strOption eq OPTION_CONFIG_REMOTE)
{
$strDefault = '/etc/{[backrest-exe]}.conf';
}
elsif ($strOption eq OPTION_COMMAND_REMOTE)
{
$strDefault = '[INSTALL-PATH]/{[backrest-exe]}';
}
elsif ($strOption eq OPTION_REPO_REMOTE_PATH)
{
$strDefault = optionDefault(OPTION_REPO_PATH);
}
else
{
if (optionTypeTest($strOption, OPTION_TYPE_BOOLEAN))
{
$strDefault = optionDefault($strOption, $strCommand) ? 'y' : 'n';
}
else
{
$strDefault = optionDefault($strOption, $strCommand);
}
}
$strCodeBlock = "default: ${strDefault}";
}
# This won't work correctly until there is some notion of dependency
# elsif (optionRequired($strOption, $strCommand))
# {
# $strCodeBlock = 'required: y';
# }
# Get the allowed range if it exists
my ($strRangeMin, $strRangeMax) = optionRange($strOption, $strCommand);
if (defined($strRangeMin))
{
$strCodeBlock .= (defined($strCodeBlock) ? "\n" : '') . "allowed: ${strRangeMin}-${strRangeMax}";
}
# Get the example
my $strExample;
if (defined($strCommand))
{
if (optionTypeTest($strOption, OPTION_TYPE_BOOLEAN))
{
if ($$oOptionHash{&CONFIG_HELP_EXAMPLE} ne 'n' && $$oOptionHash{&CONFIG_HELP_EXAMPLE} ne 'y')
{
confess &log(ERROR, "option ${strOption} example should be boolean but value is: " .
$$oOptionHash{&CONFIG_HELP_EXAMPLE});
}
$strExample = '--' . ($$oOptionHash{&CONFIG_HELP_EXAMPLE} eq 'n' ? 'no-' : '') . $strOption;
}
else
{
$strExample = "--${strOption}=" . $$oOptionHash{&CONFIG_HELP_EXAMPLE};
}
}
else
{
$strExample = "${strOption}=" . $$oOptionHash{&CONFIG_HELP_EXAMPLE};
}
$strCodeBlock .= (defined($strCodeBlock) ? "\n" : '') . "example: ${strExample}";
$oOptionElement->
nodeAdd('code-block')->valueSet($strCodeBlock);
}
1;

192
doc/lib/BackRestDoc/DocRender.pm → doc/lib/BackRestDoc/Common/DocRender.pm
View File

@ -1,7 +1,7 @@
####################################################################################################################################
# DOC RENDER MODULE
####################################################################################################################################
package BackRestDoc::DocRender;
package BackRestDoc::Common::DocRender;
use strict;
use warnings FATAL => qw(all);
@ -38,8 +38,8 @@ my $oRenderTag =
'b' => ['**', '**'],
'i' => ['_', '_'],
'bi' => ['_**', '**_'],
'ul' => ["\n", ''],
'ol' => ["\n", ''],
'ul' => ["\n", "\n"],
'ol' => ["\n", "\n"],
'li' => ['- ', "\n"],
'id' => ['`', '`'],
'file' => ['`', '`'],
@ -59,13 +59,14 @@ my $oRenderTag =
'b' => ['', ''],
'i' => ['', ''],
'bi' => ['', ''],
'ul' => ["\n", ''],
'ol' => ["\n", ''],
'ul' => ["\n", "\n"],
'ol' => ["\n", "\n"],
'li' => ['* ', "\n"],
'id' => ['', ''],
'file' => ['', ''],
'path' => ['', ''],
'cmd' => ['', ''],
'br-option' => ['', ''],
'param' => ['', ''],
'setting' => ['', ''],
'code' => ['', ''],
@ -77,7 +78,27 @@ my $oRenderTag =
'html' =>
{
'b' => ['<b>', '</b>']
'b' => ['<b>', '</b>'],
'i' => ['<i>', '</i>'],
'bi' => ['<i><b>', '</b></i>'],
'ul' => ['<ul>', '</ul>'],
'ol' => ['<ol>', '</ol>'],
'li' => ['<li>', '</li>'],
'id' => ['<span class="id">', '</span>'],
'file' => ['<span class="file">', '</span>'],
'path' => ['<span class="path">', '</span>'],
'cmd' => ['<span class="cmd">', '</span>'],
'user' => ['<span class="user">', '</span>'],
'br-option' => ['<span class="br-option">', '</span>'],
'br-setting' => ['<span class="br-setting">', '</span>'],
'pg-option' => ['<span class="pg-option">', '</span>'],
'pg-setting' => ['<span class="pg-setting">', '</span>'],
'code' => ['<id>', '</id>'],
'code-block' => ['<code-block>', '</code-block>'],
'exe' => ['<id>', '</id>'],
'setting' => ['<span class="br-setting">', '</span>'], # !!! This will need to be fixed
'backrest' => [undef, ''],
'postgres' => ['<span class="postgres">PostgreSQL</span>', '']
}
};
@ -111,6 +132,7 @@ sub new
$$oRenderTag{markdown}{exe}[0] = $self->{strExeName};
$$oRenderTag{text}{backrest}[0] = $self->{strProjectName};
$$oRenderTag{text}{exe}[0] = $self->{strExeName};
$$oRenderTag{html}{backrest}[0] = "<span class=\"backrest\">$self->{strProjectName}</span>";
# Return from function and log return values if any
return logDebugReturn
@ -166,7 +188,21 @@ sub process
$strBuffer = ('#' x $iDepth) . ' `' . $oDoc->paramGet('id') . '` ' . $strTitle;
}
if (defined($oDoc->paramGet('title', false)))
# Try to get the title param from the element (!!! this is the old style and should be removed)
my $strTitle = $oDoc->paramGet('title', false);
if (!defined($strTitle))
{
$strTitle = $oDoc->paramGet('subtitle', false);
}
# If not found then get the title element
if (!defined($strTitle) && defined($oDoc->nodeGet('title', false)))
{
$strTitle = $self->processText($oDoc->nodeGet('title')->textGet());
}
if (defined($strTitle))
{
$strBuffer = ('#' x $iDepth) . ' ';
@ -175,7 +211,7 @@ sub process
$strBuffer .= 'v' . $oDoc->paramGet('version') . ': ';
}
$strBuffer .= ($iDepth == 1 ? "${strProjectName} - " : '') . $oDoc->paramGet('title');
$strBuffer .= ($iDepth == 1 ? "${strProjectName}<br/>" : '') . $strTitle;
if (defined($oDoc->paramGet('date', false)))
{
@ -233,56 +269,6 @@ sub process
$strBuffer .= $self->processText($oDoc->textGet());
}
if ($oDoc->nameGet() eq 'config-key' || $oDoc->nameGet() eq 'option')
{
my $strError = 'config section ?, key ' . $oDoc->paramGet('id') . 'requires';
my $bRequired = defined($oDoc->fieldGet('required', false)) && $oDoc->fieldGet('required');
my $strDefault = $oDoc->fieldGet('default', false);
my $strAllow = $oDoc->fieldGet('allow', false);
my $strOverride = $oDoc->fieldGet('override', false);
my $strExample = $oDoc->fieldGet('example', false);
# !!! Temporary hack to make docs generate correctly. This should be replace with a parameter so that it can be
# changed based on the build. Maybe check the exe name by default?
if ($oDoc->paramGet('id') eq 'config')
{
$strDefault = '/etc/pg_backrest.conf';
}
if (defined($strExample))
{
if (index($strExample, '=') == -1)
{
$strExample = "=${strExample}";
}
else
{
$strExample = " ${strExample}";
}
$strExample = $oDoc->paramGet('id') . $strExample;
if (defined($oDoc->fieldGet('cmd', false)) && $oDoc->fieldGet('cmd'))
{
$strExample = '--' . $strExample;
if (index($oDoc->fieldGet('example'), ' ') != -1)
{
$strExample = "\"${strExample}\"";
}
}
}
$strBuffer .= "\n```\n" .
($bRequired ? "required: " . ($bRequired ? 'y' : 'n') . "\n" : '') .
(defined($strDefault) ? "default: ${strDefault}\n" : '') .
(defined($strAllow) ? "allow: ${strAllow}\n" : '') .
(defined($strOverride) ? "override: ${strOverride}\n" : '') .
(defined($strExample) ? "example: ${strExample}\n" : '') .
"```";
}
if ($strBuffer ne "" && $iDepth != 1 && !$bList)
{
$strBuffer = "\n\n" . $strBuffer;
@ -297,7 +283,7 @@ sub process
foreach my $oChild ($oDoc->nodeList(undef, false))
{
if ($oChild->nameGet() ne 'summary')
if ($oChild->nameGet() ne 'summary' && $oChild->nameGet() ne 'title')
{
if ($strType eq 'markdown' || $strType eq 'text')
{
@ -354,33 +340,70 @@ sub processTag
my $strType = $self->{strType};
my $strTag = $oTag->nameGet();
my $strStart = $$oRenderTag{$strType}{$strTag}[0];
my $strStop = $$oRenderTag{$strType}{$strTag}[1];
if (!defined($strStart) || !defined($strStop))
if (!defined($strTag))
{
confess "invalid type ${strType} or tag ${strTag}";
use Data::Dumper;
confess Dumper($oTag);
}
$strBuffer .= $strStart;
if ($strTag eq 'link')
{
my $strUrl = $oTag->paramGet('url', false);
if ($strTag eq 'li' || $strTag eq 'code-block')
{
$strBuffer .= $self->processText($oTag);
}
elsif (defined($oTag->valueGet()))
{
$strBuffer .= $oTag->valueGet();
if ($strType eq 'markdown')
{
if (!defined($strUrl))
{
$strUrl = '{[backrest-url-base]}/' . $oTag->paramGet('page');
}
$strBuffer = '[' . $oTag->valueGet() . '](' . $strUrl . ')';
}
elsif ($strType eq 'html')
{
if (!defined($strUrl))
{
$strUrl = $oTag->paramGet('page');
}
$strBuffer = '<a href="' . $strUrl . '">' . $oTag->valueGet() . '</a>';
}
else
{
confess "tag link not valid for type ${strType}";
}
}
else
{
foreach my $oSubTag ($oTag->nodeList(undef, false))
{
$strBuffer .= $self->processTag($oSubTag);
}
}
my $strStart = $$oRenderTag{$strType}{$strTag}[0];
my $strStop = $$oRenderTag{$strType}{$strTag}[1];
$strBuffer .= $strStop;
if (!defined($strStart) || !defined($strStop))
{
confess "invalid type ${strType} or tag ${strTag}";
}
$strBuffer .= $strStart;
if ($strTag eq 'p' || $strTag eq 'title' || $strTag eq 'li' || $strTag eq 'code-block' || $strTag eq 'summary')
{
$strBuffer .= $self->processText($oTag);
}
elsif (defined($oTag->valueGet()))
{
$strBuffer .= $oTag->valueGet();
}
else
{
foreach my $oSubTag ($oTag->nodeList(undef, false))
{
$strBuffer .= $self->processTag($oSubTag);
}
}
$strBuffer .= $strStop;
}
# Return from function and log return values if any
return logDebugReturn
@ -423,6 +446,21 @@ sub processText
$strBuffer .= $self->processTag($oNode);
}
}
#
# if ($strType eq 'html')
# {
# # $strBuffer =~ s/^\s+|\s+$//g;
#
# $strBuffer =~ s/\n/\<br\/\>\n/g;
# }
# if ($strType eq 'markdown')
# {
# $strBuffer =~ s/^\s+|\s+$//g;
$strBuffer =~ s/ +/ /g;
$strBuffer =~ s/^ //smg;
# }
# Return from function and log return values if any
return logDebugReturn

206
doc/lib/BackRestDoc/Html/DocHtmlBuilder.pm Normal file
View File

@ -0,0 +1,206 @@
####################################################################################################################################
# DOC HTML BUILDER MODULE
####################################################################################################################################
package BackRestDoc::Html::DocHtmlBuilder;
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
use Exporter qw(import);
our @EXPORT = qw();
use File::Basename qw(dirname);
use File::Copy;
use lib dirname($0) . '/../lib';
use BackRest::Common::Log;
use BackRest::Common::String;
use BackRestDoc::Html::DocHtmlElement;
####################################################################################################################################
# Operation constants
####################################################################################################################################
use constant OP_DOC_HTML_BUILDER => 'DocHtmlBuilder';
use constant OP_DOC_HTML_BUILDER_NEW => OP_DOC_HTML_BUILDER . '->new';
use constant OP_DOC_HTML_BUILDER_HTML_GET => OP_DOC_HTML_BUILDER . '->htmlGet';
use constant OP_DOC_HTML_BUILDER_HTML_RENDER => OP_DOC_HTML_BUILDER . '->htmlRender';
####################################################################################################################################
# CONSTRUCTOR
####################################################################################################################################
sub new
{
my $class = shift; # Class name
# Create the class hash
my $self = {};
bless $self, $class;
$self->{strClass} = $class;
# Assign function parameters, defaults, and log debug info
(
my $strOperation,
$self->{strName},
$self->{strTitle}
) =
logDebugParam
(
OP_DOC_HTML_BUILDER_NEW, \@_,
{name => 'strName'},
{name => 'strTitle'}
);
$self->{oBody} = new BackRestDoc::Html::DocHtmlElement(HTML_BODY);
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'self', value => $self}
);
}
####################################################################################################################################
# indent
#
# Indent html
####################################################################################################################################
sub indent
{
my $self = shift;
my $iDepth = shift;
return (' ' x $iDepth);
}
####################################################################################################################################
# lf
#
# Add a linefeed.
####################################################################################################################################
sub lf
{
my $self = shift;
return "\n";
}
####################################################################################################################################
# bodyGet
#
# Get the body element.
####################################################################################################################################
sub bodyGet
{
my $self = shift;
return $self->{oBody};
}
####################################################################################################################################
# htmlRender
#
# Render each html element.
####################################################################################################################################
sub htmlRender
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my (
$strOperation,
$oElement,
$iDepth
) =
logDebugParam
(
OP_DOC_HTML_BUILDER_HTML_RENDER, \@_,
{name => 'oElement'},
{name => 'iDepth'}
);
# Build the header
my $strHtml =
$self->indent($iDepth) . "<$oElement->{strType}" .
(defined($oElement->{strClass}) ? " class=\"$oElement->{strClass}\"": '') .
(defined($oElement->{strRef}) ? " href=\"$oElement->{strRef}\"": '') .
(defined($oElement->{strId}) ? " id=\"$oElement->{strId}\"": '') . '>';
if (defined($oElement->{strContent}))
{
$oElement->{strContent} =~ s/\n/\<br\/>\n/g;
$strHtml .= $self->lf() . trim($oElement->{strContent}) . $self->lf() . $self->indent($iDepth);
}
else
{
if (!($oElement->{strType} eq HTML_A && @{$oElement->{oyElement}} == 0))
{
$strHtml .= $self->lf();
}
foreach my $oChildElement (@{$oElement->{oyElement}})
{
$strHtml .= $self->htmlRender($oChildElement, $iDepth + 1);
}
if (!($oElement->{strType} eq HTML_A && @{$oElement->{oyElement}} == 0))
{
$strHtml .= $self->indent($iDepth);
}
}
$strHtml .= "</$oElement->{strType}>" . $self->lf();
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'strHtml', value => $strHtml, trace => true}
);
}
####################################################################################################################################
# htmlGet
#
# Generate the HTML.
####################################################################################################################################
sub htmlGet
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my ($strOperation) = logDebugParam(OP_DOC_HTML_BUILDER_HTML_GET);
# Build the header
my $strHtml =
$self->indent(0) . "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"" .
" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">" . $self->lf() .
$self->indent(0) . "<html xmlns=\"http://www.w3.org/1999/xhtml\">" . $self->lf() .
$self->indent(0) . "<head>" . $self->lf() .
$self->indent(1) . "<title>$self->{strTitle}</title>" . $self->lf() .
$self->indent(1) . "<link rel=\"stylesheet\" href=\"default.css\" type=\"text/css\"></link>" . $self->lf() .
# $self->indent(1) . "<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"></meta>" . $self->lf() .
$self->indent(1) . "<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\"></meta>" . $self->lf() .
$self->indent(1) . "<meta name=\"og:site_name\" content=\"$self->{strName}\"></meta>" . $self->lf() .
$self->indent(1) . "<meta name=\"og:title\" content=\"$self->{strTitle}\"></meta>" . $self->lf() .
$self->indent(1) . "<meta name=\"og:image\" content=\"favicon.png\"></meta>" . $self->lf() .
$self->indent(0) . "</head>" . $self->lf();
$strHtml .= $self->htmlRender($self->bodyGet(), 0);
# Complete the html
$strHtml .=
$self->indent(0) . "</html>" . $self->lf();
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'strHtml', value => $strHtml, trace => true}
);
}
1;

152
doc/lib/BackRestDoc/Html/DocHtmlElement.pm Normal file
View File

@ -0,0 +1,152 @@
####################################################################################################################################
# DOC HTML ELEMENT MODULE
####################################################################################################################################
package BackRestDoc::Html::DocHtmlElement;
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
use Exporter qw(import);
our @EXPORT = qw();
use File::Basename qw(dirname);
use File::Copy;
use Scalar::Util qw(blessed);
use lib dirname($0) . '/../lib';
use BackRest::Common::Log;
####################################################################################################################################
# Operation constants
####################################################################################################################################
use constant OP_DOC_HTML_ELEMENT => 'DocHtmlElement';
use constant OP_DOC_HTML_ELEMENT_ADD => OP_DOC_HTML_ELEMENT . '->add';
use constant OP_DOC_HTML_ELEMENT_ADD_NEW => OP_DOC_HTML_ELEMENT . '->addNew';
use constant OP_DOC_HTML_ELEMENT_NEW => OP_DOC_HTML_ELEMENT . '->new';
####################################################################################################################################
# Html Element Types
####################################################################################################################################
use constant HTML_A => 'a';
push @EXPORT, qw(HTML_A);
use constant HTML_BODY => 'body';
push @EXPORT, qw(HTML_BODY);
use constant HTML_DIV => 'div';
push @EXPORT, qw(HTML_DIV);
use constant HTML_SPAN => 'span';
push @EXPORT, qw(HTML_SPAN);
####################################################################################################################################
# CONSTRUCTOR
####################################################################################################################################
sub new
{
my $class = shift; # Class name
# Create the class hash
my $self = {};
bless $self, $class;
$self->{strClass} = $class;
# Assign function parameters, defaults, and log debug info
(
my $strOperation,
$self->{strType},
$self->{strClass},
my $oParam
) =
logDebugParam
(
OP_DOC_HTML_ELEMENT_NEW, \@_,
{name => 'strType', trace => true},
{name => 'strClass', required => false, trace => true},
{name => 'oParam', required => false, trace => true}
);
$self->{oyElement} = [];
$self->{strContent} = $$oParam{strContent};
$self->{strId} = $$oParam{strId};
$self->{strRef} = $$oParam{strRef};
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'self', value => $self}
);
}
####################################################################################################################################
# addNew
#
# Create a new element and add it.
####################################################################################################################################
sub addNew
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my (
$strOperation,
$strType,
$strClass,
$oParam
) =
logDebugParam
(
OP_DOC_HTML_ELEMENT_ADD_NEW, \@_,
{name => 'strType', trace => true},
{name => 'strClass', required => false, trace => true},
{name => 'oParam', required => false, trace => true}
);
my $oElement = new BackRestDoc::Html::DocHtmlElement($strType, $strClass, $oParam);
$self->add($oElement);
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'oElement', value => $oElement, trace => true}
);
}
####################################################################################################################################
# add
#
# Add an element.
####################################################################################################################################
sub add
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my (
$strOperation,
$oElement
) =
logDebugParam
(
OP_DOC_HTML_ELEMENT_ADD, \@_,
{name => 'oElement', trace => true}
);
if (!(blessed($oElement) && $oElement->isa('BackRestDoc::Html::DocHtmlElement')))
{
confess &log(ASSERT, 'oElement must be a valid element object');
}
push(@{$self->{oyElement}}, $oElement);
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'oElement', value => $oElement, trace => true}
);
}
1;

699
doc/lib/BackRestDoc/Html/DocHtmlPage.pm Normal file
View File

@ -0,0 +1,699 @@
####################################################################################################################################
# DOC HTML PAGE MODULE
####################################################################################################################################
package BackRestDoc::Html::DocHtmlPage;
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
use Data::Dumper;
use Exporter qw(import);
our @EXPORT = qw();
use File::Basename qw(dirname);
use File::Copy;
use Storable qw(dclone);
use lib dirname($0) . '/../lib';
use BackRest::Common::Ini;
use BackRest::Common::Log;
use BackRest::Common::String;
use BackRest::Config::ConfigHelp;
use BackRest::FileCommon;
use lib dirname($0) . '/../test/lib';
use BackRestTest::Common::ExecuteTest;
use BackRestDoc::Html::DocHtmlBuilder;
use BackRestDoc::Html::DocHtmlElement;
####################################################################################################################################
# Operation constants
####################################################################################################################################
use constant OP_DOC_HTML_PAGE => 'DocHtmlPage';
use constant OP_DOC_HTML_PAGE_BACKREST_CONFIG_PROCESS => OP_DOC_HTML_PAGE . '->backrestConfigProcess';
use constant OP_DOC_HTML_PAGE_EXECUTE => OP_DOC_HTML_PAGE . '->execute';
use constant OP_DOC_HTML_PAGE_NEW => OP_DOC_HTML_PAGE . '->new';
use constant OP_DOC_HTML_PAGE_POSTGRES_CONFIG_PROCESS => OP_DOC_HTML_PAGE . '->postgresConfigProcess';
use constant OP_DOC_HTML_PAGE_PROCESS => OP_DOC_HTML_PAGE . '->process';
use constant OP_DOC_HTML_PAGE_SECTION_PROCESS => OP_DOC_HTML_PAGE . '->sectionProcess';
####################################################################################################################################
# CONSTRUCTOR
####################################################################################################################################
sub new
{
my $class = shift; # Class name
# Create the class hash
my $self = {};
bless $self, $class;
$self->{strClass} = $class;
# Assign function parameters, defaults, and log debug info
(
my $strOperation,
$self->{oSite},
$self->{strPageId},
$self->{bExe}
) =
logDebugParam
(
OP_DOC_HTML_PAGE_NEW, \@_,
{name => 'oSite'},
{name => 'strPageId'},
{name => 'bExe', default => true}
);
#
# use Data::Dumper;
# confess Dumper(${$self->{oSite}->{oSite}}{common}{oRender});
# Copy page data to self
$self->{oPage} = ${$self->{oSite}->{oSite}}{page}{$self->{strPageId}};
$self->{oDoc} = ${$self->{oPage}}{'oDoc'};
$self->{oRender} = ${$self->{oSite}->{oSite}}{common}{oRender};
$self->{oReference} = ${$self->{oSite}->{oSite}}{common}{oReference};
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'self', value => $self}
);
}
####################################################################################################################################
# execute
####################################################################################################################################
sub execute
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$oCommand,
$iIndent
) =
logDebugParam
(
OP_DOC_HTML_PAGE_EXECUTE, \@_,
{name => 'oCommand'},
{name => 'iIndent', default => 1}
);
# Working variables
my $strOutput;
# Command variables
my $strCommand = trim($oCommand->fieldGet('exe-cmd'));
my $strUser = $oCommand->fieldGet('exe-user', false);
my $bSuppressError = defined($oCommand->fieldGet('exe-err-suppress', false)) ? $oCommand->fieldGet('exe-err-suppress') : false;
my $bSuppressStdErr = defined($oCommand->fieldGet('exe-err-suppress-stderr', false)) ?
$oCommand->fieldGet('exe-err-suppress-stderr') : false;
my $bExeSkip = defined($oCommand->fieldGet('exe-skip', false)) ? $oCommand->fieldGet('exe-skip') : false;
my $bExeOutput = defined($oCommand->fieldGet('exe-output', false)) ? $oCommand->fieldGet('exe-output') : false;
my $bExeRetry = defined($oCommand->fieldGet('exe-retry', false)) ? $oCommand->fieldGet('exe-retry') : false;
my $strExeVar = defined($oCommand->fieldGet('exe-var', false)) ? $oCommand->fieldGet('exe-var') : undef;
my $iExeExpectedError = defined($oCommand->fieldGet('exe-err-expect', false)) ? $oCommand->fieldGet('exe-err-expect') : undef;
if ($bExeRetry)
{
sleep(1);
}
$strUser = defined($strUser) ? $strUser : 'postgres';
$strCommand = $self->{oSite}->variableReplace(
($strUser eq 'vagrant' ? '' : 'sudo ' . ($strUser eq 'root' ? '' : "-u ${strUser} ")) . $strCommand);
&log(INFO, (' ' x $iIndent) . "execute: $strCommand");
if (!$bExeSkip)
{
if ($self->{bExe})
{
my $oExec = new BackRestTest::Common::ExecuteTest($strCommand,
{bSuppressError => $bSuppressError,
bSuppressStdErr => $bSuppressStdErr,
iExpectedExitStatus => $iExeExpectedError});
$oExec->begin();
$oExec->end();
if ($bExeOutput && defined($oExec->{strOutLog}) && $oExec->{strOutLog} ne '')
{
$strOutput = trim($oExec->{strOutLog});
$strOutput =~ s/^[0-9]{4}-[0-1][0-9]-[0-3][0-9] [0-2][0-9]:[0-6][0-9]:[0-6][0-9]\.[0-9]{3} T[0-9]{2} //smg;
}
if (defined($strExeVar))
{
$self->{oSite}->variableSet($strExeVar, trim($oExec->{strOutLog}));
}
if (defined($iExeExpectedError))
{
$strOutput .= trim($oExec->{strErrorLog});
}
}
elsif ($bExeOutput)
{
$strOutput = 'Output suppressed for testing';
}
}
if (defined($strExeVar) && !defined($self->{oSite}->variableGet($strExeVar)))
{
$self->{oSite}->variableSet($strExeVar, '[Unset Variable]');
}
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => '$strCommand', value => $strCommand, trace => true},
{name => '$strOutput', value => $strOutput, trace => true}
);
}
####################################################################################################################################
# process
#
# Generate the site html
####################################################################################################################################
sub process
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my $strOperation = logDebugParam(OP_DOC_HTML_PAGE_PROCESS);
# Working variables
my $oPage = $self->{oDoc};
# Initialize page
my $strTitle = ${$self->{oRender}}{strProjectName} .
(defined($oPage->paramGet('title', false)) ? ' ' . $oPage->paramGet('title') : '');
my $strSubTitle = $oPage->paramGet('subtitle', false);
my $oHtmlBuilder = new BackRestDoc::Html::DocHtmlBuilder("${$self->{oRender}}{strProjectName} - Reliable PostgreSQL Backup",
$strTitle . (defined($strSubTitle) ? " - ${strSubTitle}" : ''));
# Execute cleanup commands
if (defined($self->{oDoc}->nodeGet('cleanup', false)))
{
&log(INFO, "do cleanup");
foreach my $oExecute ($oPage->nodeGet('cleanup')->nodeList('execute'))
{
$self->execute($oExecute);
}
}
# Generate header
my $oPageHeader = $oHtmlBuilder->bodyGet()->addNew(HTML_DIV, 'page-header');
$oPageHeader->
addNew(HTML_DIV, 'page-header-title',
{strContent => $strTitle});
if (defined($strSubTitle))
{
$oPageHeader->
addNew(HTML_DIV, 'page-header-subtitle',
{strContent => $strSubTitle});
}
# Generate menu
my $oMenuBody = $oHtmlBuilder->bodyGet()->addNew(HTML_DIV, 'page-menu')->addNew(HTML_DIV, 'menu-body');
if ($self->{strPageId} ne 'index')
{
my $oPage = ${$self->{oSite}->{oSite}}{page}{'index'};
$oMenuBody->
addNew(HTML_DIV, 'menu')->
addNew(HTML_A, 'menu-link', {strContent => $$oPage{strMenuTitle}, strRef => '{[backrest-url-root]}'});
}
foreach my $strPageId(sort(keys(${$self->{oSite}->{oSite}}{page})))
{
if ($strPageId ne $self->{strPageId} && $strPageId ne 'index')
{
my $oPage = ${$self->{oSite}->{oSite}}{page}{$strPageId};
$oMenuBody->
addNew(HTML_DIV, 'menu')->
addNew(HTML_A, 'menu-link', {strContent => $$oPage{strMenuTitle}, strRef => "${strPageId}.html"});
}
}
# Generate table of contents
my $oPageTocBody;
if (!defined($oPage->paramGet('toc', false)) || $oPage->paramGet('toc') eq 'y')
{
my $oPageToc = $oHtmlBuilder->bodyGet()->addNew(HTML_DIV, 'page-toc');
$oPageToc->
addNew(HTML_DIV, 'page-toc-title',
{strContent => "Table of Contents"});
$oPageTocBody = $oPageToc->
addNew(HTML_DIV, 'page-toc-body');
}
# Generate body
my $oPageBody = $oHtmlBuilder->bodyGet()->addNew(HTML_DIV, 'page-body');
# Render sections
foreach my $oSection ($oPage->nodeList('section'))
{
my ($oChildSectionElement, $oChildSectionTocElement) =
$self->sectionProcess($oSection, undef, 1);
$oPageBody->add($oChildSectionElement);
if (defined($oPageTocBody))
{
$oPageTocBody->add($oChildSectionTocElement);
}
}
my $oPageFooter = $oHtmlBuilder->bodyGet()->
addNew(HTML_DIV, 'page-footer',
{strContent => ${$self->{oSite}->{oSite}}{common}{strFooter}});
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'strHtml', value => $oHtmlBuilder->htmlGet(), trace => true}
);
}
####################################################################################################################################
# sectionProcess
####################################################################################################################################
sub sectionProcess
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$oSection,
$strAnchor,
$iDepth
) =
logDebugParam
(
OP_DOC_HTML_PAGE_SECTION_PROCESS, \@_,
{name => 'oSection'},
{name => 'strAnchor', required => false},
{name => 'iDepth'}
);
&log(INFO, (' ' x ($iDepth - 1)) . 'process section: ' . $oSection->paramGet('id'));
if ($iDepth > 3)
{
confess &log(ASSERT, "section depth of ${iDepth} exceeds maximum");
}
# Working variables
$strAnchor = (defined($strAnchor) ? "${strAnchor}-" : '') . $oSection->paramGet('id');
my $oRender = $self->{oRender};
# Create the section toc element
my $oSectionTocElement = new BackRestDoc::Html::DocHtmlElement(HTML_DIV, "section${iDepth}-toc");
# Create the section element
my $oSectionElement = new BackRestDoc::Html::DocHtmlElement(HTML_DIV, "section${iDepth}");
# Add the section anchor
$oSectionElement->addNew(HTML_A, undef, {strId => $strAnchor});
# Add the section title to section and toc
my $strSectionTitle = $oRender->processText($oSection->nodeGet('title')->textGet());
$oSectionElement->
addNew(HTML_DIV, "section${iDepth}-title",
{strContent => $strSectionTitle});
my $oTocSectionTitleElement = $oSectionTocElement->
addNew(HTML_DIV, "section${iDepth}-toc-title");
$oTocSectionTitleElement->
addNew(HTML_A, undef,
{strContent => $strSectionTitle, strRef => "#${strAnchor}"});
# Add the section intro if it exists
if (defined($oSection->textGet(false)))
{
$oSectionElement->
addNew(HTML_DIV, "section-intro",
{strContent => $oRender->processText($oSection->textGet())});
}
# Add the section body
my $oSectionBodyElement = $oSectionElement->addNew(HTML_DIV, "section-body");
# Process each child
my $oSectionBodyExe;
foreach my $oChild ($oSection->nodeList())
{
&log(INFO, (' ' x $iDepth) . 'process child ' . $oChild->nameGet());
# Execute a command
if ($oChild->nameGet() eq 'execute-list')
{
my $oSectionBodyExecute = $oSectionBodyElement->addNew(HTML_DIV, "execute");
my $bFirst = true;
$oSectionBodyExecute->
addNew(HTML_DIV, "execute-title",
{strContent => $oRender->processText($oChild->nodeGet('title')->textGet()) . ':'});
my $oExecuteBodyElement = $oSectionBodyExecute->addNew(HTML_DIV, "execute-body");
foreach my $oExecute ($oChild->nodeList('execute'))
{
my $bExeShow = defined($oExecute->fieldGet('exe-no-show', false)) ? false : true;
my $bExeExpectedError = defined($oExecute->fieldGet('exe-err-expect', false)) ? true : false;
my ($strCommand, $strOutput) = $self->execute($oExecute, $iDepth + 1);
if ($bExeShow)
{
$oExecuteBodyElement->
addNew(HTML_DIV, "execute-body-cmd" . ($bFirst ? '-first' : ''),
{strContent => $strCommand});
if (defined($strOutput))
{
my $strHighLight = $self->{oSite}->variableReplace($oExecute->fieldGet('exe-highlight', false));
my $bHighLightOld;
my $bHighLightFound = false;
my $strHighLightOutput;
foreach my $strLine (split("\n", $strOutput))
{
my $bHighLight = defined($strHighLight) && $strLine =~ /$strHighLight/;
if (defined($bHighLightOld) && $bHighLight != $bHighLightOld)
{
$oExecuteBodyElement->
addNew(HTML_DIV, 'execute-body-output' . ($bHighLightOld ? '-highlight' : '') .
($bExeExpectedError ? '-error' : ''), {strContent => $strHighLightOutput});
undef($strHighLightOutput);
}
$strHighLightOutput .= "${strLine}\n";
$bHighLightOld = $bHighLight;
$bHighLightFound = $bHighLightFound ? true : $bHighLight ? true : false;
}
if (defined($bHighLightOld))
{
$oExecuteBodyElement->
addNew(HTML_DIV, 'execute-body-output' . ($bHighLightOld ? '-highlight' : ''),
{strContent => $strHighLightOutput});
undef($strHighLightOutput);
}
if ($self->{bExe} && defined($strHighLight) && !$bHighLightFound)
{
confess &log(ERROR, "unable to find a match for highlight: ${strHighLight}");
}
$bFirst = true;
}
}
$bFirst = false;
}
}
# Add code block
elsif ($oChild->nameGet() eq 'code-block')
{
$oSectionBodyElement->
addNew(HTML_DIV, 'code-block',
{strContent => $oChild->valueGet()});
}
# Add descriptive text
elsif ($oChild->nameGet() eq 'p')
{
$oSectionBodyElement->
addNew(HTML_DIV, 'section-body-text',
{strContent => $oRender->processText($oChild->textGet())});
}
# Add option descriptive text
elsif ($oChild->nameGet() eq 'option-description')
{
my $strOption = $oChild->paramGet("key");
my $oDescription = ${$self->{oReference}->{oConfigHash}}{&CONFIG_HELP_OPTION}{$strOption}{&CONFIG_HELP_DESCRIPTION};
if (!defined($oDescription))
{
confess &log(ERROR, "unable to find ${strOption} option in sections - try adding command?");
}
$oSectionBodyElement->
addNew(HTML_DIV, 'section-body-text',
{strContent => $oRender->processText($oDescription)});
}
# Add/remove backrest config options
elsif ($oChild->nameGet() eq 'backrest-config')
{
$oSectionBodyElement->add($self->backrestConfigProcess($oChild, $iDepth));
}
# Add/remove postgres config options
elsif ($oChild->nameGet() eq 'postgres-config')
{
$oSectionBodyElement->add($self->postgresConfigProcess($oChild, $iDepth));
}
# Add a subsection
elsif ($oChild->nameGet() eq 'section')
{
my ($oChildSectionElement, $oChildSectionTocElement) =
$self->sectionProcess($oChild, $strAnchor, $iDepth + 1);
$oSectionBodyElement->add($oChildSectionElement);
$oSectionTocElement->add($oChildSectionTocElement);
}
# Skip children that have already been processed and error on others
elsif ($oChild->nameGet() ne 'title')
{
confess &log(ASSERT, 'unable to find child type ' . $oChild->nameGet());
}
}
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'oSectionElement', value => $oSectionElement, trace => true},
{name => 'oSectionTocElement', value => $oSectionTocElement, trace => true}
);
}
####################################################################################################################################
# backrestConfigProcess
####################################################################################################################################
sub backrestConfigProcess
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$oConfig,
$iDepth
) =
logDebugParam
(
OP_DOC_HTML_PAGE_BACKREST_CONFIG_PROCESS, \@_,
{name => 'oConfig'},
{name => 'iDepth'}
);
# Get filename
my $strFile = $self->{oSite}->variableReplace($oConfig->paramGet('file'));
&log(INFO, (' ' x $iDepth) . 'process backrest config: ' . $strFile);
foreach my $oOption ($oConfig->nodeList('backrest-config-option'))
{
my $strSection = $oOption->fieldGet('backrest-config-option-section');
my $strKey = $oOption->fieldGet('backrest-config-option-key');
my $strValue = $self->{oSite}->variableReplace(trim($oOption->fieldGet('backrest-config-option-value'), false));
if (!defined($strValue))
{
delete(${$self->{config}}{$strFile}{$strSection}{$strKey});
if (keys(${$self->{config}}{$strFile}{$strSection}) == 0)
{
delete(${$self->{config}}{$strFile}{$strSection});
}
&log(INFO, (' ' x ($iDepth + 1)) . "reset ${strSection}->${strKey}");
}
else
{
${$self->{config}}{$strFile}{$strSection}{$strKey} = $strValue;
&log(INFO, (' ' x ($iDepth + 1)) . "set ${strSection}->${strKey} = ${strValue}");
}
}
# Save the ini file
executeTest("sudo chmod 777 $strFile", {bSuppressError => true});
iniSave($strFile, $self->{config}{$strFile}, true);
# Generate config element
my $oConfigElement = new BackRestDoc::Html::DocHtmlElement(HTML_DIV, "config");
$oConfigElement->
addNew(HTML_DIV, "config-title",
{strContent => $self->{oRender}->processText($oConfig->nodeGet('title')->textGet()) . ':'});
my $oConfigBodyElement = $oConfigElement->addNew(HTML_DIV, "config-body");
$oConfigBodyElement->
addNew(HTML_DIV, "config-body-title",
{strContent => "${strFile}:"});
$oConfigBodyElement->
addNew(HTML_DIV, "config-body-output",
{strContent => fileStringRead($strFile)});
executeTest("sudo chown postgres:postgres $strFile");
executeTest("sudo chmod 640 $strFile");
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'oConfigElement', value => $oConfigElement, trace => true}
);
}
####################################################################################################################################
# postgresConfigProcess
####################################################################################################################################
sub postgresConfigProcess
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$oConfig,
$iDepth
) =
logDebugParam
(
OP_DOC_HTML_PAGE_POSTGRES_CONFIG_PROCESS, \@_,
{name => 'oConfig'},
{name => 'iDepth'}
);
# Get filename
my $strFile = $self->{oSite}->variableReplace($oConfig->paramGet('file'));
if (!defined(${$self->{'pg-config'}}{$strFile}{base}))
{
${$self->{'pg-config'}}{$strFile}{base} = fileStringRead($strFile);
}
my $oConfigHash = $self->{'pg-config'}{$strFile};
my $oConfigHashNew;
if (!defined($$oConfigHash{old}))
{
$oConfigHashNew = {};
$$oConfigHash{old} = {}
}
else
{
$oConfigHashNew = dclone($$oConfigHash{old});
}
&log(INFO, (' ' x $iDepth) . 'process postgres config: ' . $strFile);
foreach my $oOption ($oConfig->nodeList('postgres-config-option'))
{
my $strKey = $oOption->paramGet('key');
my $strValue = $self->{oSite}->variableReplace(trim($oOption->valueGet()));
if ($strValue eq '')
{
delete($$oConfigHashNew{$strKey});
&log(INFO, (' ' x ($iDepth + 1)) . "reset ${strKey}");
}
else
{
$$oConfigHashNew{$strKey} = $strValue;
&log(INFO, (' ' x ($iDepth + 1)) . "set ${strKey} = ${strValue}");
}
}
# Generate config text
my $strConfig;
foreach my $strKey (sort(keys(%$oConfigHashNew)))
{
if (defined($strConfig))
{
$strConfig .= "\n";
}
$strConfig .= "${strKey} = $$oConfigHashNew{$strKey}";
}
# Save the conf file
executeTest("sudo chown vagrant $strFile");
fileStringWrite($strFile, $$oConfigHash{base} .
(defined($strConfig) ? "\n# pgBackRest Configuration\n${strConfig}" : ''));
executeTest("sudo chown postgres $strFile");
# Generate config element
my $oConfigElement = new BackRestDoc::Html::DocHtmlElement(HTML_DIV, "config");
$oConfigElement->
addNew(HTML_DIV, "config-title",
{strContent => $self->{oRender}->processText($oConfig->nodeGet('title')->textGet()) . ':'});
my $oConfigBodyElement = $oConfigElement->addNew(HTML_DIV, "config-body");
$oConfigBodyElement->
addNew(HTML_DIV, "config-body-title",
{strContent => "append to ${strFile}:"});
$oConfigBodyElement->
addNew(HTML_DIV, "config-body-output",
{strContent => defined($strConfig) ? $strConfig : '<No PgBackRest Settings>'});
$$oConfigHash{old} = $oConfigHashNew;
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'oConfigElement', value => $oConfigElement, trace => true}
);
}
1;

246
doc/lib/BackRestDoc/Html/DocHtmlSite.pm Normal file
View File

@ -0,0 +1,246 @@
####################################################################################################################################
# DOC HTML SITE MODULE
####################################################################################################################################
package BackRestDoc::Html::DocHtmlSite;
use strict;
use warnings FATAL => qw(all);
use Carp qw(confess);
use Data::Dumper;
use Exporter qw(import);
our @EXPORT = qw();
use File::Basename qw(dirname);
use File::Copy;
use POSIX qw(strftime);
use Storable qw(dclone);
use lib dirname($0) . '/../lib';
use BackRest::Common::Log;
use BackRest::Common::String;
use BackRest::FileCommon;
use BackRest::Version;
use lib dirname($0) . '/../test/lib';
use BackRestTest::Common::ExecuteTest;
use BackRestDoc::Common::DocConfig;
use BackRestDoc::Html::DocHtmlPage;
####################################################################################################################################
# Operation constants
####################################################################################################################################
use constant OP_DOC_HTML_SITE => 'DocHtmlSite';
use constant OP_DOC_HTML_SITE_NEW => OP_DOC_HTML_SITE . '->new';
use constant OP_DOC_HTML_SITE_PROCESS => OP_DOC_HTML_SITE . '->process';
####################################################################################################################################
# CONSTRUCTOR
####################################################################################################################################
sub new
{
my $class = shift; # Class name
# Create the class hash
my $self = {};
bless $self, $class;
$self->{strClass} = $class;
# Assign function parameters, defaults, and log debug info
(
my $strOperation,
$self->{oRender},
$self->{oReference},
$self->{strXmlPath},
$self->{strHtmlPath},
$self->{strCssFile},
$self->{strHtmlRoot},
$self->{bExe}
) =
logDebugParam
(
OP_DOC_HTML_SITE_NEW, \@_,
{name => 'oRender'},
{name => 'oReference'},
{name => 'strXmlPath'},
{name => 'strHtmlPath'},
{name => 'strCssFile'},
{name => 'strHtmlRoot'},
{name => 'bExe'}
);
# Remove the current html path if it exists
if (-e $self->{strHtmlPath})
{
executeTest("rm -rf $self->{strHtmlPath}/*");
}
# Else create the html path
else
{
mkdir($self->{strHtmlPath})
or confess &log(ERROR, "unable to create path $self->{strHtmlPath}");
}
# Create the footer
$self->{strFooter} = 'Copyright © 2015' . (strftime('%Y', localtime) ne '2015' ? '-' . strftime('%Y', localtime) : '') .
', The PostgreSQL Global Development Group, <a href="{[github-url-license]}">MIT License</a>. Updated ' .
strftime('%B ', localtime) . trim(strftime('%e,', localtime)) . strftime(' %Y.', localtime);
# Insert pages into the site hash
$self->{oSite} =
{
'common' =>
{
'oRender' => $self->{oRender},
'oReference' => $self->{oReference},
'strFooter' => $self->{strFooter}
},
'page' =>
{
'index' =>
{
strMenuTitle => 'Home',
oDoc => new BackRestDoc::Common::Doc("$self->{strXmlPath}/index.xml")
},
'command' =>
{
strMenuTitle => 'Commands',
oDoc => $self->{oReference}->helpCommandDocGet()
},
'configuration' =>
{
strMenuTitle => 'Configuration',
oDoc => $self->{oReference}->helpConfigDocGet()
# }
},
'user-guide' =>
{
strMenuTitle => 'User Guide',
oDoc => new BackRestDoc::Common::Doc("$self->{strXmlPath}/user-guide.xml")
}
}
};
# Create common variables
${$self->{var}}{version} = $VERSION;
${$self->{var}}{'backrest-exe'} = $self->{oRender}->{strExeName};
${$self->{var}}{'postgres'} = 'PostgreSQL';
${$self->{var}}{'backrest-url-root'} = $self->{strHtmlRoot};
${$self->{var}}{'dash'} = '-';
# Read variables from pages
foreach my $strPageId (sort(keys(${$self->{oSite}}{page})))
{
my $oPage = ${$self->{oSite}}{page}{$strPageId};
if (defined($$oPage{oDoc}->nodeGet('variable-list', false)))
{
foreach my $oVariable ($$oPage{oDoc}->nodeGet('variable-list')->nodeList('variable'))
{
my $strName = $oVariable->fieldGet('variable-name');
my $strValue = $oVariable->fieldGet('variable-value');
${$self->{var}}{$strName} = $self->variableReplace($strValue);
}
}
}
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'self', value => $self}
);
}
####################################################################################################################################
# variableReplace
#
# Replace variables in the string.
####################################################################################################################################
sub variableReplace
{
my $self = shift;
my $strBuffer = shift;
if (!defined($strBuffer))
{
return undef;
}
foreach my $strName (sort(keys(%{$self->{var}})))
{
my $strValue = $self->{var}{$strName};
$strBuffer =~ s/\{\[$strName\]\}/$strValue/g;
}
return $strBuffer;
}
####################################################################################################################################
# variableSet
#
# Set a variable to be replaced later.
####################################################################################################################################
sub variableSet
{
my $self = shift;
my $strKey = shift;
my $strValue = shift;
${$self->{var}}{$strKey} = $strValue;
}
####################################################################################################################################
# variableGet
#
# Get the current value of a variable.
####################################################################################################################################
sub variableGet
{
my $self = shift;
my $strKey = shift;
return ${$self->{var}}{$strKey};
}
####################################################################################################################################
# process
#
# Generate the site html
####################################################################################################################################
sub process
{
my $self = shift;
# Assign function parameters, defaults, and log debug info
my $strOperation = logDebugParam(OP_DOC_HTML_SITE_PROCESS);
# Copy the css file
my $strCssFileDestination = "$self->{strHtmlPath}/default.css";
copy($self->{strCssFile}, $strCssFileDestination)
or confess &log(ERROR, "unable to copy $self->{strCssFile} to ${strCssFileDestination}");
# Render pages
my $oSite = $self->{oSite};
foreach my $strPageId (sort(keys($$oSite{'page'})))
{
# Save the html page
fileStringWrite("$self->{strHtmlPath}/${strPageId}.html",
$self->variableReplace((new BackRestDoc::Html::DocHtmlPage($self, $strPageId, $self->{bExe}))->process()),
false);
}
# Return from function and log return values if any
logDebugReturn($strOperation);
}
1;

397
doc/xml/change-log.xml Normal file
View File

@ -0,0 +1,397 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE doc SYSTEM "doc.dtd">
<doc title="Change Log">
<intro>
<text></text>
</intro>
<changelog>
<changelog-release date="XXXX-XX-XX" version="0.87" title="VIENNA MILESTONE - UNDER DEVELOPMENT">
<release-feature-bullet-list>
<release-feature>
<text>Added a new user guide that covers <backrest/> basics and some advanced topics including PITR. Much more to come, but it's a start.</text>
</release-feature>
<release-feature>
<text>The website, markdown, and command-line help are now all generated from the same XML source.</text>
</release-feature>
<release-feature>
<text>The <file>backup_label.old</file> and <file>recovery.done</file> files are now excluded from backups.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-10-08" version="0.85" title="Start/Stop Commands and Minor Bug Fixes">
<release-feature-bullet-list>
<release-feature>
<text>Added new feature to allow all <backrest/> operations to be stopped or started using the <cmd>stop</cmd> and <cmd>start</cmd> commands. This prevents any <backrest/> processes from running on a system where <postgres/> is shutdown or the system needs to be quiesced for some reason.</text>
</release-feature>
<release-feature>
<text>Removed dependency on <code>IO::String</code> module.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where an error could be returned after a backup or restore completely successfully.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed. This scenario was likely if the backup process got terminated during the copy phase.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5 beta1. This may break when the control version or WAL magic changes in future versions but will be updated in each <backrest/> release to keep pace. All regression tests pass except for <setting>--target-resume</setting> tests (this functionality has changed in 9.5) and there is no testing yet for <file>.partial</file> WAL segments.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-09-14" version="0.82" title="Refactoring, Command-line Help, and Minor Bug Fixes">
<release-feature-bullet-list>
<release-feature>
<text>Fixed an issue where resumed compressed backups were not preserving existing files.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a cold backup using <setting>--no-start-stop</setting> could be started on a running <postgres/> cluster without <setting>--force</setting> specified.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a thread could be started even when none were requested.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where the <backrest/> version number was not being updated in <file>backup.info</file> and <file>archive.info</file> after an upgrade/downgrade.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where the <cmd>info</cmd> command was throwing an exception when the repository contained no stanzas. <i>Reported by Stephen Frost</i>.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where the <postgres/> <code>pg_stop_backup()</code> NOTICEs were being output to stderr. <i>Reported by Stephen Frost</i>.</text>
</release-feature>
<release-feature>
<text>Renamed <setting>recovery-setting</setting> option and section to <setting>recovery-option</setting> to be more consistent with <backrest/> naming conventions.</text>
</release-feature>
<release-feature>
<text>Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.</text>
</release-feature>
<release-feature>
<text>Code cleanup and refactoring to standardize on patterns that have evolved over time.</text>
</release-feature>
<release-feature>
<text>Added dynamic module loading to speed up commands, especially asynchronous archiving.</text>
</release-feature>
<release-feature>
<text>Expiration tests are now synthetic rather than based on actual backups. This will allow development of more advanced expiration features.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5 alpha2. This may break when the control version or WAL magic changes in future versions but will be updated in each <backrest/> release to keep pace. All regression tests pass except for <setting>--target-resume</setting> tests (this functionality has changed in 9.5) and there is no testing yet for <file>.partial</file> WAL segments.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-08-09" version="0.80" title="DBI Support, Stability, and Convenience Features">
<release-feature-bullet-list>
<release-feature>
<text>Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the <cmd>info</cmd> command. Only <id>text</id> output was affected -- <id>json</id> output reported the correct epoch values. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.</text>
</release-feature>
<release-feature>
<text>Now using Perl <code>DBI</code> and <code>DBD::Pg</code> for connections to <postgres/> rather than <cmd>psql</cmd>. The <setting>cmd-psql</setting> and <setting>cmd-psql-option</setting> settings have been removed and replaced with <setting>db-port</setting> and <setting>db-socket-path</setting>. Follow the instructions in [Installation](USERGUIDE.md#installation) to install <code>DBD::Pg</code> on your operating system.</text>
</release-feature>
<release-feature>
<text>Add [stop-auto](USERGUIDE.md#stop-auto-key) option to allow failed backups to automatically be stopped when a new backup starts.</text>
</release-feature>
<release-feature>
<text>Add [db-timeout](USERGUIDE.md#db-timeout-key) option to limit the amount of time <backrest/> will wait for pg_start_backup() and pg_stop_backup() to return.</text>
</release-feature>
<release-feature>
<text>Remove <file>pg_control</file> file at the beginning of the restore and copy it back at the very end. This prevents the possibility that a partial restore can be started by <postgres/>.</text>
</release-feature>
<release-feature>
<text>The repository is now created and updated with consistent directory and file modes. By default <id>umask</id> is set to <id>0000</id> but this can be disabled with the <setting>neutral-umask</setting> setting. <i>Reported by Cynthia Shang</i></text>
</release-feature>
<release-feature>
<text>Added checks to be sure the <setting>db-path</setting> setting is consistent with <setting>db-port</setting> by comparing the <setting>data_directory</setting> as reported by the cluster against the <setting>db-path</setting> setting and the version as reported by the cluster against the value read from <file>pg_control</file>. The <setting>db-socket-path</setting> setting is checked to be sure it is an absolute path.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5 alpha1. This may break when the control version or WAL magic changes in future versions but will be updated in each <backrest/> release to keep pace. All regression tests pass except for <setting>--target-resume</setting> tests (this functionality has changed in 9.5) and there is no testing yet for <file>.partial</file> WAL segments.</text>
</release-feature>
<release-feature>
<text>Major refactoring of the protocol layer to support future development.</text>
</release-feature>
<release-feature>
<text>Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.</text>
</release-feature>
<release-feature>
<text>Split most of <file>README.md</file> out into <file>USERGUIDE.md</file> and <file>CHANGELOG.md</file> because it was becoming unwieldy. Changed most references to "database" in the user guide to "database cluster" for clarity.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-07-13" version="0.78" title="Remove CPAN Dependencies, Stability Improvements">
<release-feature-bullet-list>
<release-feature>
<text>Removed dependency on CPAN packages for multi-threaded operation. While it might not be a bad idea to update the <code>threads</code> and <code>Thread::Queue</code> packages, it is no longer necessary.</text>
</release-feature>
<release-feature>
<text>Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.</text>
</release-feature>
<release-feature>
<text>Modified wait backoff to use a Fibonacci rather than geometric sequence. This will make wait time grow less aggressively while still giving reasonable values.</text>
</release-feature>
<release-feature>
<text>More options for regression tests and improved code to run in a variety of environments.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-06-30" version="0.77" title="CentOS/RHEL 6 Support and Protocol Improvements">
<release-feature-bullet-list>
<release-feature>
<text>Removed <file>pg_backrest_remote</file> and added the functionality to <file>pg_backrest</file> as the <cmd>remote</cmd> command.</text>
</release-feature>
<release-feature>
<text>Added file and directory syncs to the <code>File</code> object for additional safety during backup/restore and archiving. <i>Suggested by Andres Freund</i>.</text>
</release-feature>
<release-feature>
<text>Support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6. <i>Reported by Eric Radman.</i></text>
</release-feature>
<release-feature>
<text>Improved error message when backup is run without <setting>archive_command</setting> set and without <setting>--no-archive-check</setting> specified. <i>Reported by Eric Radman</i>.</text>
</release-feature>
<release-feature>
<text>Moved version number out of the <file>VERSION</file> file to <file>Version.pm</file> to better support packaging. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Replaced <code>IPC::System::Simple</code> and <code>Net::OpenSSH</code> with <code>IPC::Open3</code> to eliminate CPAN dependency for multiple operating systems.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-06-14" version="0.75" title="New Repository Format, Info Command and Experimental 9.5 Support">
<release-feature-bullet-list>
<release-feature>
<text><b>IMPORTANT NOTE</b>: This flag day release breaks compatibility with older versions of <backrest/>. The manifest format, on-disk structure, and the binary names have all changed. You must create a new repository to hold backups for this version of <backrest/> and keep your older repository for a time in case you need to do a restore. The <file>pg_backrest.conf</file> file has not changed but you'll need to change any references to <file>pg_backrest.pl</file> in cron (or elsewhere) to <file>pg_backrest</file> (without the <file>.pl</file> extension).</text>
</release-feature>
<release-feature>
<text>Add <cmd>info</cmd> command.</text>
</release-feature>
<release-feature>
<text>More efficient file ordering for <cmd>backup</cmd>. Files are copied in descending size order so a single thread does not end up copying a large file at the end. This had already been implemented for <cmd>restore</cmd>.</text>
</release-feature>
<release-feature>
<text>Logging now uses unbuffered output. This should make log files that are being written by multiple threads less chaotic. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5. This may break when the control version or WAL magic changes but will be updated in each release.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-06-01" version="0.70" title="Stability Improvements for Archiving, Improved Logging and Help">
<release-feature-bullet-list>
<release-feature>
<text>Fixed an issue where <cmd>archive-copy</cmd> would fail on an incr/diff backup when <setting>hardlink=n</setting>. In this case the <path>pg_xlog</path> path does not already exist and must be created. <i>Reported by Michael Renner</i></text>
</release-feature>
<release-feature>
<text>Allow duplicate WAL segments to be archived when the checksum matches. This is necessary for some recovery scenarios.</text>
</release-feature>
<release-feature>
<text>Allow comments/disabling in <file>pg_backrest.conf</file> using the <id>#</id> character. Only <id>#</id> characters in the forst character of the line are honored. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Better logging before <id>pg_start_backup()</id> to make it clear when the backup is waiting on a checkpoint. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Various command behavior, help and logging fixes. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Fixed an issue in async archiving where <cmd>archive-push</cmd> was not properly returning 0 when <setting>archive-max-mb</setting> was reached and moved the async check after transfer to avoid having to remove the stop file twice. Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Fixed a locking issue that could allow multiple operations of the same type against a single stanza. This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Replaced <code>JSON</code> module with <code>JSON::PP</code> which ships with core Perl.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-05-11" version="0.65" title="Improved Resume and Restore Logging, Compact Restores">
<release-feature-bullet-list>
<release-feature>
<text>Better resume support. Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses. More unit tests to verify each resume case.</text>
</release-feature>
<release-feature>
<text>Resume is now optional. Use the <setting>resume</setting> setting or <param>--no-resume</param> from the command line to disable.</text>
</release-feature>
<release-feature>
<text>More info messages during restore. Previously, most of the restore messages were debug level so not a lot was output in the log.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.</text>
</release-feature>
<release-feature>
<text>Added <setting>tablespace</setting> setting to allow tablespaces to be restored into the <path>pg_tblspc</path> path. This produces compact restores that are convenient for development, staging, etc. Currently these restores cannot be backed up as <backrest/> expects only links in the <path>pg_tblspc</path> path.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-04-21" version="0.61" title="Bug Fix for Uncompressed Remote Destination">
<release-feature-bullet-list>
<release-feature>
<text>Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination. The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-04-19" version="0.60" title="Better Version Support and WAL Improvements">
<release-feature-bullet-list>
<release-feature>
<text>Pushing duplicate WAL now generates an error. This worked before only if checksums were disabled.</text>
</release-feature>
<release-feature>
<text>Database System IDs are used to make sure that all WAL in an archive matches up. This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.</text>
</release-feature>
<release-feature>
<text>Regression tests working back to <postgres/> 8.3.</text>
</release-feature>
<release-feature>
<text>Improved threading model by starting threads early and terminating them late.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-03-25" version="0.50" title="Restore and Much More">
<release-feature-bullet-list>
<release-feature>
<text>Added restore functionality.</text>
</release-feature>
<release-feature>
<text>All options can now be set on the command-line making <file>pg_backrest.conf</file> optional.</text>
</release-feature>
<release-feature>
<text>De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.</text>
</release-feature>
<release-feature>
<text>Added option <param>--no-start-stop</param> to allow backups when Postgres is shut down. If <file>postmaster.pid</file> is present then <param>--force</param> is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.</text>
</release-feature>
<release-feature>
<text>Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplified a number of things. Issue #28 has been created for checksum deltas.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.</text>
</release-feature>
<release-feature>
<text>Removed dependency on <code>Moose</code>. It wasn't being used extensively and makes for longer startup times.</text>
</release-feature>
<release-feature>
<text>Checksum for <file>backup.manifest</file> to detect a corrupted/modified manifest.</text>
</release-feature>
<release-feature>
<text>Link <path>latest</path> always points to the last backup. This has been added for convenience and to make restores simpler.</text>
</release-feature>
<release-feature>
<text>More comprehensive unit tests in all areas.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-10-05" version="0.30" title="Core Restructuring and Unit Tests">
<release-feature-bullet-list>
<release-feature>
<text>Complete rewrite of <code>BackRest::File</code> module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.</text>
</release-feature>
<release-feature>
<text>Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.</text>
</release-feature>
<release-feature>
<text>Removed dependency on <code>Storable</code> and replaced with a custom ini file implementation.</text>
</release-feature>
<release-feature>
<text>Added much needed documentation</text>
</release-feature>
<release-feature>
<text>Numerous other changes that can only be identified with a diff.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-05-13" version="0.19" title="Improved Error Reporting/Handling">
<release-feature-bullet-list>
<release-feature>
<text>Working on improving error handling in the <code>File</code> object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).</text>
</release-feature>
<release-feature>
<text>Found and squashed a nasty bug where <code>file_copy()</code> was defaulted to ignore errors. There was also an issue in <code>file_exists()</code> that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-04-13" version="0.18" title="Return Soft Error When Archive Missing">
<release-feature-bullet-list>
<release-feature>
<text>The <cmd>archive-get</cmd> command returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets <postgres/> know that that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-04-03" version="0.17" title="Warn When Archive Directories Cannot Be Deleted">
<release-feature-bullet-list>
<release-feature>
<text>If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-04-01" version="0.16" title="RequestTTY=yes for SSH Sessions">
<release-feature-bullet-list>
<release-feature>
<text>Added <setting>RequestTTY=yes</setting> to ssh sessions. Hoping this will prevent random lockups.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-03-29" version="0.15" title="Added archive-get">
<release-feature-bullet-list>
<release-feature>
<text>Added <cmd>archive-get</cmd> functionality to aid in restores.</text>
</release-feature>
<release-feature>
<text>Added option to force a checkpoint when starting the backup, <setting>start-fast=y</setting>.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-03-26" version="0.11" title="Minor Fixes">
<release-feature-bullet-list>
<release-feature>
<text>Removed <setting>master_stderr_discard</setting> option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code.</text>
</release-feature>
<release-feature>
<text>Changed lock file conflicts on <cmd>backup</cmd> and <cmd>expire</cmd> commands to <id>ERROR</id>. They were set to <id>DEBUG</id> due to a copy-and-paste from the archive locks.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-03-05" version="0.10" title="Backup and Archiving are Functional">
<release-feature-bullet-list>
<release-feature>
<text>No restore functionality, but the backup directories are consistent <postgres/> data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.</text>
</release-feature>
<release-feature>
<text>Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.</text>
</release-feature>
<release-feature>
<text>Backups are multi-threaded, but the <code>Net::OpenSSH</code> library does not appear to be 100% thread-safe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.</text>
</release-feature>
<release-feature>
<text>Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.</text>
</release-feature>
<release-feature>
<text>The <file>backup.manifest</file> is being written as <code>Storable</code> because <code>Config::IniFile</code> does not seem to handle large files well. Would definitely like to save these as human-readable text.</text>
</release-feature>
<release-feature>
<text>Absolutely no documentation (outside the code). Well, excepting these release notes.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
</changelog>
</doc>

391
doc/xml/changelog.xml
View File

@ -1,391 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE doc SYSTEM "doc.dtd">
<doc title="Change Log">
<intro>
<text></text>
</intro>
<changelog>
<changelog-release date="XXXX-XX-XX" version="0.87" title="VIENNA MILESTONE - UNDER DEVELOPMENT">
<release-feature-bullet-list>
<release-feature>
<text></text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-10-08" version="0.85" title="Start/Stop Commands and Minor Bug Fixes">
<release-feature-bullet-list>
<release-feature>
<text>Added new feature to allow all <backrest/> operations to be stopped or started using the <cmd>stop</cmd> and <cmd>start</cmd> commands. This prevents any <backrest/> processes from running on a system where <postgres/> is shutdown or the system needs to be quiesced for some reason.</text>
</release-feature>
<release-feature>
<text>Removed dependency on <code>IO::String</code> module.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where an error could be returned after a backup or restore completely successfully.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed. This scenario was likely if the backup process got terminated during the copy phase.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5 beta1. This may break when the control version or WAL magic changes in future versions but will be updated in each <backrest/> release to keep pace. All regression tests pass except for <setting>--target-resume</setting> tests (this functionality has changed in 9.5) and there is no testing yet for <file>.partial</file> WAL segments.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-09-14" version="0.82" title="Refactoring, Command-line Help, and Minor Bug Fixes">
<release-feature-bullet-list>
<release-feature>
<text>Fixed an issue where resumed compressed backups were not preserving existing files.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a cold backup using <setting>--no-start-stop</setting> could be started on a running <postgres/> cluster without <setting>--force</setting> specified.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a thread could be started even when none were requested.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where the <backrest/> version number was not being updated in <file>backup.info</file> and <file>archive.info</file> after an upgrade/downgrade.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where the <cmd>info</cmd> command was throwing an exception when the repository contained no stanzas. <i>Reported by Stephen Frost</i>.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where the <postgres/> <code>pg_stop_backup()</code> NOTICEs were being output to stderr. <i>Reported by Stephen Frost</i>.</text>
</release-feature>
<release-feature>
<text>Renamed <setting>recovery-setting</setting> option and section to <setting>recovery-option</setting> to be more consistent with <backrest/> naming conventions.</text>
</release-feature>
<release-feature>
<text>Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.</text>
</release-feature>
<release-feature>
<text>Code cleanup and refactoring to standardize on patterns that have evolved over time.</text>
</release-feature>
<release-feature>
<text>Added dynamic module loading to speed up commands, especially asynchronous archiving.</text>
</release-feature>
<release-feature>
<text>Expiration tests are now synthetic rather than based on actual backups. This will allow development of more advanced expiration features.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5 alpha2. This may break when the control version or WAL magic changes in future versions but will be updated in each <backrest/> release to keep pace. All regression tests pass except for <setting>--target-resume</setting> tests (this functionality has changed in 9.5) and there is no testing yet for <file>.partial</file> WAL segments.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-08-09" version="0.80" title="DBI Support, Stability, and Convenience Features">
<release-feature-bullet-list>
<release-feature>
<text>Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the <cmd>info</cmd> command. Only <id>text</id> output was affected -- <id>json</id> output reported the correct epoch values. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.</text>
</release-feature>
<release-feature>
<text>Now using Perl <code>DBI</code> and <code>DBD::Pg</code> for connections to <postgres/> rather than <cmd>psql</cmd>. The <setting>cmd-psql</setting> and <setting>cmd-psql-option</setting> settings have been removed and replaced with <setting>db-port</setting> and <setting>db-socket-path</setting>. Follow the instructions in [Installation](USERGUIDE.md#installation) to install <code>DBD::Pg</code> on your operating system.</text>
</release-feature>
<release-feature>
<text>Add [stop-auto](USERGUIDE.md#stop-auto-key) option to allow failed backups to automatically be stopped when a new backup starts.</text>
</release-feature>
<release-feature>
<text>Add [db-timeout](USERGUIDE.md#db-timeout-key) option to limit the amount of time <backrest/> will wait for pg_start_backup() and pg_stop_backup() to return.</text>
</release-feature>
<release-feature>
<text>Remove <file>pg_control</file> file at the beginning of the restore and copy it back at the very end. This prevents the possibility that a partial restore can be started by <postgres/>.</text>
</release-feature>
<release-feature>
<text>The repository is now created and updated with consistent directory and file modes. By default <id>umask</id> is set to <id>0000</id> but this can be disabled with the <setting>neutral-umask</setting> setting.</text>
</release-feature>
<release-feature>
<text>Added checks to be sure the <setting>db-path</setting> setting is consistent with <setting>db-port</setting> by comparing the <setting>data_directory</setting> as reported by the cluster against the <setting>db-path</setting> setting and the version as reported by the cluster against the value read from <file>pg_control</file>. The <setting>db-socket-path</setting> setting is checked to be sure it is an absolute path.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5 alpha1. This may break when the control version or WAL magic changes in future versions but will be updated in each <backrest/> release to keep pace. All regression tests pass except for <setting>--target-resume</setting> tests (this functionality has changed in 9.5) and there is no testing yet for <file>.partial</file> WAL segments.</text>
</release-feature>
<release-feature>
<text>Major refactoring of the protocol layer to support future development.</text>
</release-feature>
<release-feature>
<text>Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.</text>
</release-feature>
<release-feature>
<text>Split most of <file>README.md</file> out into <file>USERGUIDE.md</file> and <file>CHANGELOG.md</file> because it was becoming unwieldy. Changed most references to "database" in the user guide to "database cluster" for clarity.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-07-13" version="0.78" title="Remove CPAN Dependencies, Stability Improvements">
<release-feature-bullet-list>
<release-feature>
<text>Removed dependency on CPAN packages for multi-threaded operation. While it might not be a bad idea to update the <code>threads</code> and <code>Thread::Queue</code> packages, it is no longer necessary.</text>
</release-feature>
<release-feature>
<text>Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.</text>
</release-feature>
<release-feature>
<text>Modified wait backoff to use a Fibonacci rather than geometric sequence. This will make wait time grow less aggressively while still giving reasonable values.</text>
</release-feature>
<release-feature>
<text>More options for regression tests and improved code to run in a variety of environments.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-06-30" version="0.77" title="CentOS/RHEL 6 Support and Protocol Improvements">
<release-feature-bullet-list>
<release-feature>
<text>Removed <file>pg_backrest_remote</file> and added the functionality to <file>pg_backrest</file> as the <cmd>remote</cmd> command.</text>
</release-feature>
<release-feature>
<text>Added file and directory syncs to the <code>File</code> object for additional safety during backup/restore and archiving. <i>Suggested by Andres Freund</i>.</text>
</release-feature>
<release-feature>
<text>Support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6. <i>Reported by Eric Radman.</i></text>
</release-feature>
<release-feature>
<text>Improved error message when backup is run without <setting>archive_command</setting> set and without <setting>--no-archive-check</setting> specified. <i>Reported by Eric Radman</i>.</text>
</release-feature>
<release-feature>
<text>Moved version number out of the <file>VERSION</file> file to <file>Version.pm</file> to better support packaging. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Replaced <code>IPC::System::Simple</code> and <code>Net::OpenSSH</code> with <code>IPC::Open3</code> to eliminate CPAN dependency for multiple operating systems.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-06-14" version="0.75" title="New Repository Format, Info Command and Experimental 9.5 Support">
<release-feature-bullet-list>
<release-feature>
<text><b>IMPORTANT NOTE</b>: This flag day release breaks compatibility with older versions of <backrest/>. The manifest format, on-disk structure, and the binary names have all changed. You must create a new repository to hold backups for this version of <backrest/> and keep your older repository for a time in case you need to do a restore. The <file>pg_backrest.conf</file> file has not changed but you'll need to change any references to <file>pg_backrest.pl</file> in cron (or elsewhere) to <file>pg_backrest</file> (without the <file>.pl</file> extension).</text>
</release-feature>
<release-feature>
<text>Add <cmd>info</cmd> command.</text>
</release-feature>
<release-feature>
<text>More efficient file ordering for <cmd>backup</cmd>. Files are copied in descending size order so a single thread does not end up copying a large file at the end. This had already been implemented for <cmd>restore</cmd>.</text>
</release-feature>
<release-feature>
<text>Logging now uses unbuffered output. This should make log files that are being written by multiple threads less chaotic. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Experimental support for <postgres/> 9.5. This may break when the control version or WAL magic changes but will be updated in each release.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-06-01" version="0.70" title="Stability Improvements for Archiving, Improved Logging and Help">
<release-feature-bullet-list>
<release-feature>
<text>Fixed an issue where <cmd>archive-copy</cmd> would fail on an incr/diff backup when <setting>hardlink=n</setting>. In this case the <path>pg_xlog</path> path does not already exist and must be created. <i>Reported by Michael Renner</i></text>
</release-feature>
<release-feature>
<text>Allow duplicate WAL segments to be archived when the checksum matches. This is necessary for some recovery scenarios.</text>
</release-feature>
<release-feature>
<text>Allow comments/disabling in <file>pg_backrest.conf</file> using the <id>#</id> character. Only <id>#</id> characters in the forst character of the line are honored. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Better logging before <id>pg_start_backup()</id> to make it clear when the backup is waiting on a checkpoint. <i>Suggested by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Various command behavior, help and logging fixes. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Fixed an issue in async archiving where <cmd>archive-push</cmd> was not properly returning 0 when <setting>archive-max-mb</setting> was reached and moved the async check after transfer to avoid having to remove the stop file twice. Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Fixed a locking issue that could allow multiple operations of the same type against a single stanza. This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. <i>Reported by Michael Renner</i>.</text>
</release-feature>
<release-feature>
<text>Replaced <code>JSON</code> module with <code>JSON::PP</code> which ships with core Perl.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-05-11" version="0.65" title="Improved Resume and Restore Logging, Compact Restores">
<release-feature-bullet-list>
<release-feature>
<text>Better resume support. Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses. More unit tests to verify each resume case.</text>
</release-feature>
<release-feature>
<text>Resume is now optional. Use the <setting>resume</setting> setting or <param>--no-resume</param> from the command line to disable.</text>
</release-feature>
<release-feature>
<text>More info messages during restore. Previously, most of the restore messages were debug level so not a lot was output in the log.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.</text>
</release-feature>
<release-feature>
<text>Added <setting>tablespace</setting> setting to allow tablespaces to be restored into the <path>pg_tblspc</path> path. This produces compact restores that are convenient for development, staging, etc. Currently these restores cannot be backed up as <backrest/> expects only links in the <path>pg_tblspc</path> path.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-04-21" version="0.61" title="Bug Fix for Uncompressed Remote Destination">
<release-feature-bullet-list>
<release-feature>
<text>Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination. The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-04-19" version="0.60" title="Better Version Support and WAL Improvements">
<release-feature-bullet-list>
<release-feature>
<text>Pushing duplicate WAL now generates an error. This worked before only if checksums were disabled.</text>
</release-feature>
<release-feature>
<text>Database System IDs are used to make sure that all WAL in an archive matches up. This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.</text>
</release-feature>
<release-feature>
<text>Regression tests working back to <postgres/> 8.3.</text>
</release-feature>
<release-feature>
<text>Improved threading model by starting threads early and terminating them late.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2015-03-25" version="0.50" title="Restore and Much More">
<release-feature-bullet-list>
<release-feature>
<text>Added restore functionality.</text>
</release-feature>
<release-feature>
<text>All options can now be set on the command-line making <file>pg_backrest.conf</file> optional.</text>
</release-feature>
<release-feature>
<text>De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.</text>
</release-feature>
<release-feature>
<text>Added option <param>--no-start-stop</param> to allow backups when Postgres is shut down. If <file>postmaster.pid</file> is present then <param>--force</param> is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.</text>
</release-feature>
<release-feature>
<text>Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplified a number of things. Issue #28 has been created for checksum deltas.</text>
</release-feature>
<release-feature>
<text>Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.</text>
</release-feature>
<release-feature>
<text>Removed dependency on <code>Moose</code>. It wasn't being used extensively and makes for longer startup times.</text>
</release-feature>
<release-feature>
<text>Checksum for <file>backup.manifest</file> to detect a corrupted/modified manifest.</text>
</release-feature>
<release-feature>
<text>Link <path>latest</path> always points to the last backup. This has been added for convenience and to make restores simpler.</text>
</release-feature>
<release-feature>
<text>More comprehensive unit tests in all areas.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-10-05" version="0.30" title="Core Restructuring and Unit Tests">
<release-feature-bullet-list>
<release-feature>
<text>Complete rewrite of <code>BackRest::File</code> module to use a custom protocol for remote operations and Perl native GZIP and SHA operations. Compression is performed in threads rather than forked processes.</text>
</release-feature>
<release-feature>
<text>Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.</text>
</release-feature>
<release-feature>
<text>Removed dependency on <code>Storable</code> and replaced with a custom ini file implementation.</text>
</release-feature>
<release-feature>
<text>Added much needed documentation</text>
</release-feature>
<release-feature>
<text>Numerous other changes that can only be identified with a diff.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-05-13" version="0.19" title="Improved Error Reporting/Handling">
<release-feature-bullet-list>
<release-feature>
<text>Working on improving error handling in the <code>File</code> object. This is not complete, but works well enough to find a few errors that have been causing us problems (notably, find is occasionally failing building the archive async manifest when system is under load).</text>
</release-feature>
<release-feature>
<text>Found and squashed a nasty bug where <code>file_copy()</code> was defaulted to ignore errors. There was also an issue in <code>file_exists()</code> that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-04-13" version="0.18" title="Return Soft Error When Archive Missing">
<release-feature-bullet-list>
<release-feature>
<text>The <cmd>archive-get</cmd> command returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets <postgres/> know that that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-04-03" version="0.17" title="Warn When Archive Directories Cannot Be Deleted">
<release-feature-bullet-list>
<release-feature>
<text>If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-04-01" version="0.16" title="RequestTTY=yes for SSH Sessions">
<release-feature-bullet-list>
<release-feature>
<text>Added <setting>RequestTTY=yes</setting> to ssh sessions. Hoping this will prevent random lockups.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-03-29" version="0.15" title="Added archive-get">
<release-feature-bullet-list>
<release-feature>
<text>Added <cmd>archive-get</cmd> functionality to aid in restores.</text>
</release-feature>
<release-feature>
<text>Added option to force a checkpoint when starting the backup, <setting>start-fast=y</setting>.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-03-26" version="0.11" title="Minor Fixes">
<release-feature-bullet-list>
<release-feature>
<text>Removed <setting>master_stderr_discard</setting> option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code.</text>
</release-feature>
<release-feature>
<text>Changed lock file conflicts on <cmd>backup</cmd> and <cmd>expire</cmd> commands to <id>ERROR</id>. They were set to <id>DEBUG</id> due to a copy-and-paste from the archive locks.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
<changelog-release date="2014-03-05" version="0.10" title="Backup and Archiving are Functional">
<release-feature-bullet-list>
<release-feature>
<text>No restore functionality, but the backup directories are consistent <postgres/> data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.</text>
</release-feature>
<release-feature>
<text>Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.</text>
</release-feature>
<release-feature>
<text>Backups are multi-threaded, but the <code>Net::OpenSSH</code> library does not appear to be 100% thread-safe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.</text>
</release-feature>
<release-feature>
<text>Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.</text>
</release-feature>
<release-feature>
<text>The <file>backup.manifest</file> is being written as <code>Storable</code> because <code>Config::IniFile</code> does not seem to handle large files well. Would definitely like to save these as human-readable text.</text>
</release-feature>
<release-feature>
<text>Absolutely no documentation (outside the code). Well, excepting these release notes.</text>
</release-feature>
</release-feature-bullet-list>
</changelog-release>
</changelog>
</doc>

127
doc/xml/dtd/doc.dtd
View File

@ -1,61 +1,95 @@
<!ELEMENT doc ((intro, changelog)|(intro, start, contribute, support, recognition)|(intro, install, config, operation))>
<!ATTLIST doc title CDATA #REQUIRED>
<!ELEMENT doc ((intro, changelog)|(config, operation)|(variable-list?, cleanup?, section+))>
<!ATTLIST doc title CDATA "">
<!ATTLIST doc subtitle CDATA "">
<!ATTLIST doc toc CDATA "y">
<!ELEMENT intro (text)>
<!ELEMENT start (text)>
<!ATTLIST start title CDATA #REQUIRED>
<!ELEMENT install (text, install-system-list)>
<!ATTLIST install title CDATA #REQUIRED>
<!ELEMENT install-system-list (text?, install-system+)>
<!ELEMENT install-system (text)>
<!ATTLIST install-system title CDATA #REQUIRED>
<!ELEMENT operation (text?, operation-general, command-list)>
<!ATTLIST operation title CDATA #REQUIRED>
<!ATTLIST operation title CDATA #REQUIRED>
<!ELEMENT operation-general (text, option-list)>
<!ATTLIST operation-general title CDATA #REQUIRED>
<!ELEMENT operation-general (option-list)>
<!ATTLIST operation-general title CDATA #REQUIRED>
<!ELEMENT command-list (text?, command+)>
<!ATTLIST command-list title CDATA #REQUIRED>
<!ATTLIST command-list title CDATA #REQUIRED>
<!ELEMENT command (summary, text, option-list?, command-example-list)>
<!ATTLIST command id CDATA #REQUIRED>
<!ATTLIST command id CDATA #REQUIRED>
<!ATTLIST command name CDATA #REQUIRED>
<!ELEMENT command-example-list (text?, command-example+)>
<!ATTLIST command-example-list title CDATA "Examples">
<!ATTLIST command-example-list title CDATA "Examples">
<!ELEMENT command-example (text)>
<!ATTLIST command-example title CDATA "Example">
<!ATTLIST command-example title CDATA "Example">
<!ELEMENT option-list (option+)>
<!ELEMENT option (summary, text, example?)>
<!ATTLIST option id CDATA #REQUIRED>
<!ELEMENT option (summary, text, example)>
<!ATTLIST option id CDATA #REQUIRED>
<!ATTLIST option name CDATA #REQUIRED>
<!ELEMENT config (text, config-example-list, config-section-list)>
<!ATTLIST config title CDATA #REQUIRED>
<!ELEMENT config-example-list (text?, config-example+)>
<!ATTLIST config-example-list title CDATA #REQUIRED>
<!ELEMENT config (text, config-section-list)>
<!ATTLIST config title CDATA #REQUIRED>
<!ELEMENT config-example (text)>
<!ATTLIST config-example title CDATA #REQUIRED>
<!ATTLIST config-example title CDATA #REQUIRED>
<!ELEMENT config-section-list (text?, config-section+)>
<!ATTLIST config-section-list title CDATA #REQUIRED>
<!ATTLIST config-section-list title CDATA #REQUIRED>
<!ELEMENT config-section (text, config-key-list?)>
<!ATTLIST config-section id CDATA #REQUIRED>
<!ATTLIST config-section id CDATA #REQUIRED>
<!ATTLIST config-section name CDATA #REQUIRED>
<!ELEMENT config-key-list (config-key+)>
<!ELEMENT config-key (summary, text, default?, allow?, example)>
<!ATTLIST config-key id CDATA #REQUIRED>
<!ATTLIST config-key id CDATA #REQUIRED>
<!ATTLIST config-key name CDATA #REQUIRED>
<!ELEMENT execute-list (title, execute+)>
<!ELEMENT execute (exe-cmd, exe-user?, exe-var?, exe-retry?, exe-output?, exe-no-show?, exe-highlight?, exe-skip?,
exe-err-expect?, exe-err-suppress?, exe-err-suppress-stderr?)>
<!ELEMENT exe-cmd (#PCDATA)>
<!ELEMENT exe-user (#PCDATA)>
<!ELEMENT exe-highlight (#PCDATA)>
<!ELEMENT exe-var (#PCDATA)>
<!ELEMENT exe-output EMPTY>
<!ELEMENT exe-skip EMPTY>
<!ELEMENT exe-no-show EMPTY>
<!ELEMENT exe-retry EMPTY>
<!ELEMENT exe-err-expect (#PCDATA)>
<!ELEMENT exe-err-suppress EMPTY>
<!ELEMENT exe-err-suppress-stderr EMPTY>
<!ELEMENT cleanup (execute+)>
<!ELEMENT variable-list (variable+)>
<!ELEMENT variable (variable-name, variable-value)>
<!ELEMENT variable-name (#PCDATA)>
<!ELEMENT variable-value (#PCDATA)>
<!ELEMENT backrest-config (title, backrest-config-option+)>
<!ATTLIST backrest-config file CDATA #REQUIRED>
<!ELEMENT backrest-config-option (backrest-config-option-section?, backrest-config-option-key, backrest-config-option-value?)>
<!ELEMENT backrest-config-option-section (#PCDATA)>
<!ELEMENT backrest-config-option-key (#PCDATA)>
<!ELEMENT backrest-config-option-value (#PCDATA)>
<!ELEMENT postgres-config (title, postgres-config-option+)>
<!ATTLIST postgres-config file CDATA #REQUIRED>
<!ELEMENT postgres-config-option (#PCDATA)>
<!ATTLIST postgres-config-option key CDATA #REQUIRED>
<!ELEMENT option-description EMPTY>
<!ATTLIST option-description key CDATA #REQUIRED>
<!ELEMENT section (title,((p|execute-list|backrest-config|postgres-config|option-description)+|(p*, section+)|p*))>
<!ATTLIST section id CDATA #REQUIRED>
<!ELEMENT title (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|exe|backrest|postgres|br-option|br-setting|
pg-option|pg-setting|link|user)*>
<!ELEMENT default (#PCDATA)>
<!ELEMENT allow (#PCDATA)>
@ -64,39 +98,52 @@
<!ELEMENT changelog (text?, changelog-release+)>
<!ELEMENT changelog-release (text?, release-feature-bullet-list)>
<!ATTLIST changelog-release date CDATA #REQUIRED>
<!ATTLIST changelog-release version CDATA #REQUIRED>
<!ATTLIST changelog-release title CDATA #REQUIRED>
<!ATTLIST changelog-release date CDATA #REQUIRED>
<!ATTLIST changelog-release version CDATA #REQUIRED>
<!ATTLIST changelog-release title CDATA #REQUIRED>
<!ELEMENT release-feature-bullet-list (release-feature+)>
<!ELEMENT release-feature (text)>
<!ELEMENT contribute (text)>
<!ATTLIST contribute title CDATA #REQUIRED>
<!ATTLIST contribute title CDATA #REQUIRED>
<!ELEMENT recognition (text)>
<!ATTLIST recognition title CDATA #REQUIRED>
<!ATTLIST recognition title CDATA #REQUIRED>
<!ELEMENT support (text)>
<!ATTLIST support title CDATA #REQUIRED>
<!ATTLIST support title CDATA #REQUIRED>
<!ELEMENT summary (text)>
<!ELEMENT text (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|exe|backrest|postgres)*>
<!ELEMENT summary (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|exe|backrest|postgres|br-option|br-setting|
pg-option|pg-setting|link|user)*>
<!ELEMENT p (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|exe|backrest|postgres|br-option|br-setting|
pg-option|pg-setting|link|user)*>
<!ELEMENT text (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|exe|backrest|postgres|br-option|br-setting|
pg-option|pg-setting|link|user)*>
<!ELEMENT i (#PCDATA)>
<!ELEMENT b (#PCDATA)>
<!ELEMENT bi (#PCDATA)>
<!ELEMENT ul (li+)>
<!ELEMENT ol (li+)>
<!ELEMENT li (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|exe|backrest|postgres)*>
<!ELEMENT li (#PCDATA|b|i|bi|ul|ol|id|code|code-block|file|path|cmd|param|setting|exe|backrest|postgres|br-option|br-setting|
pg-option|pg-setting|link|user)*>
<!ELEMENT id (#PCDATA)>
<!ELEMENT code (#PCDATA)>
<!ELEMENT code-block (#PCDATA|exe)*>
<!ELEMENT file (#PCDATA)>
<!ELEMENT path (#PCDATA)>
<!ELEMENT cmd (#PCDATA)>
<!ELEMENT user (#PCDATA)>
<!ELEMENT param (#PCDATA)>
<!ELEMENT setting (#PCDATA)>
<!ELEMENT br-option (#PCDATA)>
<!ELEMENT br-setting (#PCDATA)>
<!ELEMENT pg-option (#PCDATA)>
<!ELEMENT pg-setting (#PCDATA)>
<!ELEMENT exe EMPTY>
<!ELEMENT backrest EMPTY>
<!ELEMENT postgres EMPTY>
<!ELEMENT link (#PCDATA)>
<!ATTLIST link url CDATA "">
<!ATTLIST link page CDATA "">

125
doc/xml/index.xml Normal file
View File

@ -0,0 +1,125 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE doc SYSTEM "doc.dtd">
<doc subtitle="Reliable {[postgres]} Backup &amp; Restore" toc="n">
<variable-list>
<!-- Variables used by the rest of the script -->
<variable>
<variable-name>github-url-base</variable-name>
<variable-value>https://github.com/pgmasters/backrest</variable-value>
</variable>
<variable>
<variable-name>github-url-master</variable-name>
<variable-value>{[github-url-base]}/blob/master</variable-value>
</variable>
<variable>
<variable-name>github-url-issues</variable-name>
<variable-value>{[github-url-base]}/issues</variable-value>
</variable>
<variable>
<variable-name>github-url-change-log</variable-name>
<variable-value>{[github-url-master]}/CHANGELOG.md</variable-value>
</variable>
<variable>
<variable-name>github-url-license</variable-name>
<variable-value>{[github-url-master]}/LICENSE</variable-value>
</variable>
<variable>
<variable-name>backrest-url-base</variable-name>
<variable-value>http://www.pgbackrest.org/backrest</variable-value>
</variable>
<variable>
<variable-name>backrest-page-user-guide</variable-name>
<variable-value>user-guide.html</variable-value>
</variable>
<variable>
<variable-name>backrest-page-configuration</variable-name>
<variable-value>configuration.html</variable-value>
</variable>
<variable>
<variable-name>backrest-page-command</variable-name>
<variable-value>command.html</variable-value>
</variable>
<variable>
<variable-name>crunchy-url-base</variable-name>
<variable-value>http://www.crunchydatasolutions.com</variable-value>
</variable>
<variable>
<variable-name>crunchy-url-cbm</variable-name>
<variable-value>{[crunchy-url-base]}/crunchy-backup-manager</variable-value>
</variable>
<variable>
<variable-name>resonate-url-base</variable-name>
<variable-value>http://www.resonate.com</variable-value>
</variable>
</variable-list>
<section id="introduction">
<title>Introduction</title>
<p><backrest/> aims to be a simple, reliable backup and restore system that can seamlessly scale up to the largest databases and workloads.
Primary <backrest/> features:
<ul>
<li>Local or remote backup</li>
<li>Multi-threaded backup/restore for performance</li>
<li>Checksums</li>
<li>Safe backups (checks that logs required for consistency are present before backup completes)</li>
<li>Full, differential, and incremental backups</li>
<li>Backup rotation (and minimum retention rules with optional separate retention for archive)</li>
<li>In-stream compression/decompression</li>
<li>Archiving and retrieval of logs for replicas/restores built in</li>
<li>Async archiving for very busy systems (including space limits)</li>
<li>Backup directories are consistent <postgres/> clusters (when hardlinks are on and compression is off)</li>
<li>Tablespace support</li>
<li>Restore delta option</li>
<li>Restore using timestamp/size or checksum</li>
<li>Restore remapping base/tablespaces</li>
<li>Support for <postgres/> >= 8.3</li>
</ul>Instead of relying on traditional backup tools like tar and rsync, <backrest/> implements all backup features internally and uses a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows for better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security.
<backrest/> is hosted at <link url="{[github-url-base]}">GitHub</link> and follows the gitflow model of development. This means that the master branch contains only the release history, i.e. each commit represents a single release and release tags are always from the master branch. The dev branch contains a single commit for each feature or fix and more accurately depicts the development history. Actual development is done on feature (dev_*) branches and squashed into dev after regression tests have passed. In this model dev is considered stable and can be released at any time. As such, the dev branch does not have any special version modifiers.</p>
</section>
<section id="getting-started">
<title>Getting Started</title>
<p><backrest/> strives to be easy to configure and operate:
<ul>
<li><link page="{[backrest-page-user-guide]}">User guide</link> for Ubuntu 12.04 &amp; 14.04 / <postgres/> 9.4.</li>
<li><link page="{[backrest-page-command]}">Command reference</link> for command-line operations.</li>
<li><link page="{[backrest-page-configuration]}">Configuration reference</link> for creating rich <backrest/> configurations.</li>
</ul></p>
</section>
<section id="contributing">
<title>Contributing</title>
<p>Contributions to <backrest/> are always welcome!
Code fixes or new features can be submitted via pull requests. Ideas for new features and improvements to existing functionality or documentation can be <link url="{[github-url-issues]}">submitted as issues</link>.
Bug reports should be <link url="{[github-url-issues]}">submitted as issues</link>. Please provide as much information as possible to aid in determining the cause of the problem.
You will always receive credit in the <link url="{[github-url-change-log]}">change log</link> for your contributions.</p>
</section>
<section id="support">
<title>Support</title>
<p><backrest/> is completely free and open source under the <link url="{[github-url-license]}">MIT</link> license. You may use it for personal or commercial purposes without any restrictions whatsoever. Bug reports are taken very seriously and will be addressed as quickly as possible.
Creating a robust disaster recovery policy with proper replication and backup strategies can be a very complex and daunting task. You may find that you need help during the architecture phase and ongoing support to ensure that your enterprise continues running smoothly.
<link url="{[crunchy-url-base]}">Crunchy Data</link> provides packaged versions of <backrest/> for major operating systems and expert full life-cycle commercial support for <backrest/> and all things <postgres/>. <link url="{[crunchy-url-base]}">Crunchy Data</link> is committed to providing open source solutions with no vendor lock-in so cross-compatibility with the community version of <backrest/> is always strictly maintained.
Please visit <link url="{[crunchy-url-cbm]}">Crunchy Backup Manager</link> for more information.</p>
</section>
<section id="recognition">
<title>Recognition</title>
<p>Primary recognition goes to Stephen Frost for all his valuable advice and criticism during the development of <backrest/>.
<link url="{[crunchy-url-base]}">Crunchy Data</link> has contributed significant time and resources to <backrest/> and continues to actively support development. <link url="{[resonate-url-base]}">Resonate</link> also contributed to the development of <backrest/> and allowed early (but well tested) versions to be installed as their primary <postgres/> backup solution.</p>
</section>
</doc>

67
doc/xml/readme.xml
View File

@ -1,67 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE doc SYSTEM "doc.dtd">
<doc title="Simple Postgres Backup &amp; Restore">
<intro>
<text><backrest/> aims to be a simple backup and restore system that can seamlessly scale up to the largest databases and workloads.
Primary <backrest/> features:
<ul>
<li>Local or remote backup</li>
<li>Multi-threaded backup/restore for performance</li>
<li>Checksums</li>
<li>Safe backups (checks that logs required for consistency are present before backup completes)</li>
<li>Full, differential, and incremental backups</li>
<li>Backup rotation (and minimum retention rules with optional separate retention for archive)</li>
<li>In-stream compression/decompression</li>
<li>Archiving and retrieval of logs for replicas/restores built in</li>
<li>Async archiving for very busy systems (including space limits)</li>
<li>Backup directories are consistent <postgres/> clusters (when hardlinks are on and compression is off)</li>
<li>Tablespace support</li>
<li>Restore delta option</li>
<li>Restore using timestamp/size or checksum</li>
<li>Restore remapping base/tablespaces</li>
<li>Support for <postgres/> >= 8.3</li>
</ul>
Instead of relying on traditional backup tools like tar and rsync, <backrest/> implements all backup features internally and uses a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows for better solutions to database-specific backup issues. The custom remote protocol limits the types of connections that are required to perform a backup which increases security.
<backrest/> uses the gitflow model of development. This means that the master branch contains only the release history, i.e. each commit represents a single release and release tags are always from the master branch. The dev branch contains a single commit for each feature or fix and more accurately depicts the development history. Actual development is done on feature (dev_*) branches and squashed into dev after regression tests have passed. In this model dev is considered stable and can be released at any time. As such, the dev branch does not have any special version modifiers.</text>
</intro>
<start title="Getting Started">
<text><backrest/> strives to be easy to configure and operate:
* [Installation instructions](USERGUIDE.md#installation) for major operating systems.
* [Sample configurations](USERGUIDE.md#examples) that cover most basic use cases.
* [Command guide](USERGUIDE.md#commands) for command-line operations.
* [Settings documentation](USERGUIDE.md#setttings) for creating complex configurations and more detail on options.</text>
</start>
<contribute title="Contributing">
<text>Contributions to <backrest/> are always welcome!
Code fixes or new features can be submitted via pull requests. Ideas for new features and improvements to existing functionality or documentation can be [submitted as issues](http://github.com/pgmasters/backrest/issues).
Bug reports should be [submitted as issues](http://github.com/pgmasters/backrest/issues). Please provide as much information as possible to aid in determining the cause of the problem.
You will always receive credit in the [change log](https://github.com/pgmasters/backrest/blob/master/CHANGELOG.md) for your contributions.</text>
</contribute>
<support title="Support">
<text><backrest/> is completely free and open source under the [MIT](https://github.com/pgmasters/backrest/blob/master/LICENSE) license. You may use it for personal or commercial purposes without any restrictions whatsoever. Bug reports are taken very seriously and will be addressed as quickly as possible.
Creating a robust disaster recovery policy with proper replication and backup strategies can be a very complex and daunting task. You may find that you need help during the architecture phase and ongoing support to ensure that your enterprise continues running smoothly.
[Crunchy Data](http://www.crunchydatasolutions.com) provides packaged versions of <backrest/> for major operating systems and expert full life-cycle commercial support for <backrest/> and all things <postgres/>. [Crunchy Data](http://www.crunchydatasolutions.com) is committed to providing open source solutions with no vendor lock-in so cross-compatibility with the community version of <backrest/> is always strictly maintained.
Please visit [Crunchy Backup Manager](http://crunchydatasolutions.com/crunchy-backup-manager) for more information.</text>
</support>
<recognition title="Recognition">
<text>Primary recognition goes to Stephen Frost for all his valuable advice and criticism during the development of <backrest/>.
[Crunchy Data](http://www.crunchydatasolutions.com) has contributed significant time and resources to <backrest/> and continues to actively support development. [Resonate](http://www.resonate.com) also contributed to the development of <backrest/> and allowed early (but well tested) versions to be installed as their primary <postgres/> backup solution.</text>
</recognition>
</doc>

1655
doc/xml/reference.xml
View File

File diff suppressed because it is too large Load Diff

804
doc/xml/user-guide.xml Normal file
View File

@ -0,0 +1,804 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE doc SYSTEM "doc.dtd">
<doc title="User Guide" subtitle="Ubuntu 12.04 &amp; 14.04 / {[postgres]} 9.4">
<variable-list>
<!-- Variables used by the rest of the script -->
<variable>
<variable-name>perl-lib-path</variable-name>
<variable-value>/usr/lib/perl5</variable-value>
</variable>
<variable>
<variable-name>perl-bin-path</variable-name>
<variable-value>/usr/bin</variable-value>
</variable>
<variable>
<variable-name>backrest-repo-path</variable-name>
<variable-value>/var/lib/backrest</variable-value>
</variable>
<variable>
<variable-name>postgres-cluster-demo</variable-name>
<variable-value>demo</variable-value>
</variable>
<variable>
<variable-name>backrest-config-demo</variable-name>
<variable-value>/etc/{[backrest-exe]}.conf</variable-value>
</variable>
<variable>
<variable-name>postgres-config-demo</variable-name>
<variable-value>/etc/postgresql/9.4/{[postgres-cluster-demo]}/postgresql.conf</variable-value>
</variable>
<variable>
<variable-name>db-path</variable-name>
<variable-value>/var/lib/postgresql/9.4/{[postgres-cluster-demo]}</variable-value>
</variable>
<!-- Commands for various operations -->
<variable>
<variable-name>cmd-backup-last</variable-name>
<variable-value>ls -1 /var/lib/backrest/backup/demo | tail -3 | head -1</variable-value>
</variable>
<!-- Data used to demonstrate backup/restore operations -->
<variable>
<variable-name>test-table-data</variable-name>
<variable-value>Very important data</variable-value>
</variable>
</variable-list>
<!-- Commands used to clean the environment before or after doc generation -->
<cleanup>
<execute>
<exe-cmd>pg_dropcluster {[dash]}-stop 9.4 {[postgres-cluster-demo]}</exe-cmd>
<exe-err-suppress/>
</execute>
<execute>
<exe-cmd>apt-get remove -y libdbd-pg-perl libdbi-perl libnet-daemon-perl libplrpc-perl</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>chmod 777 /home/vagrant</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>rm {[backrest-config-demo]}</exe-cmd>
<exe-user>root</exe-user>
<exe-err-suppress/>
</execute>
<execute>
<exe-cmd>chmod 777 /etc</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>rm -rf {[backrest-repo-path]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>rm -rf {[perl-lib-path]}/BackRest</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>rm {[perl-bin-path]}/{[backrest-exe]}</exe-cmd>
<exe-user>root</exe-user>
<exe-err-suppress/>
</execute>
<execute>
<exe-cmd>rm -rf /home/vagrant/backrest-release-{[version]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>mkdir /home/vagrant/backrest-release-{[version]}</exe-cmd>
<exe-user>vagrant</exe-user>
</execute>
<execute>
<exe-cmd>cp -r /backrest/bin /home/vagrant/backrest-release-{[version]}</exe-cmd>
<exe-user>vagrant</exe-user>
</execute>
<execute>
<exe-cmd>cp -r /backrest/lib /home/vagrant/backrest-release-{[version]}</exe-cmd>
<exe-user>vagrant</exe-user>
</execute>
</cleanup>
<!-- SECTION => INTRODUCTION -->
<section id="introduction">
<title>Introduction</title>
<p>This user guide is intended to be followed sequentially from beginning to end &amp;mdash; each section depends on the last. For example the <link page="#backup">Backup</link> section relies on setup that is performed in the <link page="#quickstart">Quick Start</link> section. Once you have <backrest/> up and running it possible to skip around but it is recommended to follow the user guide in order the first time through.</p>
<p>Although the examples are targeted at Ubuntu and <postgres/> 9.4 they will also work fine on Debian and it should be fairly easy to apply this guide to any Unix distribution and <postgres/> version. The only OS-specific commands are those to create, start, stop, and drop <postgres/> clusters. The <backrest/> commands will be the same on any Unix system though the locations to install Perl libraries and executables may vary. Configuring archiving is different on <postgres/> versions &amp;lt;= 8.4 and configuration information can be found in the <postgres/> documentation.</p>
<p>A somewhat novel approach is taken to documentation in this user guide. Each command is run on a virtual machine when the documentation is built from the XML source. This means you can have a high confidence that the commands work correctly in the order presented. Output is captured and displayed below the command when appropriate. If the output is not included it is because it was deemed not relevant or was considered a distraction from the narrative.</p>
<p>All commands are intended to be run as an unprivileged user that has sudo privileges for both the <user>root</user> and <user>postgres</user> users. It's also possible to run the commands directly as their respective users without modification though in that case you can also strip off the <cmd>sudo</cmd> commands if you like.</p>
</section>
<!-- SECTION => CONCEPTS -->
<section id="concept">
<title>Concepts</title>
<p>The following concepts are defined as they are relevant to <backrest/>, <postgres/>, and this user guide.</p>
<!-- SECTION => CONCEPTS - BACKUP -->
<section id="backup">
<title>Backup</title>
<p>A backup is a consistent copy of a database cluster that can be restored to recover from a hardware failure, to perform Point-In-Time Recovery, or to bring up a new replica.</p>
<p><b>Full Backup</b>: <backrest/> copies the entire contents of the database cluster to the backup server. The first backup of the database cluster is always a Full Backup. <backrest/> is always able to restore a full backup directly. The full backup does not depend on any files outside of the full backup for consistency.</p>
<p><b>Differential Backup</b>: <backrest/> copies only those database cluster files that have changed since the last full backup. <backrest/> restores a differential backup by copying all of the files in the chosen differential backup and the appropriate unchanged files from the previous full backup. The advantage of a differential backup is that it requires less disk space than a full backup, however, the differential backup and the full backup must both be valid to restore the differential backup.</p>
<p><b>Incremental Backup</b>: <backrest/> copies only those database cluster file that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup). As an incremental backup only includes those files changed since the prior backup, they are generally much smaller than full or differential backups. As with the differential backup, the incremental backup depends on other backups to be valid to restore the incremental backup. Since the incremental backup includes only those files since the last backup, all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup must all be valid to perform a restore of the incremental backup. If no differential backup exists then all prior incremental backups back to the prior full backup, which must exist, and the full backup itself must be valid to restore the incremental backup.</p>
</section>
<!-- SECTION => CONCEPTS - RESTORE -->
<section id="restore">
<title>Restore</title>
<p>A restore is the act of copying a backup to a system where it will be started as a live database cluster. A restore requires the backup files and one or more WAL segments in order to work correctly.</p>
</section>
<!-- SECTION => CONCEPTS - WAL -->
<section id="wal">
<title>Write Ahead Log (WAL)</title>
<p>WAL is the mechanism by which <postgres/> ensures that no committed changes are lost. Transactions are written sequentially to the WAL and a transaction is considered to be committed when those writes are flushed to disk. Afterwards, a background process writes the changes into the main database cluster files (also known as the heap). In the event of a crash, the WAL is replayed to make the database consistent.</p>
<p>WAL is conceptually infinite but in practice is broken up into individual 16MB files called segments. WAL segments follow the naming convention <id>0000000100000A1E000000FE</id> where the first 8 hexadecimal digits represent the timeline and the next 16 digits are the WAL segment sequence number.</p>
<p>A valid backup will always include <i>at least</i> one WAL segment even if no writes were made to the database between backups.</p>
</section>
</section>
<!-- SECTION => INSTALLATION -->
<section id="installation">
<title>Installation</title>
<p><backrest/> is written in Perl which is included with Ubuntu by default. A few additional modules are required which are all available as packages.</p>
<execute-list>
<title>Install required Perl modules</title>
<execute>
<exe-cmd>apt-get install libdbd-pg-perl libdbi-perl libnet-daemon-perl libplrpc-perl</exe-cmd>
<exe-user>root</exe-user>
<exe-output/>
</execute>
</execute-list>
<p>No Debian/Ubuntu packages are currently available for <backrest/> but it is easy to download the source and install manually.</p>
<execute-list>
<title>Download version <id>{[version]}</id> of <backrest/></title>
<execute>
<exe-cmd>wget -O - https://github.com/pgmasters/backrest/archive/release/{[version]}.tar.gz | tar zxv -C ~</exe-cmd>
<exe-user>vagrant</exe-user>
<exe-skip/>
</execute>
</execute-list>
<execute-list>
<title>Install <backrest/></title>
<execute>
<exe-cmd>cp -r ~/backrest-release-{[version]}/lib/BackRest {[perl-lib-path]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>find /usr/lib/perl5/BackRest -type f -exec chmod 644 {} +</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>find /usr/lib/perl5/BackRest -type d -exec chmod 755 {} +</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>cp ~/backrest-release-{[version]}/bin/{[backrest-exe]} {[perl-bin-path]}/{[backrest-exe]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>chmod 755 {[perl-bin-path]}/{[backrest-exe]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
</execute-list>
<p><backrest/> should now be properly installed but it is best to check. If any dependencies were missed then you will get an error when running <backrest/> from the command line.</p>
<execute-list>
<title>Make sure the installation worked</title>
<execute>
<exe-cmd>{[backrest-exe]}</exe-cmd>
<exe-user>vagrant</exe-user>
<exe-output/>
</execute>
</execute-list>
</section>
<!-- SECTION => QUICKSTART -->
<section id="quickstart">
<title>Quick Start</title>
<p>The Quick Start section will cover basic configuration of <backrest/> and <postgres/> and introduce the <cmd>backup</cmd>, <cmd>restore</cmd>, and <cmd>info</cmd> commands.</p>
<!-- SECTION => QUICKSTART - SETUP DEMO CLUSTER -->
<section id="setup-demo-cluster">
<title>Setup Demo Cluster</title>
<p>You'll need to create a demo cluster to run the example commands in this user guide. This step is optional, but you may need to adjust commands in the user guide to work with your environment if you choose not to create the demo cluster.</p>
<execute-list>
<title>Create and start the demo cluster</title>
<execute>
<exe-cmd>pg_createcluster 9.4 {[postgres-cluster-demo]} {[dash]}-start</exe-cmd>
<exe-user>root</exe-user>
<exe-output/>
</execute>
</execute-list>
</section>
<!-- SECTION => QUICKSTART - CONFIGURE STANZA -->
<section id="configure-stanza">
<title>Configure Cluster Stanza</title>
<option-description key="stanza"/>
<p>Demo describes the purpose of this cluster accurately so that will also make a good stanza name.</p>
<p><backrest/> needs to know where the base data directory for the <postgres/> cluster is located. The path can be requested from <postgres/> directly but in a recovery scenario the <postgres/> process will not be available. During backups the value supplied to <backrest/> will be compared against the path that <postgres/> is running on and they must be equal or the backup will return an error. Make sure that <br-option>db-path</br-option> is exactly equal to <pg-option>data_directory</pg-option> in <file>postgresql.conf</file>.</p>
<p>By default Ubuntu stores clusters in <path>/var/lib/postgresql/[VERSION]/[CLUSTER-NAME]</path> so it is easy to determine the correct path for the data directory.</p>
<backrest-config file="{[backrest-config-demo]}">
<title>Configure the <postgres/> cluster data directory</title>
<backrest-config-option>
<backrest-config-option-section>demo</backrest-config-option-section>
<backrest-config-option-key>db-path</backrest-config-option-key>
<backrest-config-option-value>{[db-path]}</backrest-config-option-value>
</backrest-config-option>
</backrest-config>
</section>
<!-- SECTION => QUICKSTART - CREATE REPOSITORY -->
<section id="create-repository">
<title>Create the Repository</title>
<p>For this simple configuration the repository will be stored on the same host as the <postgres/> server. This is the simplest configuration and is useful in cases where external backup software is employed to backup the database host.</p>
<execute-list>
<title>Create the <backrest/> repository</title>
<execute>
<exe-cmd>mkdir {[backrest-repo-path]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>chmod 750 {[backrest-repo-path]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>chown postgres:postgres {[backrest-repo-path]}</exe-cmd>
<exe-user>root</exe-user>
</execute>
</execute-list>
<p>The repository path must be configured so <backrest/> knows where to find it.</p>
<backrest-config file="{[backrest-config-demo]}">
<title>Configure the <backrest/> repository path</title>
<backrest-config-option>
<backrest-config-option-section>global:general</backrest-config-option-section>
<backrest-config-option-key>repo-path</backrest-config-option-key>
<backrest-config-option-value>{[backrest-repo-path]}</backrest-config-option-value>
</backrest-config-option>
</backrest-config>
</section>
<!-- SECTION => QUICKSTART - CONFIGURE ARCHIVING -->
<section id="configure-archiving">
<title>Configure Archiving</title>
<p>Backing up a running <postgres/> cluster requires WAL archiving to be enabled.</p>
<postgres-config file="{[postgres-config-demo]}">
<title>Configure <postgres/> settings</title>
<postgres-config-option key="archive_command">'{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} archive-push %p'</postgres-config-option>
<postgres-config-option key="archive_mode">on</postgres-config-option>
<postgres-config-option key="wal_level">archive</postgres-config-option>
</postgres-config>
<p>The <pg-option>wal_level</pg-option> setting must be set to <pg-setting>archive</pg-setting> at a minimum but <pg-setting>hot_standby</pg-setting> and <pg-setting>logical</pg-setting> also work fine for backups. Setting <pg-option>wal_level</pg-option> to <pg-setting>hot_standy</pg-setting> is a good idea even if you do not currently run a hot standby since one can be added later without restarting the primary cluster.</p>
<p>The <postgres/> cluster must be restarted after making these changes and before performing a backup.</p>
<execute-list>
<title>Restart the {[postgres-cluster-demo]} cluster</title>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} restart</exe-cmd>
</execute>
</execute-list>
<!-- It would be good if the info command showed xlogs so we don't have to examine directories
<execute-list>
<execute>
<exe-cmd>psql -c "select pg_switch_xlog()"</exe-cmd>
<exe-output/>
</execute>
</execute-list>
<execute-list>
<execute>
<exe-cmd>ls -lahR {[backrest-repo-path]}/archive/{[postgres-cluster-demo]}/9.4-1</exe-cmd>
<exe-retry/>
<exe-output/>
</execute>
</execute-list> -->
</section>
<!-- SECTION => QUICKSTART - PERFORM BACKUP -->
<section id="perform-backup">
<title>Perform a Backup</title>
<p>To perform a backup of the <postgres/> cluster run <backrest/> with the <cmd>backup</cmd> command.</p>
<execute-list>
<title>Backup the {[postgres-cluster-demo]} cluster</title>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} backup</exe-cmd>
<exe-output/>
</execute>
</execute-list>
<p>By default <backrest/> will attempt to perform an incremental backup. However, an incremental backup must be based on a full backup and since no full backup existed <backrest/> ran a full backup instead.</p>
<p>The <br-option>type</br-option> option can be used to specify and full or differential backup.</p>
<execute-list>
<title>Differential backup of the {[postgres-cluster-demo]} cluster</title>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-type=diff backup {[dash]}-log-level-console=info</exe-cmd>
<exe-output/>
</execute>
</execute-list>
<p>This time there was no warning because a full backup already existed. Like incremental backups, differential backups must be based on a full backup. An incremental backup can be performed by running the <cmd>backup</cmd> command with <br-setting>{[dash]}-type=incr</br-setting>.</p>
<p>Use the <cmd>info</cmd> command to get information about backups.</p>
<execute-list>
<title>Get info for the {[postgres-cluster-demo]} cluster</title>
<execute>
<exe-cmd>{[backrest-exe]} info</exe-cmd>
<exe-output/>
<exe-highlight>(oldest|latest) backup label</exe-highlight>
</execute>
</execute-list>
<p>The oldest and newest backups are shown in the info output. The oldest backup will <i>always</i> be a full backup (indicated by an <id>F</id> at the end of the label) but the newest backup can be full, differential (ends with <id>D</id>), or incremental (ends with <id>I</id>).</p>
<p>More information about the <cmd>backup</cmd> command can be found in the <link page="#backup">Backup</link> section.</p>
</section>
<!-- SECTION => QUICKSTART - PERFORM RESTORE -->
<section id="perform-restore">
<title>Restore a Backup</title>
<p>Backups can protect you from a number of disaster scenarios, the most common of which are hardware failure and data corruption. The easiest way to simulate data corruption is to remove an important <postgres/> cluster file.</p>
<execute-list>
<title>Stop the {[postgres-cluster-demo]} cluster and delete the <file>pg_control</file> file</title>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} stop</exe-cmd>
</execute>
<execute>
<exe-cmd>rm {[db-path]}/global/pg_control</exe-cmd>
</execute>
</execute-list>
<p>Starting the cluster without this important file will result in an error.</p>
<execute-list>
<title>Attempt to start the corrupted {[postgres-cluster-demo]} cluster</title>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} start</exe-cmd>
<exe-output/>
<exe-err-expect>1</exe-err-expect>
</execute>
</execute-list>
<p>To restore a backup of the <postgres/> cluster run <backrest/> with the <cmd>restore</cmd> command. The cluster needs to be stopped (in this case it is already stopped) and all files must be removed from the <postgres/> data directory.</p>
<execute-list>
<title>Remove old files from {[postgres-cluster-demo]} cluster</title>
<execute>
<exe-cmd>find {[db-path]} -mindepth 1 -delete</exe-cmd>
</execute>
</execute-list>
<execute-list>
<title>Restore the {[postgres-cluster-demo]} cluster and start <postgres/></title>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} restore</exe-cmd>
</execute>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} start</exe-cmd>
</execute>
</execute-list>
<p>This time the cluster started successfully since the <cmd>restore</cmd> replaced the missing <file>pg_control</file> file.</p>
<p>More information about the <cmd>restore</cmd> command can be found in the <link page="#restore">Restore</link> section.</p>
</section>
</section>
<!-- SECTION => BACKUP -->
<section id="backup">
<title>Backup</title>
<p>The Backup section introduces additional <cmd>backup</cmd> command features.</p>
<!-- SECTION => BACKUP - START-FAST -->
<section id="option-start-fast">
<title>Fast Start Option</title>
<p>By default <backrest/> will wait for the next regularly scheduled checkpoint before starting a backup. Depending on the <pg-option>checkpoint_timeout</pg-option> and <pg-option>checkpoint_segments</pg-option> settings in <postgres/> it may be quite some time before a checkpoint completes and the backup can begin.</p>
<execute-list>
<title>Incremental backup of the {[postgres-cluster-demo]} cluster with the regularly scheduled checkpoint</title>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-type=incr backup {[dash]}-log-level-console=info</exe-cmd>
<exe-output/>
<exe-highlight>backup begins after the next regular checkpoint completes</exe-highlight>
</execute>
</execute-list>
<p>By setting <br-setting>start-fast</br-setting> on the command-line or in <file>{[backrest-config-demo]}</file> an immediate checkpoint is requested and the backup will start more quickly. This is convenient for testing and for ad-hoc backups. For instance, if a backup is being taken at the beginning of a release window it makes no sense to wait for a checkpoint. Since regularly scheduled backups generally only happen once per day it is unlikely that enabling the <br-option>start-fast</br-option> in <file>{[backrest-config-demo]}</file> will negatively affect performance, however for high-volume transactional systems you may want to pass <br-setting>{[dash]}-start-fast</br-setting> on the command-line instead.</p>
<backrest-config file="{[backrest-config-demo]}">
<title>Enable the <br-option>start-fast</br-option> option</title>
<backrest-config-option>
<backrest-config-option-section>global:backup</backrest-config-option-section>
<backrest-config-option-key>start-fast</backrest-config-option-key>
<backrest-config-option-value>y</backrest-config-option-value>
</backrest-config-option>
</backrest-config>
<execute-list>
<title>Incremental backup of the {[postgres-cluster-demo]} cluster with an immediate checkpoint</title>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-type=incr backup {[dash]}-log-level-console=info</exe-cmd>
<exe-output/>
<exe-highlight>backup begins after the requested immediate checkpoint completes</exe-highlight>
</execute>
</execute-list>
</section>
<!-- SECTION => BACKUP - STOP-AUTO -->
<section id="option-stop-auto">
<title>Automatic Stop Option</title>
<p>Sometimes <backrest/> will exit unexpectedly and the backup in progress on the <postgres/> cluster will not be properly stopped. <backrest/> exits as quickly as possible when an error occurs so that the cause can be reported accurately and is not masked by another problem that could happen during a more extensive cleanup.</p>
<p>Here an error in intentionally caused by removing repository permissions.</p>
<execute-list>
<title>Revoke write privileges in the <backrest/> repository and attempt a backup</title>
<execute>
<exe-cmd>chmod 550 {[backrest-repo-path]}/temp</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-type=incr backup {[dash]}-log-level-console=info</exe-cmd>
<exe-output/>
<exe-highlight>ERROR:</exe-highlight>
<exe-err-expect>199</exe-err-expect>
</execute>
</execute-list>
<p>Even when the permissions are fixed <backrest/> will still be unable to perform a backup because the <postgres/> cluster is stuck in backup mode.</p>
<execute-list>
<title>Restore write privileges in the <backrest/> repository and attempt a backup</title>
<execute>
<exe-cmd>chmod 750 {[backrest-repo-path]}/temp</exe-cmd>
<exe-user>root</exe-user>
</execute>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-type=incr backup {[dash]}-log-level-console=info</exe-cmd>
<exe-output/>
<exe-highlight>ERROR:</exe-highlight>
<exe-err-expect>132</exe-err-expect>
</execute>
</execute-list>
<p>Enabling the <br-option>stop-auto</br-option> option allows <backrest/> to stop the current backup if it detects that no other <backrest/> backup process is running.</p>
<backrest-config file="{[backrest-config-demo]}">
<title>Enable the <br-option>stop-auto</br-option> option</title>
<backrest-config-option>
<backrest-config-option-section>global:backup</backrest-config-option-section>
<backrest-config-option-key>stop-auto</backrest-config-option-key>
<backrest-config-option-value>y</backrest-config-option-value>
</backrest-config-option>
</backrest-config>
<p>Now <backrest/> will stop the old backup and start a new one so the process completes successfully.</p>
<execute-list>
<title>Perform an incremental backup</title>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-type=incr backup {[dash]}-log-level-console=info</exe-cmd>
<exe-output/>
<exe-highlight>cluster is already in backup mode|backup begins after the requested immediate checkpoint completes</exe-highlight>
</execute>
</execute-list>
<p>Although useful this feature may not be appropriate when another third-party backup solution is being used to take online backups as <backrest/> will not recognize that the other software is running and may terminate a backup started by that software. However, it would be unusual to run more than one third-party backup solution at the same time so this is not likely to be a problem.</p>
<p>Note that <id>pg_dump</id> and <id>pg_base_backup</id> do not take online backups so are not affected. It is safe to run them in conjunction with <backrest/>.</p>
</section>
</section>
<!-- SECTION => RESTORE -->
<section id="restore">
<title>Restore</title>
<p>The Restore section introduces additional <cmd>restore</cmd> command features.</p>
<!-- SECTION => RESTORE - PITR -->
<section id="pitr">
<title>Point-in-Time Recovery (PITR)</title>
<p>The <link page="#quickstart-perform-restore">restore example</link> in <link page="#quickstart">Quick Start</link> performed default recovery, which is to play all the way to the end of the WAL stream. In the case of a hardware failure this is probably the most appropriate action but for data corruption scenarios (whether machine or human in origin) there is a better alternative called Point-in-Time Recovery (PITR).</p>
<p>PITR allows the WAL to be played from the last backup to a specified time, transaction id, or recovery point. For common recovery scenarios time-based recovery is arguably the most useful. A common recovery scenario is to restore a table or data was accidentally dropped or deleted. Recovering a dropped table is more dramatic so that's the example given here but deleted data would be recovered in exactly the same way.</p>
<execute-list>
<title>Create table with very important data</title>
<execute>
<exe-cmd>
psql -c "begin;
create table important_table (message text);
insert into important_table values ('{[test-table-data]}');
commit;
select * from important_table";
</exe-cmd>
<exe-output/>
<exe-highlight>{[test-table-data]}</exe-highlight>
</execute>
</execute-list>
<p>It is important to represent the time as reckoned by <postgres/> and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.</p>
<execute-list>
<title>Get time from <postgres/></title>
<execute>
<exe-cmd>
psql -Atc "select current_timestamp"
</exe-cmd>
<exe-var>time-recovery-timestamp</exe-var>
<exe-output/>
</execute>
</execute-list>
<p>Now that the time has been recorded the table is dropped. In practice finding the exact time that the table was dropped is a lot harder than in this example. It may not be possible to find the exact time, but some forensic work should be able to get you close.</p>
<execute-list>
<title>Drop the important table</title>
<execute>
<exe-cmd>psql -c "begin;
drop table important_table;
commit;
select * from important_table;"</exe-cmd>
<exe-output/>
<exe-highlight>does not exist</exe-highlight>
<exe-err-expect>1</exe-err-expect>
</execute>
</execute-list>
<p>Now the restore can be performed with time-based recovery to <id>{[time-recovery-timestamp]}</id> to bring back the missing table.</p>
<execute-list>
<title>Stop <postgres/>, restore the {[postgres-cluster-demo]} cluster to <id>{[time-recovery-timestamp]}</id>, and display <file>recovery.conf</file></title>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} stop</exe-cmd>
</execute>
<execute>
<exe-cmd>rm /var/log/postgresql/postgresql-9.4-{[postgres-cluster-demo]}.log</exe-cmd>
<exe-no-show/>
</execute>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} --type=time "--target={[time-recovery-timestamp]}" --delta restore</exe-cmd>
</execute>
<execute>
<exe-cmd>cat /var/lib/postgresql/9.4/{[postgres-cluster-demo]}/recovery.conf</exe-cmd>
<exe-output/>
</execute>
</execute-list>
<p>The <file>recovery.conf</file> file has been automatically generated by <backrest/> so <postgres/> can be started immediately. Once <postgres/> has finished recovery the table will exist again and can be queried.</p>
<execute-list>
<title>Start <postgres/> and check that the important table exists</title>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} start</exe-cmd>
</execute>
<execute>
<exe-cmd>psql -c "select * from important_table"</exe-cmd>
<exe-output/>
<exe-highlight>{[test-table-data]}</exe-highlight>
</execute>
</execute-list>
<p>The <postgres/> log also contains valuable information. It will indicate the time and transaction where the recovery stopped and also give the time of the last transaction to be applied.</p>
<execute-list>
<title>Examine the <postgres/> log output</title>
<execute>
<exe-cmd>cat /var/log/postgresql/postgresql-9.4-{[postgres-cluster-demo]}.log</exe-cmd>
<exe-output/>
<exe-highlight>recovery stopping before|last completed transaction|starting point-in-time recovery</exe-highlight>
</execute>
</execute-list>
<p>This example was rigged to give the correct result. If a backup after the required time is chosen then <postgres/> will not be able to recover the lost table. <postgres/> can only play forward, not backward. To demonstrate this the important table must be dropped (again).</p>
<execute-list>
<title>Drop the important table (again)</title>
<execute>
<exe-cmd>psql -c "begin;
drop table important_table;
commit;
select * from important_table;"</exe-cmd>
<exe-output/>
<exe-highlight>does not exist</exe-highlight>
<exe-err-expect>1</exe-err-expect>
</execute>
</execute-list>
<p>Now take a new backup and attempt the recovery from the new backup.</p>
<execute-list>
<title>Perform a backup then attempt recovery from that backup</title>
<execute>
<exe-cmd>{[cmd-backup-last]}</exe-cmd>
<exe-var>backup-last</exe-var>
<exe-no-show/>
</execute>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-type=incr backup</exe-cmd>
</execute>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} stop</exe-cmd>
</execute>
<execute>
<exe-cmd>rm /var/log/postgresql/postgresql-9.4-{[postgres-cluster-demo]}.log</exe-cmd>
<exe-no-show/>
</execute>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} --type=time "--target={[time-recovery-timestamp]}" --delta restore</exe-cmd>
</execute>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} start</exe-cmd>
</execute>
<execute>
<exe-cmd>psql -c "select * from important_table"</exe-cmd>
<exe-output/>
<exe-highlight>does not exist</exe-highlight>
<exe-err-expect>1</exe-err-expect>
</execute>
</execute-list>
<p>Looking at the log output it's not obvious that recovery failed to restore the table. The key is to look for the presence of the "recovery stopping before..." and "last completed transaction..." log messages. If they are not present then the recovery to the specified point-in-time was not successful.</p>
<execute-list>
<title>Examine the <postgres/> log output to discover the recovery was not successful</title>
<execute>
<exe-cmd>cat /var/log/postgresql/postgresql-9.4-{[postgres-cluster-demo]}.log</exe-cmd>
<exe-output/>
<exe-highlight>starting point-in-time recovery</exe-highlight>
</execute>
</execute-list>
<p>Using an earlier backup will allow <postgres/> to play forward to the correct time again. The default behavior for restore is to use the most recent backup but an earlier backup can be specified with the <br-option>--set</br-option> option.</p>
<execute-list>
<title>Stop <postgres/>, YADA YADA! </title>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} stop</exe-cmd>
</execute>
<execute>
<exe-cmd>rm /var/log/postgresql/postgresql-9.4-{[postgres-cluster-demo]}.log</exe-cmd>
<exe-no-show/>
</execute>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} --type=time "--target={[time-recovery-timestamp]}" --set={[backup-last]} --delta restore</exe-cmd>
</execute>
<execute>
<exe-cmd>pg_ctlcluster 9.4 {[postgres-cluster-demo]} start</exe-cmd>
</execute>
<execute>
<exe-cmd>psql -c "select * from important_table"</exe-cmd>
<exe-output/>
<!-- <exe-highlight>{[message-cluster-before-time]}</exe-highlight> -->
</execute>
</execute-list>
<p>Now the the log output will contain the expected "recovery stopping before..." and "last completed transaction..." messages showing that the recovery was successful.</p>
<execute-list>
<title>Examine the <postgres/> log output for log messages indicating success</title>
<execute>
<exe-cmd>cat /var/log/postgresql/postgresql-9.4-{[postgres-cluster-demo]}.log</exe-cmd>
<exe-output/>
<exe-highlight>recovery stopping before|last completed transaction|starting point-in-time recovery</exe-highlight>
</execute>
</execute-list>
</section>
</section>
<!-- SECTION => INFO -->
<!-- <section id="info">
<title>Repository Information</title>
<p></p>
<execute-list>
<title>Get detailed information</title>
<execute>
<exe-cmd>{[backrest-exe]} {[dash]}-stanza={[postgres-cluster-demo]} {[dash]}-output=json info</exe-cmd>
<exe-output/>
</execute>
</execute-list>
</section> -->
</doc>

61
lib/BackRest/Config/Config.pm
View File

@ -930,8 +930,7 @@ my %oOptionRule =
&CMD_ARCHIVE_PUSH => true,
&CMD_BACKUP => true,
&CMD_INFO => true,
&CMD_RESTORE => true,
&CMD_EXPIRE => true
&CMD_RESTORE => true
},
},
@ -1993,6 +1992,35 @@ sub optionRequired
push @EXPORT, qw(optionRequired);
####################################################################################################################################
# optionType
#
# Get the option type.
####################################################################################################################################
sub optionType
{
my $strOption = shift;
return $oOptionRule{$strOption}{&OPTION_RULE_TYPE};
}
push @EXPORT, qw(optionType);
####################################################################################################################################
# optionTypeTest
#
# Test the option type.
####################################################################################################################################
sub optionTypeTest
{
my $strOption = shift;
my $strType = shift;
return optionType($strOption) eq $strType;
}
push @EXPORT, qw(optionTypeTest);
####################################################################################################################################
# optionDefault
#
@ -2015,6 +2043,31 @@ sub optionDefault
push @EXPORT, qw(optionDefault);
####################################################################################################################################
# optionRange
#
# Gets the allowed setting range for the option if it exists.
####################################################################################################################################
sub optionRange
{
my $strOption = shift;
my $strCommand = shift;
# Get the command rule
my $oCommandRule = optionCommandRule($strOption, $strCommand);
# Check for default in command
if (defined($oCommandRule) && defined($$oCommandRule{&OPTION_RULE_ALLOW_RANGE}))
{
return $$oCommandRule{&OPTION_RULE_ALLOW_RANGE}[0], $$oCommandRule{&OPTION_RULE_ALLOW_RANGE}[1];
}
# If defined return, else try to grab the global default
return $oOptionRule{$strOption}{&OPTION_RULE_ALLOW_RANGE}[0], $oOptionRule{$strOption}{&OPTION_RULE_ALLOW_RANGE}[1];
}
push @EXPORT, qw(optionRange);
####################################################################################################################################
# optionSource
#
@ -2388,7 +2441,9 @@ push @EXPORT, qw(commandWrite);
####################################################################################################################################
sub commandHashGet
{
use Storable qw(dclone);
require Storable;
Storable->import();
return dclone(\%oCommandHash);
}

22
lib/BackRest/Config/ConfigHelp.pm
View File

@ -21,28 +21,28 @@ use BackRest::Config::Config;
####################################################################################################################################
# Help types
####################################################################################################################################
use constant CONFIG_HELP_COMMAND => 'command';
use constant CONFIG_HELP_COMMAND => 'command';
push @EXPORT, qw(CONFIG_HELP_COMMAND);
use constant CONFIG_HELP_CURRENT => 'current';
use constant CONFIG_HELP_CURRENT => 'current';
push @EXPORT, qw(CONFIG_HELP_CURRENT);
use constant CONFIG_HELP_DEFAULT => 'default';
use constant CONFIG_HELP_DEFAULT => 'default';
push @EXPORT, qw(CONFIG_HELP_DEFAULT);
use constant CONFIG_HELP_DESCRIPTION => 'description';
use constant CONFIG_HELP_DESCRIPTION => 'description';
push @EXPORT, qw(CONFIG_HELP_DESCRIPTION);
use constant CONFIG_HELP_OPTION => 'option';
use constant CONFIG_HELP_OPTION => 'option';
push @EXPORT, qw(CONFIG_HELP_OPTION);
use constant CONFIG_HELP_SECTION => 'section';
use constant CONFIG_HELP_SECTION => 'section';
push @EXPORT, qw(CONFIG_HELP_SECTION);
use constant CONFIG_HELP_SUMMARY => 'summary';
use constant CONFIG_HELP_SUMMARY => 'summary';
push @EXPORT, qw(CONFIG_HELP_SUMMARY);
use constant CONFIG_HELP_SOURCE => 'source';
use constant CONFIG_HELP_SOURCE => 'source';
push @EXPORT, qw(CONFIG_HELP_SOURCE);
use constant CONFIG_HELP_SOURCE_DEFAULT => 'default';
use constant CONFIG_HELP_SOURCE_DEFAULT => 'default';
push @EXPORT, qw(CONFIG_HELP_SOURCE_DEFAULT);
use constant CONFIG_HELP_SOURCE_SECTION => CONFIG_HELP_SECTION;
use constant CONFIG_HELP_SOURCE_SECTION => CONFIG_HELP_SECTION;
push @EXPORT, qw(CONFIG_HELP_SOURCE_SECTION);
use constant CONFIG_HELP_SOURCE_COMMAND => CONFIG_HELP_COMMAND;
use constant CONFIG_HELP_SOURCE_COMMAND => CONFIG_HELP_COMMAND;
push @EXPORT, qw(CONFIG_HELP_SOURCE_COMMAND);
####################################################################################################################################

201
lib/BackRest/Config/ConfigHelpData.pm
View File

@ -30,7 +30,7 @@ my $oConfigHelpData =
"Archive WAL segments asynchronously.",
description =>
"WAL segments will be copied to the local repo, then a process will be forked to compress the segment and " .
"transfer it to the remote repo if configured. Control will be returned to PostgreSQL as soon as the WAL " .
"transfer it to the remote repo if configured. Control will be returned to PostgreSQL as soon as the WAL " .
"segment is copied locally."
},
@ -42,7 +42,7 @@ my $oConfigHelpData =
summary =>
"Check that WAL segments are present in the archive before backup completes.",
description =>
"Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a " .
"Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a " .
"good idea to leave this as the default unless you are using another method for archiving."
},
@ -55,13 +55,13 @@ my $oConfigHelpData =
"Copy WAL segments needed for consistency to the backup.",
description =>
"This slightly paranoid option protects against corruption or premature expiration in the WAL segment archive " .
"by storing the WAL segments directly in the backup. PITR won't be possible without the WAL segment " .
"archive and this option also consumes more space.\n" .
"by storing the WAL segments directly in the backup. PITR won't be possible without the WAL segment archive " .
"and this option also consumes more space.\n" .
"\n" .
"Even though WAL segments will be restored with the backup, PostgreSQL will ignore them if a recovery.conf file " .
"exists and instead use archive_command to fetch WAL segments. Specifying type=none when restoring will " .
"not create recovery.conf and force PostgreSQL to use the WAL segments in pg_xlog. This will get the " .
"database cluster to a consistent state."
"exists and instead use archive_command to fetch WAL segments. Specifying type=none when restoring will not " .
"create recovery.conf and force PostgreSQL to use the WAL segments in pg_xlog. This will get the database " .
"cluster to a consistent state."
},
# ARCHIVE-MAX-MB Option Help
@ -79,11 +79,11 @@ my $oConfigHelpData =
"* A stop file will be written in the lock directory and no more archive files will be backed up until it is " .
"removed.\n" .
"\n" .
"If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. " .
"A new backup will be required to regain full restore capability.\n" .
"If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. " .
"A ne backup will be required to regain full restore capability.\n" .
"\n" .
"The purpose of this feature is to prevent the log volume from filling up at which point Postgres will stop " .
"completely. Better to lose the backup than have PostgreSQL go down.\n" .
"completely. Better to lose the backup than have PostgreSQL go down.\n" .
"\n" .
"To start normal archiving again you'll need to remove the stop file which will be located at " .
"\${repo-path}/lock/\${stanza}-archive.stop where \${repo-path} is the path set in the general section, and " .
@ -111,8 +111,8 @@ my $oConfigHelpData =
summary =>
"Backup host user when backup-host is set.",
description =>
"Defines the user that will be used for operations on the backup server. Preferably this is not the postgres " .
"user but rather some other user like backrest. If PostgreSQL runs on the backup server the postgres user " .
"Defines the user that will be used for operations on the backup server. Preferably this is not the postgres " .
"user but rather some other user like backrest. If PostgreSQL runs on the backup server the postgres user " .
"can be placed in the backrest group so it has read permissions on the repository without being able to " .
"damage the contents accidentally."
},
@ -125,8 +125,8 @@ my $oConfigHelpData =
summary =>
"Buffer size for file operations.",
description =>
"Set the buffer size used for copy, compress, and uncompress functions. A maximum of 3 buffers will be in use " .
"at a time per thread. An additional maximum of 256K per thread may be used for zlib buffers."
"Set the buffer size used for copy, compress, and uncompress functions. A maximum of 3 buffers will be in use " .
"at a time per thread. An additional maximum of 256K per thread may be used for zlib buffers."
},
# CMD-REMOTE Option Help
@ -137,7 +137,7 @@ my $oConfigHelpData =
summary =>
"pgBackRest exe path on the remote host.",
description =>
"Required only if the path to pg_backrest is different on the local and remote systems. If not defined, the " .
"Required only if the path to pg_backrest is different on the local and remote systems. If not defined, the " .
"remote exe path will be set the same as the local exe path."
},
@ -147,9 +147,9 @@ my $oConfigHelpData =
{
section => 'general',
summary =>
"Use file compression.",
"Use gzip file compression.",
description =>
"Enable gzip compression. Backup files are compatible with command-line gzip tools."
"Backup files are compatible with command-line gzip tools."
},
# COMPRESS-LEVEL Option Help
@ -172,10 +172,9 @@ my $oConfigHelpData =
"Compression level for network transfer when compress=n.",
description =>
"Sets the zlib level to be used for protocol compression when compress=n and the database cluster is not on the " .
"same host as the backup. Protocol compression is used to reduce network traffic but can be disabled by " .
"setting compress-level-network=0. When compress=y the compress-level-network setting is ignored and " .
"compress-level is used instead so that the file is only compressed once. SSH compression is always " .
"disabled."
"same host as the backup. Protocol compression is used to reduce network traffic but can be disabled by " .
"setting compress-level-network=0. When compress=y the compress-level-network setting is ignored and " .
"compress-level is used instead so that the file is only compressed once. SSH compression is always disabled."
},
# CONFIG Option Help
@ -196,7 +195,7 @@ my $oConfigHelpData =
summary =>
"pgBackRest remote configuration file.",
description =>
"Sets the location of the remote configuration file. This is only required if the remote configuration file is " .
"Sets the location of the remote configuration file. This is only required if the remote configuration file is " .
"in a different location than the local configuration file."
},
@ -219,7 +218,7 @@ my $oConfigHelpData =
summary =>
"Cluster data directory.",
description =>
"This should be the same as the data_directory setting in postgresql.conf. Even though this value can be read " .
"This should be the same as the data_directory setting in postgresql.conf. Even though this value can be read " .
"from postgresql.conf or the database cluster it is prudent to set it in case those resources are not " .
"available during a restore or cold backup scenario.\n" .
"\n" .
@ -235,7 +234,7 @@ my $oConfigHelpData =
summary =>
"Cluster port.",
description =>
"Port that PostgreSQL is running on. This usually does not need to be specified as most database clusters run " .
"Port that PostgreSQL is running on. This usually does not need to be specified as most database clusters run " .
"on the default port."
},
@ -245,9 +244,9 @@ my $oConfigHelpData =
{
section => 'stanza',
summary =>
"cluster unix socket path.",
"Cluster unix socket path.",
description =>
"The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look " .
"The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look " .
"in the standard location for your OS so there usually no need to specify this setting unless the socket " .
"directory was explicitly modified with the unix_socket_directory setting in postgressql.conf."
},
@ -260,8 +259,8 @@ my $oConfigHelpData =
summary =>
"Database query timeout.",
description =>
"Sets the timeout for queries against the database. This includes the pg_start_backup() and pg_stop_backup() " .
"functions which can each take a substantial amount of time. Because of this the timeout should be kept " .
"Sets the timeout for queries against the database. This includes the pg_start_backup() and pg_stop_backup() " .
"functions which can each take a substantial amount of time. Because of this the timeout should be kept " .
"high unless you know that these functions will return quickly (i.e. if you have set startfast=y and you " .
"know that the database cluster will not generate many WAL segments during the backup)."
},
@ -274,9 +273,9 @@ my $oConfigHelpData =
summary =>
"Cluster host logon user when db-host is set.",
description =>
"This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this " .
"to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, " .
"the default."
"This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to " .
"work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the " .
"default."
},
# HARDLINK Option Help
@ -287,8 +286,8 @@ my $oConfigHelpData =
summary =>
"Hardlink files between backups.",
description =>
"Enable hard-linking of files in differential and incremental backups to their full backups. This gives the " .
"appearance that each backup is a full backup. Be careful, though, because modifying files that are " .
"Enable hard-linking of files in differential and incremental backups to their full backups. This gives the " .
"appearance that each backup is a full backup. Be careful, though, because modifying files that are " .
"hard-linked can affect all the backups in the set."
},
@ -336,8 +335,8 @@ my $oConfigHelpData =
summary =>
"Manifest save threshold during backup.",
description =>
"Defines how often the manifest will be saved during a backup (in bytes). Saving the manifest is important " .
"because it stores the checksums and allows the resume function to work efficiently. The actual threshold " .
"Defines how often the manifest will be saved during a backup (in bytes). Saving the manifest is important " .
"because it stores the checksums and allows the resume function to work efficiently. The actual threshold " .
"used is 1% of the backup size or manifest-save-threshold, whichever is greater."
},
@ -349,8 +348,8 @@ my $oConfigHelpData =
summary =>
"Use a neutral umask.",
description =>
"Sets the umask to 0000 so modes in the repository as created in a sensible way. The default directory mode is " .
"0750 and default file mode is 0640. The lock and log directories set the directory and file mode to 0770 " .
"Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is " .
"0750 and default file mode is 0640. The lock and log directories set the directory and file mode to 0770 " .
"and 0660 respectively.\n" .
"\n" .
"To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on " .
@ -365,25 +364,25 @@ my $oConfigHelpData =
summary =>
"Repository path where WAL segments, backups, logs, etc are stored.",
description =>
"The repository serves as both storage and working area for pgBackRest. In a simple installation where the " .
"backups are stored locally to the database server there will be only one repository which will contain " .
"The repository serves as both storage and working area for pgBackRest. In a simple installation where the " .
"backups are stored locally on the database server there will be only one repository which will contain " .
"everything: backups, archives, logs, locks, etc.\n" .
"\n" .
"If the backups are being done remotely then the backup server's repository will contain backups, archives, " .
"locks and logs while the database server's repository will contain only locks and logs. However, if " .
"locks and logs while the database server's repository will contain only locks and logs. However, if " .
"asynchronous archiving is enabled then the database server's repository will also contain a spool " .
"directory for archive logs that have not yet been pushed to the remote repository.\n" .
"\n" .
"Each system where pgBackRest is installed should have a repository directory configured. Storage requirements " .
"vary based on usage. The main backup repository will need the most space as it contains both backups and " .
"WAL segments for whatever retention you have specified. The database repository only needs significant " .
"Each system where pgBackRest is installed should have a repository directory configured. Storage requirements " .
"vary based on usage. The main backup repository will need the most space as it contains both backups and " .
"WAL segments for whatever retention you have specified. The database repository only needs significant " .
"space if asynchronous archiving is enabled and then it will act as an overflow for WAL segments and might " .
"need to be large depending on your database activity.\n" .
"\n" .
"If you are new to backup then it will be difficult to estimate in advance how much space you'll need. The " .
"best thing to do it take some backups then record the size of different types of backups (full/incr/diff) " .
"and measure the amount of WAL generated per day. This will give you a general idea of how much space " .
"you'll need, though of course requirements will change over time as your database evolves."
"If you are new to backup then it will be difficult to estimate in advance how much space you'll need. The best " .
"thing to do is take some backups then record the size of different types of backups (full/incr/diff) and " .
"measure the amount of WAL generated per day. This will give you a general idea of how much space you'll " .
"need, though of course requirements will change over time as your database evolves."
},
# REPO-REMOTE-PATH Option Help
@ -394,8 +393,8 @@ my $oConfigHelpData =
summary =>
"Remote repository path where WAL segments, backups, logs, etc are stored.",
description =>
"The remote repository is relative to the current installation of pgBackRest. On a database server the backup " .
"server will be remote and visa versa for the backup server where the database server will be remote. This " .
"The remote repository is relative to the current installation of pgBackRest. On a database server the backup " .
"server will be remote and vice versa for the backup server where the database server will be remote. This " .
"option is only required if the remote repository has a different path than the local repository."
},
@ -407,8 +406,8 @@ my $oConfigHelpData =
summary =>
"Allow resume of failed backup.",
description =>
"Defines whether the resume feature is enabled. Resume can greatly reduce the amount of time required to run a " .
"backup after a previous backup of the same type has failed. It adds complexity, however, so it may be " .
"Defines whether the resume feature is enabled. Resume can greatly reduce the amount of time required to run a " .
"backup after a previous backup of the same type has failed. It adds complexity, however, so it may be " .
"desirable to disable in environments that do not require the feature."
},
@ -420,12 +419,12 @@ my $oConfigHelpData =
summary =>
"Number of backups worth of WAL to retain.",
description =>
"Number of backups worth of archive log to keep. If this is set less than your backup retention then be sure " .
"Number of backups worth of archive log to keep. If this is set less than your backup retention then be sure " .
"you set archive-copy=y or you won't be able to restore some older backups.\n" .
"\n" .
"For example, if retention-archive=2 and retention-full=4, then any backups older than the most recent two full " .
"backups will not have WAL segments in the archive to make them consistent. To solve this, set " .
"archive-copy=y and use type=none when restoring. This issue will be addressed in a future release but for " .
"backups will not have WAL segments in the archive to make them consistent. To solve this, set " .
"archive-copy=y and use type=none when restoring. This issue will be addressed in a future release but for " .
"now be careful with this setting."
},
@ -438,10 +437,10 @@ my $oConfigHelpData =
"Backup type for WAL retention.",
description =>
"If set to full, then pgBackRest will keep archive logs for the number of full backups defined by " .
"retention-archive. If set to diff (differential), then pgBackRest will keep archive logs for the number " .
"of differential backups defined by retention-archive.\n" .
"retention-archive. If set to diff (differential), then pgBackRest will keep archive logs for the number of " .
"differential backups defined by retention-archive.\n" .
"\n" .
"If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs " .
"If not defined then archive logs will be kept indefinitely. In general it is not useful to keep archive logs " .
"that are older than the oldest backup but there may occasionally be reasons for doing so."
},
@ -454,7 +453,7 @@ my $oConfigHelpData =
"Number of differential backups to retain.",
description =>
"When a differential backup expires, all incremental backups associated with the differential backup will also " .
"expire. When not defined all differential backups will be kept."
"expire. When not defined all differential backups will be kept."
},
# RETENTION-FULL Option Help
@ -466,7 +465,7 @@ my $oConfigHelpData =
"Number of full backups to retain.",
description =>
"When a full backup expires, all differential and incremental backups associated with the full backup will also " .
"expire. When not defined then all full backups will be kept."
"expire. When not defined then all full backups will be kept."
},
# STANZA Option Help
@ -477,11 +476,14 @@ my $oConfigHelpData =
"Command stanza.",
description =>
"A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will " .
"be backed up, archiving options, etc. Most db servers will only have one Postgres database cluster and " .
"be backed up, archiving options, etc. Most db servers will only have one Postgres database cluster and " .
"therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to " .
"be backed up.\n" .
"\n" .
"Examples of how to configure a stanza can be found in the `configuration examples` section."
"It is tempting to name the stanza after the primary cluster but a better name describes the databases " .
"contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more " .
"appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather " .
"than the local cluster name, such as main or prod."
},
# START-FAST Option Help
@ -492,8 +494,8 @@ my $oConfigHelpData =
summary =>
"Force a checkpoint to start backup quickly.",
description =>
"Forces a checkpoint (by passing true to the fast parameter of pg_start_backup()) so the backup begins " .
"immediately. Otherwise the backup will start after the next regular checkpoint.\n" .
"Forces a checkpoint (by passing y to the fast parameter of pg_start_backup()) so the backup begins " .
"immediately. Otherwise the backup will start after the next regular checkpoint.\n" .
"\n" .
"This feature only works in PostgreSQL <= 8.3."
},
@ -512,8 +514,8 @@ my $oConfigHelpData =
"This feature relies on pg_is_in_backup() so only works on PostgreSQL >= 9.3.\n" .
"\n" .
"The setting is disabled by default because it assumes that pgBackRest is the only process doing exclusive " .
"online backups. It depends on an advisory lock that only pgBackRest sets so it may abort other processes " .
"that do exclusive online backups. Note that base_backup and pg_dump are safe to use with this setting " .
"online backups. It depends on an advisory lock that only pgBackRest sets so it may abort other processes " .
"that do exclusive online backups. Note that base_backup and pg_dump are safe to use with this setting " .
"because they do not call pg_start_backup() so are not exclusive."
},
@ -526,8 +528,8 @@ my $oConfigHelpData =
"Restore tablespaces into original or remapped paths.",
description =>
"Defines whether tablespaces will be be restored into their original (or remapped) paths or stored directly " .
"under the pg_tblspc path. Disabling this setting produces compact restores that are convenient for " .
"development, staging, etc. Currently these restores cannot be backed up as pgBackRest expects only links " .
"under the pg_tblspc path. Disabling this setting produces compact restores that are convenient for " .
"development, staging, etc. Currently these restores cannot be backed up as pgBackRest expects only links " .
"in the pg_tblspc path. If no tablespaces are present this this setting has no effect."
},
@ -552,7 +554,7 @@ my $oConfigHelpData =
"Max time a thread can run.",
description =>
"This limits the amount of time (in seconds) that a thread might be stuck due to unforeseen issues executing " .
"the command. Has no affect when thread-max=1."
"the command. Has no affect when thread-max=1."
}
},
@ -661,11 +663,11 @@ my $oConfigHelpData =
"Force a cold backup.",
description =>
"When used with --no-start-stop a backup will be run even if pgBackRest thinks that PostgreSQL is " .
"running. This option should be used with extreme care as it will likely result in a bad backup.\n" .
"running. This option should be used with extreme care as it will likely result in a bad backup.\n" .
"\n" .
"There are some scenarios where a backup might still be desirable under these conditions. For example, " .
"There are some scenarios where a backup might still be desirable under these conditions. For example, " .
"if a server crashes and the database cluster volume can only be mounted read-only, it would be a " .
"good idea to take a backup even if postmaster.pid is present. In this case it would be better to " .
"good idea to take a backup even if postmaster.pid is present. In this case it would be better to " .
"revert to the prior backup and replay WAL, but possibly there is a very important transaction in a " .
"WAL segment that did not get archived."
},
@ -684,10 +686,10 @@ my $oConfigHelpData =
"Perform cold backup.",
description =>
"This option prevents pgBackRest from running pg_start_backup() and pg_stop_backup() on the database " .
"cluster. In order for this to work PostgreSQL should be shut down and pgBackRest will generate an " .
"cluster. In order for this to work PostgreSQL should be shut down and pgBackRest will generate an " .
"error if it is not.\n" .
"\n" .
"The purpose of this option is to allow cold backups. The pg_xlog directory is copied as-is and " .
"The purpose of this option is to allow cold backups. The pg_xlog directory is copied as-is and " .
"archive-check is automatically disabled for the backup."
},
@ -728,13 +730,12 @@ my $oConfigHelpData =
summary =>
"Expire backups that exceed retention.",
description =>
"pgBackRest does backup rotation but is not concerned with when the backups were created. If two full backups " .
"pgBackRest does backup rotation but is not concerned with when the backups were created. If two full backups " .
"are configured for retention, pgBackRest will keep two full backups no matter whether they occur two hours " .
"or two weeks apart.",
option =>
{
'config-remote' => 'section',
'log-level-console' => 'section',
'log-level-file' => 'section',
'repo-path' => 'section',
@ -753,9 +754,9 @@ my $oConfigHelpData =
summary =>
"Get help.",
description =>
"Three levels of help are provided. If no command is specified then general help will be displayed. If a " .
"Three levels of help are provided. If no command is specified then general help will be displayed. If a " .
"command is specified then a full description of the command will be displayed along with a list of valid " .
"options. If an option is specified in addition to a command then the a full description of the option as " .
"options. If an option is specified in addition to a command then the a full description of the option as " .
"it applies to the command will be displayed.",
},
@ -767,11 +768,11 @@ my $oConfigHelpData =
summary =>
"Retrieve information about backups.",
description =>
"The info command operates on a single stanza or all stanzas. Text output is the default and gives a " .
"human-readable summary of backups for the stanza(s) requested. This format is subject to change with any " .
"The info command operates on a single stanza or all stanzas. Text output is the default and gives a " .
"human-readable summary of backups for the stanza(s) requested. This format is subject to change with any " .
"release.\n" .
"\n" .
"For machine-readable output use --output=json. The JSON output contains far more information than the text " .
"For machine-readable output use --output=json. The JSON output contains far more information than the text " .
"output, however this feature is currently experimental so the format may change between versions.",
option =>
@ -836,7 +837,7 @@ my $oConfigHelpData =
summary =>
"Restore using delta.",
description =>
"By default the PostgreSQL data and tablespace directories are expected to be present but empty. This " .
"By default the PostgreSQL data and tablespace directories are expected to be present but empty. This " .
"option performs a delta restore using checksums."
},
@ -847,8 +848,8 @@ my $oConfigHelpData =
summary =>
"Force a restore.",
description =>
"By itself this option forces the PostgreSQL data and tablespace paths to be completely overwritten. " .
"In combination with --delta a timestamp/size delta will be performed instead of using checksums."
"By itself this option forces the PostgreSQL data and tablespace paths to be completely overwritten. In " .
"combination with --delta a timestamp/size delta will be performed instead of using checksums."
},
'log-level-console' => 'section',
@ -863,14 +864,14 @@ my $oConfigHelpData =
"Set an option in recovery.conf.",
description =>
"See http://www.postgresql.org/docs/X.X/static/recovery-config.html for details on recovery.conf " .
"options (replace X.X with your PostgreSQL version). This option can be used multiple times.\n" .
"options (replace X.X with your PostgreSQL version). This option can be used multiple times.\n" .
"\n" .
"Note: The restore_command option will be automatically generated but can be overridden with this " .
"option. Be careful about specifying your own restore_command as pgBackRest is designed to handle " .
"this for you. Target Recovery options (recovery_target_name, recovery_target_time, etc.) are " .
"option. Be careful about specifying your own restore_command as pgBackRest is designed to handle " .
"this for you. Target Recovery options (recovery_target_name, recovery_target_time, etc.) are " .
"generated automatically by pgBackRest and should not be set with this option.\n" .
"\n" .
"Recovery settings can also be set in the restore:recovery-option section of pg_backrest.conf. For " .
"Recovery settings can also be set in the restore:recovery-option section of pg_backrest.conf. For " .
"example:\n" .
"\n" .
"[restore:recovery-option]\n" .
@ -891,7 +892,7 @@ my $oConfigHelpData =
summary =>
"Backup set to restore.",
description =>
"The backup set to be restored. latest will restore the latest backup, otherwise provide the name of " .
"The backup set to be restored. latest will restore the latest backup, otherwise provide the name of " .
"the backup to restore."
},
@ -905,12 +906,12 @@ my $oConfigHelpData =
summary =>
"Modify a tablespace path.",
description =>
"Moves a tablespace to a new location during the restore. This is useful when tablespace locations are " .
"Moves a tablespace to a new location during the restore. This is useful when tablespace locations are " .
"not the same on a replica, or an upgraded system has different mount points.\n" .
"\n" .
"Since PostgreSQL 9.2 tablespace locations are not stored in pg_tablespace so moving tablespaces can be " .
"done with impunity. However, moving a tablespace to the data_directory is not recommended and may " .
"cause problems. For more information on moving tablespaces " .
"done with impunity. However, moving a tablespace to the data_directory is not recommended and may " .
"cause problems. For more information on moving tablespaces " .
"http://www.databasesoup.com/2013/11/moving-tablespaces.html is a good resource."
},
@ -932,8 +933,8 @@ my $oConfigHelpData =
"Stop just before the recovery target is reached.",
description =>
"Defines whether recovery to the target would be exclusive (the default is inclusive) and is only valid " .
"when --type is time or xid. For example, using --target-exclusive would exclude the contents of " .
"transaction 1007 when --type=xid and --target=1007. See the recovery_target_inclusive option in " .
"when --type is time or xid. For example, using --target-exclusive would exclude the contents of " .
"transaction 1007 when --type=xid and --target=1007. See the recovery_target_inclusive option in " .
"the PostgreSQL docs for more information."
},
@ -944,7 +945,7 @@ my $oConfigHelpData =
summary =>
"Resume when recovery target is reached.",
description =>
"Specifies whether recovery should resume when the recovery target is reached. See " .
"Specifies whether recovery should resume when the recovery target is reached. See " .
"pause_at_recovery_target in the PostgreSQL docs for more information."
},
@ -976,7 +977,7 @@ my $oConfigHelpData =
"* time - recover to the time specified in --target.\n" .
"* preserve - preserve the existing recovery.conf file.\n" .
"* none - no recovery.conf file is written so PostgreSQL will attempt to achieve consistency using WAL " .
"segments present in pg_xlog. Provide the required WAL segments or use the archive-copy setting to " .
"segments present in pg_xlog. Provide the required WAL segments or use the archive-copy setting to " .
"include them with the backup."
}
}
@ -990,7 +991,7 @@ my $oConfigHelpData =
"Allow pgBackRest processes to run.",
description =>
"If the pgBackRest processes were previously stopped using the stop command then they can be started again " .
"using the start command. Note that this will not immediately start up any pgBackRest processes but they " .
"using the start command. Note that this will not immediately start up any pgBackRest processes but they " .
"are allowed to run.",
option =>
@ -1007,8 +1008,8 @@ my $oConfigHelpData =
summary =>
"Stop pgBackRest processes from running.",
description =>
"Does not allow any new pgBackRest processes to run. By default running processes will be allowed to complete " .
"successfully. Use the --force option to terminate running processes.\n" .
"Does not allow any new pgBackRest processes to run. By default running processes will be allowed to complete " .
"successfully. Use the --force option to terminate running processes.\n" .
"\n" .
"pgBackRest processes will return an error if they are run after the stop command completes.",
@ -1022,8 +1023,8 @@ my $oConfigHelpData =
"Force all pgBackRest processes to stop.",
description =>
"This option will send TERM signals to all running pgBackRest processes to effect a graceful but " .
"immediate shutdown. Note that this will also shutdown processes that were initiated on another " .
"system but have remotes running on the current system. For instance, if a backup was started on " .
"immediate shutdown. Note that this will also shutdown processes that were initiated on another " .
"system but have remotes running on the current system. For instance, if a backup was started on " .
"the backup server then running stop --force on the database server will shutdown the backup " .
"process on the backup server."
},

56
lib/BackRest/FileCommon.pm
View File

@ -23,6 +23,7 @@ use BackRest::Common::Log;
use constant OP_FILE_COMMON => 'FileCommon';
use constant OP_FILE_COMMON_PATH_SYNC => OP_FILE_COMMON . '::filePathSync';
use constant OP_FILE_COMMON_STRING_READ => OP_FILE_COMMON . '::fileStringRead';
use constant OP_FILE_COMMON_STRING_WRITE => OP_FILE_COMMON . '::fileStringWrite';
####################################################################################################################################
@ -61,6 +62,59 @@ sub filePathSync
push @EXPORT, qw(filePathSync);
####################################################################################################################################
# fileStringRead
#
# Read the specified file as a string.
####################################################################################################################################
sub fileStringRead
{
# Assign function parameters, defaults, and log debug info
my
(
$strOperation,
$strFileName
) =
logDebugParam
(
OP_FILE_COMMON_STRING_READ, \@_,
{name => 'strFileName', trace => true}
);
# Open the file for writing
sysopen(my $hFile, $strFileName, O_RDONLY)
or confess &log(ERROR, "unable to open ${strFileName}");
# Read the string
my $iBytesRead;
my $iBytesTotal = 0;
my $strContent;
do
{
$iBytesRead = sysread($hFile, $strContent, 65536, $iBytesTotal);
if (!defined($iBytesRead))
{
confess &log(ERROR, "unable to read string from ${strFileName}: $!", ERROR_FILE_READ);
}
$iBytesTotal += $iBytesRead;
}
while ($iBytesRead != 0);
close($hFile);
# Return from function and log return values if any
return logDebugReturn
(
$strOperation,
{name => 'strContent', value => $strContent, trace => true}
);
}
push @EXPORT, qw(fileStringRead);
####################################################################################################################################
# fileStringWrite
#
@ -90,7 +144,7 @@ sub fileStringWrite
# Write the string
syswrite($hFile, $strContent)
or confess "unable to write string: $!";
or confess &log(ERROR, "unable to write string to ${strFileName}: $!", ERROR_FILE_WRITE);
# Sync and close ini file
if ($bSync)

2
lib/BackRest/Manifest.pm
View File

@ -457,6 +457,8 @@ sub build
# Skip certain files during backup
if (($strName =~ /^pg\_xlog\/.*/ && !$bNoStartStop) || # pg_xlog/ - this will be reconstructed
$strName =~ /^postmaster\.pid$/ || # postmaster.pid - to avoid confusing postgres when restoring
$strName =~ /^backup\_label\.old$/ || # backup_label.old - old backup labels are not useful
$strName =~ /^recovery\.done$/ || # recovery.done - doesn't make sense to backup this file
$strName =~ /^recovery\.conf$/) # recovery.conf - doesn't make sense to backup this file
{
next;

3
test/lib/BackRestTest/BackupCommonTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# BackupCommonTest.pl - Common code for backup unit tests
# BackupCommonTest.pm - Common code for backup unit tests
####################################################################################################################################
package BackRestTest::BackupCommonTest;

3
test/lib/BackRestTest/BackupTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# BackupTest.pl - Unit Tests for BackRest::Backup and BackRest::Restore
# BackupTest.pm - Unit Tests for BackRest::Backup and BackRest::Restore
####################################################################################################################################
package BackRestTest::BackupTest;

6
test/lib/BackRestTest/Common/ExecuteTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# ExecuteTest.pl - Module to execute external commands
# ExecuteTest.pm - Module to execute external commands
####################################################################################################################################
package BackRestTest::Common::ExecuteTest;
@ -64,6 +63,7 @@ sub new
# Set defaults
$self->{bRemote} = defined($self->{bRemote}) ? $self->{bRemote} : false;
$self->{bSuppressError} = defined($self->{bSuppressError}) ? $self->{bSuppressError} : false;
$self->{bSuppressStdErr} = defined($self->{bSuppressStdErr}) ? $self->{bSuppressStdErr} : false;
$self->{bShowOutput} = defined($self->{bShowOutput}) ? $self->{bShowOutput} : false;
$self->{iExpectedExitStatus} = defined($self->{iExpectedExitStatus}) ? $self->{iExpectedExitStatus} : 0;
@ -233,7 +233,7 @@ sub end
}
}
if ($self->{strErrorLog} ne '')
if ($self->{strErrorLog} ne '' && !$self->{bSuppressStdErr} && !$self->{bSuppressError})
{
confess &log(ERROR, "output found on STDERR:\n$self->{strErrorLog}");
}

3
test/lib/BackRestTest/Common/LogTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# LogTest.pl - Capture the output of commands to compare them with an expected version
# LogTest.pm - Capture the output of commands to compare them with an expected version
####################################################################################################################################
package BackRestTest::Common::LogTest;

1
test/lib/BackRestTest/CommonTest.pm
View File

@ -1,4 +1,3 @@
#!/usr/bin/perl
####################################################################################################################################
# CommonTest.pm - Common globals used for testing
####################################################################################################################################

3
test/lib/BackRestTest/CompareTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# CompareTest.pl - Performance comparison tests between rsync and backrest
# CompareTest.pm - Performance comparison tests between rsync and backrest
####################################################################################################################################
package BackRestTest::CompareTest;

3
test/lib/BackRestTest/ConfigTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# ConfigTest.pl - Unit Tests for BackRest::Param and BackRest::Config::Config
# ConfigTest.pm - Unit Tests for BackRest::Param and BackRest::Config::Config
####################################################################################################################################
package BackRestTest::ConfigTest;

3
test/lib/BackRestTest/ExpireCommonTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# ExpireCommonTest.pl - Common code for expire tests
# ExpireCommonTest.pm - Common code for expire tests
####################################################################################################################################
package BackRestTest::ExpireCommonTest;

3
test/lib/BackRestTest/FileTest.pm
View File

@ -1,6 +1,5 @@
#!/usr/bin/perl
####################################################################################################################################
# FileTest.pl - Unit Tests for BackRest::File
# FileTest.pm - Unit Tests for BackRest::File
####################################################################################################################################
package BackRestTest::FileTest;

1
test/lib/BackRestTest/HelpTest.pm
View File

@ -1,4 +1,3 @@
#!/usr/bin/perl
####################################################################################################################################
# HelpTest.pm - Unit Tests for help
####################################################################################################################################

1
test/lib/BackRestTest/IniTest.pm
View File

@ -1,4 +1,3 @@
#!/usr/bin/perl
####################################################################################################################################
# IniTest.pm - Unit Tests for ini load and save
####################################################################################################################################

12
test/vm/Vagrantfile vendored
View File

@ -5,7 +5,7 @@ Vagrant.configure(2) do |config|
end
config.vm.define "u12" do |u12|
u12.vm.box = "ubuntu/precise64"
u12.vm.box = "boxcutter/ubuntu1204"
u12.vm.provider :virtualbox do |vb|
vb.name = "backrest-test-ubuntu-12.04"
@ -36,15 +36,15 @@ Vagrant.configure(2) do |config|
/backrest/test/vm/ssh/setup-cm.sh
# Install required Perl modules
apt-get install -y libdbd-pg-perl
apt-get install libdbi-perl libdbd-pg-perl
# Install Perl modules required for building the docs
apt-get install -y libxml-checker-perl
apt-get install -y libxml-checker-perl libxml-writer-perl
SHELL
end
config.vm.define "u14" do |u14|
u14.vm.box = "ubuntu/trusty64"
u14.vm.box = "boxcutter/ubuntu1404"
u14.vm.provider :virtualbox do |vb|
vb.name = "backrest-test-ubuntu-14.04"
@ -75,10 +75,10 @@ Vagrant.configure(2) do |config|
/backrest/test/vm/ssh/setup-cm.sh
# Install required Perl modules
apt-get install -y libdbd-pg-perl
apt-get install libdbi-perl libdbd-pg-perl
# Install Perl modules required for building the docs
apt-get install -y libxml-checker-perl
apt-get install -y libxml-checker-perl libxml-writer-perl
SHELL
end