1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

657 Commits

Author SHA1 Message Date
David Steele
206415d4c7 Fixed an issue that could cause compression to abort on growing files.
Reported by Jesper St John, Aleksandr Rogozin.
2017-08-30 16:34:05 -04:00
David Steele
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process.
Reported by William Cox.
2017-08-30 08:41:46 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
8d3436bce7 Merge release 2 container build changes. 2017-08-19 09:02:24 -04:00
David Steele
7dab459cac Simplified smart logic for C Library and package builds. 2017-08-19 08:48:13 -04:00
Cynthia Shang
1be69d5ac7 Fixed document generation to include section summaries on the Configuration page.
Fixed by Cynthia Shang.
2017-08-16 16:54:35 -04:00
David Steele
1d2b08ce94 Move contributor list to the end of release.xml for convenience. 2017-08-11 18:43:21 -04:00
David Steele
4c2c612b6f Ignore auto-generated C code created by release 2. 2017-08-11 18:38:40 -04:00
David Steele
4331e09376 Drain stderr during test process execution as well as termination to prevent lockups if there is a lot of output. 2017-08-11 18:31:25 -04:00
David Steele
f8de73674c Updated Docker build in Vagrantfile. 2017-08-10 20:52:29 -04:00
David Steele
ac1574448e Changed log test order to ignore unimportant log errors while shutting down PostgreSQL. 2017-08-10 18:11:29 -04:00
David Steele
84caca0f34 v1.22: Fixed S3 Retry
Bug Fixes:

* Fixed authentication issue in S3 retry.
2017-08-09 11:50:02 -04:00
David Steele
61c38f5808 Fixed authentication issue in S3 retry. 2017-08-09 11:27:09 -04:00
David Steele
cbc026418c v1.21: Improved Info Output and SSH Port Option
Bug Fixes:

* The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
* Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang.)

Features:

* Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang.)
* Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang.)

Refactoring:

* Retry when S3 returns an internal error (500).
* Add bIgnoreMissing parameter to Local->manifest().
2017-08-08 18:09:07 -04:00
David Steele
038d47bcc0 Retry when S3 returns an internal error (500). 2017-08-08 17:15:01 -04:00
David Steele
961f7033d1 Generate global fake cert in containers for testing. 2017-08-08 16:26:44 -04:00
Cynthia Shang
b6da509c9f Consolidate stanza-create and stanza-upgrade tests into new stanza test.
Contributed by Cynthia Shang.
2017-08-04 16:10:51 -04:00
Cynthia Shang
f9bdc092ba Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade.
Fixed by Cynthia Shang.
2017-08-04 16:03:55 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports.
Contributed by Cynthia Shang.
2017-08-01 10:34:53 -04:00
David Steele
cf3110b0d8 Use new Travis Trusty image. 2017-07-26 19:22:54 -04:00
David Steele
14d6974d1b Use Google DNS in test environment for consistency. 2017-07-26 13:40:09 -04:00
David Steele
97c4d9ec18 Reduce log verbosity in doc.pl.
Reduce verbosity by only logging sections that contain an execute list directly or in a child section.
2017-07-26 10:22:22 -04:00
David Steele
df7d633683 Improvements to command/command-line help:
* Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
* Fix description of --online based on the command context.
* Remove vestigial repository options from backup command.
2017-07-26 09:33:10 -04:00
David Steele
156fd4d54d Add bIgnoreMissing parameter to Local->manifest(). 2017-07-25 12:44:38 -04:00
David Steele
49449feab3 Add creation of /etc/pgbackrest.conf to manual installation instructions. 2017-07-24 21:14:42 -04:00
David Steele
cbb749e12a Debian/Ubuntu documentation now builds on Ubuntu 16. 2017-07-24 21:12:05 -04:00
David Steele
d5c1f02c72 Include archive_status directory in online backups.
The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do.

Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log.

Reported by Stephen Frost.
2017-07-24 07:57:47 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command.
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
2017-07-21 17:32:41 -04:00
David Steele
33d769e147 v1.20: Critical 8.3/8.4 Bug Fix
Bug Fixes:

* Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
* Fixed missing flag in C library build that resulted in a mismatched binary on 32-bit systems. (Reported by Adrian Vondendriesch.)

Features:

* Add s3-repo-ca-path and s3-repo-ca-file options to accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs. (Suggested by Scott Frazer.)

Refactoring:

* Harden protocol handshake to handle race conditions.
* Fixed misleading error message when a file was opened for write in a missing directory.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
* Rename Archive modules to remove redundancy.
* Improve S3 error reporting.
* Minor optimizations to package loads and ordering for archive-get and archive-push commands.
2017-06-27 22:42:31 -04:00
David Steele
884413b251 Remove mount and user from host cache-key because it can vary by system. 2017-06-27 22:27:48 -04:00
David Steele
7759a94fa2 Add Debian 9 to test suite. 2017-06-27 18:29:48 -04:00
David Steele
9e9286ee20 Add performance module and basic performance test for archive-push. 2017-06-27 18:00:39 -04:00
David Steele
9e248a6627 Minor optimizations to package loads and ordering for archive-get and archive-push commands. 2017-06-27 17:54:30 -04:00
David Steele
f3b62d2d67 Fixed misleading error message when a file was opened for write in a missing directory. 2017-06-27 17:07:12 -04:00
David Steele
2310e423e9 Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
The integration tests that were supposed to prevent this regression did not work as intended.  They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace.  All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid.

However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions.  Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore.

The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again.

Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 16:47:40 -04:00
David Steele
fdabf33604 Revert removal of Ubuntu 12.04 and PostgreSQL 8.3. 2017-06-27 15:58:02 -04:00
David Steele
001cff9eb9 Automatically check that all supported PostgreSQL versions are being tested on a single default VM. 2017-06-27 15:48:34 -04:00
David Steele
7a1385cc27 Add documentation builds to CI. 2017-06-24 18:04:02 -04:00
David Steele
7a9b1f6993 Container build improvements:
* More optimized container suite that greatly improves build time.
* Added static Debian packages for Devel::Cover to reduce build time.
* Add deprecated state for containers. Deprecated containers may only be used to build packages.
* Remove Debian 8 from CI because it does not provide additional coverage over Ubuntu 14.04 and Ubuntu 16.04.
2017-06-24 10:59:00 -04:00
David Steele
53184a7b30 Update CentOS 6 documentation to build on PostgreSQL 9.5. 2017-06-23 10:30:22 -04:00
David Steele
3d224ad8fa Improve S3 server implementation in documentation. 2017-06-23 10:10:48 -04:00
David Steele
eb4325fa9a Remove exhaustive version list from Stable Releases TOC. 2017-06-22 19:04:52 -04:00
David Steele
a10a22127d Consolidate archive-get test into new archive test. 2017-06-22 18:51:39 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options.
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
2017-06-22 18:22:49 -04:00
David Steele
a1a5d18aa4 Balance database versions between VMs to minimize test duration. 2017-06-22 14:44:21 -04:00
David Steele
d3054d919b Fixed missing flag in C library build.
The missing flag resulted in a mismatched binary on 32-bit systems.

Reported by Adrian Vondendriesch.
2017-06-22 12:57:10 -04:00
David Steele
f8849d7359 Remove patch directory before Debian package builds.
Patches should be merged into master before trying a new build after a release with patches.
2017-06-22 12:37:21 -04:00
David Steele
f596702c5b Improve S3 error reporting. 2017-06-21 20:46:49 -04:00
David Steele
8b9965abcb Remove process-max option.
Parallelism is now tested in a more targeted manner and the high level option is no longer needed.
2017-06-21 19:21:16 -04:00
David Steele
cc0e3f843e Remove support for PostgreSQL 8.3. 2017-06-21 17:24:36 -04:00
David Steele
73274fc608 Remove Ubuntu 12.04 from CI now that it is EOL (it can still be run manually). 2017-06-21 17:02:01 -04:00
David Steele
c69baa2de1 Install sudo in base containers rather than on demand. 2017-06-21 16:51:41 -04:00
David Steele
64ec7eb490 Eliminate redundancy in real tests. 2017-06-21 16:07:13 -04:00
David Steele
2337be8636 Consolidate archive-push tests into new archive test. 2017-06-21 08:38:53 -04:00
David Steele
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
David Steele
692b1e274a Split full module into mock and real to allow better test combinations and save time in CI. 2017-06-19 18:55:00 -04:00
David Steele
f6d4457d58 Full/Synthetic test refactor.
* Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
2017-06-15 15:32:10 -04:00
David Steele
269ae4556d Fix timeouts in ExecuteTest to speed multi-process testing. 2017-06-15 15:20:02 -04:00
David Steele
6e29f616b3 Harden protocol handshake to handle race conditions. 2017-06-15 15:12:28 -04:00
David Steele
c10c5bb0ef v1.19: S3 Support
Bug Fixes:

* Fixed the info command so the WAL archive min/max displayed is for the current database version. (Fixed by Cynthia Shang.)
* Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable. (Fixed by Cynthia Shang.)
* Fixed config warnings raised from a remote process causing errors in the master process. (Fixed by Cynthia Shang.)

Features:

* Amazon S3 repository support. (Reviewed by Cynthia Shang.)

Refactoring:

* Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
* Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
* Move modules in Protocol directory in subdirectories.
* Move backup modules into Backup directory.
2017-06-12 11:36:00 -04:00
David Steele
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
David Steele
de7fc37f88 Storage and IO layer refactor:
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 17:51:41 -04:00
Cynthia Shang
7e982f05f5 Fixed config warnings raised from a remote process causing errors in the master process.
Fixed by Cynthia Shang.
2017-05-16 11:28:51 -04:00
David Steele
74053ed842 Fixed missing sudo in installation section.
Fixed by Lætitia.
2017-05-16 11:15:31 -04:00
David Steele
5296747271 Move backup modules into Backup directory. 2017-05-15 16:01:00 -04:00
David Steele
2a23157bf2 Move modules in Protocol directory in subdirectories. 2017-05-15 11:12:14 -04:00
David Steele
33c87b2a1c Remove flapping protocol timeout test that will be replaced in the upcoming storage patch. 2017-05-15 10:24:46 -04:00
David Steele
c31da3f961 Moved test and env modules to new directories to avoid namespace conflicts with common tests. 2017-05-12 16:43:04 -04:00
David Steele
4049d59c9e Fixed issue with --dry-run requiring --vm-out to work properly. 2017-05-12 16:27:06 -04:00
Cynthia Shang
7d0c50fcd6 Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable.
Fixed by Cynthia Shang.
2017-05-12 16:05:44 -04:00
Cynthia Shang
5bf4f52e3d Fixed the info command so the WAL archive min/max displayed is for the current database version.
Fixed by Cynthia Shang.
2017-05-12 15:49:14 -04:00
David Steele
64be6400fe Changed invalid max-archive-mb option in configuration reference to archive-queue-max. 2017-04-17 06:39:29 -04:00
David Steele
979610a07d Fixed an undefined variable when a module had no uncoverable code exceptions. 2017-04-13 20:10:28 -04:00
David Steele
77ef4fde20 Set --vm-max=2 for CI. 2017-04-13 07:42:45 -04:00
David Steele
76161bd845 v1.18: Stanza Upgrade, Refactoring, and Locking Improvements
Bug Fixes:

* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)

Features:

* Added the stanza-upgrade command to provide a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. (Contributed by Cynthia Shang.)
* Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. (Contributed by Cynthia Shang.)

Refactoring:

* Simplify locking scheme. Now, only the master process will hold write locks (for archive-push and backup commands) and not all local and remote worker processes as before.
* Refactor Ini.pm to facilitate testing.
* Do not set timestamps of files in the backup directories to match timestamps in the cluster directory. This was originally done to enable backup resume, but that process is now implemented with checksums.
* Improved error message when the restore command detects the presence of postmaster.pid. (Suggested by Yogesh Sharma.)
* Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. (Suggested by Yogesh Sharma.)
* The backup and restore commands no longer copy via temp files. In both cases the files are checksummed on resume so there's no danger of partial copies.
* Allow functions to accept optional parameters as a hash.
* Refactor File->list() and fileList() to accept optional parameters.
* Refactor backupLabel() and add unit tests.
* Silence some perl critic warnings. (Contributed by Cynthia Shang.)
2017-04-12 19:17:39 -04:00
David Steele
5c635e0f0a Go back to using static user for documentation.
Making this dynamic in commit 5d2e792 broke doc builds from cache.  The long-term solution is to create a special user for doc builds but that’s beyond the scope of this release.
2017-04-12 18:36:33 -04:00
David Steele
f207dc7123 Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions.
Suggested by Yogesh Sharma.
2017-04-12 12:41:34 -04:00
David Steele
f5f8d51366 Improved error message when the restore command detects the presence of postmaster.pid.
Suggested by Yogesh Sharma.
2017-04-12 09:53:07 -04:00
David Steele
b0119914d4 Do not set timestamps of files in the backup directories to match timestamps in the cluster directory.
This was originally done to enable backup resume, but that process is now implemented with checksums.
2017-04-12 09:22:51 -04:00
David Steele
fe02b40293 Backup and restore commands no longer copy via temp files.
In both cases the files are checksummed on resume so there's no danger of partial copies.
2017-04-12 09:05:57 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings.
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
2017-04-10 17:23:39 -04:00
David Steele
8546571461 Ignore clock skew in container libc/package builds using make.
It is common for containers to have clock skew so the build process takes care of this issue independently.
2017-04-10 17:20:21 -04:00
Cynthia Shang
b63ede5614 Added ability to test warning messages.
Contributed by Cynthia Shang.
2017-04-10 13:53:19 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm.
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
2017-04-10 13:24:45 -04:00
David Steele
367e06f4be Coverage testing always enabled on Debian-based containers.
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
2017-04-10 12:31:30 -04:00
David Steele
eb4918d018 Fixed locking bug and refactoring.
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
2017-04-07 10:31:49 -04:00
David Steele
edfb0962b6 Update wording for release note sections. 2017-04-06 21:47:43 -04:00
Cynthia Shang
b620e6b282 Improvements and additional tests for stanza-upgrade.
Contributed by Cynthia Shang.
2017-04-06 19:36:37 -04:00
David Steele
0da216c034 Require description in every call to testResult() and make iWaitSeconds optional. 2017-04-05 10:50:32 -04:00
David Steele
c7b61a2002 Refactor backupLabel() and add unit tests. 2017-04-05 10:16:16 -04:00
David Steele
b2f43b5601 Allow functions to accept optional parameters as a hash.
Refactor File->list() and fileList() to accept optional parameters.
2017-04-04 21:17:19 -04:00
David Steele
9eae36a702 Fixed flapping archive stop tests. 2017-04-04 21:15:22 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command.
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
2017-04-03 10:42:55 -04:00
Cynthia Shang
af7e4f4b4b Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section.
Contributed by Cynthia Shang.
2017-03-22 19:03:28 -04:00
David Steele
5d2e79242d Fixed user change in docs missed in vagrant upgrade. 2017-03-16 19:50:59 -04:00
David Steele
cbd879a983 Updated vagrant to new version and image. 2017-03-14 22:05:17 -04:00
David Steele
cc9f8fc4b6 v1.17: Page Checksum Bug Fix
Bug Fixes:

* Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. (Reported by Stephen Frost.)
2017-03-13 16:54:51 -04:00
David Steele
b0947fdaa8 Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings.
Reported by Stephen Frost.
2017-03-13 15:50:13 -04:00
David Steele
3cec6702e8 v1.16: Page Checksum Improvements, CI, and Package Testing
Bug Fixes:

* Fixed an issue where tables over 1GB would report page checksum warnings after the first segment. (Reported by Stephen Frost.)
* Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. (Reported by blogh.)

Refactoring:

* Improved the code and tests for fileManifest() to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-03-02 20:15:14 -05:00
David Steele
02730526fc Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned.
Reported by blogh.
2017-03-02 13:50:29 -05:00
David Steele
5957f8566f Archive test improvements:
* Fixed flapping archive-stop test with testResult() retries.
* Added final test of archive contents to archive-push test.
2017-03-02 11:47:23 -05:00
David Steele
5eb585c9ba Improved truncation when outputting errors logs in the ExecuteTest module. 2017-03-01 14:38:05 -05:00
David Steele
c2666cc7f3 Fixed an issue where tables over 1GB would report page checksum errors after the first segment.
Reported by Stephen Frost.
2017-03-01 14:15:46 -05:00
David Steele
8161ab1c50 Temporarily disable flapping keep-alive test. 2017-02-26 12:58:48 -05:00
David Steele
31d2d790bd Added --retry option. 2017-02-21 22:10:02 -05:00
David Steele
14ca4dac6f Improved the code and tests for fileManifest().
Done to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-02-21 11:56:55 -05:00
David Steele
01312f6260 Improved formatting of testResult() output. 2017-02-21 11:47:45 -05:00
David Steele
86fee3da18 Continuous integration using travis-ci. 2017-02-21 08:59:23 -05:00
David Steele
56543f7a99 Added package builds to test suite and other improvements:
* Automated builds of Debian packages for all supported distributions.
* Added --dev option to aggregate commonly used dev options.
* Added --no-package option to skip package builds.
* C library and packages are built by default, added -smart option to rebuild only when file changes are detected.
* The --libc-only option has been changed to --build-only now that packages builds have been added.
2017-02-17 22:31:16 -05:00
David Steele
ac7ff755b3 Container executions now load the user's environment. 2017-02-16 17:18:48 -05:00
David Steele
2d7d9626bb v1.15: Refactoring and Bug Fixes
Bug Fixes:

* Fixed a regression introduced in v1.13 that could cause backups to fail if files were removed (e.g. tables dropped) while the manifest was being built. (Reported by Navid Golpayegani.)

Refactoring:

* Refactor FileCommon::fileManifest() and FileCommon::fileStat to be more modular to allow complete branch/statement level coverage testing.
2017-02-13 20:00:30 -05:00
David Steele
e2ac7e1ea6 Fixed a regression introduced in v1.13 that could cause backups to fail.
This happened if files were removed (e.g. tables dropped) while the manifest was being built.

Reported by Navid Golpayegani.
2017-02-13 19:59:14 -05:00
David Steele
845c6112bf v1.14: Bug Fixes
Bug Fixes:

* Fixed an issue where an archive-push error would not be retried and would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted). (Reported by Jens Wilke.)
* Fixed a race condition in parallel archiving where creation of new paths generated an error when multiple processes attempted to do so at the same time. (Reported by Jens Wilke.)

Refactoring:

* Improved performance of wal archive min/max provided by the info command. (Suggested by Jens Wilke.)
2017-02-13 10:07:04 -05:00
David Steele
a54018d573 Improved performance of wal archive min/max provided by the info command.
Suggested by Jens Wilke.
2017-02-13 10:03:02 -05:00
David Steele
adcc7cb954 Fixed a race condition in parallel archiving.
Creation of new paths generated an error when multiple processes attempted to do so at the same time.

Reported by Jens Wilke.
2017-02-13 09:58:50 -05:00
David Steele
bc2bb040c5 Fixed an issue where an archive-push error would not be retried.
It would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted).

Reported by Jens Wilke.
2017-02-13 09:57:10 -05:00
David Steele
12041f73e2 Fixed syntax from d68b48d that was not backwards compatible to Perl 5.10. 2017-02-12 21:09:34 -05:00
David Steele
d68b48d19b Improved support for --require option and section depends now default to the previous section. 2017-02-11 10:26:54 -05:00
David Steele
498f52da09 Improvements to documentation engine:
* Documentation can now be built with reusable blocks to reduce duplication.
* Added ability to pass options to containers within the documentation.
* Add proper tag to slightly emphasize proper nouns.
2017-02-10 10:22:05 -05:00
David Steele
aab763d558 Additional notes on the async spool directory. 2017-02-08 09:37:17 -05:00
David Steele
b813bfaff6 Updated async archiving documentation.
More accurately describe how the new method works and how it differs from the old method.

Suggested by Jens Wilke.
2017-02-08 08:41:43 -05:00
David Steele
41eba95155 v1.13: Parallel Archiving, Stanza Create, Improved Info and Check
IMPORTANT NOTE: The new implementation of asynchronous archiving no longer copies WAL to a separate queue. If there is any WAL left over in the old queue after upgrading to 1.13, it will be abandoned and not pushed to the repository.

To prevent this outcome, stop archiving by setting archive_command = false. Next, drain the async queue by running pgbackrest --stanza=[stanza-name] archive-push and wait for the process to complete. Check that the queue in [spool-path]/archive/[stanza-name]/out is empty. Finally, install 1.13 and restore the original archive_command.

IMPORTANT NOTE: The stanza-create command is not longer optional and must be executed before backup or archiving can be performed on a new stanza. Pre-existing stanzas do not require stanza-create to be executed.

Bug Fixes:

* Fixed const assignment giving compiler warning in C library. (Fixed by Adrian Vondendriesch.)
* Fixed a few directory syncs that were missed for the --repo-sync option.
* Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner(). (Reported by Leonardo Avellar.)
* Fixed an issue where protocol mismatch errors did not output the expected value.
* Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination.

Features:

* Improved, multi-process implementation of asynchronous archiving.
* Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. (Contributed by Cynthia Shang.)
* Improved check command to run on a standby, though only basic checks are done because pg_switch_xlog() cannot be executed on a replica. (Contributed by Cynthia Shang.)
* Added archive and backup WAL ranges to the info command.
* Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2. (Contributed by blogh.)
* Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations. (Suggested by Michael Vitale.)

Refactoring:

* Refactor File and BackupCommon modules to improve test coverage.
* Return proper error code when unable to convert a relative path to an absolute path. (Suggested by Yogesh Sharma.)
* Log file banner is not output until the first log entry is written. (Suggested by Jens Wilke.)
* Moved File->manifest() into the FileCommon.pm module.
* Moved the Archive modules to the Archive directory and split the archive-get and archive-push commands into separate modules.
* Split the check command out of the Archive.pm module.
* Allow logging to be suppressed via logDisable() and logEnable().
* Allow for locks to be taken more than once in the same process without error.
* Lock directories can be created when more than one directory level is required.
* Clean up optionValid()/optionTest() logic in Lock.pm.
* Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic.
* Represent .gz extension with a constant.
* Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads.
* Refactor process IO and process master/minion code out from the common protocol code.
* Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
* Remove Intel-specific optimization from C library build flags. (Contributed by Adrian Vondendriesch.)
* Removed --lock option. This option was introduced before the lock directory could be located outside the repository and is now obsolete.
* Added --log-timestamp option to allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
* Fixed alignment issues with multiline logging.
2017-02-05 20:23:03 -05:00
David Steele
0daeace1cd Fixed alignment issues with multiline logging. 2017-02-05 19:58:52 -05:00
David Steele
cded35791d Added --log-timestamp option.
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
2017-02-05 10:59:49 -05:00
David Steele
55feae645e Log file banner is not output until the first log entry is written.
Suggested by Jens Wilke.
2017-02-05 10:37:54 -05:00
David Steele
2237c3fc1b Added documentation for parallel archive-push. 2017-02-05 10:35:53 -05:00
David Steele
8b5f37c2a8 Added archive and backup WAL ranges to the info command. 2017-02-05 10:31:29 -05:00
David Steele
1666ee177f Fixed --no-online tests to suppress expected errors. 2017-02-02 20:44:42 -05:00
David Steele
efdf64506a Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations.
Suggested by Michael Vitale.
2017-02-02 20:42:45 -05:00
David Steele
3e54aad88f Removed --lock option.
This option was introduced before the lock directory could be located outside the repository and is now obsolete.
2017-01-31 10:38:45 -05:00
David Steele
7418559a67 Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination. 2017-01-31 10:36:59 -05:00
Adrian Vondendriesch
9b92edddbe Remove Intel-specific optimization from C library build flags.
Contributed by Adrian Vondendriesch.
2017-01-30 17:06:35 -05:00
David Steele
6b99d770b5 Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN. 2017-01-30 13:59:00 -05:00
David Steele
43ada9fb33 Improved, multi-process implementation of asynchronous archiving. 2017-01-27 11:02:27 -05:00
David Steele
2a9ba877e1 Refactor process IO and process master/minion code out from the common protocol code. 2017-01-27 10:06:16 -05:00
David Steele
0163d3b3e0 Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads. 2017-01-27 10:04:41 -05:00
David Steele
2553c7a998 Represent .gz extension with a constant. 2017-01-27 10:02:00 -05:00
David Steele
f3ccf3b897 Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic. 2017-01-27 09:56:26 -05:00
David Steele
6620594b77 Various improvements in Lock.pm:
* Allow for locks to be taken more than once in the same process without error.
* Lock directories can be created when more than one directory level is required.
* Clean up optionValid()/optionTest() logic in Lock.pm.
2017-01-27 09:53:46 -05:00
David Steele
18db178ac3 Various improvements to the test suite:
* Allow logging to be suppressed via logDisable() and logEnable().
* Added more flexibility in initializing and cleaning up after modules and tests.
* testResult() suppresses logging and reports exceptions.
* testException() allows messages to be matched with regular expressions.
* Refactor name/locations of common modules that setup test environments.
2017-01-27 09:42:30 -05:00
David Steele
e876a222fc Fixed an issue where protocol mismatch errors did not output the expected value. 2017-01-24 14:58:55 -05:00
Benoit Lobréau (blogh)
b6fa146f9b Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2.
Contributed by blogh.
2017-01-22 19:29:56 -05:00
David Steele
7d1347cdec Split the archive-get and archive-push commands into separate modules. 2017-01-10 19:59:32 -05:00
David Steele
155c808163 Split the check command out of the Archive.pm module. 2017-01-10 19:54:51 -05:00
David Steele
b58045f575 Moved the Archive modules to the Archive directory. 2017-01-10 19:51:20 -05:00
David Steele
2ab63f27b2 C Library builds only run when C library has actually changed. 2017-01-10 13:12:53 -05:00
David Steele
0b4055e588 Moved File->manifest() into the FileCommon.pm module. 2017-01-10 13:08:03 -05:00
David Steele
0e4f51c271 Added integration for testing coverage with Devel::Cover. 2017-01-09 20:49:04 -05:00
David Steele
39744a4f1d Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner().
Reported by Leonardo Avellar.
2017-01-05 15:19:56 -05:00
David Steele
4ff2714bf0 Return proper error code when unable to convert a relative path to an absolute path.
Suggested by Yogesh Sharma.
2017-01-04 18:39:50 -05:00
David Steele
e3d63c0252 Allow multiple --module, --test, and --run options to be used for test.pl. 2017-01-04 18:29:13 -05:00
David Steele
b91ca11329 Added expect log expression to replace year subdirectories in backup.history. 2017-01-04 10:09:21 -05:00
David Steele
314ef2c8b3 Update LICENSE.txt for 2017. 2017-01-04 10:05:43 -05:00
Cynthia Shang
d5ee7d3535 Added documentation to the User Guide for the process-max option.
Contributed by Cynthia Shang.
2016-12-23 16:50:02 -05:00
David Steele
83beab7ec3 Refactor File module to improve test coverage.
Added unit tests for low-level functions in the BackupCommon module.
2016-12-23 11:43:26 -05:00
David Steele
5d3473b52d Refactor File module to improve test coverage. 2016-12-23 08:30:34 -05:00
David Steele
6b2666a9d7 Split test modules into separate files.
Makes the code more maintainable. Tests are dynamically loaded by name rather than requiring an if-else block.
2016-12-23 08:22:59 -05:00
Cynthia Shang
56144c99c0 Improved check command to run on a standby.
Only basic checks are done because pg_switch_xlog() cannot be executed on a replica.

Contributed by Cynthia Shang.
2016-12-21 14:35:20 -05:00
Cynthia Shang
974a02dbe4 Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory.
Contributed by Cynthia Shang.
2016-12-20 16:52:20 -05:00
David Steele
aecc0f737e Fixed a few directory syncs that were missed for the --repo-sync option. 2016-12-19 14:40:22 -05:00
Adrian Vondendriesch
4e1b74688d Fixed const assignment giving compiler warning in C library.
Fixed by Adrian Vondendriesch.
2016-12-13 08:22:36 -05:00
David Steele
8e3464da80 v1.12: Page Checksums, Configuration, and Bug Fixes
IMPORTANT NOTE: In prior releases it was possible to specify options on the command-line that were invalid for the current command without getting an error. An error will now be generated for invalid options so it is important to carefully check command-line options in your environment to prevent disruption.

Bug Fixes:

* Fixed an issue where options that were invalid for the specified command could be provided on the command-line without generating an error. The options were ignored and did not cause any change in behavior, but it did lead to some confusion. Invalid options will now generate an error. (Reported by Nikhilchandra Kulkarni.)
* Fixed an issue where internal symlinks were not being created for tablespaces in the repository. This issue was only apparent when trying to bring up clusters in-place manually using filesystem snapshots and did not affect normal backup and restore.
* Fixed an issue that prevented errors from being output to the console before the logging system was initialized, i.e. while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not. (Reported by Adrian Vondendriesch.)
* Fixed an issue where the db-port option specified on the backup server would not be properly passed to the remote unless it was from the first configured database. (Reported by Michael Vitale.)

Features:

* Added the --checksum-page option to allow pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster. (Suggested by Stephen Frost.)
* Added the --repo-link option to allow internal symlinks to be suppressed when the repository is located on a filesystem that does not support symlinks. This does not affect any pgBackRest functionality, but the convenience link latest will not be created and neither will internal tablespace symlinks, which will affect the ability to bring up clusters in-place manually using filesystem snapshots.
* Added the --repo-sync option to allow directory syncs in the repository to be disabled for file systems that do not support them, e.g. NTFS.
* Added a predictable log entry to signal that a command has completed successfully. For example a backup ends successfully with: INFO: backup command end: completed successfully. (Suggested by Jens Wilke.)

Refactoring:

* Abstracted code to determine which database cluster is the master and which are standbys. (Contributed by Cynthia Shang.)
* Improved consistency and flexibility of the protocol layer by using JSON for all messages.
* File copy protocol now accepts a function that can do additional processing on the copy buffers and return a result to the calling process.
* Improved IO->bufferRead to always return requested number of bytes until EOF.
* For simplicity, the pg_control file is now copied with the rest of the files instead of by itself of at the end of the process. The backup command does not require this behavior and the restore copies to a temporary file which is renamed at the end of the restore.
* Simplified the result hash of File->manifest(), Db->tablespaceMapGet(), and Db->databaseMapGet().
* Improved errors returned from child processes by removing redundant error level and code.
* Code cleanup in preparation for improved stanza-create command. (Contributed by Cynthia Shang.)
* Improved parameter/result logging in debug/trace functions.
2016-12-12 19:18:27 -05:00
David Steele
36a5349b1c Added the --checksum-page option.
This option allows pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster.
2016-12-12 18:54:07 -05:00
David Steele
6dd0829033 File copy protocol now accepts a function that can do additional processing on the copy buffers and return a result to the calling process. 2016-12-12 18:47:17 -05:00
David Steele
b4884e5a0b Fixed an issue where the db-port option specified on the backup server would not be properly passed to the remote unless it was from the first configured database.
Reported by Michael Vitale.
2016-12-10 14:05:28 -05:00
David Steele
1a5fa920e6 Improved parameter/result logging in debug/trace functions. 2016-12-10 09:15:20 -05:00
David Steele
e61ef6ebce Add more realistic data files to synthetic backup and restore tests. 2016-12-10 09:13:55 -05:00
David Steele
eff7b46eb1 Various improvements to validation of backup and restore. 2016-12-10 09:11:12 -05:00
David Steele
f818901c0e Added the --repo-sync option to allow directory syncs in the repository to be disabled.
This is for file systems that do not support them, e.g. NTFS
2016-12-10 09:06:45 -05:00
David Steele
36378c47bd Added Retention to QuickStart section. 2016-12-04 18:11:23 -05:00
David Steele
896984848b Improved errors returned from child processes by removing redundant error level and code. 2016-12-04 17:51:00 -05:00
David Steele
817592837c Added a predictable log entry to signal that a command has completed successfully.
For example a backup ends successfully with: INFO: backup command end: completed successfully.
2016-12-04 14:44:53 -05:00
David Steele
f823a25ac2 Fixed an issue where options that were invalid for the specified command could be provided on the command-line without generating an error.
* The options were ignored and did not cause any change in behavior, but it did lead to some confusion. Invalid options will now generate an error.
* Removed erroneous --no-config option in help test module.
* Changed the --no-fork test option to --fork with negation to match all other boolean parameters.
2016-12-04 09:25:40 -05:00
David Steele
d18cd7ad0f Fixed an issue that prevented errors from being output to the console before the logging system was initialized.
That is, while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not.
2016-12-03 18:01:17 -05:00
David Steele
2e01e6b9e1 Improved consistency and flexibility of the protocol layer by using JSON for all messages. 2016-12-03 17:34:51 -05:00
Cynthia Shang
b7f1738200 Code cleanup in preparation for improved stanza-create command.
Contributed by Cynthia Shang.
2016-12-03 10:28:08 -05:00
David Steele
005fc08795 Added the --repo-link option.
Allow internal symlinks to be suppressed when the repository is located on a filesystem that does not support symlinks. This does not affect any pgBackRest functionality, but the convenience link latest will not be created and neither will internal tablespace symlinks, which will affect the ability to bring up clusters in-place manually using filesystem snapshots.
2016-12-03 10:23:26 -05:00
David Steele
6ae4d7de81 Fixed a bug where internal symlinks were not being created for tablespaces in the repository.
This issue was only apparent when trying to bring up clusters in-place with filesystem snapshots and did not affect normal backup and restore.
2016-12-03 09:56:05 -05:00
Cynthia Shang
22cd29771f Update control and wal test files to 9.4 with matching system identifiers.
Contributed by Cynthia Shang.
2016-12-01 09:33:14 -05:00
David Steele
a850335015 Simplified the result hash of File->manifest(), Db->tablespaceMapGet(), and Db->databaseMapGet(). 2016-11-30 14:36:39 -05:00
David Steele
09df07efb7 For simplicity, the pg_control file is now copied with the rest of the files instead of by itself of at the end of the process.
The backup command does not require this behavior and the restore copies to a temporary file which is renamed at the end of the restore.
2016-11-30 14:23:05 -05:00
David Steele
4e9f8da9a6 Improved IO->bufferRead to always return requested number of bytes until EOF. 2016-11-30 14:18:12 -05:00
Cynthia Shang
e951c4f9c2 Abstracted code to determine which database cluster is the master and which are standbys.
Contributed by Cynthia Shang.
2016-11-30 14:15:11 -05:00
David Steele
d101e9c84e Improved exception handling in file unit tests. 2016-11-30 14:03:58 -05:00
David Steele
4df3eabf6d Builds in release.pl now remove all docker containers to get consistent IP address assignments. 2016-11-22 17:31:27 -05:00
David Steele
dbd16d25b9 Fixed regression in section links introduced in v1.10.
This was introduced in an effort to make the html output XHTML 1.0 STRICT compliant because the standard does not allow / characters in anchors.

However, the / characters were changed to . in the anchors but not in the links.  For now revert the anchors to / so further though can be given to this issue.
2016-11-22 17:29:24 -05:00
David Steele
c9b49b0d7e Doc engine improvements.
Bug Fixes:

* Fixed and issue that suppressed exceptions in PDF builds.

Features:

* Allow a source to be included as a section so large documents can be broken up.
* Added section link support to Markdown output.
* Added list support to PDF output.
* Added include option to explicitly build sources (complements the exclude option though both cannot be used in the same invocation).
* Added keyword-add option to add keywords without overriding the default keyword.
* Added debug option to doc.pl to easily add the debug keyword to documentation builds.
* Added pre option to doc.pl to easily add the pre keyword to documentation builds.

Refactoring:

* Improvements to markdown rendering.
* Remove code dependency on project variable, instead use title param.
2016-11-17 16:35:11 -05:00
David Steele
54b1a784b5 v1.11: Bug Fix for Asynchronous Archiving Efficiency
Bug Fixes:

* Fixed an issue where asynchronous archiving was transferring one file per execution instead of transferring files in batches. This regression was introduced in v1.09 and affected efficiency only, all WAL segments were correctly archived in asynchronous mode. (Reported by Stephen Frost.)
2016-11-17 09:57:59 -05:00
David Steele
dbb9d80dab Fixed an issue where asynchronous archiving was transferring one file per execution instead of transferring files in batches.
This regression was introduced in v1.09 and affected efficiency only, all WAL segments were correctly archived in asynchronous mode.

Reported by Stephen Frost.
2016-11-17 09:39:21 -05:00
David Steele
06cac30c33 v1.10: Stanza Creation and Minor Bug Fixes
Bug Fixes:

* Fixed an issue where a backup could error if no changes were made to a database between backups and only pg_control changed.
* Fixed an issue where tablespace paths with the same prefix would cause an invalid link error. (Reported by Nikhilchandra Kulkarni.)

Features:

* Added the stanza-create command to formalize creation of stanzas in the repository. (Contributed by Cynthia Shang.)

Refactoring:

* Removed extraneous use lib directives from Perl modules. (Suggested by Devrim Gündüz.)
2016-11-08 10:47:47 +00:00
David Steele
9d9c438f59 Doc engine improvements.
Bug Fixes:

* Fixed missing variable replacements.
* Removed hard-coded host names from configuration file paths.

Documentation Features:

* Allow command-line length to be configured using cmd-line-len param.
* Added compact param to allow CSS to be embedded in HTML file.
* Added pretty param to produce HTML with proper indenting.
* Only generate HTML menu when required and don't require index page.
* Assign numbers to sections by default.
* VM mount points are now optional.
2016-11-07 17:06:35 +02:00
David Steele
dd621081b9 Fixed an issue where tablespace paths with the same prefix would cause an invalid link error.
Reported by Nikhilchandra Kulkarni.
2016-11-07 16:37:16 +02:00
David Steele
21ca412702 Fixed an issue where a backup could error if no changes were made to a database between backups and only pg_control changed. 2016-11-05 19:22:30 +02:00
David Steele
f43e5bc52d Removed extraneous use lib directives from Perl modules.
Suggested by Devrim Gündüz.
2016-11-04 13:56:26 +02:00
Cynthia Shang
8d1880ca4a Added the stanza-create command to formalize creation of stanzas in the repository.
Contributed by Cynthia Shang.
2016-10-14 12:21:47 +01:00
David Steele
1e0ca3056a v1.09: 9.6 Support, Configurability, and Bug Fixes
Bug Fixes:

* Fixed the check command to prevent an error message from being logged if the backup directory does not exist. (Fixed by Cynthia Shang.)
* Fixed error message to properly display the archive command when an invalid archive command is detected. (Reported by Jason O'Donnell.)
* Fixed an issue where the async archiver would not be started if archive-push did not have enough space to queue a new WAL segment. This meant that the queue would never be cleared without manual intervention (such as calling archive-push directly). PostgreSQL now receives errors when there is not enough space to store new WAL segments but the async process will still be started so that space is eventually freed. (Reported by Jens Wilke.)
* Fixed a remote timeout that occurred when a local process generated checksums (during resume or restore) but did not copy files, allowing the remote to go idle. (Reported by Jens Wilke.)

Features:

* Non-exclusive backups will automatically be used on PostgreSQL 9.6.
* Added the cmd-ssh option to allow the ssh client to be specified. (Suggested by Jens Wilke.)
* Added the log-level-stderr option to control whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. (Suggested by Sascha Biberhofer.)
* Set application_name to "pgBackRest [command]" for database connections. (Suggested by Jens Wilke.)
* Check that archive_mode is enabled when archive-check option enabled.

Refactoring:

* Clarified error message when unable to acquire pgBackRest advisory lock to make it clear that it is not a PostgreSQL backup lock. (Suggested by Jens Wilke.)
* pgBackRest version number included in command start INFO log output.
* Process ID logged for local process start/stop INFO log output.
2016-10-10 17:35:58 -04:00
David Steele
49a5e8a3f5 Fixed a remote timeout.
The timeout occurred when a local process generated checksums (during resume or restore) but did not copy files, allowing the remote to go idle.

Reported by Jens Wilke.
2016-10-07 08:51:10 -04:00
David Steele
41669a826e Added --dev option to doc.pl to easily add dev keyword to documentation builds. 2016-10-05 21:22:42 -04:00
David Steele
16d0e767ef Set application_name to "pgBackRest [command]" for database connections.
Suggested by Jens Wilke.
2016-10-05 21:15:10 -04:00
David Steele
0a4099fff8 Clarified error message when unable to acquire pgBackRest advisory lock to make it clear that it is not a PostgreSQL backup lock.
Suggested by Jens Wilke.
2016-10-05 21:13:29 -04:00