1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

436 Commits

Author SHA1 Message Date
David Steele
6f5186f9e6 Convert config tests into C unit tests. 2017-10-16 11:13:51 -04:00
David Steele
9d78948a14 Improvements to page checksum module and tests:
* Page checksum module uses new C error handler.
* Convert page checksum tests into C unit tests.
2017-10-16 11:03:06 -04:00
David Steele
80758f0023 Add test macros for C results and errors. 2017-10-16 10:54:35 -04:00
David Steele
904b83747c Perl error handler recognizes errors thrown from the C library. 2017-10-16 10:47:31 -04:00
David Steele
f1e739b4aa Add C error handler.
Basic try ... catch ... finally pattern to simplify error handling in C.
2017-10-16 10:09:56 -04:00
David Steele
10dfbd90b5 Add C unit test infrastructure. 2017-10-12 12:55:48 -04:00
David Steele
08d6f14603 Add LibC.template.pm to simplify LibC module generation. 2017-10-10 18:21:44 -04:00
David Steele
abea4d1fd5 Remove Debian test repo after PostgreSQL 10 release. 2017-10-05 12:33:13 -04:00
David Steele
d89b7d5862 Warnings in C builds treated as errors. 2017-09-30 10:48:25 -04:00
David Steele
8efcc38304 Improvements in C codebase:
* Update C naming conventions.
* Use int datatype wherever possible.
* Better separation of C source from Perl interface.
2017-09-30 10:44:03 -04:00
David Steele
d0bf946857 v1.24: New Backup Exclusions
Bug Fixes:

* Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. (Reported by Uspen.)
* Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. (Reported by Uspen.)

Features:

* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-28 19:06:41 -04:00
David Steele
40a85e34f1 Improve C library smart build.
Ignore changes outside of /lib/pgBackRest/Config.
2017-09-20 19:57:05 -04:00
David Steele
4d6c56b4d7 Open log file after async process is completely separated from the main process.
This prevents the main process from also logging to the file.

Suggested by Jens Wilke.
2017-09-20 08:08:36 -04:00
David Steele
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
David Steele
c857015a38 Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. 2017-09-19 10:03:53 -04:00
David Steele
d36c52271f Break up LibC.xs into separate module files. 2017-09-17 15:52:19 -04:00
David Steele
20af8ecf50 Dynamically generate list of files for C library build. 2017-09-17 14:49:23 -04:00
David Steele
7c95a2f4de Updates to installation documentation.
* Add trusted SSH configuration.
* Add full installation where required and remove doc containers that included parts of the installation.
2017-09-08 21:50:00 -07:00
David Steele
4bcef7020e Rename master to primary in documentation to align with PostgreSQL convention. 2017-09-07 10:59:40 -07:00
David Steele
6343fdd584 Additional backup exclusions.
* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-04 08:26:57 -04:00
David Steele
df1ec0aaeb v1.23: Multiple Standbys and PostgreSQL 10 Support
Bug Fixes:

* Fixed an issue that could cause compression to abort on growing files. (Reported by Jesper St John, Aleksandr Rogozin.)
* Fixed an issue with keep-alives not being sent to the remote from the local process. (Reported by William Cox.)

Features:

* Up to seven standbys can be configured for backup from standby. (Contributed by Cynthia Shang.)
* PostgreSQL 10 support.
* Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways. (Suggested by Victor Gdalevich.)

Refactoring:

* Configuration rules are now pulled from the C library when present.
* Increase HTTP timeout for S3.
* Add HTTP retries to harden against transient S3 network errors.
2017-09-03 17:42:00 -04:00
David Steele
eea2ccc3ab Add HTTP retries to harden against transient S3 network errors. 2017-09-03 16:48:41 -04:00
David Steele
80e5e8db01 Increase HTTP timeout for S3. 2017-09-01 19:16:57 -04:00
David Steele
75511d2481 Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways.
Suggested by Victor Gdalevich.
2017-09-01 15:07:20 -04:00
David Steele
fcb7c6fd1d PostgreSQL 10 support. 2017-09-01 12:29:34 -04:00
Cynthia Shang
36e576b483 Up to seven standbys can be configured for backup from standby.`
Contributed by Cynthia Shang.
2017-08-31 19:15:44 -04:00
David Steele
206415d4c7 Fixed an issue that could cause compression to abort on growing files.
Reported by Jesper St John, Aleksandr Rogozin.
2017-08-30 16:34:05 -04:00
David Steele
81ffd09445 Fixed an issue with keep-alives not being sent to the remote from the local process.
Reported by William Cox.
2017-08-30 08:41:46 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
8d3436bce7 Merge release 2 container build changes. 2017-08-19 09:02:24 -04:00
David Steele
7dab459cac Simplified smart logic for C Library and package builds. 2017-08-19 08:48:13 -04:00
Cynthia Shang
1be69d5ac7 Fixed document generation to include section summaries on the Configuration page.
Fixed by Cynthia Shang.
2017-08-16 16:54:35 -04:00
David Steele
1d2b08ce94 Move contributor list to the end of release.xml for convenience. 2017-08-11 18:43:21 -04:00
David Steele
4c2c612b6f Ignore auto-generated C code created by release 2. 2017-08-11 18:38:40 -04:00
David Steele
4331e09376 Drain stderr during test process execution as well as termination to prevent lockups if there is a lot of output. 2017-08-11 18:31:25 -04:00
David Steele
f8de73674c Updated Docker build in Vagrantfile. 2017-08-10 20:52:29 -04:00
David Steele
ac1574448e Changed log test order to ignore unimportant log errors while shutting down PostgreSQL. 2017-08-10 18:11:29 -04:00
David Steele
84caca0f34 v1.22: Fixed S3 Retry
Bug Fixes:

* Fixed authentication issue in S3 retry.
2017-08-09 11:50:02 -04:00
David Steele
61c38f5808 Fixed authentication issue in S3 retry. 2017-08-09 11:27:09 -04:00
David Steele
cbc026418c v1.21: Improved Info Output and SSH Port Option
Bug Fixes:

* The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
* Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang.)

Features:

* Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang.)
* Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang.)

Refactoring:

* Retry when S3 returns an internal error (500).
* Add bIgnoreMissing parameter to Local->manifest().
2017-08-08 18:09:07 -04:00
David Steele
038d47bcc0 Retry when S3 returns an internal error (500). 2017-08-08 17:15:01 -04:00
David Steele
961f7033d1 Generate global fake cert in containers for testing. 2017-08-08 16:26:44 -04:00
Cynthia Shang
b6da509c9f Consolidate stanza-create and stanza-upgrade tests into new stanza test.
Contributed by Cynthia Shang.
2017-08-04 16:10:51 -04:00
Cynthia Shang
f9bdc092ba Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade.
Fixed by Cynthia Shang.
2017-08-04 16:03:55 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports.
Contributed by Cynthia Shang.
2017-08-01 10:34:53 -04:00
David Steele
cf3110b0d8 Use new Travis Trusty image. 2017-07-26 19:22:54 -04:00
David Steele
14d6974d1b Use Google DNS in test environment for consistency. 2017-07-26 13:40:09 -04:00
David Steele
97c4d9ec18 Reduce log verbosity in doc.pl.
Reduce verbosity by only logging sections that contain an execute list directly or in a child section.
2017-07-26 10:22:22 -04:00
David Steele
df7d633683 Improvements to command/command-line help:
* Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
* Fix description of --online based on the command context.
* Remove vestigial repository options from backup command.
2017-07-26 09:33:10 -04:00
David Steele
156fd4d54d Add bIgnoreMissing parameter to Local->manifest(). 2017-07-25 12:44:38 -04:00
David Steele
49449feab3 Add creation of /etc/pgbackrest.conf to manual installation instructions. 2017-07-24 21:14:42 -04:00
David Steele
cbb749e12a Debian/Ubuntu documentation now builds on Ubuntu 16. 2017-07-24 21:12:05 -04:00
David Steele
d5c1f02c72 Include archive_status directory in online backups.
The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do.

Also fixed log checking after PostgreSQL shuts down to include FATAL messages and disallow immediate shutdowns which can throw FATAL errors in the log.

Reported by Stephen Frost.
2017-07-24 07:57:47 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command.
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
2017-07-21 17:32:41 -04:00
David Steele
33d769e147 v1.20: Critical 8.3/8.4 Bug Fix
Bug Fixes:

* Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
* Fixed missing flag in C library build that resulted in a mismatched binary on 32-bit systems. (Reported by Adrian Vondendriesch.)

Features:

* Add s3-repo-ca-path and s3-repo-ca-file options to accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs. (Suggested by Scott Frazer.)

Refactoring:

* Harden protocol handshake to handle race conditions.
* Fixed misleading error message when a file was opened for write in a missing directory.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
* Rename Archive modules to remove redundancy.
* Improve S3 error reporting.
* Minor optimizations to package loads and ordering for archive-get and archive-push commands.
2017-06-27 22:42:31 -04:00
David Steele
884413b251 Remove mount and user from host cache-key because it can vary by system. 2017-06-27 22:27:48 -04:00
David Steele
7759a94fa2 Add Debian 9 to test suite. 2017-06-27 18:29:48 -04:00
David Steele
9e9286ee20 Add performance module and basic performance test for archive-push. 2017-06-27 18:00:39 -04:00
David Steele
9e248a6627 Minor optimizations to package loads and ordering for archive-get and archive-push commands. 2017-06-27 17:54:30 -04:00
David Steele
f3b62d2d67 Fixed misleading error message when a file was opened for write in a missing directory. 2017-06-27 17:07:12 -04:00
David Steele
2310e423e9 Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
The integration tests that were supposed to prevent this regression did not work as intended.  They verified the contents of a table in the (supposedly) restored tablespace, deleted the table, and then deleted the tablespace.  All of this was deemed sufficient to prove that the tablespace had been restored correctly and was valid.

However, PostgreSQL will happily recreate a tablespace on the basis of a single full-page write, at least in the affected versions.  Since writes to the test table were replayed from WAL with each recovery, all the tests passed even though the tablespace was missing after the restore.

The tests have been updated to include direct comparisons against the file system and a new table that is not replayed after a restore because it is created before the backup and never modified again.

Versions ≥ 9.0 were not affected due to numerous synthetic integration tests that verify backups and restores file by file.
2017-06-27 16:47:40 -04:00
David Steele
fdabf33604 Revert removal of Ubuntu 12.04 and PostgreSQL 8.3. 2017-06-27 15:58:02 -04:00
David Steele
001cff9eb9 Automatically check that all supported PostgreSQL versions are being tested on a single default VM. 2017-06-27 15:48:34 -04:00
David Steele
7a1385cc27 Add documentation builds to CI. 2017-06-24 18:04:02 -04:00
David Steele
7a9b1f6993 Container build improvements:
* More optimized container suite that greatly improves build time.
* Added static Debian packages for Devel::Cover to reduce build time.
* Add deprecated state for containers. Deprecated containers may only be used to build packages.
* Remove Debian 8 from CI because it does not provide additional coverage over Ubuntu 14.04 and Ubuntu 16.04.
2017-06-24 10:59:00 -04:00
David Steele
53184a7b30 Update CentOS 6 documentation to build on PostgreSQL 9.5. 2017-06-23 10:30:22 -04:00
David Steele
3d224ad8fa Improve S3 server implementation in documentation. 2017-06-23 10:10:48 -04:00
David Steele
eb4325fa9a Remove exhaustive version list from Stable Releases TOC. 2017-06-22 19:04:52 -04:00
David Steele
a10a22127d Consolidate archive-get test into new archive test. 2017-06-22 18:51:39 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options.
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
2017-06-22 18:22:49 -04:00
David Steele
a1a5d18aa4 Balance database versions between VMs to minimize test duration. 2017-06-22 14:44:21 -04:00
David Steele
d3054d919b Fixed missing flag in C library build.
The missing flag resulted in a mismatched binary on 32-bit systems.

Reported by Adrian Vondendriesch.
2017-06-22 12:57:10 -04:00
David Steele
f8849d7359 Remove patch directory before Debian package builds.
Patches should be merged into master before trying a new build after a release with patches.
2017-06-22 12:37:21 -04:00
David Steele
f596702c5b Improve S3 error reporting. 2017-06-21 20:46:49 -04:00
David Steele
8b9965abcb Remove process-max option.
Parallelism is now tested in a more targeted manner and the high level option is no longer needed.
2017-06-21 19:21:16 -04:00
David Steele
cc0e3f843e Remove support for PostgreSQL 8.3. 2017-06-21 17:24:36 -04:00
David Steele
73274fc608 Remove Ubuntu 12.04 from CI now that it is EOL (it can still be run manually). 2017-06-21 17:02:01 -04:00
David Steele
c69baa2de1 Install sudo in base containers rather than on demand. 2017-06-21 16:51:41 -04:00
David Steele
64ec7eb490 Eliminate redundancy in real tests. 2017-06-21 16:07:13 -04:00
David Steele
2337be8636 Consolidate archive-push tests into new archive test. 2017-06-21 08:38:53 -04:00
David Steele
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
David Steele
692b1e274a Split full module into mock and real to allow better test combinations and save time in CI. 2017-06-19 18:55:00 -04:00
David Steele
f6d4457d58 Full/Synthetic test refactor.
* Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
2017-06-15 15:32:10 -04:00
David Steele
269ae4556d Fix timeouts in ExecuteTest to speed multi-process testing. 2017-06-15 15:20:02 -04:00
David Steele
6e29f616b3 Harden protocol handshake to handle race conditions. 2017-06-15 15:12:28 -04:00
David Steele
c10c5bb0ef v1.19: S3 Support
Bug Fixes:

* Fixed the info command so the WAL archive min/max displayed is for the current database version. (Fixed by Cynthia Shang.)
* Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable. (Fixed by Cynthia Shang.)
* Fixed config warnings raised from a remote process causing errors in the master process. (Fixed by Cynthia Shang.)

Features:

* Amazon S3 repository support. (Reviewed by Cynthia Shang.)

Refactoring:

* Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
* Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
* Move modules in Protocol directory in subdirectories.
* Move backup modules into Backup directory.
2017-06-12 11:36:00 -04:00
David Steele
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
David Steele
de7fc37f88 Storage and IO layer refactor:
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 17:51:41 -04:00
Cynthia Shang
7e982f05f5 Fixed config warnings raised from a remote process causing errors in the master process.
Fixed by Cynthia Shang.
2017-05-16 11:28:51 -04:00
David Steele
74053ed842 Fixed missing sudo in installation section.
Fixed by Lætitia.
2017-05-16 11:15:31 -04:00
David Steele
5296747271 Move backup modules into Backup directory. 2017-05-15 16:01:00 -04:00
David Steele
2a23157bf2 Move modules in Protocol directory in subdirectories. 2017-05-15 11:12:14 -04:00
David Steele
33c87b2a1c Remove flapping protocol timeout test that will be replaced in the upcoming storage patch. 2017-05-15 10:24:46 -04:00
David Steele
c31da3f961 Moved test and env modules to new directories to avoid namespace conflicts with common tests. 2017-05-12 16:43:04 -04:00
David Steele
4049d59c9e Fixed issue with --dry-run requiring --vm-out to work properly. 2017-05-12 16:27:06 -04:00
Cynthia Shang
7d0c50fcd6 Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable.
Fixed by Cynthia Shang.
2017-05-12 16:05:44 -04:00
Cynthia Shang
5bf4f52e3d Fixed the info command so the WAL archive min/max displayed is for the current database version.
Fixed by Cynthia Shang.
2017-05-12 15:49:14 -04:00
David Steele
64be6400fe Changed invalid max-archive-mb option in configuration reference to archive-queue-max. 2017-04-17 06:39:29 -04:00
David Steele
979610a07d Fixed an undefined variable when a module had no uncoverable code exceptions. 2017-04-13 20:10:28 -04:00
David Steele
77ef4fde20 Set --vm-max=2 for CI. 2017-04-13 07:42:45 -04:00
David Steele
76161bd845 v1.18: Stanza Upgrade, Refactoring, and Locking Improvements
Bug Fixes:

* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)

Features:

* Added the stanza-upgrade command to provide a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. (Contributed by Cynthia Shang.)
* Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. (Contributed by Cynthia Shang.)

Refactoring:

* Simplify locking scheme. Now, only the master process will hold write locks (for archive-push and backup commands) and not all local and remote worker processes as before.
* Refactor Ini.pm to facilitate testing.
* Do not set timestamps of files in the backup directories to match timestamps in the cluster directory. This was originally done to enable backup resume, but that process is now implemented with checksums.
* Improved error message when the restore command detects the presence of postmaster.pid. (Suggested by Yogesh Sharma.)
* Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. (Suggested by Yogesh Sharma.)
* The backup and restore commands no longer copy via temp files. In both cases the files are checksummed on resume so there's no danger of partial copies.
* Allow functions to accept optional parameters as a hash.
* Refactor File->list() and fileList() to accept optional parameters.
* Refactor backupLabel() and add unit tests.
* Silence some perl critic warnings. (Contributed by Cynthia Shang.)
2017-04-12 19:17:39 -04:00
David Steele
f207dc7123 Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions.
Suggested by Yogesh Sharma.
2017-04-12 12:41:34 -04:00
David Steele
f5f8d51366 Improved error message when the restore command detects the presence of postmaster.pid.
Suggested by Yogesh Sharma.
2017-04-12 09:53:07 -04:00
David Steele
b0119914d4 Do not set timestamps of files in the backup directories to match timestamps in the cluster directory.
This was originally done to enable backup resume, but that process is now implemented with checksums.
2017-04-12 09:22:51 -04:00
David Steele
fe02b40293 Backup and restore commands no longer copy via temp files.
In both cases the files are checksummed on resume so there's no danger of partial copies.
2017-04-12 09:05:57 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings.
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
2017-04-10 17:23:39 -04:00
David Steele
8546571461 Ignore clock skew in container libc/package builds using make.
It is common for containers to have clock skew so the build process takes care of this issue independently.
2017-04-10 17:20:21 -04:00
Cynthia Shang
b63ede5614 Added ability to test warning messages.
Contributed by Cynthia Shang.
2017-04-10 13:53:19 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm.
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
2017-04-10 13:24:45 -04:00
David Steele
367e06f4be Coverage testing always enabled on Debian-based containers.
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
2017-04-10 12:31:30 -04:00
David Steele
eb4918d018 Fixed locking bug and refactoring.
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
2017-04-07 10:31:49 -04:00
David Steele
edfb0962b6 Update wording for release note sections. 2017-04-06 21:47:43 -04:00
David Steele
0da216c034 Require description in every call to testResult() and make iWaitSeconds optional. 2017-04-05 10:50:32 -04:00
David Steele
c7b61a2002 Refactor backupLabel() and add unit tests. 2017-04-05 10:16:16 -04:00
David Steele
b2f43b5601 Allow functions to accept optional parameters as a hash.
Refactor File->list() and fileList() to accept optional parameters.
2017-04-04 21:17:19 -04:00
David Steele
9eae36a702 Fixed flapping archive stop tests. 2017-04-04 21:15:22 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command.
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
2017-04-03 10:42:55 -04:00
Cynthia Shang
af7e4f4b4b Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section.
Contributed by Cynthia Shang.
2017-03-22 19:03:28 -04:00
David Steele
cbd879a983 Updated vagrant to new version and image. 2017-03-14 22:05:17 -04:00
David Steele
cc9f8fc4b6 v1.17: Page Checksum Bug Fix
Bug Fixes:

* Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. (Reported by Stephen Frost.)
2017-03-13 16:54:51 -04:00
David Steele
b0947fdaa8 Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings.
Reported by Stephen Frost.
2017-03-13 15:50:13 -04:00
David Steele
3cec6702e8 v1.16: Page Checksum Improvements, CI, and Package Testing
Bug Fixes:

* Fixed an issue where tables over 1GB would report page checksum warnings after the first segment. (Reported by Stephen Frost.)
* Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. (Reported by blogh.)

Refactoring:

* Improved the code and tests for fileManifest() to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-03-02 20:15:14 -05:00
David Steele
02730526fc Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned.
Reported by blogh.
2017-03-02 13:50:29 -05:00
David Steele
5957f8566f Archive test improvements:
* Fixed flapping archive-stop test with testResult() retries.
* Added final test of archive contents to archive-push test.
2017-03-02 11:47:23 -05:00
David Steele
5eb585c9ba Improved truncation when outputting errors logs in the ExecuteTest module. 2017-03-01 14:38:05 -05:00
David Steele
c2666cc7f3 Fixed an issue where tables over 1GB would report page checksum errors after the first segment.
Reported by Stephen Frost.
2017-03-01 14:15:46 -05:00
David Steele
8161ab1c50 Temporarily disable flapping keep-alive test. 2017-02-26 12:58:48 -05:00
David Steele
31d2d790bd Added --retry option. 2017-02-21 22:10:02 -05:00
David Steele
14ca4dac6f Improved the code and tests for fileManifest().
Done to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-02-21 11:56:55 -05:00
David Steele
01312f6260 Improved formatting of testResult() output. 2017-02-21 11:47:45 -05:00
David Steele
86fee3da18 Continuous integration using travis-ci. 2017-02-21 08:59:23 -05:00
David Steele
56543f7a99 Added package builds to test suite and other improvements:
* Automated builds of Debian packages for all supported distributions.
* Added --dev option to aggregate commonly used dev options.
* Added --no-package option to skip package builds.
* C library and packages are built by default, added -smart option to rebuild only when file changes are detected.
* The --libc-only option has been changed to --build-only now that packages builds have been added.
2017-02-17 22:31:16 -05:00
David Steele
ac7ff755b3 Container executions now load the user's environment. 2017-02-16 17:18:48 -05:00
David Steele
2d7d9626bb v1.15: Refactoring and Bug Fixes
Bug Fixes:

* Fixed a regression introduced in v1.13 that could cause backups to fail if files were removed (e.g. tables dropped) while the manifest was being built. (Reported by Navid Golpayegani.)

Refactoring:

* Refactor FileCommon::fileManifest() and FileCommon::fileStat to be more modular to allow complete branch/statement level coverage testing.
2017-02-13 20:00:30 -05:00
David Steele
e2ac7e1ea6 Fixed a regression introduced in v1.13 that could cause backups to fail.
This happened if files were removed (e.g. tables dropped) while the manifest was being built.

Reported by Navid Golpayegani.
2017-02-13 19:59:14 -05:00
David Steele
845c6112bf v1.14: Bug Fixes
Bug Fixes:

* Fixed an issue where an archive-push error would not be retried and would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted). (Reported by Jens Wilke.)
* Fixed a race condition in parallel archiving where creation of new paths generated an error when multiple processes attempted to do so at the same time. (Reported by Jens Wilke.)

Refactoring:

* Improved performance of wal archive min/max provided by the info command. (Suggested by Jens Wilke.)
2017-02-13 10:07:04 -05:00
David Steele
a54018d573 Improved performance of wal archive min/max provided by the info command.
Suggested by Jens Wilke.
2017-02-13 10:03:02 -05:00
David Steele
adcc7cb954 Fixed a race condition in parallel archiving.
Creation of new paths generated an error when multiple processes attempted to do so at the same time.

Reported by Jens Wilke.
2017-02-13 09:58:50 -05:00
David Steele
bc2bb040c5 Fixed an issue where an archive-push error would not be retried.
It would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted).

Reported by Jens Wilke.
2017-02-13 09:57:10 -05:00
David Steele
d68b48d19b Improved support for --require option and section depends now default to the previous section. 2017-02-11 10:26:54 -05:00
David Steele
498f52da09 Improvements to documentation engine:
* Documentation can now be built with reusable blocks to reduce duplication.
* Added ability to pass options to containers within the documentation.
* Add proper tag to slightly emphasize proper nouns.
2017-02-10 10:22:05 -05:00
David Steele
b813bfaff6 Updated async archiving documentation.
More accurately describe how the new method works and how it differs from the old method.

Suggested by Jens Wilke.
2017-02-08 08:41:43 -05:00
David Steele
41eba95155 v1.13: Parallel Archiving, Stanza Create, Improved Info and Check
IMPORTANT NOTE: The new implementation of asynchronous archiving no longer copies WAL to a separate queue. If there is any WAL left over in the old queue after upgrading to 1.13, it will be abandoned and not pushed to the repository.

To prevent this outcome, stop archiving by setting archive_command = false. Next, drain the async queue by running pgbackrest --stanza=[stanza-name] archive-push and wait for the process to complete. Check that the queue in [spool-path]/archive/[stanza-name]/out is empty. Finally, install 1.13 and restore the original archive_command.

IMPORTANT NOTE: The stanza-create command is not longer optional and must be executed before backup or archiving can be performed on a new stanza. Pre-existing stanzas do not require stanza-create to be executed.

Bug Fixes:

* Fixed const assignment giving compiler warning in C library. (Fixed by Adrian Vondendriesch.)
* Fixed a few directory syncs that were missed for the --repo-sync option.
* Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner(). (Reported by Leonardo Avellar.)
* Fixed an issue where protocol mismatch errors did not output the expected value.
* Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination.

Features:

* Improved, multi-process implementation of asynchronous archiving.
* Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. (Contributed by Cynthia Shang.)
* Improved check command to run on a standby, though only basic checks are done because pg_switch_xlog() cannot be executed on a replica. (Contributed by Cynthia Shang.)
* Added archive and backup WAL ranges to the info command.
* Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2. (Contributed by blogh.)
* Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations. (Suggested by Michael Vitale.)

Refactoring:

* Refactor File and BackupCommon modules to improve test coverage.
* Return proper error code when unable to convert a relative path to an absolute path. (Suggested by Yogesh Sharma.)
* Log file banner is not output until the first log entry is written. (Suggested by Jens Wilke.)
* Moved File->manifest() into the FileCommon.pm module.
* Moved the Archive modules to the Archive directory and split the archive-get and archive-push commands into separate modules.
* Split the check command out of the Archive.pm module.
* Allow logging to be suppressed via logDisable() and logEnable().
* Allow for locks to be taken more than once in the same process without error.
* Lock directories can be created when more than one directory level is required.
* Clean up optionValid()/optionTest() logic in Lock.pm.
* Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic.
* Represent .gz extension with a constant.
* Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads.
* Refactor process IO and process master/minion code out from the common protocol code.
* Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
* Remove Intel-specific optimization from C library build flags. (Contributed by Adrian Vondendriesch.)
* Removed --lock option. This option was introduced before the lock directory could be located outside the repository and is now obsolete.
* Added --log-timestamp option to allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
* Fixed alignment issues with multiline logging.
2017-02-05 20:23:03 -05:00
David Steele
0daeace1cd Fixed alignment issues with multiline logging. 2017-02-05 19:58:52 -05:00
David Steele
cded35791d Added --log-timestamp option.
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
2017-02-05 10:59:49 -05:00
David Steele
55feae645e Log file banner is not output until the first log entry is written.
Suggested by Jens Wilke.
2017-02-05 10:37:54 -05:00
David Steele
8b5f37c2a8 Added archive and backup WAL ranges to the info command. 2017-02-05 10:31:29 -05:00
David Steele
1666ee177f Fixed --no-online tests to suppress expected errors. 2017-02-02 20:44:42 -05:00
David Steele
efdf64506a Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations.
Suggested by Michael Vitale.
2017-02-02 20:42:45 -05:00
David Steele
3e54aad88f Removed --lock option.
This option was introduced before the lock directory could be located outside the repository and is now obsolete.
2017-01-31 10:38:45 -05:00