1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

306 Commits

Author SHA1 Message Date
David Steele
76161bd845 v1.18: Stanza Upgrade, Refactoring, and Locking Improvements
Bug Fixes:

* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)

Features:

* Added the stanza-upgrade command to provide a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. (Contributed by Cynthia Shang.)
* Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. (Contributed by Cynthia Shang.)

Refactoring:

* Simplify locking scheme. Now, only the master process will hold write locks (for archive-push and backup commands) and not all local and remote worker processes as before.
* Refactor Ini.pm to facilitate testing.
* Do not set timestamps of files in the backup directories to match timestamps in the cluster directory. This was originally done to enable backup resume, but that process is now implemented with checksums.
* Improved error message when the restore command detects the presence of postmaster.pid. (Suggested by Yogesh Sharma.)
* Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. (Suggested by Yogesh Sharma.)
* The backup and restore commands no longer copy via temp files. In both cases the files are checksummed on resume so there's no danger of partial copies.
* Allow functions to accept optional parameters as a hash.
* Refactor File->list() and fileList() to accept optional parameters.
* Refactor backupLabel() and add unit tests.
* Silence some perl critic warnings. (Contributed by Cynthia Shang.)
2017-04-12 19:17:39 -04:00
David Steele
f207dc7123 Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions.
Suggested by Yogesh Sharma.
2017-04-12 12:41:34 -04:00
David Steele
f5f8d51366 Improved error message when the restore command detects the presence of postmaster.pid.
Suggested by Yogesh Sharma.
2017-04-12 09:53:07 -04:00
David Steele
b0119914d4 Do not set timestamps of files in the backup directories to match timestamps in the cluster directory.
This was originally done to enable backup resume, but that process is now implemented with checksums.
2017-04-12 09:22:51 -04:00
David Steele
fe02b40293 Backup and restore commands no longer copy via temp files.
In both cases the files are checksummed on resume so there's no danger of partial copies.
2017-04-12 09:05:57 -04:00
Cynthia Shang
21a000d002 Silence some perl critic warnings.
* Variables::RequireNegativeIndice
* Variables::ProhibitUnusedVariables
* Variables::ProhibitAugmentedAssignmentInDeclaration
* Modules::ProhibitConditionalUseStatements
2017-04-10 17:23:39 -04:00
Cynthia Shang
b63ede5614 Added ability to test warning messages.
Contributed by Cynthia Shang.
2017-04-10 13:53:19 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm.
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
2017-04-10 13:24:45 -04:00
David Steele
367e06f4be Coverage testing always enabled on Debian-based containers.
* Full coverage is verified when specified.
* Modules marked with partial coverage will error if they are actually fully covered.
* Simplified test representation is DefineTest.
* Added new representation for queries in DefineTest and added API functions.
* Update modules using DefineTest to use new API.
2017-04-10 12:31:30 -04:00
David Steele
eb4918d018 Fixed locking bug and refactoring.
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
2017-04-07 10:31:49 -04:00
Cynthia Shang
b620e6b282 Improvements and additional tests for stanza-upgrade.
Contributed by Cynthia Shang.
2017-04-06 19:36:37 -04:00
David Steele
c7b61a2002 Refactor backupLabel() and add unit tests. 2017-04-05 10:16:16 -04:00
David Steele
b2f43b5601 Allow functions to accept optional parameters as a hash.
Refactor File->list() and fileList() to accept optional parameters.
2017-04-04 21:17:19 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command.
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
2017-04-03 10:42:55 -04:00
Cynthia Shang
af7e4f4b4b Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section.
Contributed by Cynthia Shang.
2017-03-22 19:03:28 -04:00
David Steele
cc9f8fc4b6 v1.17: Page Checksum Bug Fix
Bug Fixes:

* Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. (Reported by Stephen Frost.)
2017-03-13 16:54:51 -04:00
David Steele
b0947fdaa8 Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings.
Reported by Stephen Frost.
2017-03-13 15:50:13 -04:00
David Steele
3cec6702e8 v1.16: Page Checksum Improvements, CI, and Package Testing
Bug Fixes:

* Fixed an issue where tables over 1GB would report page checksum warnings after the first segment. (Reported by Stephen Frost.)
* Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. (Reported by blogh.)

Refactoring:

* Improved the code and tests for fileManifest() to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-03-02 20:15:14 -05:00
David Steele
02730526fc Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned.
Reported by blogh.
2017-03-02 13:50:29 -05:00
David Steele
c2666cc7f3 Fixed an issue where tables over 1GB would report page checksum errors after the first segment.
Reported by Stephen Frost.
2017-03-01 14:15:46 -05:00
David Steele
14ca4dac6f Improved the code and tests for fileManifest().
Done to prevent a possible race condition when files are removed by the database while the manifest is being built.
2017-02-21 11:56:55 -05:00
David Steele
2d7d9626bb v1.15: Refactoring and Bug Fixes
Bug Fixes:

* Fixed a regression introduced in v1.13 that could cause backups to fail if files were removed (e.g. tables dropped) while the manifest was being built. (Reported by Navid Golpayegani.)

Refactoring:

* Refactor FileCommon::fileManifest() and FileCommon::fileStat to be more modular to allow complete branch/statement level coverage testing.
2017-02-13 20:00:30 -05:00
David Steele
e2ac7e1ea6 Fixed a regression introduced in v1.13 that could cause backups to fail.
This happened if files were removed (e.g. tables dropped) while the manifest was being built.

Reported by Navid Golpayegani.
2017-02-13 19:59:14 -05:00
David Steele
845c6112bf v1.14: Bug Fixes
Bug Fixes:

* Fixed an issue where an archive-push error would not be retried and would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted). (Reported by Jens Wilke.)
* Fixed a race condition in parallel archiving where creation of new paths generated an error when multiple processes attempted to do so at the same time. (Reported by Jens Wilke.)

Refactoring:

* Improved performance of wal archive min/max provided by the info command. (Suggested by Jens Wilke.)
2017-02-13 10:07:04 -05:00
David Steele
a54018d573 Improved performance of wal archive min/max provided by the info command.
Suggested by Jens Wilke.
2017-02-13 10:03:02 -05:00
David Steele
adcc7cb954 Fixed a race condition in parallel archiving.
Creation of new paths generated an error when multiple processes attempted to do so at the same time.

Reported by Jens Wilke.
2017-02-13 09:58:50 -05:00
David Steele
bc2bb040c5 Fixed an issue where an archive-push error would not be retried.
It would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted).

Reported by Jens Wilke.
2017-02-13 09:57:10 -05:00
David Steele
41eba95155 v1.13: Parallel Archiving, Stanza Create, Improved Info and Check
IMPORTANT NOTE: The new implementation of asynchronous archiving no longer copies WAL to a separate queue. If there is any WAL left over in the old queue after upgrading to 1.13, it will be abandoned and not pushed to the repository.

To prevent this outcome, stop archiving by setting archive_command = false. Next, drain the async queue by running pgbackrest --stanza=[stanza-name] archive-push and wait for the process to complete. Check that the queue in [spool-path]/archive/[stanza-name]/out is empty. Finally, install 1.13 and restore the original archive_command.

IMPORTANT NOTE: The stanza-create command is not longer optional and must be executed before backup or archiving can be performed on a new stanza. Pre-existing stanzas do not require stanza-create to be executed.

Bug Fixes:

* Fixed const assignment giving compiler warning in C library. (Fixed by Adrian Vondendriesch.)
* Fixed a few directory syncs that were missed for the --repo-sync option.
* Fixed an issue where a missing user/group on restore could cause an "uninitialized value" error in File->owner(). (Reported by Leonardo Avellar.)
* Fixed an issue where protocol mismatch errors did not output the expected value.
* Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination.

Features:

* Improved, multi-process implementation of asynchronous archiving.
* Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. (Contributed by Cynthia Shang.)
* Improved check command to run on a standby, though only basic checks are done because pg_switch_xlog() cannot be executed on a replica. (Contributed by Cynthia Shang.)
* Added archive and backup WAL ranges to the info command.
* Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2. (Contributed by blogh.)
* Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations. (Suggested by Michael Vitale.)

Refactoring:

* Refactor File and BackupCommon modules to improve test coverage.
* Return proper error code when unable to convert a relative path to an absolute path. (Suggested by Yogesh Sharma.)
* Log file banner is not output until the first log entry is written. (Suggested by Jens Wilke.)
* Moved File->manifest() into the FileCommon.pm module.
* Moved the Archive modules to the Archive directory and split the archive-get and archive-push commands into separate modules.
* Split the check command out of the Archive.pm module.
* Allow logging to be suppressed via logDisable() and logEnable().
* Allow for locks to be taken more than once in the same process without error.
* Lock directories can be created when more than one directory level is required.
* Clean up optionValid()/optionTest() logic in Lock.pm.
* Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic.
* Represent .gz extension with a constant.
* Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads.
* Refactor process IO and process master/minion code out from the common protocol code.
* Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
* Remove Intel-specific optimization from C library build flags. (Contributed by Adrian Vondendriesch.)
* Removed --lock option. This option was introduced before the lock directory could be located outside the repository and is now obsolete.
* Added --log-timestamp option to allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
* Fixed alignment issues with multiline logging.
2017-02-05 20:23:03 -05:00
David Steele
0daeace1cd Fixed alignment issues with multiline logging. 2017-02-05 19:58:52 -05:00
David Steele
cded35791d Added --log-timestamp option.
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
2017-02-05 10:59:49 -05:00
David Steele
55feae645e Log file banner is not output until the first log entry is written.
Suggested by Jens Wilke.
2017-02-05 10:37:54 -05:00
David Steele
2237c3fc1b Added documentation for parallel archive-push. 2017-02-05 10:35:53 -05:00
David Steele
8b5f37c2a8 Added archive and backup WAL ranges to the info command. 2017-02-05 10:31:29 -05:00
David Steele
3e54aad88f Removed --lock option.
This option was introduced before the lock directory could be located outside the repository and is now obsolete.
2017-01-31 10:38:45 -05:00
David Steele
7418559a67 Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination. 2017-01-31 10:36:59 -05:00
David Steele
6b99d770b5 Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN. 2017-01-30 13:59:00 -05:00
David Steele
43ada9fb33 Improved, multi-process implementation of asynchronous archiving. 2017-01-27 11:02:27 -05:00
David Steele
2a9ba877e1 Refactor process IO and process master/minion code out from the common protocol code. 2017-01-27 10:06:16 -05:00
David Steele
0163d3b3e0 Allow empty files to be created with FileCommon::fileStringWrite() and use temp files to avoid partial reads. 2017-01-27 10:04:41 -05:00
David Steele
2553c7a998 Represent .gz extension with a constant. 2017-01-27 10:02:00 -05:00
David Steele
f3ccf3b897 Added Exception::exceptionCode() and Exception::exceptionMessage() to simplify error handling logic. 2017-01-27 09:56:26 -05:00
David Steele
6620594b77 Various improvements in Lock.pm:
* Allow for locks to be taken more than once in the same process without error.
* Lock directories can be created when more than one directory level is required.
* Clean up optionValid()/optionTest() logic in Lock.pm.
2017-01-27 09:53:46 -05:00
David Steele
18db178ac3 Various improvements to the test suite:
* Allow logging to be suppressed via logDisable() and logEnable().
* Added more flexibility in initializing and cleaning up after modules and tests.
* testResult() suppresses logging and reports exceptions.
* testException() allows messages to be matched with regular expressions.
* Refactor name/locations of common modules that setup test environments.
2017-01-27 09:42:30 -05:00
David Steele
e876a222fc Fixed an issue where protocol mismatch errors did not output the expected value. 2017-01-24 14:58:55 -05:00
Benoit Lobréau (blogh)
b6fa146f9b Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2.
Contributed by blogh.
2017-01-22 19:29:56 -05:00
David Steele
7d1347cdec Split the archive-get and archive-push commands into separate modules. 2017-01-10 19:59:32 -05:00
David Steele
155c808163 Split the check command out of the Archive.pm module. 2017-01-10 19:54:51 -05:00
David Steele
b58045f575 Moved the Archive modules to the Archive directory. 2017-01-10 19:51:20 -05:00
David Steele
0b4055e588 Moved File->manifest() into the FileCommon.pm module. 2017-01-10 13:08:03 -05:00
David Steele
0e4f51c271 Added integration for testing coverage with Devel::Cover. 2017-01-09 20:49:04 -05:00