1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1714 Commits

Author SHA1 Message Date
David Steele
88d9bd3d37 Improvements in Db module.
* Separated out connect() function.
* Allow executeSql() calls that do not return data.
* Improve error handling.
2016-08-15 17:44:51 -04:00
David Steele
1e0f15f425 Improve error message for links that reference links in manifest build. 2016-08-15 17:23:37 -04:00
David Steele
f9fa1270b2 Fixed #236: Recursive user tablespace symlink.
A tablespace link that referenced another link would not produce an error, but instead skip the tablespace entirely.
2016-08-15 17:11:45 -04:00
David Steele
35931305d7 Update control version for PostgreSQL beta4. 2016-08-15 16:48:24 -04:00
David Steele
17b79d6279 Database version refactoring.
* Refactor db version constants into a separate module.
* Update synthetic backup tests to PostgreSQL 9.4.
2016-08-11 22:35:24 -04:00
David Steele
bff262ac47 Removed all OP_* function constants that were used only for debugging, not in the protocol, and replaced with __PACKAGE__. 2016-08-11 17:32:28 -04:00
David Steele
9f19e13095 v1.05: Bug Fix for Tablespace Link Checking
Bug Fixes:

* Fixed an issue where tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not. Also hardened relative path checking a bit. (Reported by Chris Fort.)
2016-08-09 10:06:21 -04:00
David Steele
34afe5e85b Fixed issue with tablespace link checking.
* Tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not.
* Also hardened relative path checking a bit.
2016-08-09 09:05:27 -04:00
Cynthia Shang
af790be34f Added documentation for scheduling backups with cron.
Contributed by Cynthia Shang.
2016-08-02 12:55:09 -04:00
David Steele
730262678a Improved rendering of spaces in code blocks. 2016-08-02 12:49:39 -04:00
Cynthia Shang
f0c9728921 Moved the backlog from the pgBackRest website to the GitHub repository wiki.
Contributed by Cynthia Shang.
2016-08-02 11:38:12 -04:00
David Steele
277934447b v1.04: Various Bug Fixes
Bug Fixes:

* Fixed an issue an where an extraneous remote was created causing threaded backup/restore to possibly timeout and/or throw a lock conflict. (Reported by Michael Vitale.)
* Fixed an issue where db-path was not required for the check command so an assert was raised when it was missing rather than a polite error message. (Reported by Michael Vitale.)
* Fixed check command to throw an error when database version/id does not match that of the archive. (Fixed by Cynthia Shang.)
* Fixed an issue where a remote could try to start its own remote when the backup-host option was not present in pgbackrest.conf on the database server. (Reported by Lardière Sébastien.)
* Fixed an issue where the contents of pg_xlog were being backed up if the directory was symlinked. This didn't cause any issues during restore but was a waste of space.
* Fixed an invalid log() call in lock routines.

Features:

* Experimental support for non-exclusive backups in PostgreSQL 9.6 beta3. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.

Refactoring:

* Enhancements to the protocol layer for improved reliability and error handling.
* All remote types now take locks. The exceptions date to when the test harness and pgBackRest were running in the same VM and no longer apply.
* Exceptions are now passed back from threads as messages when possible rather than raised directly.
* Temp files created during backup are now placed in the same directory as the target file.
* Output lock file name when a lock cannot be acquired to aid in debugging.
* Reduce calls to protocolGet() in backup/restore.
* Suppress banners on SSH protocol connections.
* Improved remote error messages to identify the host where the error was raised.
2016-07-30 09:42:35 -04:00
David Steele
f2b3dda647 Fixed an issue where a remote could try to start its own remote.
This is a better approach than 93320b8 (reverted in this commit) because it ensures that the remote type will be none so any functions that utilize optionRemoteTypeTest will work correctly.

This bug was only an issue when backup-host was not properly configured on the database host.
2016-07-30 08:49:26 -04:00
David Steele
94a4cb4123 Disable control master for older OS versions where it is less stable. 2016-07-30 08:43:38 -04:00
David Steele
1b6b26956a Reduce the frequency that certain tests are run to save time in regression. 2016-07-29 18:46:19 -04:00
David Steele
ac23716d26 Enhancements to the protocol layer for improved reliability and error handling. 2016-07-29 18:46:19 -04:00
David Steele
b447863218 Exceptions are now passed back from threads as messages when possible rather than raised directly. 2016-07-29 18:46:19 -04:00
David Steele
93320b865e Fixed a issue where a remote could try to start its own remote. 2016-07-29 18:45:37 -04:00
David Steele
af5965a6ef Fixed an invalid log call in lock routines. 2016-07-29 18:45:23 -04:00
David Steele
a3b8808f94 Fixed an issue where the contents of pg_xlog were being copied if the directory was symlinked. 2016-07-29 18:44:53 -04:00
David Steele
ef33febfb7 Temp files created during backup are now placed in the same directory as the target file. 2016-07-29 18:43:59 -04:00
David Steele
95bbc8ad54 Fixed an issue where multi-threaded tests were not being run when requested. 2016-07-29 18:02:16 -04:00
Cynthia Shang
6e9d5f5bd3 Fixed check command to throw an error when database version/id does not match that of the archive.
Fixed by Cynthia Shang.
2016-07-29 14:02:11 -04:00
David Steele
f95d4af64a Closed #220: Installation documentation misinformation.
Updated information about package availability on supported platforms.
2016-07-26 18:50:58 -04:00
David Steele
ee80b7b255 Added clarification on why the default for the backrest-user option is backrest. 2016-07-26 18:30:25 -04:00
David Steele
1844756990 Closed #209: Don't use "localtime" for HTML-docs.
HTML footer dates are statically created in English in order to be reproducible.
2016-07-26 17:40:42 -04:00
David Steele
273d1c08d8 Improved remote error messages to identify the host where the error was raised. 2016-07-26 16:57:38 -04:00
David Steele
bc5c04ffe7 Updated control/catalog/WAL versions for 9.6 beta3. 2016-07-26 16:39:01 -04:00
David Steele
b6ac6b0f1c Suppress banners on SSH protocol connections. 2016-07-26 16:28:43 -04:00
David Steele
f31ec20fcb Reduce calls to protocolGet() in backup/restore. 2016-07-26 16:21:14 -04:00
David Steele
98a67792ec Output lock file name when a lock cannot be acquired to aid in debugging. 2016-07-26 16:14:15 -04:00
David Steele
d96b1a0dbc All remote types now take locks.
The exceptions date to when the test harness and pgBackRest were running in the same VM and no longer apply.
2016-07-26 16:12:38 -04:00
David Steele
095351f145 Fixed #219: Unable to acquire restore lock.
Fixed an issue an where an extraneous remote was created causing threaded backup/restore to possibly timeout and/or throw a lock conflict.
2016-07-26 16:08:05 -04:00
David Steele
5893c3cd2e Fixed an issue where db-path was not required for the check command so an assert was thrown when it was missing rather than a polite error message.
Reported by Michael Vitale.
2016-07-19 15:26:41 -04:00
David Steele
9ae6c01432 Fixed a version checking issue in test.pl. 2016-07-19 15:22:20 -04:00
David Steele
6e52808065 Added release.pl to make releases reproducible. 2016-07-02 11:08:25 -04:00
David Steele
1b62354dcb v1.03: Check Command and Bug Fixes
Bug Fixes:

* Fixed an issue where keep-alives could be starved out by lots of small files during multi-threaded backup. They were also completely absent from single/multi-threaded backup resume and restore checksumming. (Reported by Janice Parkinson, Chris Barber.)
* Fixed an issue where the expire command would refuse to run when explicitly called from the command line if the db-host option was set. This was not an issue when expire was run automatically after a backup (Reported by Chris Barber.)
* Fixed an issue where validation was being running on archive_command even when the archive-check option was disabled.

Features:

* Added check command to validate that pgBackRest is configured correctly for archiving and backups. (Contributed by Cynthia Shang.)
* Added the protocol-timeout option. Previously protocol-timeout was set as db-timeout + 30 seconds.
* Failure to shutdown remotes at the end of the backup no longer throws an exception. Instead a warning is generated that recommends a higher protocol-timeout.
* Experimental support for non-exclusive backups in PostgreSQL 9.6 beta2. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.

Refactoring:

* The pg_xlogfile_name() function is no longer used to construct WAL filenames from LSNs. While this function is convenient it is not available on a standby. Instead, the archive is searched for the LSN in order to find the timeline. If due to some misadventure the LSN appears on multiple timelines then an error will be thrown, whereas before this condition would have passed unnoticed.
* Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the backup-host and db-host options which are used to determine locality.
* Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
* Changed version variable to a constant. It had originally been designed to play nice with a specific packaging tool but that tool was never used.
2016-07-02 10:22:52 -04:00
David Steele
186ca18659 Update WAL/catalog versions for 9.5 beta 2. 2016-07-02 09:19:10 -04:00
David Steele
418039027b Closed #213: Archive command should not be validate when archive-check=n 2016-07-02 09:02:55 -04:00
Cynthia Shang
9f9396943f Added documentation for asynchronous archiving.
Also removed some obsolete sections.

Contributed by Cynthia Shang.
2016-06-30 19:38:28 -04:00
David Steele
304d890f6c Release note edits. 2016-06-27 23:07:47 -04:00
David Steele
9516db4ad6 Closed #209: Dont' use "localtime" for HTML-docs.
The release-date-static var has been added to allow the html footer to use the last release date instead of localtime().
2016-06-27 20:37:25 -04:00
David Steele
cc2a8777d5 User/group permissions improvements.
Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
2016-06-26 21:01:20 -04:00
David Steele
b571218930 Minor refactor of version variable to a constant.
It had originally been designed to play nice with a specific packaging tool but that tool was never used.
2016-06-26 20:53:45 -04:00
David Steele
6ee485cc76 Added more keep-alives.
Keep-alives were also missing from multi-threaded backup/restore and single-threaded restore.

Found by Chris Barber.
2016-06-24 13:16:07 -04:00
David Steele
bd25223fd6 Rename test paths for clarity.
This was worked out as part of the test suite refactor [c8f806a] but not committed with it because of the large number of expect logs changes involved.  Keeping them separate made it easier to audit the changes in the refactor.
2016-06-24 10:54:31 -04:00
David Steele
c8f806a293 Major refactor of the test suite.
* Make the code more modular and object-oriented.
* Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 08:12:58 -04:00
David Steele
23a3911830 Stop using pg_xlogfile_name().
The pg_xlogfile_name() function is no longer used to construct WAL filenames from LSNs. While this function is convenient it is not available on a standby. Instead, the archive is searched for the LSN in order to find the timeline. If due to some misadventure the LSN appears on multiple timelines then an error will be thrown, whereas before this condition would have passed unnoticed.
2016-06-24 08:06:20 -04:00
David Steele
41b27c0456 Protocol timeout option and keep-alive fixes.
* Fixed an issue where keep-alives could be starved out by lots of small files during multi-threaded operation and were completely absent during single-threaded operation when resuming from a previous incomplete backup.

Reported by Janice Parkinson.

* Added the protocol-timeout option. Previously protocol-timeout was set as db-timeout + 30 seconds.
* Failure to shutdown remotes at the end of the backup no longer throws an exception. A warning is still generated that recommends a higher protocol-timeout.
2016-06-22 18:01:18 -04:00
David Steele
4cff5bb0e6 Fixed pgBackRest config sections being blank in the output when not loaded from cache. 2016-06-18 10:59:45 -04:00
David Steele
012405a33b Closed #207: Expire fails with unhandled exception.
* Fixed an issue where the expire command would refuse to run when explicitly called from the command line if the db-host option was set.  This was not an issue when expire was run after a backup, which is the usual case.
* Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the backup-host and db-host options which are used to determine locality.

Reported by Chris Barber.
2016-06-18 09:55:00 -04:00
David Steele
e988b96ece Fixed issue in doc info output with --require.
Section names were being repeated in the info output when multiple --require options depended on the same sections.
2016-06-18 09:22:26 -04:00
Cynthia Shang
7e45ed8366 Added check command.
The check command validates that pgBackRest is configured correctly for archiving and backups.

Contributed by Cynthia Shang.
2016-06-12 09:13:46 -04:00
David Steele
7c9eaf7210 Add cmd-description tag.
Allow command summaries to be inserted anywhere in the documentation to avoid duplication.
2016-06-12 09:01:56 -04:00
David Steele
f1e92804ce Docker container improvements.
* Containers now use a squid proxy for apt/yum to speed builds.
* Obsolete containers are removed by the <br-option>--vm-force</br-option> option.
* Greatly reduced the quantity of Docker containers built by default. Containers are only built for PostgreSQL versions specified in db-minimal and those required to build documentation. Additional containers can be built with --db-version=all or by specifying a version, e.g. --db-version=9.4.
2016-06-12 09:00:16 -04:00
David Steele
3d761b726a Update TeX Live to 2016 version. 2016-06-10 21:22:36 -04:00
David Steele
5660f69859 Fixed DTDs not being loaded from test.pl. 2016-06-06 13:27:52 -04:00
David Steele
3f7ca876c2 PostgreSQL version is now a variable to allow multi-version documentation. 2016-06-04 12:24:31 -04:00
David Steele
bfed78ef42 New, consolidated implementation for link rendering. 2016-06-04 12:04:09 -04:00
David Steele
1ddfb1e453 Fixed pgBackRest-specific xml that was loaded for non-pgBackRest projects. 2016-06-04 11:40:33 -04:00
David Steele
8ddfab0cbc Fixed DTD search path.
The search path that did not work properly when --doc-path was used.
2016-06-04 11:38:23 -04:00
David Steele
e796f563c4 Add cmd-extra field.
Allow hidden options to be added to a command. This allows certain commands (like apt-get) to be forced during the build without making that a part of the documentation.
2016-06-03 20:07:28 -04:00
David Steele
36cf0b18f5 Update installation documentation.
* Recommended install location for pgBackRest modules is now /usr/share/perl5 since /usr/lib/perl5 has been removed from the search path in newer versions of Perl.

* Added instructions for removing prior versions of pgBackRest.
2016-06-03 18:59:12 -04:00
David Steele
bfaee7380d v1.02: Bug Fix for Perl 5.22
* Fix usage of sprintf() due to new constraints in Perl 5.22. Parameters not referenced in the format string are no longer allowed. (Fixed by Adrian Vondendriesch.)
2016-06-02 11:10:24 -04:00
David Steele
eaa981c5aa Added execution cache for document generation.
Added an execution cache so that documentation can be generated without setting up the full container environment. This is useful for packaging, keeps the documentation consistent for a release, and speeds up generation when no changes are made in the execution list.
2016-06-02 09:32:56 -04:00
David Steele
df6086bd24 Added a man page to document generation. 2016-06-02 09:25:12 -04:00
David Steele
9a9f26a96f Some word smithing in the introduction. 2016-05-27 18:36:25 -04:00
David Steele
174b16165b Replaced overzealous perl -cW check.
It failed on Perl 5.22 so replaced with with perl -cw.
2016-05-26 16:57:35 -04:00
David Steele
379ab1b353 Improve log directory/file creation.
Log directory create and file open now using FileCommon functions which produce more detailed error messages on failure.
2016-05-26 15:04:18 -04:00
David Steele
e4e885d602 Add notes for Ubuntu 16.04 upgrade. 2016-05-26 14:22:26 -04:00
David Steele
cf344c144f Fixed absolute paths that were used for the PDF logo. 2016-05-26 14:17:50 -04:00
Adrian Vondendriesch
b19143b6ea Fixed syntax in doc code that was not compatible with Perl 5.2X. 2016-05-26 14:07:29 -04:00
Adrian Vondendriesch
3d437d9278 Fix usage of sprintf() for Perl 5.22.
Parameters not referenced in the format string are no longer allowed.
2016-05-26 11:23:52 -04:00
David Steele
c8d68bcf2d More detailed release notes.
Release notes are now broken into sections so that bugs, features, and refactors are clearly delineated.  An "Additional Notes" section has been added for changes to documentation and the test suite that do not affect the core code.
2016-05-26 10:34:10 -04:00
David Steele
0fb8bcbfb7 Lists can now be used outside of p and text tags.
This allows for more flexible document structuring.
2016-05-26 09:34:03 -04:00
Cynthia Shang
5a85122841 Moved change log to website.
The change log was the last piece of documentation to be rendered in Markdown only.  Wrote a converter so the document can be output by the standard renderers.  The change log will now be located on the website and has been renamed to "Releases".
2016-05-26 09:20:55 -04:00
David Steele
e2094c3d31 Remove function constants and pass strings directly to logDebugParam().
The function names were only used once so creating constants for them was wasteful.
2016-05-26 09:09:42 -04:00
David Steele
c604b6e657 Test release version using change-log.xml instead of CHANGELOG.md.
This is in preparation for moving the change log to the website and removing it from Github.
2016-05-23 19:33:02 -04:00
David Steele
3af48f1d78 Initialize local variable variable immediately, per Perl Critic. 2016-05-20 11:18:51 -04:00
Cynthia Shang
c9632e9874 Closed #195: Added type from source node in manifest.xml.
This parameter `allows the doc code to determine if special handling is needed for the xml document.
2016-05-19 17:10:34 -04:00
Christoph Berg
3a309256fa Closed #193: Fix perl warnings in doc/ tree.
Somewhere between perl 5.14 and 5.20, constructs like this:
    perl -e '$a = {}; keys $a'
started to throw warnings:
    keys on reference is experimental at -e line 1.

Fix by adding a bunch of %{} and @{} casts.
2016-05-19 10:25:04 -04:00
David Steele
b710c01630 v1.01: Enhanced Info, Selective Restore, and 9.6 Support
* Enhanced text output of `info` command to include timestamps, sizes, and the reference list for all backups. Contributed by Cynthia Shang.

* Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.

* Experimental support for non-exclusive backups in PostgreSQL 9.6 beta1. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2016-05-17 01:04:01 -04:00
David Steele
0451d3afdd Support for non-exclusive backups in PostgreSQL 9.6. 2016-05-16 17:59:26 -04:00
Cynthia Shang
28c5e54b09 Added SEO and sharing features to the website.
Descriptions wordsmithed (for better or worse) by committer.
2016-05-16 17:01:48 -04:00
David Steele
0c320e7df7 Allow selective restore of databases from a cluster backup.
This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 09:21:39 -04:00
Cynthia Shang
60b901948a Enhanced text output of info command to include timestamps, sizes, and the reference list for all backups. 2016-05-06 10:25:46 -04:00
David Steele
c477d6f899 Next development cycle. 2016-05-05 18:13:50 -04:00
Cynthia Shang
6cd1c42ddc Add usage documentation for test.pl. 2016-05-03 16:28:20 -04:00
Cynthia Shang
7b030c5173 User Guide clarifications. 2016-05-03 16:12:44 -04:00
Cynthia Shang
afc8a76ee9 Fixed typos in user guide. 2016-04-25 16:33:08 -04:00
David Steele
226d266d08 Added current stable release to documentation. 2016-04-17 11:43:33 -04:00
David Steele
0d34c34cea Added feature backlog.
This backlog is intended to hold long-lived feature requests that clutter up the issues area on Github.
2016-04-17 11:27:19 -04:00
David Steele
7bdfa7839a Minor wording change: problems => challenges. 2016-04-17 09:32:03 -04:00
David Steele
7bd9b28ff4 Expanded feature bullet points into a new section with detailed explanations.
Copy editing by John Harvey (@crunchyjohn), Jason O'Donnell (@Dwaligon), and Stephen Frost (@sfrost).
2016-04-17 08:23:09 -04:00
David Steele
ed20c2eda3 Close #172: Unable to unpack Int64 when running on 32-bit OS
Added a note to documentation that only 64-bit distributions are supported.  It seems unlikely that anybody would be running a production server on anything else these days so we'll wait for a field report before taking further action.
2016-04-16 16:38:44 -04:00
David Steele
dee3e86ff8 README.md is now rendered by new Markdown code. 2016-04-16 11:55:23 -04:00
David Steele
ad8ad11e4f Update URLs for new Github org. 2016-04-16 11:11:29 -04:00
David Steele
9ab0dd80f7 v1.00: New Repository Format and Configuration Scheme, Link Support
* This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, configuration scheme, and the exe/path names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. Restores from the prior repository will require the prior version of pgBackRest but because of name changes it is possible to have 1.00 and a prior version of pgBackRest installed at the same time. See the notes below for more detailed information on what has changed.

* Implemented a new configuration scheme which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. A local repository is no longer required.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally when migrating to 1.00.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest is the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.

* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 23:04:20 -04:00
David Steele
9457e15347 New manifest format.
* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 22:50:02 -04:00
Cynthia Shang
9a7d8b26cc Fixed some inaccurate statements in the user guide.
Fixed typo in documentation code comments.
2016-04-14 13:50:58 -04:00
David Steele
18fd25233b New simpler configuration and consistent project/exe/path naming.
* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required.

* Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 09:30:54 -04:00
David Steele
885797e4b5 Migrated many functions from File.pm to FileCommon.pm.
This makes make the migrated file functions available to parts of the code that don't have access to a File object.  They still exist as wrappers in the File object to support remote calls.
2016-04-13 19:09:35 -04:00
David Steele
be8487dbad Changed some TBD comments from urgent to whenever. 2016-04-13 19:05:31 -04:00
David Steele
723abadb75 Change spacing in expect logs to accommodate DETAIL level. 2016-04-13 18:55:34 -04:00
David Steele
aacc369e4f Next development cycle. 2016-04-06 09:41:33 -04:00
David Steele
5307c858a7 v0.92: Command-line Repository Path Fix
* Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf.  Reported by Jan Wieck.
2016-04-06 09:14:06 -04:00
David Steele
1ce9961f8a Closed #183: Options --repo-path and --repo-remote-path ignored in archive-push.
Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf.
2016-04-06 08:57:46 -04:00
John K. Harvey
1bbf3e6db4 Fixed #181: Missing semicolon violates code style. 2016-03-22 22:18:33 -04:00
David Steele
1d4779cfe6 Next development cycle. 2016-03-22 10:23:20 -04:00
David Steele
aea14c8553 v0.91: Tablespace Bug Fix and Minor Enhancements
* **IMPORTANT BUG FIX FOR TABLESPACES**: A change to the repository format was accidentally introduced in 0.90 which means the on-disk backup was no longer a valid PostgreSQL cluster when the backup contained tablespaces. This only affected users who directly copied the backups to restore PostgreSQL clusters rather than using the restore command. However, the fix breaks compatibility with older backups that contain tablespaces no matter how they are being restored (pgBackRest will throw errors and refuse to restore). New full backups should be taken immediately after installing version 0.91 for any clusters that contain tablespaces. If older backups need to be restored then use a version of pgBackRest that matches the backup version. _Reported by Evan Benoit_.

* Copy global/pg_control last during backups.

* Write .info and .manifest files to temp before moving them to their final locations and fsync'ing.

* Rename --no-start-stop option to --no-online.

* Static source analysis using Perl-Critic, currently passes on gentle.
2016-03-22 10:16:42 -04:00
David Steele
d489914173 Closed #180: Backup format regression in v0.90. 2016-03-22 09:57:39 -04:00
David Steele
c669dc01e9 Fixed url references in latex documentation to use \href instead of \hyperref. 2016-03-10 10:58:07 -05:00
Brian Faherty
87dd0a127f Clarification for delta restore documentation to indicate that files may be removed. 2016-03-09 11:27:08 -05:00
David Steele
004067a5fa Correct retention-diff example value in documentation. 2016-03-08 09:29:03 -05:00
David Steele
d4c46acf48 Working on #174: Get Perl Critic to pass on stern
Added single exception for the violation of BuiltinFunctions::ProhibitStringyEval and removed from policy file.
2016-02-23 09:39:37 -05:00
David Steele
048571e23f Closed #173: Add static source code analysis
Perl Critic added and passes on gentle.  A policy file has been created with some permanent exceptions and a list of policies to be fixed in approximately the order they should be fixed in.
2016-02-23 09:25:22 -05:00
David Steele
d35ab82a83 Closed #167: Rename --no-start-stop option to --no-online. 2016-02-19 19:32:02 -05:00
David Steele
fb63a981cb Closed #166: Copy global/pg_control last during backup. 2016-02-14 20:42:11 -05:00
David Steele
f360d5072e Next development cycle. 2016-02-07 11:20:43 -05:00
David Steele
631535ae4f v0.90: 9.5 Support, Various Enhancements, and Minor Bug Fixes
* Fixed an issue where specifying --no-archive-check would throw a configuration error. Reported by Jason O'Donnell.

* Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail.

* Fixed an issue where document generation failed because some OSs are not tolerant of having multiple installed versions of PostgreSQL. A separate VM is now created for each version. Also added a sleep after database starts during document generation to ensure the database is running before the next command runs. Reported by John Harvey.

* The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will not be possible for the backups that fall outside of archive retention.

* When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a restore. This only applies to PostgreSQL >= 9.0 -- prior versions of PostgreSQL could not share a tablespace directory.

* Generate an error when archive-check=y but archivecommand does not execute pgbackrest. Contributed by Jason O'Donnell.

* Improved error message when repo-path or repo-remote-path does not exist.

* Added checks for --delta and --force restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of PGVERSION or backup.manifest (left over from an aborted restore). If neither file is found then --delta and --force will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.

* When restore --set=latest (the default) the actual backup restored will be output to the log.

* Support for PostgreSQL 9.5 partial WAL segments and recoverytargetaction setting. The archivemode = 'always' setting is not yet supported.

* Support for recoverytarget = 'immediate' recovery setting introduced in PostgreSQL 9.4.

* The following tablespace checks have been added: paths or files in pgtblspc, relative links in pgtblspc, tablespaces in $PGDATA. All three will generate errors.
2016-02-07 10:47:16 -05:00
David Steele
de9b6d5133 Closed #113: Better error message when repo does not exist. 2016-02-06 16:49:12 -05:00
David Steele
f8b47997a3 Closed #127: More sanity checking for --delta restores
Added checks for `--delta` and `--force` restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of `PG_VERSION` or `backup.manifest` (left over from an aborted restore). If neither is found then `--delta` and `--force` will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
2016-02-05 23:03:29 -05:00
David Steele
5c815e4fc0 Closed #58: Get catalog number for better tablespace copying
When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a `--delta` restore. This only applies to PostgreSQL >= 9.0 -- before that only one PostgreSQL version could use a tablespace.
2016-02-02 14:33:15 -05:00
David Steele
24a1ad31d0 Minor rewording in user guide. 2016-02-02 11:57:46 -05:00
David Steele
02405e851c Closed #163: Document Generation Issue
Fixed an issue where document generation failed because some OSs are not tolerant of having multiple installed versions of PostgreSQL. A separate VM is now created for each version. Also added a sleep after database starts during document generation to ensure the database is running before the next command runs.

Reported by John Harvey.
2016-01-22 18:01:21 -05:00
David Steele
bacdd5583d User guide highlighting fixes for #93. 2016-01-22 14:38:48 -05:00
David Steele
e42b8de618 Updated Copyright for 2016. 2016-01-20 21:45:03 -05:00
David Steele
83b1a3cb44 Closed #150: Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail. 2016-01-20 21:28:40 -05:00
David Steele
747346db67 Closed #162: When restore --set=latest (the default) the actual backup restored will be output to the log. 2016-01-20 18:34:42 -05:00
David Steele
2d38876bc4 Closed #93: The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will still not be possible for backups that fall outside of archive retention. 2016-01-20 17:24:44 -05:00
David Steele
4d12cc233e Fix for typo in error message. 2016-01-15 20:31:29 -05:00
David Steele
e91e481f3d Closed #19: The following tablespace checks have been added: paths or files in pg_tblspc, relative links in pg_tblspc, tablespaces in $PGDATA. All three will generate errors. 2016-01-15 07:07:00 -05:00
David Steele
65d0c7dc3b Updated Crunchy URL. 2016-01-13 22:48:35 -05:00
David Steele
a0a3d1e97e Closed #122: 9.5 Integration
The archive_mode = always setting is not included but is covered by issue #125.
2016-01-13 20:35:12 -05:00
Jason O'Donnell
c7502f341b Generate an error when archive-check=y but archive_command does not execute pg_backrest. 2016-01-11 09:50:45 -05:00
David Steele
81a2fde21f Fixed typos in test documentation. 2016-01-10 21:07:07 -05:00
David Steele
29102b2887 Added error when archive_mode=always is configured. 2016-01-09 23:08:54 -05:00
David Steele
7f0f8823df Added support for partial WAL segments. 2016-01-09 19:37:28 -05:00
David Steele
e72165a689 Provisional support for PostgreSQL 9.5. No support for archive_mode = 'always' or .partial WAL on a standby. 2016-01-09 10:11:03 -05:00
David Steele
c8f863fbab Text execution improvements:
1) Tests for all operating systems can now be run with a single command.
2) Tests can be run in parallel with --process-max.
3) Container generation now integrated into test.pl
4) Some basic test documentation.
2016-01-09 08:21:53 -05:00
David Steele
6f6c796fbf Fixed an issue where specifying --no-archive-check would throw a configuration error. 2016-01-03 20:21:30 -05:00
David Steele
7f630aec49 Switched to using docker for all unit tests as this allows for greater automation. 2015-12-29 13:57:10 -05:00
David Steele
91d1e94076 Next development cycle. 2015-12-24 12:49:47 -05:00
David Steele
50a62fab6d v0.89: Timeout Bug Fix and Restore Read-Only Repositories
* Fixed an issue where longer-running backups/restores would timeout when remote and threaded. Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work. The error message for timeouts was also improved to make debugging easier.

* Allow restores to be performed on a read-only repository by using --no-lock and --log-level-file=off. The --no-lock option can only be used with restores.

* Minor styling changes, clarifications and rewording in the user guide.

* The dev branch has been renamed to master and for the time being the master branch has renamed to release, though it will probably be removed at some point -- thus ends the gitflow experiment for pgBackRest. It is recommended that any forks get re-forked and clones get re-cloned.
2015-12-24 11:32:29 -05:00
David Steele
ba098d7b91 Fixed an issue where longer-running backups/restores would timeout when remote and threaded.
Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work.  The error messages for timeouts was also improved to make debugging easier.
2015-12-24 10:32:25 -05:00
David Steele
b0a6954671 A number of doc improvements:
1) Started on a general markdown renderer
2) Internal links now work in PDF
3) Improvements to PDF styling
4) Some comment and formatting fixes
5) User guide edits.
2015-12-23 11:04:26 -05:00
David Steele
fdc99e30bd Allow restores to be run against a read-only repository.
Two things needed to be changed:
1) Don't open a log file when log-level-file=off
2) New --no-lock option to suppress lock file creation for restores.
2015-12-23 10:48:33 -05:00
David Steele
4925756dee Merge branch 'master' into dev after release 0.88 2015-11-22 18:16:54 -05:00
David Steele
e04c21b012 v0.88: Documentation and Minor Bug Fixes
* Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication.

* Fixed an issue where the start/stop commands required the --config option.

* Fixed an issue where log files were being overwritten instead of appended.

* Fixed an issue where backup-user was not optional.

* Symlinks are no longer created in backup directories in the repository. These symlinks could point virtually anywhere and potentially be dangerous. Symlinks are still recreated during a restore.

* Added better messaging for backup expiration. Full and differential backup expirations are logged on a single line along with a list of all dependent backups expired.

* Archive retention is automatically set to full backup retention if not explicitly configured.
2015-11-22 17:27:30 -05:00
David Steele
fa05715dec Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication. 2015-11-22 16:44:01 -05:00
David Steele
8ddfdcdd3b Various fixes and features implemented during doc development
* Better messaging for expiration.
* Fixed already stopped message.
* retention-archive and retention-archive-type now use retention-full and 'full' when not specified.
* Fixed issue where backup-user was required (should default to backrest).
* ExecuteTest now supports retries.
* Fixed issue where log test was not comparing test logs.
* Fixed issue where test logs would not match for ssh connection errors
2015-11-22 14:02:14 -05:00
David Steele
7ff9fc996c Release notes for fixes and rebuild command-line help. 2015-11-20 11:03:06 -05:00
David Steele
e5cf1d84a7 CSS Tweaks. 2015-10-29 17:45:29 +01:00
David Steele
2f83362f32 Minor user guide fix. 2015-10-28 13:02:35 +01:00
David Steele
a498becf11 Merge branch 'master' into dev after release 0.87 2015-10-28 11:29:04 +01:00
David Steele
2813c4d4eb v0.87: Website and User Guide
* Added a new user guide that covers pgBackRest basics and some advanced topics including PITR. Much more to come, but it's a start.

* The website, markdown, and command-line help are now all generated from the same XML source.

* The backup_label.old and recovery.done files are now excluded from backups.
2015-10-28 11:15:38 +01:00
David Steele
32ce87d56d Fixed minor path issue in README.md. 2015-10-28 10:19:33 +01:00
David Steele
57a06ba3b1 Implemented issue #148: Website with basic user guide. 2015-10-28 10:10:36 +01:00
David Steele
3dc0142244 Renamed userguide.xml to reference.xml so it doesn't conflict with the new user guide. 2015-10-08 14:36:39 -04:00
David Steele
994ead1f05 Removed IO::String dependency from docs. 2015-10-08 13:47:51 -04:00
David Steele
0bcbe21520 Merge branch 'master' into dev after release 0.85 2015-10-08 12:44:55 -04:00
David Steele
097eb7ca41 v0.85: Start/Stop Commands and Minor Bug Fixes
* Added new feature to allow all pgBackRest operations to be stopped or started using the stop and start commands.  This prevents any pgBackRest processes from running on a system where PostgreSQL is shutdown or the system needs to be quiesced for some reason.

* Removed dependency on IO::String module.

* Fixed an issue where an error could be returned after a backup or restore completely successfully.

* Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed.  This scenario was likely if the backup process got terminated during the copy phase.

* Experimental support for PostgreSQL 9.5 beta1.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
2015-10-08 12:34:50 -04:00
David Steele
1f9d6c330a Updated WAL magic for PostgreSQL 9.5 beta1. 2015-10-08 12:22:57 -04:00
David Steele
ddd4c14773 Implemented issue #129: Stop/Start Commands.
Implemented issue #136: Add stderr back into protocol layer.

There were also miscellaneous bug fixes and improvements to buffering in IO.pm.
2015-10-08 11:43:56 -04:00
David Steele
e488f7bb19 Fixed issue #144: Error on resume when temp files (file.tmp.*) present. 2015-09-22 21:07:49 -04:00
David Steele
210f8ef723 Merge branch 'master' into dev after release 0.82 2015-09-14 12:28:22 -04:00
David Steele
9be15d00f8 v0.82: Refactoring, Command-line Help, and Minor Bug Fixes
* Fixed an issue where resumed compressed backups were not preserving existing files.

* Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.

* Fixed an issue where a cold backup using --no-start-stop could be started on a running PostgreSQL cluster without --force specified.

* Fixed an issue where a thread could be started even when none were requested.

* Fixed an issue where the pgBackRest version number was not being updated in backup.info and archive.info after an upgrade/downgrade.

* Fixed an issue where the info command was throwing an exception when the repository contained no stanzas.  Reported by Stephen Frost.

* Fixed an issue where the PostgreSQL pg_stop_backup() NOTICEs were being output to stderr.  Reported by Stephen Frost.

* Renamed recovery-setting option and section to recovery-option to be more consistent with pgBackRest naming conventions.

* Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.

* Code cleanup and refactoring to standardize on patterns that have evolved over time.

* Added dynamic module loading to speed up commands, especially asynchronous archiving.

* Expiration tests are now synthetic rather than based on actual backups.  This will allow development of more advanced expiration features.

* Experimental support for PostgreSQL 9.5 alpha2.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
2015-09-14 11:18:50 -04:00
David Steele
e0dafbff7f Improvements to issue #132: Improved command-line help.
Regression tests are now more comprehensive by default.
Better handling for errors in safeExit().
Release notes.
2015-09-09 15:40:54 -04:00
David Steele
adc5568b15 Fixed issue #132: Improved command-line help.
Some platform specific fixes and added regression tests to detect issues.
2015-09-08 12:58:13 -04:00
David Steele
ac3c0d43ab Implemented issue #132: Improved command-line help.
Implemented issue #133: Dynamic module loading where possible.
2015-09-08 07:31:24 -04:00
David Steele
ee55817fce Refactoring documentation generator. 2015-09-02 18:55:04 -04:00
David Steele
bf3ef15ccc Implemented issue #117: Refactor expiration tests to be purely synthetic 2015-09-01 19:05:10 -04:00
David Steele
6a9377a0a9 Code cleanup and refactoring to standardize on patterns that have evolved over time. 2015-08-29 14:20:46 -04:00
David Steele
83473eb9fb Merge branch 'master' into dev after release 0.80 2015-08-09 11:29:00 -04:00
David Steele
a59284a4cc v0.80: DBI Support, Stability, and Convenience Features
* Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the info command.  Only text output was affected -- json output reported the correct epoch values.  Reported by Michael Renner.

* Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.

* Now using Perl DBI and DBD::Pg for connections to PostgreSQL rather than psql.  The cmd-psql and cmd-psql-option settings have been removed and replaced with db-port and db-socket-path.

* Add stop-auto option to allow failed backups to automatically be stopped when a new backup starts.

* Add db-timeout option to limit the amount of time pgBackRest will wait for pg_start_backup() and pg_stop_backup() to return.

* Remove pg_control file at the beginning of the restore and copy it back at the very end.  This prevents the possibility that a partial restore can be started by PostgreSQL.

* The repository is now created and updated with consistent directory and file modes.  By default umask is set to 0000 but this can be disabled with the neutral-umask setting.

* Added checks to be sure the db-path setting is consistent with db-port by comparing the data_directory as reported by the cluster against the db-path setting and the version as reported by the cluster against the value read from pg_control.  The db-socket-path setting is checked to be sure it is an absolute path.

* Experimental support for PostgreSQL 9.5 alpha1.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.

* Major refactoring of the protocol layer to support future development.

* Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.

* Split most of README.md out into USERGUIDE.md and CHANGELOG.md because it was becoming unwieldy.  Changed most references to "database" in the user guide to "database cluster" for clarity.
2015-08-09 11:20:43 -04:00
David Steele
65d9de4ee7 Document issue #114: Better description of repo-path. 2015-08-09 09:18:58 -04:00
David Steele
45259a487f Change log for issue #94: backup_stop needs a timeout. 2015-08-08 18:36:36 -04:00
David Steele
1ab6239b4e Implemented issue #118: Clean up database vs. database cluster in the docs. 2015-08-08 18:32:41 -04:00
David Steele
e82afac885 Implemented issue #94: backup_stop needs a timeout.
The timeout applies to all queries agains the cluster during a backup.
2015-08-08 18:22:38 -04:00
David Steele
8c0508789e Implemented issue #25: Option to call stop_backup with every backup. 2015-08-08 17:11:20 -04:00
David Steele
2edf5d4bf7 Split most of README.md out into CHANGELOG.md and USERGUIDE.md.
Added release dates to change log.
2015-08-07 14:43:53 -04:00
David Steele
4e7bd4468a Worked on issue #122: 9.5 Integration.
Most tests are working now.  What's not working:

1) --target-resume option fails because pause_on_recovery setting was removed.  Need to implement to the new 9.5 option and make that work with older versions in a consistent way.
2) No tests for the new .partial WAL segments that can be generated on timeline switch.
2015-08-06 16:36:55 -04:00
David Steele
adb8a00925 Improvements to release notes. 2015-08-06 00:24:02 -04:00
David Steele
8b57188bc1 Fixed issue #120: Wrong "oldest backup timestamp" in info output. 2015-08-05 23:37:46 -04:00
David Steele
a10dd8ba98 Implemented issue #121: Check data from db against what's passed to backrest. 2015-08-05 22:05:45 -04:00
David Steele
ba4e5fc4cb Release notes for issue #20: Convert Postgres calls from psql to DBD::Pg. 2015-08-05 13:35:19 -04:00
David Steele
23e6267adc Implemented issue #20: Convert Postgres calls from psql to DBD::Pg. 2015-08-05 12:32:12 -04:00
David Steele
f8a2da9400 Work on issue #48: Abandon threads and go to processes
* Major refactoring of the protocol layer to support this work.
* Fixed protocol issue that was preventing ssh errors (especially connect) from being logged.
2015-08-05 08:43:41 -04:00
David Steele
97c036973c Implemented issue #119: Make sure all repo permissions are 640/750. 2015-07-23 13:11:38 -04:00
David Steele
f3199d13b1 Implement issue #115: When restoring copy pg_control last. 2015-07-16 11:12:48 -04:00
David Steele
702afeb6f0 Merge branch 'master' into dev after release 0.78 2015-07-13 15:37:50 -04:00
David Steele
c568f5e727 v0.78: Remove CPAN dependencies, stability improvements
* Removed dependency on CPAN packages for multi-threaded operation.  While it might not be a bad idea to update the threads and Thread::Queue packages, it is no longer necessary.

* Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.

* Modified wait backoffs to use a Fibonacci sequence rather than geometric.  This will make then grow less aggressively while still giving reasonable wait times.

* More options for regression tests and improved code to run in a variety of environments.
2015-07-13 14:44:32 -04:00
David Steele
89c0f23f37 Update release notes and some test fixes. 2015-07-13 12:13:07 -04:00
David Steele
e9ce4efedf Removed dependency on dequeue_timed() which eliminates the CPAN install.
Added vagrant config for CentOS 6.
2015-07-10 09:20:28 -04:00
David Steele
1df8c30549 Swapped config and operation sections to hopefully make the docs flow a little better. 2015-07-09 10:21:02 -04:00
David Steele
e799361e90 Clarification for CPAN requirements. 2015-07-09 10:11:49 -04:00
David Steele
bb443dd098 Renamed pg_backrest.pl to pg_backrest in the docs. 2015-07-06 17:59:44 -04:00
David Steele
38f48be3cf Improvements to help to make it more dynamic depending on environment.
Changed PgBackRest to pgBackRest.
2015-07-02 10:05:13 -04:00
David Steele
12932dc54c Merge branch 'master' into dev after release 0.77 2015-06-30 14:57:17 -04:00
David Steele
b1ded135df v0.77: CentOS/RHEL 6 support and protocol improvements
* Removed pg_backrest_remote and added the functionality to pg_backrest as remote command.

* Added file and directory syncs to the File object for additional safety during backup/restore and archiving.  Suggested by Andres Freund.

* Support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6.  Found by Eric Radman.

* Improved error message when backup is run without archive_command set and without --no-archive-check specified.  Found by Eric Radman.

* Moved version number out of the VERSION file to Version.pm to better support packaging.  Suggested by Michael Renner.

* Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple distros.
2015-06-30 14:47:58 -04:00
David Steele
7248795b91 Work on issue #48: Abandon threads and go to processes
Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple distros.  Using open3 will also be used for local processes so it make sense to switch now.
2015-06-29 22:07:42 -04:00
David Steele
f210fe99c3 Implemented issue #109: Move VERSION into source code.
Also stopped replacing FORMAT number which explains the large number of test log changes.  FORMAT should change very rarely and cause test log failures when it does.
2015-06-22 13:11:07 -04:00
David Steele
af58016871 Release notes for issue #108: Incompatibility with Perl 5.10.1. 2015-06-22 10:04:58 -04:00
David Steele
fdd9e8eb17 Improved release notes on fsync to include restore. 2015-06-22 09:58:01 -04:00
David Steele
3f841fcd95 Improved issue #110: 'db-version' is required but not defined.
Improved the error message and added hints.
2015-06-22 09:51:16 -04:00
David Steele
61dfeca3c1 Fixed issue #108: Incompatibility with Perl 5.10.1 2015-06-21 12:06:13 -04:00
David Steele
49f1a3e481 Document issue #108: Incompatibility with Perl 5.10.1 2015-06-19 14:27:44 -04:00
David Steele
0695c0e7af Document issue #108: Incompatibility with Perl 5.10.1 2015-06-19 14:02:26 -04:00
David Steele
a5d9d6d84d Added file and directory syncs to the File object for additional safety during backup and archiving. 2015-06-17 12:53:33 -04:00
David Steele
9511f9c35c Implemented issue #106: Roll pg_backup_remote into pg_backrest 2015-06-17 11:26:07 -04:00
David Steele
0ba4029253 Update version to 0.80. 2015-06-16 16:21:24 -04:00
David Steele
da8b337ee8 Added supported versions of PostreSQL. 2015-06-16 15:44:27 -04:00
David Steele
1afbab010f v0.75: New repository format, info command and experimental 9.5 support
* IMPORTANT NOTE: This flag day release breaks compatibility with older versions of PgBackRest.  The manifest format, on-disk structure, and the binary names have all changed.  You must create a new repository to hold backups for this version of PgBackRest and keep your older repository for a time in case you need to do a restore.  The `pg_backrest.conf` file has not changed but you'll need to change any references to `pg_backrest.pl` in cron (or elsewhere) to `pg_backrest` (without the `.pl` extension).

* Add info command.

* More efficient file ordering for backup.  Files are copied in descending size order so a single thread does not end up copying a large file at the end.  This had already been implemented for restore.

* Logging now uses unbuffered output.  This should make log files that are being written by multiple threads less chaotic.  Suggested by Michael Renner.

* Experimental support for PostgreSQL 9.5.  This may break when the control version or WAL magic changes but will be updated in each release.
2015-06-14 20:59:32 -04:00
David Steele
b365d9aacc Updated release notes for 0.75. 2015-06-14 11:48:33 -04:00
David Steele
b865070edd Experimental 9.5 support. Unit tests are not working yet. 2015-06-14 10:12:36 -04:00
David Steele
0b6f81a812 Implemented issue #50: Reverse ordering by file size. 2015-06-14 09:36:58 -04:00
David Steele
6c807f2ddd Fixed release notes for info command. 2015-06-13 18:30:10 -04:00
David Steele
148836fe44 Implemented issue #26: Info command.
* Includes updating the manifest to format 4.  It turns out the manifest and .info files were not very good for providing information.  A format update was required anyway so worked through the backlog of changes that would require a format change.

* Multiple database versions are now supported in the archive.  Does't actually work yet but the structure should be good.

* Tests use more constants now that test logs can catch name regressions.
2015-06-13 18:25:49 -04:00
David Steele
5a73b1a240 Fixed minor typos in the documentation. 2015-06-10 15:54:24 -04:00
David Steele
aca4511350 Documentation for issue #93: Prevent misconfigured retention-archive setting
Not an actual fix - just some documentation to (hopefully) keep users from getting bitten until then.
2015-06-10 15:44:13 -04:00
David Steele
af98f164cd Fixed issue #97: Interleaved log lines during parallel operations. 2015-06-02 09:42:56 -04:00
David Steele
7e06e53a69 Merge branch 'master' into dev after release 0.70 2015-06-01 09:39:02 -04:00
David Steele
f83f0fa54d v0.70: Stability improvements for archiving, improved logging and help
* Fixed an issue where archive-copy would fail on an incr/diff backup when hardlink=n.  In this case the pg_xlog path does not already exist and must be created. Reported by Michael Renner

* Allow duplicate WAL segments to be archived when the checksum matches.  This is necessary for some recovery scenarios.

* Allow comments/disabling in pg_backrest.conf using #.  Suggested by Michael Renner.

* Better logging before pg_start_backup() to make it clear when the backup is waiting on a checkpoint.  Suggested by Michael Renner.

* Various command behavior, help and logging fixes.  Reported by Michael Renner.

* Fixed an issue in async archiving where archive-push was not properly returning 0 when archive-max-mb was reached and moved the async check after transfer to avoid having to remove the stop file twice.  Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong.  Reported by Michael Renner.

* Fixed a locking issue that could allow multiple operations of the same type against a single stanza.  This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. Reported by Michael Renner.

* Replaced JSON module with JSON::PP which ships with core Perl.
2015-06-01 09:24:38 -04:00
David Steele
e7d2d704fe Fixed issue #91: Race condition in async archive-push. 2015-05-29 16:31:12 -04:00
David Steele
5fce709ae7 Document issue #88: Thoughts on repo & release management. 2015-05-29 12:33:38 -04:00
David Steele
d321ef0b6d Implement issue #89: Make confess backtraces log-level dependent.
ASSERTs still dump stack traces to the console and file in all cases.  ERRORs only dump stack traces to the file when the file log level is DEBUG or TRACE.
2015-05-29 12:26:31 -04:00
David Steele
13e4eec629 Implement issue #90: Move archive-max-mb archive-stop checks to end of archive-push. 2015-05-29 11:41:19 -04:00
David Steele
639c27e113 Release notes for issue #68: Allow duplicate WAL to be pushed if the checksum matches. 2015-05-29 08:18:12 -04:00
David Steele
3083e5fc7e Fix for issue #83: Provide more helpful error message during archive-stop situations.
Fix for issue #84: archive-async in combination with archive-max-mb doesn't work as documented.
Unit tests for archive stop.
2015-05-28 13:17:34 -04:00
David Steele
a4974f8996 Fix for issue #70: command line help improvements 2015-05-26 15:29:42 -04:00
David Steele
ca0c866d3b Implement issue #71: log a message when waiting for a checkpoint
Some other log improvements
2015-05-26 12:26:59 -04:00
David Steele
5c43e7035a Implement issue #79: Support comment characters in config file 2015-05-26 11:36:03 -04:00
David Steele
e9099b99aa Updated required modules. Minor doc fixes. 2015-05-26 10:01:05 -04:00
David Steele
d5335b40e8 Fix for issue #80: enabling archive-copy causes failing differential & incremental backups 2015-05-22 14:49:14 -04:00
David Steele
5ada7fb5ad Use JSON::PP instead of JSON since it is shipped with core Perl. 2015-05-12 15:44:10 -04:00
David Steele
ab4efce5d1 Merge branch 'master' into dev after release 0.65 2015-05-11 18:36:02 -04:00
David Steele
49fe40850f v0.65: Improved resume and restore logging, compact restores
* Better resume support.  Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses.  More unit tests to verify each resume case.

* Resume is now optional.  Use the `resume` setting or `--no-resume` from the command line to disable.

* More info messages during restore.  Previously, most of the restore messages were debug level so not a lot was output in the log.

* Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.

* Added `tablespace` setting to allow tablespaces to be restored into the `pg_tblspc` path.  This produces compact restores that are convenient for development, staging, etc.  Currently these restores cannot be backed up as PgBackRest expects only links in the `pg_tblspc` path.
2015-05-11 18:29:40 -04:00
David Steele
9aa41ee76b Added tablespace setting to allow compact database restores for development, staging, etc. 2015-05-08 18:34:27 -04:00
David Steele
a0d985b7ad Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path. 2015-05-07 17:37:25 -05:00
David Steele
1ac4b781fd Better info logging for restore. Most of the messages were debug before and some important ones were missing. 2015-05-07 15:56:56 -06:00
David Steele
095a9a0b83 Added resume setting to allow disabling resume feature. 2015-05-07 10:29:30 -06:00
David Steele
328c2ca5c0 Much better resume:
1) Re-checksums files that have checksums in the manifest
2) Recopies files that do not have a checksum
3) Saves the manifest at regular intervals to preserve checksums
4) Unit tests for all cases (that I can think of)
2015-05-06 18:24:34 -06:00
David Steele
1d1c7e47d1 Doc fixes for archive-max-mb setting. 2015-05-01 19:13:12 -04:00
David Steele
b0db7c0497 Fixed case error in Perl module name. 2015-04-29 17:43:28 -04:00
David Steele
87f69508e3 Set version to 0.75 2015-04-21 10:48:01 -04:00
David Steele
6987cfaa53 Merge branch 'master' into dev after release 0.61 2015-04-21 09:46:28 -04:00
David Steele
a9b9ec2492 v0.61: bug fix for uncompressed remote destination
* Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination.  The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
2015-04-21 09:45:03 -04:00
David Steele
4b51f5dc50 Removed unused font. 2015-04-19 18:00:51 -04:00
David Steele
5185d49873 v0.60: better version support and WAL improvements
* Pushing duplicate WAL now generates an error.  This worked before only if checksums were disabled.

* Database System IDs are used to make sure that all WAL in an archive matches up.  This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.

* Regression tests working back to PostgreSQL 8.3.

* Improved threading model by starting threads early and terminating them late.
2015-04-19 17:53:29 -04:00
David Steele
f1cc1d6123 Refactored backup manifest update by centralizing the code. 2015-04-11 15:02:04 -04:00
David Steele
7081c8b867 New model where threads are created early and destroyed late.
Backups now work like restores in terms of how jobs are queued.
Split out BackupFile and RestoreFile for easier multi-threading/processing.
2015-04-07 07:34:37 -04:00
David Steele
0b27c74999 Increased version to 0.60 and added release notes. 2015-04-01 16:28:16 -04:00
David Steele
9caccc2ce2 Doc fixes. 2015-03-29 13:35:36 -04:00
David Steele
b37d59832f v0.50: restore and much more
* Added restore functionality.

* All options can now be set on the command-line making pg_backrest.conf optional.

* De/compression is now performed without threads and checksum/size is calculated in stream.  That means file checksums are no longer optional.

* Added option `--no-start-stop` to allow backups when Postgres is shut down.  If `postmaster.pid` is present then `--force` is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created).  This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.

* Fixed broken checksums and now they work with normal and resumed backups.  Finally realized that checksums and checksum deltas should be functionally separated and this simplied a number of things.  Issue #28 has been created for checksum deltas.

* Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.

* Removed dependency on Moose.  It wasn't being used extensively and makes for longer startup times.

* Checksum for backup.manifest to detect corrupted/modified manifest.

* Link `latest` always points to the last backup.  This has been added for convenience and to make restores simpler.

* More comprehensive unit tests in all areas.
2015-03-25 15:15:55 -04:00
David Steele
82fa497511 Last doc fixes. 2015-03-25 14:43:21 -04:00
David Steele
4a6f4e2340 Getting docs in shape for the next release. 2015-03-24 18:50:54 -04:00
David Steele
1729fa1c51 Added unit tests for expiration. 2015-03-17 18:31:05 -04:00
David Steele
f4f31302f0 Broke archive-required into two options: backup-wal-check and backup-wal-store. 2015-03-16 14:45:53 -04:00
David Steele
258fb9c6e2 More work on automated docs. Merging this to go back to some feature/bug work for a while. 2015-03-16 14:01:01 -04:00
David Steele
7675a11ded First pass at building automated docs for markdown/html. This works pretty well, but the config sections of doc.xml still require too much maintenance. With the new Config code, it should be possible to generate those sections automatically. 2015-03-08 14:05:41 -04:00
David Steele
f115e01b71 More work on docs. 2015-03-04 22:38:27 -05:00
David Steele
8dc3ba184d More work on docs. 2015-03-04 20:09:26 -05:00
David Steele
163f898262 HTML docs look to be in a workable format. 2015-03-04 18:50:02 -05:00
David Steele
bfb2c05357 Started on new HTML docs. The markdown is not cutting it. 2015-03-04 00:36:54 -05:00