1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

1518 Commits

Author SHA1 Message Date
David Steele
ad8ad11e4f Update URLs for new Github org. 2016-04-16 11:11:29 -04:00
David Steele
9ab0dd80f7 v1.00: New Repository Format and Configuration Scheme, Link Support
* This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, configuration scheme, and the exe/path names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. Restores from the prior repository will require the prior version of pgBackRest but because of name changes it is possible to have 1.00 and a prior version of pgBackRest installed at the same time. See the notes below for more detailed information on what has changed.

* Implemented a new configuration scheme which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. A local repository is no longer required.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally when migrating to 1.00.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest is the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.

* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 23:04:20 -04:00
David Steele
9457e15347 New manifest format.
* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 22:50:02 -04:00
Cynthia Shang
9a7d8b26cc Fixed some inaccurate statements in the user guide.
Fixed typo in documentation code comments.
2016-04-14 13:50:58 -04:00
David Steele
18fd25233b New simpler configuration and consistent project/exe/path naming.
* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required.

* Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 09:30:54 -04:00
David Steele
885797e4b5 Migrated many functions from File.pm to FileCommon.pm.
This makes make the migrated file functions available to parts of the code that don't have access to a File object.  They still exist as wrappers in the File object to support remote calls.
2016-04-13 19:09:35 -04:00
David Steele
be8487dbad Changed some TBD comments from urgent to whenever. 2016-04-13 19:05:31 -04:00
David Steele
723abadb75 Change spacing in expect logs to accommodate DETAIL level. 2016-04-13 18:55:34 -04:00
David Steele
aacc369e4f Next development cycle. 2016-04-06 09:41:33 -04:00
David Steele
5307c858a7 v0.92: Command-line Repository Path Fix
* Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf.  Reported by Jan Wieck.
2016-04-06 09:14:06 -04:00
David Steele
1ce9961f8a Closed #183: Options --repo-path and --repo-remote-path ignored in archive-push.
Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf.
2016-04-06 08:57:46 -04:00
John K. Harvey
1bbf3e6db4 Fixed #181: Missing semicolon violates code style. 2016-03-22 22:18:33 -04:00
David Steele
1d4779cfe6 Next development cycle. 2016-03-22 10:23:20 -04:00
David Steele
aea14c8553 v0.91: Tablespace Bug Fix and Minor Enhancements
* **IMPORTANT BUG FIX FOR TABLESPACES**: A change to the repository format was accidentally introduced in 0.90 which means the on-disk backup was no longer a valid PostgreSQL cluster when the backup contained tablespaces. This only affected users who directly copied the backups to restore PostgreSQL clusters rather than using the restore command. However, the fix breaks compatibility with older backups that contain tablespaces no matter how they are being restored (pgBackRest will throw errors and refuse to restore). New full backups should be taken immediately after installing version 0.91 for any clusters that contain tablespaces. If older backups need to be restored then use a version of pgBackRest that matches the backup version. _Reported by Evan Benoit_.

* Copy global/pg_control last during backups.

* Write .info and .manifest files to temp before moving them to their final locations and fsync'ing.

* Rename --no-start-stop option to --no-online.

* Static source analysis using Perl-Critic, currently passes on gentle.
2016-03-22 10:16:42 -04:00
David Steele
d489914173 Closed #180: Backup format regression in v0.90. 2016-03-22 09:57:39 -04:00
David Steele
c669dc01e9 Fixed url references in latex documentation to use \href instead of \hyperref. 2016-03-10 10:58:07 -05:00
Brian Faherty
87dd0a127f Clarification for delta restore documentation to indicate that files may be removed. 2016-03-09 11:27:08 -05:00
David Steele
004067a5fa Correct retention-diff example value in documentation. 2016-03-08 09:29:03 -05:00
David Steele
d4c46acf48 Working on #174: Get Perl Critic to pass on stern
Added single exception for the violation of BuiltinFunctions::ProhibitStringyEval and removed from policy file.
2016-02-23 09:39:37 -05:00
David Steele
048571e23f Closed #173: Add static source code analysis
Perl Critic added and passes on gentle.  A policy file has been created with some permanent exceptions and a list of policies to be fixed in approximately the order they should be fixed in.
2016-02-23 09:25:22 -05:00
David Steele
d35ab82a83 Closed #167: Rename --no-start-stop option to --no-online. 2016-02-19 19:32:02 -05:00
David Steele
fb63a981cb Closed #166: Copy global/pg_control last during backup. 2016-02-14 20:42:11 -05:00
David Steele
f360d5072e Next development cycle. 2016-02-07 11:20:43 -05:00
David Steele
631535ae4f v0.90: 9.5 Support, Various Enhancements, and Minor Bug Fixes
* Fixed an issue where specifying --no-archive-check would throw a configuration error. Reported by Jason O'Donnell.

* Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail.

* Fixed an issue where document generation failed because some OSs are not tolerant of having multiple installed versions of PostgreSQL. A separate VM is now created for each version. Also added a sleep after database starts during document generation to ensure the database is running before the next command runs. Reported by John Harvey.

* The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will not be possible for the backups that fall outside of archive retention.

* When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a restore. This only applies to PostgreSQL >= 9.0 -- prior versions of PostgreSQL could not share a tablespace directory.

* Generate an error when archive-check=y but archivecommand does not execute pgbackrest. Contributed by Jason O'Donnell.

* Improved error message when repo-path or repo-remote-path does not exist.

* Added checks for --delta and --force restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of PGVERSION or backup.manifest (left over from an aborted restore). If neither file is found then --delta and --force will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.

* When restore --set=latest (the default) the actual backup restored will be output to the log.

* Support for PostgreSQL 9.5 partial WAL segments and recoverytargetaction setting. The archivemode = 'always' setting is not yet supported.

* Support for recoverytarget = 'immediate' recovery setting introduced in PostgreSQL 9.4.

* The following tablespace checks have been added: paths or files in pgtblspc, relative links in pgtblspc, tablespaces in $PGDATA. All three will generate errors.
2016-02-07 10:47:16 -05:00
David Steele
de9b6d5133 Closed #113: Better error message when repo does not exist. 2016-02-06 16:49:12 -05:00
David Steele
f8b47997a3 Closed #127: More sanity checking for --delta restores
Added checks for `--delta` and `--force` restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of `PG_VERSION` or `backup.manifest` (left over from an aborted restore). If neither is found then `--delta` and `--force` will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
2016-02-05 23:03:29 -05:00
David Steele
5c815e4fc0 Closed #58: Get catalog number for better tablespace copying
When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a `--delta` restore. This only applies to PostgreSQL >= 9.0 -- before that only one PostgreSQL version could use a tablespace.
2016-02-02 14:33:15 -05:00
David Steele
24a1ad31d0 Minor rewording in user guide. 2016-02-02 11:57:46 -05:00
David Steele
02405e851c Closed #163: Document Generation Issue
Fixed an issue where document generation failed because some OSs are not tolerant of having multiple installed versions of PostgreSQL. A separate VM is now created for each version. Also added a sleep after database starts during document generation to ensure the database is running before the next command runs.

Reported by John Harvey.
2016-01-22 18:01:21 -05:00
David Steele
bacdd5583d User guide highlighting fixes for #93. 2016-01-22 14:38:48 -05:00
David Steele
e42b8de618 Updated Copyright for 2016. 2016-01-20 21:45:03 -05:00
David Steele
83b1a3cb44 Closed #150: Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail. 2016-01-20 21:28:40 -05:00
David Steele
747346db67 Closed #162: When restore --set=latest (the default) the actual backup restored will be output to the log. 2016-01-20 18:34:42 -05:00
David Steele
2d38876bc4 Closed #93: The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will still not be possible for backups that fall outside of archive retention. 2016-01-20 17:24:44 -05:00
David Steele
4d12cc233e Fix for typo in error message. 2016-01-15 20:31:29 -05:00
David Steele
e91e481f3d Closed #19: The following tablespace checks have been added: paths or files in pg_tblspc, relative links in pg_tblspc, tablespaces in $PGDATA. All three will generate errors. 2016-01-15 07:07:00 -05:00
David Steele
65d0c7dc3b Updated Crunchy URL. 2016-01-13 22:48:35 -05:00
David Steele
a0a3d1e97e Closed #122: 9.5 Integration
The archive_mode = always setting is not included but is covered by issue #125.
2016-01-13 20:35:12 -05:00
Jason O'Donnell
c7502f341b Generate an error when archive-check=y but archive_command does not execute pg_backrest. 2016-01-11 09:50:45 -05:00
David Steele
81a2fde21f Fixed typos in test documentation. 2016-01-10 21:07:07 -05:00
David Steele
29102b2887 Added error when archive_mode=always is configured. 2016-01-09 23:08:54 -05:00
David Steele
7f0f8823df Added support for partial WAL segments. 2016-01-09 19:37:28 -05:00
David Steele
e72165a689 Provisional support for PostgreSQL 9.5. No support for archive_mode = 'always' or .partial WAL on a standby. 2016-01-09 10:11:03 -05:00
David Steele
c8f863fbab Text execution improvements:
1) Tests for all operating systems can now be run with a single command.
2) Tests can be run in parallel with --process-max.
3) Container generation now integrated into test.pl
4) Some basic test documentation.
2016-01-09 08:21:53 -05:00
David Steele
6f6c796fbf Fixed an issue where specifying --no-archive-check would throw a configuration error. 2016-01-03 20:21:30 -05:00
David Steele
7f630aec49 Switched to using docker for all unit tests as this allows for greater automation. 2015-12-29 13:57:10 -05:00
David Steele
91d1e94076 Next development cycle. 2015-12-24 12:49:47 -05:00
David Steele
50a62fab6d v0.89: Timeout Bug Fix and Restore Read-Only Repositories
* Fixed an issue where longer-running backups/restores would timeout when remote and threaded. Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work. The error message for timeouts was also improved to make debugging easier.

* Allow restores to be performed on a read-only repository by using --no-lock and --log-level-file=off. The --no-lock option can only be used with restores.

* Minor styling changes, clarifications and rewording in the user guide.

* The dev branch has been renamed to master and for the time being the master branch has renamed to release, though it will probably be removed at some point -- thus ends the gitflow experiment for pgBackRest. It is recommended that any forks get re-forked and clones get re-cloned.
2015-12-24 11:32:29 -05:00
David Steele
ba098d7b91 Fixed an issue where longer-running backups/restores would timeout when remote and threaded.
Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work.  The error messages for timeouts was also improved to make debugging easier.
2015-12-24 10:32:25 -05:00
David Steele
b0a6954671 A number of doc improvements:
1) Started on a general markdown renderer
2) Internal links now work in PDF
3) Improvements to PDF styling
4) Some comment and formatting fixes
5) User guide edits.
2015-12-23 11:04:26 -05:00
David Steele
fdc99e30bd Allow restores to be run against a read-only repository.
Two things needed to be changed:
1) Don't open a log file when log-level-file=off
2) New --no-lock option to suppress lock file creation for restores.
2015-12-23 10:48:33 -05:00
David Steele
4925756dee Merge branch 'master' into dev after release 0.88 2015-11-22 18:16:54 -05:00
David Steele
e04c21b012 v0.88: Documentation and Minor Bug Fixes
* Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication.

* Fixed an issue where the start/stop commands required the --config option.

* Fixed an issue where log files were being overwritten instead of appended.

* Fixed an issue where backup-user was not optional.

* Symlinks are no longer created in backup directories in the repository. These symlinks could point virtually anywhere and potentially be dangerous. Symlinks are still recreated during a restore.

* Added better messaging for backup expiration. Full and differential backup expirations are logged on a single line along with a list of all dependent backups expired.

* Archive retention is automatically set to full backup retention if not explicitly configured.
2015-11-22 17:27:30 -05:00
David Steele
fa05715dec Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication. 2015-11-22 16:44:01 -05:00
David Steele
8ddfdcdd3b Various fixes and features implemented during doc development
* Better messaging for expiration.
* Fixed already stopped message.
* retention-archive and retention-archive-type now use retention-full and 'full' when not specified.
* Fixed issue where backup-user was required (should default to backrest).
* ExecuteTest now supports retries.
* Fixed issue where log test was not comparing test logs.
* Fixed issue where test logs would not match for ssh connection errors
2015-11-22 14:02:14 -05:00
David Steele
7ff9fc996c Release notes for fixes and rebuild command-line help. 2015-11-20 11:03:06 -05:00
David Steele
e5cf1d84a7 CSS Tweaks. 2015-10-29 17:45:29 +01:00
David Steele
2f83362f32 Minor user guide fix. 2015-10-28 13:02:35 +01:00
David Steele
a498becf11 Merge branch 'master' into dev after release 0.87 2015-10-28 11:29:04 +01:00
David Steele
2813c4d4eb v0.87: Website and User Guide
* Added a new user guide that covers pgBackRest basics and some advanced topics including PITR. Much more to come, but it's a start.

* The website, markdown, and command-line help are now all generated from the same XML source.

* The backup_label.old and recovery.done files are now excluded from backups.
2015-10-28 11:15:38 +01:00
David Steele
32ce87d56d Fixed minor path issue in README.md. 2015-10-28 10:19:33 +01:00
David Steele
57a06ba3b1 Implemented issue #148: Website with basic user guide. 2015-10-28 10:10:36 +01:00
David Steele
3dc0142244 Renamed userguide.xml to reference.xml so it doesn't conflict with the new user guide. 2015-10-08 14:36:39 -04:00
David Steele
994ead1f05 Removed IO::String dependency from docs. 2015-10-08 13:47:51 -04:00
David Steele
0bcbe21520 Merge branch 'master' into dev after release 0.85 2015-10-08 12:44:55 -04:00
David Steele
097eb7ca41 v0.85: Start/Stop Commands and Minor Bug Fixes
* Added new feature to allow all pgBackRest operations to be stopped or started using the stop and start commands.  This prevents any pgBackRest processes from running on a system where PostgreSQL is shutdown or the system needs to be quiesced for some reason.

* Removed dependency on IO::String module.

* Fixed an issue where an error could be returned after a backup or restore completely successfully.

* Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed.  This scenario was likely if the backup process got terminated during the copy phase.

* Experimental support for PostgreSQL 9.5 beta1.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
2015-10-08 12:34:50 -04:00
David Steele
1f9d6c330a Updated WAL magic for PostgreSQL 9.5 beta1. 2015-10-08 12:22:57 -04:00
David Steele
ddd4c14773 Implemented issue #129: Stop/Start Commands.
Implemented issue #136: Add stderr back into protocol layer.

There were also miscellaneous bug fixes and improvements to buffering in IO.pm.
2015-10-08 11:43:56 -04:00
David Steele
e488f7bb19 Fixed issue #144: Error on resume when temp files (file.tmp.*) present. 2015-09-22 21:07:49 -04:00
David Steele
210f8ef723 Merge branch 'master' into dev after release 0.82 2015-09-14 12:28:22 -04:00
David Steele
9be15d00f8 v0.82: Refactoring, Command-line Help, and Minor Bug Fixes
* Fixed an issue where resumed compressed backups were not preserving existing files.

* Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.

* Fixed an issue where a cold backup using --no-start-stop could be started on a running PostgreSQL cluster without --force specified.

* Fixed an issue where a thread could be started even when none were requested.

* Fixed an issue where the pgBackRest version number was not being updated in backup.info and archive.info after an upgrade/downgrade.

* Fixed an issue where the info command was throwing an exception when the repository contained no stanzas.  Reported by Stephen Frost.

* Fixed an issue where the PostgreSQL pg_stop_backup() NOTICEs were being output to stderr.  Reported by Stephen Frost.

* Renamed recovery-setting option and section to recovery-option to be more consistent with pgBackRest naming conventions.

* Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.

* Code cleanup and refactoring to standardize on patterns that have evolved over time.

* Added dynamic module loading to speed up commands, especially asynchronous archiving.

* Expiration tests are now synthetic rather than based on actual backups.  This will allow development of more advanced expiration features.

* Experimental support for PostgreSQL 9.5 alpha2.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
2015-09-14 11:18:50 -04:00
David Steele
e0dafbff7f Improvements to issue #132: Improved command-line help.
Regression tests are now more comprehensive by default.
Better handling for errors in safeExit().
Release notes.
2015-09-09 15:40:54 -04:00
David Steele
adc5568b15 Fixed issue #132: Improved command-line help.
Some platform specific fixes and added regression tests to detect issues.
2015-09-08 12:58:13 -04:00
David Steele
ac3c0d43ab Implemented issue #132: Improved command-line help.
Implemented issue #133: Dynamic module loading where possible.
2015-09-08 07:31:24 -04:00
David Steele
ee55817fce Refactoring documentation generator. 2015-09-02 18:55:04 -04:00
David Steele
bf3ef15ccc Implemented issue #117: Refactor expiration tests to be purely synthetic 2015-09-01 19:05:10 -04:00
David Steele
6a9377a0a9 Code cleanup and refactoring to standardize on patterns that have evolved over time. 2015-08-29 14:20:46 -04:00
David Steele
83473eb9fb Merge branch 'master' into dev after release 0.80 2015-08-09 11:29:00 -04:00
David Steele
a59284a4cc v0.80: DBI Support, Stability, and Convenience Features
* Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the info command.  Only text output was affected -- json output reported the correct epoch values.  Reported by Michael Renner.

* Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.

* Now using Perl DBI and DBD::Pg for connections to PostgreSQL rather than psql.  The cmd-psql and cmd-psql-option settings have been removed and replaced with db-port and db-socket-path.

* Add stop-auto option to allow failed backups to automatically be stopped when a new backup starts.

* Add db-timeout option to limit the amount of time pgBackRest will wait for pg_start_backup() and pg_stop_backup() to return.

* Remove pg_control file at the beginning of the restore and copy it back at the very end.  This prevents the possibility that a partial restore can be started by PostgreSQL.

* The repository is now created and updated with consistent directory and file modes.  By default umask is set to 0000 but this can be disabled with the neutral-umask setting.

* Added checks to be sure the db-path setting is consistent with db-port by comparing the data_directory as reported by the cluster against the db-path setting and the version as reported by the cluster against the value read from pg_control.  The db-socket-path setting is checked to be sure it is an absolute path.

* Experimental support for PostgreSQL 9.5 alpha1.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.

* Major refactoring of the protocol layer to support future development.

* Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.

* Split most of README.md out into USERGUIDE.md and CHANGELOG.md because it was becoming unwieldy.  Changed most references to "database" in the user guide to "database cluster" for clarity.
2015-08-09 11:20:43 -04:00
David Steele
65d9de4ee7 Document issue #114: Better description of repo-path. 2015-08-09 09:18:58 -04:00
David Steele
45259a487f Change log for issue #94: backup_stop needs a timeout. 2015-08-08 18:36:36 -04:00
David Steele
1ab6239b4e Implemented issue #118: Clean up database vs. database cluster in the docs. 2015-08-08 18:32:41 -04:00
David Steele
e82afac885 Implemented issue #94: backup_stop needs a timeout.
The timeout applies to all queries agains the cluster during a backup.
2015-08-08 18:22:38 -04:00
David Steele
8c0508789e Implemented issue #25: Option to call stop_backup with every backup. 2015-08-08 17:11:20 -04:00
David Steele
2edf5d4bf7 Split most of README.md out into CHANGELOG.md and USERGUIDE.md.
Added release dates to change log.
2015-08-07 14:43:53 -04:00
David Steele
4e7bd4468a Worked on issue #122: 9.5 Integration.
Most tests are working now.  What's not working:

1) --target-resume option fails because pause_on_recovery setting was removed.  Need to implement to the new 9.5 option and make that work with older versions in a consistent way.
2) No tests for the new .partial WAL segments that can be generated on timeline switch.
2015-08-06 16:36:55 -04:00
David Steele
adb8a00925 Improvements to release notes. 2015-08-06 00:24:02 -04:00
David Steele
8b57188bc1 Fixed issue #120: Wrong "oldest backup timestamp" in info output. 2015-08-05 23:37:46 -04:00
David Steele
a10dd8ba98 Implemented issue #121: Check data from db against what's passed to backrest. 2015-08-05 22:05:45 -04:00
David Steele
ba4e5fc4cb Release notes for issue #20: Convert Postgres calls from psql to DBD::Pg. 2015-08-05 13:35:19 -04:00
David Steele
23e6267adc Implemented issue #20: Convert Postgres calls from psql to DBD::Pg. 2015-08-05 12:32:12 -04:00
David Steele
f8a2da9400 Work on issue #48: Abandon threads and go to processes
* Major refactoring of the protocol layer to support this work.
* Fixed protocol issue that was preventing ssh errors (especially connect) from being logged.
2015-08-05 08:43:41 -04:00
David Steele
97c036973c Implemented issue #119: Make sure all repo permissions are 640/750. 2015-07-23 13:11:38 -04:00
David Steele
f3199d13b1 Implement issue #115: When restoring copy pg_control last. 2015-07-16 11:12:48 -04:00
David Steele
702afeb6f0 Merge branch 'master' into dev after release 0.78 2015-07-13 15:37:50 -04:00
David Steele
c568f5e727 v0.78: Remove CPAN dependencies, stability improvements
* Removed dependency on CPAN packages for multi-threaded operation.  While it might not be a bad idea to update the threads and Thread::Queue packages, it is no longer necessary.

* Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.

* Modified wait backoffs to use a Fibonacci sequence rather than geometric.  This will make then grow less aggressively while still giving reasonable wait times.

* More options for regression tests and improved code to run in a variety of environments.
2015-07-13 14:44:32 -04:00
David Steele
89c0f23f37 Update release notes and some test fixes. 2015-07-13 12:13:07 -04:00
David Steele
e9ce4efedf Removed dependency on dequeue_timed() which eliminates the CPAN install.
Added vagrant config for CentOS 6.
2015-07-10 09:20:28 -04:00
David Steele
1df8c30549 Swapped config and operation sections to hopefully make the docs flow a little better. 2015-07-09 10:21:02 -04:00
David Steele
e799361e90 Clarification for CPAN requirements. 2015-07-09 10:11:49 -04:00
David Steele
bb443dd098 Renamed pg_backrest.pl to pg_backrest in the docs. 2015-07-06 17:59:44 -04:00
David Steele
38f48be3cf Improvements to help to make it more dynamic depending on environment.
Changed PgBackRest to pgBackRest.
2015-07-02 10:05:13 -04:00
David Steele
12932dc54c Merge branch 'master' into dev after release 0.77 2015-06-30 14:57:17 -04:00
David Steele
b1ded135df v0.77: CentOS/RHEL 6 support and protocol improvements
* Removed pg_backrest_remote and added the functionality to pg_backrest as remote command.

* Added file and directory syncs to the File object for additional safety during backup/restore and archiving.  Suggested by Andres Freund.

* Support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6.  Found by Eric Radman.

* Improved error message when backup is run without archive_command set and without --no-archive-check specified.  Found by Eric Radman.

* Moved version number out of the VERSION file to Version.pm to better support packaging.  Suggested by Michael Renner.

* Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple distros.
2015-06-30 14:47:58 -04:00
David Steele
7248795b91 Work on issue #48: Abandon threads and go to processes
Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple distros.  Using open3 will also be used for local processes so it make sense to switch now.
2015-06-29 22:07:42 -04:00
David Steele
f210fe99c3 Implemented issue #109: Move VERSION into source code.
Also stopped replacing FORMAT number which explains the large number of test log changes.  FORMAT should change very rarely and cause test log failures when it does.
2015-06-22 13:11:07 -04:00
David Steele
af58016871 Release notes for issue #108: Incompatibility with Perl 5.10.1. 2015-06-22 10:04:58 -04:00
David Steele
fdd9e8eb17 Improved release notes on fsync to include restore. 2015-06-22 09:58:01 -04:00
David Steele
3f841fcd95 Improved issue #110: 'db-version' is required but not defined.
Improved the error message and added hints.
2015-06-22 09:51:16 -04:00
David Steele
61dfeca3c1 Fixed issue #108: Incompatibility with Perl 5.10.1 2015-06-21 12:06:13 -04:00
David Steele
49f1a3e481 Document issue #108: Incompatibility with Perl 5.10.1 2015-06-19 14:27:44 -04:00
David Steele
0695c0e7af Document issue #108: Incompatibility with Perl 5.10.1 2015-06-19 14:02:26 -04:00
David Steele
a5d9d6d84d Added file and directory syncs to the File object for additional safety during backup and archiving. 2015-06-17 12:53:33 -04:00
David Steele
9511f9c35c Implemented issue #106: Roll pg_backup_remote into pg_backrest 2015-06-17 11:26:07 -04:00
David Steele
0ba4029253 Update version to 0.80. 2015-06-16 16:21:24 -04:00
David Steele
da8b337ee8 Added supported versions of PostreSQL. 2015-06-16 15:44:27 -04:00
David Steele
1afbab010f v0.75: New repository format, info command and experimental 9.5 support
* IMPORTANT NOTE: This flag day release breaks compatibility with older versions of PgBackRest.  The manifest format, on-disk structure, and the binary names have all changed.  You must create a new repository to hold backups for this version of PgBackRest and keep your older repository for a time in case you need to do a restore.  The `pg_backrest.conf` file has not changed but you'll need to change any references to `pg_backrest.pl` in cron (or elsewhere) to `pg_backrest` (without the `.pl` extension).

* Add info command.

* More efficient file ordering for backup.  Files are copied in descending size order so a single thread does not end up copying a large file at the end.  This had already been implemented for restore.

* Logging now uses unbuffered output.  This should make log files that are being written by multiple threads less chaotic.  Suggested by Michael Renner.

* Experimental support for PostgreSQL 9.5.  This may break when the control version or WAL magic changes but will be updated in each release.
2015-06-14 20:59:32 -04:00
David Steele
b365d9aacc Updated release notes for 0.75. 2015-06-14 11:48:33 -04:00
David Steele
b865070edd Experimental 9.5 support. Unit tests are not working yet. 2015-06-14 10:12:36 -04:00
David Steele
0b6f81a812 Implemented issue #50: Reverse ordering by file size. 2015-06-14 09:36:58 -04:00
David Steele
6c807f2ddd Fixed release notes for info command. 2015-06-13 18:30:10 -04:00
David Steele
148836fe44 Implemented issue #26: Info command.
* Includes updating the manifest to format 4.  It turns out the manifest and .info files were not very good for providing information.  A format update was required anyway so worked through the backlog of changes that would require a format change.

* Multiple database versions are now supported in the archive.  Does't actually work yet but the structure should be good.

* Tests use more constants now that test logs can catch name regressions.
2015-06-13 18:25:49 -04:00
David Steele
5a73b1a240 Fixed minor typos in the documentation. 2015-06-10 15:54:24 -04:00
David Steele
aca4511350 Documentation for issue #93: Prevent misconfigured retention-archive setting
Not an actual fix - just some documentation to (hopefully) keep users from getting bitten until then.
2015-06-10 15:44:13 -04:00
David Steele
af98f164cd Fixed issue #97: Interleaved log lines during parallel operations. 2015-06-02 09:42:56 -04:00
David Steele
7e06e53a69 Merge branch 'master' into dev after release 0.70 2015-06-01 09:39:02 -04:00
David Steele
f83f0fa54d v0.70: Stability improvements for archiving, improved logging and help
* Fixed an issue where archive-copy would fail on an incr/diff backup when hardlink=n.  In this case the pg_xlog path does not already exist and must be created. Reported by Michael Renner

* Allow duplicate WAL segments to be archived when the checksum matches.  This is necessary for some recovery scenarios.

* Allow comments/disabling in pg_backrest.conf using #.  Suggested by Michael Renner.

* Better logging before pg_start_backup() to make it clear when the backup is waiting on a checkpoint.  Suggested by Michael Renner.

* Various command behavior, help and logging fixes.  Reported by Michael Renner.

* Fixed an issue in async archiving where archive-push was not properly returning 0 when archive-max-mb was reached and moved the async check after transfer to avoid having to remove the stop file twice.  Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong.  Reported by Michael Renner.

* Fixed a locking issue that could allow multiple operations of the same type against a single stanza.  This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. Reported by Michael Renner.

* Replaced JSON module with JSON::PP which ships with core Perl.
2015-06-01 09:24:38 -04:00
David Steele
e7d2d704fe Fixed issue #91: Race condition in async archive-push. 2015-05-29 16:31:12 -04:00
David Steele
5fce709ae7 Document issue #88: Thoughts on repo & release management. 2015-05-29 12:33:38 -04:00
David Steele
d321ef0b6d Implement issue #89: Make confess backtraces log-level dependent.
ASSERTs still dump stack traces to the console and file in all cases.  ERRORs only dump stack traces to the file when the file log level is DEBUG or TRACE.
2015-05-29 12:26:31 -04:00
David Steele
13e4eec629 Implement issue #90: Move archive-max-mb archive-stop checks to end of archive-push. 2015-05-29 11:41:19 -04:00
David Steele
639c27e113 Release notes for issue #68: Allow duplicate WAL to be pushed if the checksum matches. 2015-05-29 08:18:12 -04:00
David Steele
3083e5fc7e Fix for issue #83: Provide more helpful error message during archive-stop situations.
Fix for issue #84: archive-async in combination with archive-max-mb doesn't work as documented.
Unit tests for archive stop.
2015-05-28 13:17:34 -04:00
David Steele
a4974f8996 Fix for issue #70: command line help improvements 2015-05-26 15:29:42 -04:00
David Steele
ca0c866d3b Implement issue #71: log a message when waiting for a checkpoint
Some other log improvements
2015-05-26 12:26:59 -04:00
David Steele
5c43e7035a Implement issue #79: Support comment characters in config file 2015-05-26 11:36:03 -04:00
David Steele
e9099b99aa Updated required modules. Minor doc fixes. 2015-05-26 10:01:05 -04:00
David Steele
d5335b40e8 Fix for issue #80: enabling archive-copy causes failing differential & incremental backups 2015-05-22 14:49:14 -04:00
David Steele
5ada7fb5ad Use JSON::PP instead of JSON since it is shipped with core Perl. 2015-05-12 15:44:10 -04:00
David Steele
ab4efce5d1 Merge branch 'master' into dev after release 0.65 2015-05-11 18:36:02 -04:00
David Steele
49fe40850f v0.65: Improved resume and restore logging, compact restores
* Better resume support.  Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses.  More unit tests to verify each resume case.

* Resume is now optional.  Use the `resume` setting or `--no-resume` from the command line to disable.

* More info messages during restore.  Previously, most of the restore messages were debug level so not a lot was output in the log.

* Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.

* Added `tablespace` setting to allow tablespaces to be restored into the `pg_tblspc` path.  This produces compact restores that are convenient for development, staging, etc.  Currently these restores cannot be backed up as PgBackRest expects only links in the `pg_tblspc` path.
2015-05-11 18:29:40 -04:00
David Steele
9aa41ee76b Added tablespace setting to allow compact database restores for development, staging, etc. 2015-05-08 18:34:27 -04:00
David Steele
a0d985b7ad Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path. 2015-05-07 17:37:25 -05:00
David Steele
1ac4b781fd Better info logging for restore. Most of the messages were debug before and some important ones were missing. 2015-05-07 15:56:56 -06:00
David Steele
095a9a0b83 Added resume setting to allow disabling resume feature. 2015-05-07 10:29:30 -06:00
David Steele
328c2ca5c0 Much better resume:
1) Re-checksums files that have checksums in the manifest
2) Recopies files that do not have a checksum
3) Saves the manifest at regular intervals to preserve checksums
4) Unit tests for all cases (that I can think of)
2015-05-06 18:24:34 -06:00
David Steele
1d1c7e47d1 Doc fixes for archive-max-mb setting. 2015-05-01 19:13:12 -04:00
David Steele
b0db7c0497 Fixed case error in Perl module name. 2015-04-29 17:43:28 -04:00
David Steele
87f69508e3 Set version to 0.75 2015-04-21 10:48:01 -04:00
David Steele
6987cfaa53 Merge branch 'master' into dev after release 0.61 2015-04-21 09:46:28 -04:00
David Steele
a9b9ec2492 v0.61: bug fix for uncompressed remote destination
* Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination.  The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
2015-04-21 09:45:03 -04:00
David Steele
4b51f5dc50 Removed unused font. 2015-04-19 18:00:51 -04:00
David Steele
5185d49873 v0.60: better version support and WAL improvements
* Pushing duplicate WAL now generates an error.  This worked before only if checksums were disabled.

* Database System IDs are used to make sure that all WAL in an archive matches up.  This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.

* Regression tests working back to PostgreSQL 8.3.

* Improved threading model by starting threads early and terminating them late.
2015-04-19 17:53:29 -04:00
David Steele
f1cc1d6123 Refactored backup manifest update by centralizing the code. 2015-04-11 15:02:04 -04:00
David Steele
7081c8b867 New model where threads are created early and destroyed late.
Backups now work like restores in terms of how jobs are queued.
Split out BackupFile and RestoreFile for easier multi-threading/processing.
2015-04-07 07:34:37 -04:00
David Steele
0b27c74999 Increased version to 0.60 and added release notes. 2015-04-01 16:28:16 -04:00
David Steele
9caccc2ce2 Doc fixes. 2015-03-29 13:35:36 -04:00
David Steele
b37d59832f v0.50: restore and much more
* Added restore functionality.

* All options can now be set on the command-line making pg_backrest.conf optional.

* De/compression is now performed without threads and checksum/size is calculated in stream.  That means file checksums are no longer optional.

* Added option `--no-start-stop` to allow backups when Postgres is shut down.  If `postmaster.pid` is present then `--force` is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created).  This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.

* Fixed broken checksums and now they work with normal and resumed backups.  Finally realized that checksums and checksum deltas should be functionally separated and this simplied a number of things.  Issue #28 has been created for checksum deltas.

* Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.

* Removed dependency on Moose.  It wasn't being used extensively and makes for longer startup times.

* Checksum for backup.manifest to detect corrupted/modified manifest.

* Link `latest` always points to the last backup.  This has been added for convenience and to make restores simpler.

* More comprehensive unit tests in all areas.
2015-03-25 15:15:55 -04:00
David Steele
82fa497511 Last doc fixes. 2015-03-25 14:43:21 -04:00
David Steele
4a6f4e2340 Getting docs in shape for the next release. 2015-03-24 18:50:54 -04:00
David Steele
1729fa1c51 Added unit tests for expiration. 2015-03-17 18:31:05 -04:00
David Steele
f4f31302f0 Broke archive-required into two options: backup-wal-check and backup-wal-store. 2015-03-16 14:45:53 -04:00
David Steele
258fb9c6e2 More work on automated docs. Merging this to go back to some feature/bug work for a while. 2015-03-16 14:01:01 -04:00
David Steele
7675a11ded First pass at building automated docs for markdown/html. This works pretty well, but the config sections of doc.xml still require too much maintenance. With the new Config code, it should be possible to generate those sections automatically. 2015-03-08 14:05:41 -04:00
David Steele
f115e01b71 More work on docs. 2015-03-04 22:38:27 -05:00
David Steele
8dc3ba184d More work on docs. 2015-03-04 20:09:26 -05:00
David Steele
163f898262 HTML docs look to be in a workable format. 2015-03-04 18:50:02 -05:00
David Steele
bfb2c05357 Started on new HTML docs. The markdown is not cutting it. 2015-03-04 00:36:54 -05:00