1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

492 Commits

Author SHA1 Message Date
David Steele
5ada189a92 Backup from a standby cluster.
A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the master.
2016-08-25 11:25:46 -04:00
David Steele
d0b6f78b20 More flexible configuration for databases
Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the master. This means no configuration changes for backup are required after failing over from a master to standby when a separate backup server is used.
2016-08-24 12:39:27 -04:00
David Steele
cd6278e5af Revert some backup exclusions until they have been tested more thoroughly. 2016-08-24 12:27:48 -04:00
David Steele
9c13ca1d09 mprove backup log messages to indicated which host the files are being copied from. 2016-08-24 12:01:08 -04:00
David Steele
f1412baccf Exclude directories during backup that are cleaned, recreated, or zeroed by PostgreSQL at startup.
These include (depending on the version where they were introduced): pgsql_tmp, pg_dynshmem, pg_notify, pg_replslot, pg_serial, pg_snapshots, pg_stat_tmp, pg_subtrans. The postgresql.auto.conf.tmp file is now excluded in addition to files that were already excluded: backup_label.old, postmaster.opts, postmaster.pid, recovery.conf, recovery.done.
2016-08-16 09:35:16 -04:00
David Steele
332055971e Added hints to error message when relative paths are detected in archive-push or archive-get. 2016-08-15 20:54:17 -04:00
David Steele
b25f10949a Fixed #229: Check fails on target db host.
Options that should not allow multiple values could be specified multiple times in pgbackrest.conf without an error being raised.
2016-08-15 20:15:17 -04:00
David Steele
88d9bd3d37 Improvements in Db module.
* Separated out connect() function.
* Allow executeSql() calls that do not return data.
* Improve error handling.
2016-08-15 17:44:51 -04:00
David Steele
1e0f15f425 Improve error message for links that reference links in manifest build. 2016-08-15 17:23:37 -04:00
David Steele
f9fa1270b2 Fixed #236: Recursive user tablespace symlink.
A tablespace link that referenced another link would not produce an error, but instead skip the tablespace entirely.
2016-08-15 17:11:45 -04:00
David Steele
35931305d7 Update control version for PostgreSQL beta4. 2016-08-15 16:48:24 -04:00
David Steele
17b79d6279 Database version refactoring.
* Refactor db version constants into a separate module.
* Update synthetic backup tests to PostgreSQL 9.4.
2016-08-11 22:35:24 -04:00
David Steele
bff262ac47 Removed all OP_* function constants that were used only for debugging, not in the protocol, and replaced with __PACKAGE__. 2016-08-11 17:32:28 -04:00
David Steele
9f19e13095 v1.05: Bug Fix for Tablespace Link Checking
Bug Fixes:

* Fixed an issue where tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not. Also hardened relative path checking a bit. (Reported by Chris Fort.)
2016-08-09 10:06:21 -04:00
David Steele
34afe5e85b Fixed issue with tablespace link checking.
* Tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not.
* Also hardened relative path checking a bit.
2016-08-09 09:05:27 -04:00
David Steele
277934447b v1.04: Various Bug Fixes
Bug Fixes:

* Fixed an issue an where an extraneous remote was created causing threaded backup/restore to possibly timeout and/or throw a lock conflict. (Reported by Michael Vitale.)
* Fixed an issue where db-path was not required for the check command so an assert was raised when it was missing rather than a polite error message. (Reported by Michael Vitale.)
* Fixed check command to throw an error when database version/id does not match that of the archive. (Fixed by Cynthia Shang.)
* Fixed an issue where a remote could try to start its own remote when the backup-host option was not present in pgbackrest.conf on the database server. (Reported by Lardière Sébastien.)
* Fixed an issue where the contents of pg_xlog were being backed up if the directory was symlinked. This didn't cause any issues during restore but was a waste of space.
* Fixed an invalid log() call in lock routines.

Features:

* Experimental support for non-exclusive backups in PostgreSQL 9.6 beta3. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.

Refactoring:

* Enhancements to the protocol layer for improved reliability and error handling.
* All remote types now take locks. The exceptions date to when the test harness and pgBackRest were running in the same VM and no longer apply.
* Exceptions are now passed back from threads as messages when possible rather than raised directly.
* Temp files created during backup are now placed in the same directory as the target file.
* Output lock file name when a lock cannot be acquired to aid in debugging.
* Reduce calls to protocolGet() in backup/restore.
* Suppress banners on SSH protocol connections.
* Improved remote error messages to identify the host where the error was raised.
2016-07-30 09:42:35 -04:00
David Steele
f2b3dda647 Fixed an issue where a remote could try to start its own remote.
This is a better approach than 93320b8 (reverted in this commit) because it ensures that the remote type will be none so any functions that utilize optionRemoteTypeTest will work correctly.

This bug was only an issue when backup-host was not properly configured on the database host.
2016-07-30 08:49:26 -04:00
David Steele
ac23716d26 Enhancements to the protocol layer for improved reliability and error handling. 2016-07-29 18:46:19 -04:00
David Steele
b447863218 Exceptions are now passed back from threads as messages when possible rather than raised directly. 2016-07-29 18:46:19 -04:00
David Steele
93320b865e Fixed a issue where a remote could try to start its own remote. 2016-07-29 18:45:37 -04:00
David Steele
af5965a6ef Fixed an invalid log call in lock routines. 2016-07-29 18:45:23 -04:00
David Steele
a3b8808f94 Fixed an issue where the contents of pg_xlog were being copied if the directory was symlinked. 2016-07-29 18:44:53 -04:00
David Steele
ef33febfb7 Temp files created during backup are now placed in the same directory as the target file. 2016-07-29 18:43:59 -04:00
Cynthia Shang
6e9d5f5bd3 Fixed check command to throw an error when database version/id does not match that of the archive.
Fixed by Cynthia Shang.
2016-07-29 14:02:11 -04:00
David Steele
273d1c08d8 Improved remote error messages to identify the host where the error was raised. 2016-07-26 16:57:38 -04:00
David Steele
bc5c04ffe7 Updated control/catalog/WAL versions for 9.6 beta3. 2016-07-26 16:39:01 -04:00
David Steele
b6ac6b0f1c Suppress banners on SSH protocol connections. 2016-07-26 16:28:43 -04:00
David Steele
f31ec20fcb Reduce calls to protocolGet() in backup/restore. 2016-07-26 16:21:14 -04:00
David Steele
98a67792ec Output lock file name when a lock cannot be acquired to aid in debugging. 2016-07-26 16:14:15 -04:00
David Steele
095351f145 Fixed #219: Unable to acquire restore lock.
Fixed an issue an where an extraneous remote was created causing threaded backup/restore to possibly timeout and/or throw a lock conflict.
2016-07-26 16:08:05 -04:00
David Steele
5893c3cd2e Fixed an issue where db-path was not required for the check command so an assert was thrown when it was missing rather than a polite error message.
Reported by Michael Vitale.
2016-07-19 15:26:41 -04:00
David Steele
1b62354dcb v1.03: Check Command and Bug Fixes
Bug Fixes:

* Fixed an issue where keep-alives could be starved out by lots of small files during multi-threaded backup. They were also completely absent from single/multi-threaded backup resume and restore checksumming. (Reported by Janice Parkinson, Chris Barber.)
* Fixed an issue where the expire command would refuse to run when explicitly called from the command line if the db-host option was set. This was not an issue when expire was run automatically after a backup (Reported by Chris Barber.)
* Fixed an issue where validation was being running on archive_command even when the archive-check option was disabled.

Features:

* Added check command to validate that pgBackRest is configured correctly for archiving and backups. (Contributed by Cynthia Shang.)
* Added the protocol-timeout option. Previously protocol-timeout was set as db-timeout + 30 seconds.
* Failure to shutdown remotes at the end of the backup no longer throws an exception. Instead a warning is generated that recommends a higher protocol-timeout.
* Experimental support for non-exclusive backups in PostgreSQL 9.6 beta2. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.

Refactoring:

* The pg_xlogfile_name() function is no longer used to construct WAL filenames from LSNs. While this function is convenient it is not available on a standby. Instead, the archive is searched for the LSN in order to find the timeline. If due to some misadventure the LSN appears on multiple timelines then an error will be thrown, whereas before this condition would have passed unnoticed.
* Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the backup-host and db-host options which are used to determine locality.
* Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
* Changed version variable to a constant. It had originally been designed to play nice with a specific packaging tool but that tool was never used.
2016-07-02 10:22:52 -04:00
David Steele
186ca18659 Update WAL/catalog versions for 9.5 beta 2. 2016-07-02 09:19:10 -04:00
David Steele
418039027b Closed #213: Archive command should not be validate when archive-check=n 2016-07-02 09:02:55 -04:00
David Steele
cc2a8777d5 User/group permissions improvements.
Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
2016-06-26 21:01:20 -04:00
David Steele
b571218930 Minor refactor of version variable to a constant.
It had originally been designed to play nice with a specific packaging tool but that tool was never used.
2016-06-26 20:53:45 -04:00
David Steele
6ee485cc76 Added more keep-alives.
Keep-alives were also missing from multi-threaded backup/restore and single-threaded restore.

Found by Chris Barber.
2016-06-24 13:16:07 -04:00
David Steele
c8f806a293 Major refactor of the test suite.
* Make the code more modular and object-oriented.
* Multiple Docker containers can now be created for a single test to simulate more realistic environments.
2016-06-24 08:12:58 -04:00
David Steele
23a3911830 Stop using pg_xlogfile_name().
The pg_xlogfile_name() function is no longer used to construct WAL filenames from LSNs. While this function is convenient it is not available on a standby. Instead, the archive is searched for the LSN in order to find the timeline. If due to some misadventure the LSN appears on multiple timelines then an error will be thrown, whereas before this condition would have passed unnoticed.
2016-06-24 08:06:20 -04:00
David Steele
41b27c0456 Protocol timeout option and keep-alive fixes.
* Fixed an issue where keep-alives could be starved out by lots of small files during multi-threaded operation and were completely absent during single-threaded operation when resuming from a previous incomplete backup.

Reported by Janice Parkinson.

* Added the protocol-timeout option. Previously protocol-timeout was set as db-timeout + 30 seconds.
* Failure to shutdown remotes at the end of the backup no longer throws an exception. A warning is still generated that recommends a higher protocol-timeout.
2016-06-22 18:01:18 -04:00
David Steele
012405a33b Closed #207: Expire fails with unhandled exception.
* Fixed an issue where the expire command would refuse to run when explicitly called from the command line if the db-host option was set.  This was not an issue when expire was run after a backup, which is the usual case.
* Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the backup-host and db-host options which are used to determine locality.

Reported by Chris Barber.
2016-06-18 09:55:00 -04:00
Cynthia Shang
7e45ed8366 Added check command.
The check command validates that pgBackRest is configured correctly for archiving and backups.

Contributed by Cynthia Shang.
2016-06-12 09:13:46 -04:00
David Steele
bfaee7380d v1.02: Bug Fix for Perl 5.22
* Fix usage of sprintf() due to new constraints in Perl 5.22. Parameters not referenced in the format string are no longer allowed. (Fixed by Adrian Vondendriesch.)
2016-06-02 11:10:24 -04:00
David Steele
df6086bd24 Added a man page to document generation. 2016-06-02 09:25:12 -04:00
David Steele
379ab1b353 Improve log directory/file creation.
Log directory create and file open now using FileCommon functions which produce more detailed error messages on failure.
2016-05-26 15:04:18 -04:00
Adrian Vondendriesch
3d437d9278 Fix usage of sprintf() for Perl 5.22.
Parameters not referenced in the format string are no longer allowed.
2016-05-26 11:23:52 -04:00
David Steele
b1c779e606 Next development cycle. 2016-05-26 11:04:19 -04:00
David Steele
b710c01630 v1.01: Enhanced Info, Selective Restore, and 9.6 Support
* Enhanced text output of `info` command to include timestamps, sizes, and the reference list for all backups. Contributed by Cynthia Shang.

* Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.

* Experimental support for non-exclusive backups in PostgreSQL 9.6 beta1. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
2016-05-17 01:04:01 -04:00
David Steele
0451d3afdd Support for non-exclusive backups in PostgreSQL 9.6. 2016-05-16 17:59:26 -04:00
David Steele
9b5a27f657 Add Manifest->addFile().
Some files need to be added to the manifest after the initial build.  This is currently done in only one place but usage will expand in the future so the functionality has been encapsulated in addFile().
2016-05-14 10:39:56 -04:00
David Steele
77b01e980f Make the initial capital for hints consistent.
Some hints started with a capital letter and others did not. Make them all lower case for consistency.
2016-05-14 10:36:35 -04:00
David Steele
512d006346 Refactor database version identification for archive and backup commands.
Added database version constants and changed version identification code to use hash tables instead of if-else.  Propagated the db version constants to the rest of the code and in passing fixed some path/filename constants.

Added new regression tests to check that specific files are never copied.
2016-05-14 10:33:12 -04:00
David Steele
4d9920cc48 Fix null and linefeed handling in Db->executeSql().
The join() used was not able to handle nulls and was replaced by a loop.  An injudicious trim was removed when the source of extra linefeeds was determined to be an additional loop execution that was not handled correctly.
2016-05-14 10:29:35 -04:00
David Steele
0c320e7df7 Allow selective restore of databases from a cluster backup.
This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue.
2016-05-11 09:21:39 -04:00
Cynthia Shang
60b901948a Enhanced text output of info command to include timestamps, sizes, and the reference list for all backups. 2016-05-06 10:25:46 -04:00
David Steele
7b64843221 Added trim() to process id read from lock file.
The prior way worked fine but added extraneous linefeeds to the log output.
2016-05-06 09:37:01 -04:00
David Steele
c477d6f899 Next development cycle. 2016-05-05 18:13:50 -04:00
David Steele
9ab0dd80f7 v1.00: New Repository Format and Configuration Scheme, Link Support
* This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, configuration scheme, and the exe/path names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. Restores from the prior repository will require the prior version of pgBackRest but because of name changes it is possible to have 1.00 and a prior version of pgBackRest installed at the same time. See the notes below for more detailed information on what has changed.

* Implemented a new configuration scheme which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. A local repository is no longer required.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally when migrating to 1.00.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest is the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.

* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 23:04:20 -04:00
David Steele
9457e15347 New manifest format.
* All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.

* Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.

* Added detail log level which will output more information than info without being as verbose as debug.
2016-04-14 22:50:02 -04:00
Cynthia Shang
9a7d8b26cc Fixed some inaccurate statements in the user guide.
Fixed typo in documentation code comments.
2016-04-14 13:50:58 -04:00
David Steele
18fd25233b New simpler configuration and consistent project/exe/path naming.
* The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. Otherwise, a local repository is no longer required.

* Implemented a new config format which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case.

* The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally.

* The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest.

* Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest would be the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.

* Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the lock directory.

* Executable filename changed from pg_backrest to pgbackrest.
2016-04-14 09:30:54 -04:00
David Steele
885797e4b5 Migrated many functions from File.pm to FileCommon.pm.
This makes make the migrated file functions available to parts of the code that don't have access to a File object.  They still exist as wrappers in the File object to support remote calls.
2016-04-13 19:09:35 -04:00
David Steele
be8487dbad Changed some TBD comments from urgent to whenever. 2016-04-13 19:05:31 -04:00
David Steele
0e4fdda6d8 Improved error handling when remote closes unexpectedly.
In conditions where an error is known to have occurred wait to try and capture the error in the first call that detects the error.  Due to timing sometimes the error could be caught later, which worked, but it made the functionality inconsistent in testing.
2016-04-12 15:50:25 -04:00
David Steele
aacc369e4f Next development cycle. 2016-04-06 09:41:33 -04:00
David Steele
5307c858a7 v0.92: Command-line Repository Path Fix
* Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf.  Reported by Jan Wieck.
2016-04-06 09:14:06 -04:00
David Steele
1ce9961f8a Closed #183: Options --repo-path and --repo-remote-path ignored in archive-push.
Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf.
2016-04-06 08:57:46 -04:00
David Steele
1d4779cfe6 Next development cycle. 2016-03-22 10:23:20 -04:00
David Steele
aea14c8553 v0.91: Tablespace Bug Fix and Minor Enhancements
* **IMPORTANT BUG FIX FOR TABLESPACES**: A change to the repository format was accidentally introduced in 0.90 which means the on-disk backup was no longer a valid PostgreSQL cluster when the backup contained tablespaces. This only affected users who directly copied the backups to restore PostgreSQL clusters rather than using the restore command. However, the fix breaks compatibility with older backups that contain tablespaces no matter how they are being restored (pgBackRest will throw errors and refuse to restore). New full backups should be taken immediately after installing version 0.91 for any clusters that contain tablespaces. If older backups need to be restored then use a version of pgBackRest that matches the backup version. _Reported by Evan Benoit_.

* Copy global/pg_control last during backups.

* Write .info and .manifest files to temp before moving them to their final locations and fsync'ing.

* Rename --no-start-stop option to --no-online.

* Static source analysis using Perl-Critic, currently passes on gentle.
2016-03-22 10:16:42 -04:00
David Steele
d489914173 Closed #180: Backup format regression in v0.90. 2016-03-22 09:57:39 -04:00
David Steele
0b317d9040 Fix minor bug in protocol compression.
This erroneous last caused a warning (which threw an error) and masked the error in decompression.  It was found when accidentally attempting to decompress an already-decompressed file, so not a big deal in practice which is probably why it hug around for so long.
2016-02-27 10:11:58 -05:00
David Steele
048571e23f Closed #173: Add static source code analysis
Perl Critic added and passes on gentle.  A policy file has been created with some permanent exceptions and a list of policies to be fixed in approximately the order they should be fixed in.
2016-02-23 09:25:22 -05:00
David Steele
d35ab82a83 Closed #167: Rename --no-start-stop option to --no-online. 2016-02-19 19:32:02 -05:00
David Steele
fb63a981cb Closed #166: Copy global/pg_control last during backup. 2016-02-14 20:42:11 -05:00
David Steele
053f25d870 Closed #170: Write .info and .manifest files to temp before moving to final location and syncing 2016-02-11 21:42:27 -05:00
David Steele
f360d5072e Next development cycle. 2016-02-07 11:20:43 -05:00
David Steele
631535ae4f v0.90: 9.5 Support, Various Enhancements, and Minor Bug Fixes
* Fixed an issue where specifying --no-archive-check would throw a configuration error. Reported by Jason O'Donnell.

* Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail.

* Fixed an issue where document generation failed because some OSs are not tolerant of having multiple installed versions of PostgreSQL. A separate VM is now created for each version. Also added a sleep after database starts during document generation to ensure the database is running before the next command runs. Reported by John Harvey.

* The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will not be possible for the backups that fall outside of archive retention.

* When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a restore. This only applies to PostgreSQL >= 9.0 -- prior versions of PostgreSQL could not share a tablespace directory.

* Generate an error when archive-check=y but archivecommand does not execute pgbackrest. Contributed by Jason O'Donnell.

* Improved error message when repo-path or repo-remote-path does not exist.

* Added checks for --delta and --force restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of PGVERSION or backup.manifest (left over from an aborted restore). If neither file is found then --delta and --force will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.

* When restore --set=latest (the default) the actual backup restored will be output to the log.

* Support for PostgreSQL 9.5 partial WAL segments and recoverytargetaction setting. The archivemode = 'always' setting is not yet supported.

* Support for recoverytarget = 'immediate' recovery setting introduced in PostgreSQL 9.4.

* The following tablespace checks have been added: paths or files in pgtblspc, relative links in pgtblspc, tablespaces in $PGDATA. All three will generate errors.
2016-02-07 10:47:16 -05:00
David Steele
de9b6d5133 Closed #113: Better error message when repo does not exist. 2016-02-06 16:49:12 -05:00
David Steele
f8b47997a3 Closed #127: More sanity checking for --delta restores
Added checks for `--delta` and `--force` restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of `PG_VERSION` or `backup.manifest` (left over from an aborted restore). If neither is found then `--delta` and `--force` will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
2016-02-05 23:03:29 -05:00
David Steele
5c815e4fc0 Closed #58: Get catalog number for better tablespace copying
When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a `--delta` restore. This only applies to PostgreSQL >= 9.0 -- before that only one PostgreSQL version could use a tablespace.
2016-02-02 14:33:15 -05:00
David Steele
83b1a3cb44 Closed #150: Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail. 2016-01-20 21:28:40 -05:00
David Steele
747346db67 Closed #162: When restore --set=latest (the default) the actual backup restored will be output to the log. 2016-01-20 18:34:42 -05:00
David Steele
2d38876bc4 Closed #93: The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will still not be possible for backups that fall outside of archive retention. 2016-01-20 17:24:44 -05:00
David Steele
904a8c956f Refactor of expiration to use backup.info for a list of what needs to be expired rather than looking at the backup directory. 2016-01-19 07:39:29 -05:00
David Steele
65cce69dcc Closed #131: Offline backups make expire command fail. 2016-01-15 23:37:17 -05:00
David Steele
4d12cc233e Fix for typo in error message. 2016-01-15 20:31:29 -05:00
David Steele
e91e481f3d Closed #19: The following tablespace checks have been added: paths or files in pg_tblspc, relative links in pg_tblspc, tablespaces in $PGDATA. All three will generate errors. 2016-01-15 07:07:00 -05:00
David Steele
a0a3d1e97e Closed #122: 9.5 Integration
The archive_mode = always setting is not included but is covered by issue #125.
2016-01-13 20:35:12 -05:00
Jason O'Donnell
c7502f341b Generate an error when archive-check=y but archive_command does not execute pg_backrest. 2016-01-11 09:50:45 -05:00
David Steele
29102b2887 Added error when archive_mode=always is configured. 2016-01-09 23:08:54 -05:00
David Steele
7f0f8823df Added support for partial WAL segments. 2016-01-09 19:37:28 -05:00
David Steele
e72165a689 Provisional support for PostgreSQL 9.5. No support for archive_mode = 'always' or .partial WAL on a standby. 2016-01-09 10:11:03 -05:00
David Steele
c8f863fbab Text execution improvements:
1) Tests for all operating systems can now be run with a single command.
2) Tests can be run in parallel with --process-max.
3) Container generation now integrated into test.pl
4) Some basic test documentation.
2016-01-09 08:21:53 -05:00
David Steele
6f6c796fbf Fixed an issue where specifying --no-archive-check would throw a configuration error. 2016-01-03 20:21:30 -05:00
David Steele
91d1e94076 Next development cycle. 2015-12-24 12:49:47 -05:00
David Steele
50a62fab6d v0.89: Timeout Bug Fix and Restore Read-Only Repositories
* Fixed an issue where longer-running backups/restores would timeout when remote and threaded. Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work. The error message for timeouts was also improved to make debugging easier.

* Allow restores to be performed on a read-only repository by using --no-lock and --log-level-file=off. The --no-lock option can only be used with restores.

* Minor styling changes, clarifications and rewording in the user guide.

* The dev branch has been renamed to master and for the time being the master branch has renamed to release, though it will probably be removed at some point -- thus ends the gitflow experiment for pgBackRest. It is recommended that any forks get re-forked and clones get re-cloned.
2015-12-24 11:32:29 -05:00
David Steele
ba098d7b91 Fixed an issue where longer-running backups/restores would timeout when remote and threaded.
Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work.  The error messages for timeouts was also improved to make debugging easier.
2015-12-24 10:32:25 -05:00
David Steele
b0a6954671 A number of doc improvements:
1) Started on a general markdown renderer
2) Internal links now work in PDF
3) Improvements to PDF styling
4) Some comment and formatting fixes
5) User guide edits.
2015-12-23 11:04:26 -05:00
David Steele
fdc99e30bd Allow restores to be run against a read-only repository.
Two things needed to be changed:
1) Don't open a log file when log-level-file=off
2) New --no-lock option to suppress lock file creation for restores.
2015-12-23 10:48:33 -05:00
David Steele
fe9c66591f Fixed issue #159: Duplicated log section entries in config file
This would cause a warning when run with perl -cw but appears to have been harmless otherwise.
2015-12-21 14:29:54 -05:00
David Steele
fa05715dec Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication. 2015-11-22 16:44:01 -05:00
David Steele
8ddfdcdd3b Various fixes and features implemented during doc development
* Better messaging for expiration.
* Fixed already stopped message.
* retention-archive and retention-archive-type now use retention-full and 'full' when not specified.
* Fixed issue where backup-user was required (should default to backrest).
* ExecuteTest now supports retries.
* Fixed issue where log test was not comparing test logs.
* Fixed issue where test logs would not match for ssh connection errors
2015-11-22 14:02:14 -05:00
David Steele
7ff9fc996c Release notes for fixes and rebuild command-line help. 2015-11-20 11:03:06 -05:00
David Steele
4c5470e049 Implemented issue #156: Don't create backup symlinks in the repository. 2015-11-20 10:55:28 -05:00
David Steele
11f724bdfc Fixed issue #155: Start/Stop do not automatically read config file. 2015-11-20 10:28:58 -05:00
David Steele
d89cd955d0 Fixed issue #154: Log files are overwritten rather than appended. 2015-11-20 09:18:12 -05:00
David Steele
a498becf11 Merge branch 'master' into dev after release 0.87 2015-10-28 11:29:04 +01:00
David Steele
2813c4d4eb v0.87: Website and User Guide
* Added a new user guide that covers pgBackRest basics and some advanced topics including PITR. Much more to come, but it's a start.

* The website, markdown, and command-line help are now all generated from the same XML source.

* The backup_label.old and recovery.done files are now excluded from backups.
2015-10-28 11:15:38 +01:00
David Steele
57a06ba3b1 Implemented issue #148: Website with basic user guide. 2015-10-28 10:10:36 +01:00
David Steele
0bcbe21520 Merge branch 'master' into dev after release 0.85 2015-10-08 12:44:55 -04:00
David Steele
097eb7ca41 v0.85: Start/Stop Commands and Minor Bug Fixes
* Added new feature to allow all pgBackRest operations to be stopped or started using the stop and start commands.  This prevents any pgBackRest processes from running on a system where PostgreSQL is shutdown or the system needs to be quiesced for some reason.

* Removed dependency on IO::String module.

* Fixed an issue where an error could be returned after a backup or restore completely successfully.

* Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed.  This scenario was likely if the backup process got terminated during the copy phase.

* Experimental support for PostgreSQL 9.5 beta1.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
2015-10-08 12:34:50 -04:00
David Steele
1f9d6c330a Updated WAL magic for PostgreSQL 9.5 beta1. 2015-10-08 12:22:57 -04:00
David Steele
ddd4c14773 Implemented issue #129: Stop/Start Commands.
Implemented issue #136: Add stderr back into protocol layer.

There were also miscellaneous bug fixes and improvements to buffering in IO.pm.
2015-10-08 11:43:56 -04:00
David Steele
e488f7bb19 Fixed issue #144: Error on resume when temp files (file.tmp.*) present. 2015-09-22 21:07:49 -04:00
David Steele
a3d82eef98 Document code in Db.pm. 2015-09-16 15:23:19 -05:00
David Steele
9be15d00f8 v0.82: Refactoring, Command-line Help, and Minor Bug Fixes
* Fixed an issue where resumed compressed backups were not preserving existing files.

* Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.

* Fixed an issue where a cold backup using --no-start-stop could be started on a running PostgreSQL cluster without --force specified.

* Fixed an issue where a thread could be started even when none were requested.

* Fixed an issue where the pgBackRest version number was not being updated in backup.info and archive.info after an upgrade/downgrade.

* Fixed an issue where the info command was throwing an exception when the repository contained no stanzas.  Reported by Stephen Frost.

* Fixed an issue where the PostgreSQL pg_stop_backup() NOTICEs were being output to stderr.  Reported by Stephen Frost.

* Renamed recovery-setting option and section to recovery-option to be more consistent with pgBackRest naming conventions.

* Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.

* Code cleanup and refactoring to standardize on patterns that have evolved over time.

* Added dynamic module loading to speed up commands, especially asynchronous archiving.

* Expiration tests are now synthetic rather than based on actual backups.  This will allow development of more advanced expiration features.

* Experimental support for PostgreSQL 9.5 alpha2.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
2015-09-14 11:18:50 -04:00
David Steele
e0dafbff7f Improvements to issue #132: Improved command-line help.
Regression tests are now more comprehensive by default.
Better handling for errors in safeExit().
Release notes.
2015-09-09 15:40:54 -04:00
David Steele
d7e3be1ebf Fixed issue #138: Fix --no-start-stop working on running db without --force.
Unable to reproduce this anymore.  It seems to have been fixed with the last round of config changes.  Add regression tests to make sure it doesn't happen again.
2015-09-08 18:29:13 -04:00
David Steele
b17bf31fb6 Implement issue #130: Info files should be updated with new version when pgBackRest is upgraded. 2015-09-08 16:58:00 -04:00
David Steele
59b2261a8e Fix issue #137: Info command throws an exception if no stanzas exist. 2015-09-08 16:44:47 -04:00
David Steele
bb6963cf3b Implement issue #135: Suppress PostgreSQL notices. 2015-09-08 14:01:19 -04:00
David Steele
adc5568b15 Fixed issue #132: Improved command-line help.
Some platform specific fixes and added regression tests to detect issues.
2015-09-08 12:58:13 -04:00
David Steele
ac3c0d43ab Implemented issue #132: Improved command-line help.
Implemented issue #133: Dynamic module loading where possible.
2015-09-08 07:31:24 -04:00
David Steele
0913072c45 Removed unused parameter from Info::new(). 2015-09-02 19:07:21 -04:00
David Steele
bf3ef15ccc Implemented issue #117: Refactor expiration tests to be purely synthetic 2015-09-01 19:05:10 -04:00
David Steele
6a9377a0a9 Code cleanup and refactoring to standardize on patterns that have evolved over time. 2015-08-29 14:20:46 -04:00
David Steele
83473eb9fb Merge branch 'master' into dev after release 0.80 2015-08-09 11:29:00 -04:00
David Steele
a59284a4cc v0.80: DBI Support, Stability, and Convenience Features
* Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the info command.  Only text output was affected -- json output reported the correct epoch values.  Reported by Michael Renner.

* Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.

* Now using Perl DBI and DBD::Pg for connections to PostgreSQL rather than psql.  The cmd-psql and cmd-psql-option settings have been removed and replaced with db-port and db-socket-path.

* Add stop-auto option to allow failed backups to automatically be stopped when a new backup starts.

* Add db-timeout option to limit the amount of time pgBackRest will wait for pg_start_backup() and pg_stop_backup() to return.

* Remove pg_control file at the beginning of the restore and copy it back at the very end.  This prevents the possibility that a partial restore can be started by PostgreSQL.

* The repository is now created and updated with consistent directory and file modes.  By default umask is set to 0000 but this can be disabled with the neutral-umask setting.

* Added checks to be sure the db-path setting is consistent with db-port by comparing the data_directory as reported by the cluster against the db-path setting and the version as reported by the cluster against the value read from pg_control.  The db-socket-path setting is checked to be sure it is an absolute path.

* Experimental support for PostgreSQL 9.5 alpha1.  This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace.  All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.

* Major refactoring of the protocol layer to support future development.

* Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.

* Split most of README.md out into USERGUIDE.md and CHANGELOG.md because it was becoming unwieldy.  Changed most references to "database" in the user guide to "database cluster" for clarity.
2015-08-09 11:20:43 -04:00
David Steele
7e49e4596f Removed dead function wait_for_file(). 2015-08-08 20:17:33 -04:00
David Steele
e82afac885 Implemented issue #94: backup_stop needs a timeout.
The timeout applies to all queries agains the cluster during a backup.
2015-08-08 18:22:38 -04:00
David Steele
8c0508789e Implemented issue #25: Option to call stop_backup with every backup. 2015-08-08 17:11:20 -04:00
David Steele
61c729b37b Worked on issue #122: 9.5 Integration.
Skip copying tablespace_map instead of deleting it later.
Cleaned up the skip code for pg_control as well.
2015-08-07 10:21:08 -04:00
David Steele
4e7bd4468a Worked on issue #122: 9.5 Integration.
Most tests are working now.  What's not working:

1) --target-resume option fails because pause_on_recovery setting was removed.  Need to implement to the new 9.5 option and make that work with older versions in a consistent way.
2) No tests for the new .partial WAL segments that can be generated on timeline switch.
2015-08-06 16:36:55 -04:00
David Steele
8b57188bc1 Fixed issue #120: Wrong "oldest backup timestamp" in info output. 2015-08-05 23:37:46 -04:00
David Steele
a10dd8ba98 Implemented issue #121: Check data from db against what's passed to backrest. 2015-08-05 22:05:45 -04:00
David Steele
23e6267adc Implemented issue #20: Convert Postgres calls from psql to DBD::Pg. 2015-08-05 12:32:12 -04:00
David Steele
f8a2da9400 Work on issue #48: Abandon threads and go to processes
* Major refactoring of the protocol layer to support this work.
* Fixed protocol issue that was preventing ssh errors (especially connect) from being logged.
2015-08-05 08:43:41 -04:00
David Steele
021afa8046 Ensure that info output is terminated by a linefeed.
On some systems the JSON->encode() function was adding a linefeed and on others it was not.  This was causing regression test failures in in the test logs and may have also been inconvenient for users.
2015-08-01 17:26:15 -04:00
David Steele
1b0f997f59 Implemented issue #119: Make sure all repo permissions are 640/750.
Updated remaining test logs.
2015-07-24 18:05:06 -04:00
David Steele
97c036973c Implemented issue #119: Make sure all repo permissions are 640/750. 2015-07-23 13:11:38 -04:00
David Steele
f3199d13b1 Implement issue #115: When restoring copy pg_control last. 2015-07-16 11:12:48 -04:00
David Steele
c568f5e727 v0.78: Remove CPAN dependencies, stability improvements
* Removed dependency on CPAN packages for multi-threaded operation.  While it might not be a bad idea to update the threads and Thread::Queue packages, it is no longer necessary.

* Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.

* Modified wait backoffs to use a Fibonacci sequence rather than geometric.  This will make then grow less aggressively while still giving reasonable wait times.

* More options for regression tests and improved code to run in a variety of environments.
2015-07-13 14:44:32 -04:00
David Steele
ea7914d980 Wait now uses a Fibonacci backoff rather than exponential. 2015-07-11 19:07:28 -04:00
David Steele
e9ce4efedf Removed dependency on dequeue_timed() which eliminates the CPAN install.
Added vagrant config for CentOS 6.
2015-07-10 09:20:28 -04:00
David Steele
8b7fd99637 Added vagrant configuration for Ubuntu 12.04.
Changes to regression tests to work better with VMs.
2015-07-06 21:00:48 -04:00
David Steele
38f48be3cf Improvements to help to make it more dynamic depending on environment.
Changed PgBackRest to pgBackRest.
2015-07-02 10:05:13 -04:00
David Steele
b1ded135df v0.77: CentOS/RHEL 6 support and protocol improvements
* Removed pg_backrest_remote and added the functionality to pg_backrest as remote command.

* Added file and directory syncs to the File object for additional safety during backup/restore and archiving.  Suggested by Andres Freund.

* Support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6.  Found by Eric Radman.

* Improved error message when backup is run without archive_command set and without --no-archive-check specified.  Found by Eric Radman.

* Moved version number out of the VERSION file to Version.pm to better support packaging.  Suggested by Michael Renner.

* Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple distros.
2015-06-30 14:47:58 -04:00
David Steele
7248795b91 Work on issue #48: Abandon threads and go to processes
Replaced IPC::System::Simple and Net::OpenSSH with IPC::Open3 to eliminate CPAN dependency for multiple distros.  Using open3 will also be used for local processes so it make sense to switch now.
2015-06-29 22:07:42 -04:00
David Steele
dcad0f7007 Merged issue #111: Print the real exception when DBH->connect fails.
Author: Eric Radman
2015-06-22 14:20:56 -04:00
David Steele
f210fe99c3 Implemented issue #109: Move VERSION into source code.
Also stopped replacing FORMAT number which explains the large number of test log changes.  FORMAT should change very rarely and cause test log failures when it does.
2015-06-22 13:11:07 -04:00
David Steele
3f841fcd95 Improved issue #110: 'db-version' is required but not defined.
Improved the error message and added hints.
2015-06-22 09:51:16 -04:00
David Steele
61dfeca3c1 Fixed issue #108: Incompatibility with Perl 5.10.1 2015-06-21 12:06:13 -04:00
David Steele
06c2904212 Change operation constants and functions to command. 2015-06-18 16:55:09 -04:00
David Steele
4e2c14623b Work on issue #48: Abandon threads and go to processes
More separation of the protocol and remote layers than was done in issue #106.
Settings are passed to the remote via command-line parameters rather than in the protocol.
2015-06-18 15:39:30 -04:00
David Steele
22e126eda7 Fix for logic that removes trailing spaces from code. 2015-06-17 15:40:12 -04:00
David Steele
a050b76fa9 Implement issue #107: Better debug commands. 2015-06-17 15:33:58 -04:00
David Steele
a5d9d6d84d Added file and directory syncs to the File object for additional safety during backup and archiving. 2015-06-17 12:53:33 -04:00
David Steele
9511f9c35c Implemented issue #106: Roll pg_backup_remote into pg_backrest 2015-06-17 11:26:07 -04:00
David Steele
1afbab010f v0.75: New repository format, info command and experimental 9.5 support
* IMPORTANT NOTE: This flag day release breaks compatibility with older versions of PgBackRest.  The manifest format, on-disk structure, and the binary names have all changed.  You must create a new repository to hold backups for this version of PgBackRest and keep your older repository for a time in case you need to do a restore.  The `pg_backrest.conf` file has not changed but you'll need to change any references to `pg_backrest.pl` in cron (or elsewhere) to `pg_backrest` (without the `.pl` extension).

* Add info command.

* More efficient file ordering for backup.  Files are copied in descending size order so a single thread does not end up copying a large file at the end.  This had already been implemented for restore.

* Logging now uses unbuffered output.  This should make log files that are being written by multiple threads less chaotic.  Suggested by Michael Renner.

* Experimental support for PostgreSQL 9.5.  This may break when the control version or WAL magic changes but will be updated in each release.
2015-06-14 20:59:32 -04:00
David Steele
b865070edd Experimental 9.5 support. Unit tests are not working yet. 2015-06-14 10:12:36 -04:00
David Steele
0b6f81a812 Implemented issue #50: Reverse ordering by file size. 2015-06-14 09:36:58 -04:00
David Steele
57640d1ac3 Implemented issue #96: Executable names 2015-06-14 08:37:37 -04:00
David Steele
148836fe44 Implemented issue #26: Info command.
* Includes updating the manifest to format 4.  It turns out the manifest and .info files were not very good for providing information.  A format update was required anyway so worked through the backlog of changes that would require a format change.

* Multiple database versions are now supported in the archive.  Does't actually work yet but the structure should be good.

* Tests use more constants now that test logs can catch name regressions.
2015-06-13 18:25:49 -04:00
David Steele
af98f164cd Fixed issue #97: Interleaved log lines during parallel operations. 2015-06-02 09:42:56 -04:00
David Steele
f83f0fa54d v0.70: Stability improvements for archiving, improved logging and help
* Fixed an issue where archive-copy would fail on an incr/diff backup when hardlink=n.  In this case the pg_xlog path does not already exist and must be created. Reported by Michael Renner

* Allow duplicate WAL segments to be archived when the checksum matches.  This is necessary for some recovery scenarios.

* Allow comments/disabling in pg_backrest.conf using #.  Suggested by Michael Renner.

* Better logging before pg_start_backup() to make it clear when the backup is waiting on a checkpoint.  Suggested by Michael Renner.

* Various command behavior, help and logging fixes.  Reported by Michael Renner.

* Fixed an issue in async archiving where archive-push was not properly returning 0 when archive-max-mb was reached and moved the async check after transfer to avoid having to remove the stop file twice.  Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong.  Reported by Michael Renner.

* Fixed a locking issue that could allow multiple operations of the same type against a single stanza.  This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. Reported by Michael Renner.

* Replaced JSON module with JSON::PP which ships with core Perl.
2015-06-01 09:24:38 -04:00
David Steele
95d62a82d9 Minor log changes for formatting and consistency. 2015-05-31 09:50:42 -04:00
David Steele
e7d2d704fe Fixed issue #91: Race condition in async archive-push. 2015-05-29 16:31:12 -04:00
David Steele
5fd4f3bd5b Improvements to log testing.
Replaced tokens are now numbered to give a more accurate rendition of the logs.
Some formatting improvements.
2015-05-29 14:39:43 -04:00
David Steele
d321ef0b6d Implement issue #89: Make confess backtraces log-level dependent.
ASSERTs still dump stack traces to the console and file in all cases.  ERRORs only dump stack traces to the file when the file log level is DEBUG or TRACE.
2015-05-29 12:26:31 -04:00
David Steele
13e4eec629 Implement issue #90: Move archive-max-mb archive-stop checks to end of archive-push. 2015-05-29 11:41:19 -04:00
David Steele
7ace66e544 Implement issue #68: Allow duplicate WAL to be pushed if the checksum matches. 2015-05-29 08:12:39 -04:00
David Steele
6d2f2d3f91 Fix for issue #74: More helpful message when targeting missing stanzas. 2015-05-28 20:52:42 -04:00
David Steele
3083e5fc7e Fix for issue #83: Provide more helpful error message during archive-stop situations.
Fix for issue #84: archive-async in combination with archive-max-mb doesn't work as documented.
Unit tests for archive stop.
2015-05-28 13:17:34 -04:00
David Steele
a4974f8996 Fix for issue #70: command line help improvements 2015-05-26 15:29:42 -04:00
David Steele
ca0c866d3b Implement issue #71: log a message when waiting for a checkpoint
Some other log improvements
2015-05-26 12:26:59 -04:00
David Steele
5c43e7035a Implement issue #79: Support comment characters in config file 2015-05-26 11:36:03 -04:00
David Steele
d5335b40e8 Fix for issue #80: enabling archive-copy causes failing differential & incremental backups 2015-05-22 14:49:14 -04:00
David Steele
5ada7fb5ad Use JSON::PP instead of JSON since it is shipped with core Perl. 2015-05-12 15:44:10 -04:00
David Steele
49fe40850f v0.65: Improved resume and restore logging, compact restores
* Better resume support.  Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses.  More unit tests to verify each resume case.

* Resume is now optional.  Use the `resume` setting or `--no-resume` from the command line to disable.

* More info messages during restore.  Previously, most of the restore messages were debug level so not a lot was output in the log.

* Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.

* Added `tablespace` setting to allow tablespaces to be restored into the `pg_tblspc` path.  This produces compact restores that are convenient for development, staging, etc.  Currently these restores cannot be backed up as PgBackRest expects only links in the `pg_tblspc` path.
2015-05-11 18:29:40 -04:00
David Steele
9aa41ee76b Added tablespace setting to allow compact database restores for development, staging, etc. 2015-05-08 18:34:27 -04:00
David Steele
a0d985b7ad Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path. 2015-05-07 17:37:25 -05:00
David Steele
1ac4b781fd Better info logging for restore. Most of the messages were debug before and some important ones were missing. 2015-05-07 15:56:56 -06:00
David Steele
095a9a0b83 Added resume setting to allow disabling resume feature. 2015-05-07 10:29:30 -06:00
David Steele
328c2ca5c0 Much better resume:
1) Re-checksums files that have checksums in the manifest
2) Recopies files that do not have a checksum
3) Saves the manifest at regular intervals to preserve checksums
4) Unit tests for all cases (that I can think of)
2015-05-06 18:24:34 -06:00
David Steele
56588f6fdd Log testing can now be enabled for certain deterministic tests. This works by comparing the generated logs against a previous copy. Currently only enabled for the backup/synthetic tests. 2015-05-05 11:08:48 -06:00
David Steele
174896e797 Centralized wait functions. 2015-04-23 20:52:41 -04:00
David Steele
2dac171642 More modular locking - this is the first part of a series of improvements. 2015-04-22 16:39:53 -04:00
David Steele
87f69508e3 Set version to 0.75 2015-04-21 10:48:01 -04:00
David Steele
a9b9ec2492 v0.61: bug fix for uncompressed remote destination
* Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination.  The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
2015-04-21 09:45:03 -04:00
David Steele
5185d49873 v0.60: better version support and WAL improvements
* Pushing duplicate WAL now generates an error.  This worked before only if checksums were disabled.

* Database System IDs are used to make sure that all WAL in an archive matches up.  This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.

* Regression tests working back to PostgreSQL 8.3.

* Improved threading model by starting threads early and terminating them late.
2015-04-19 17:53:29 -04:00
David Steele
b37d59832f v0.50: restore and much more
* Added restore functionality.

* All options can now be set on the command-line making pg_backrest.conf optional.

* De/compression is now performed without threads and checksum/size is calculated in stream.  That means file checksums are no longer optional.

* Added option `--no-start-stop` to allow backups when Postgres is shut down.  If `postmaster.pid` is present then `--force` is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created).  This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.

* Fixed broken checksums and now they work with normal and resumed backups.  Finally realized that checksums and checksum deltas should be functionally separated and this simplied a number of things.  Issue #28 has been created for checksum deltas.

* Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.

* Removed dependency on Moose.  It wasn't being used extensively and makes for longer startup times.

* Checksum for backup.manifest to detect corrupted/modified manifest.

* Link `latest` always points to the last backup.  This has been added for convenience and to make restores simpler.

* More comprehensive unit tests in all areas.
2015-03-25 15:15:55 -04:00
David Steele
4bc4d97f2b v0.30: core restructuring and unit tests
* Complete rewrite of BackRest::File module to use a custom protocol for remote operations and Perl native GZIP and SHA operations.  Compression is performed in threads rather than forked processes.

* Fairly comprehensive unit tests for all the basic operations.  More work to be done here for sure, but then there is always more work to be done on unit tests.

* Removed dependency on Storable and replaced with a custom ini file implementation.

* Added much needed documentation (see INSTALL.md).

* Numerous other changes that can only be identified with a diff.
2014-10-05 19:49:30 -04:00