1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-12 10:04:14 +02:00
Commit Graph

4553 Commits

Author SHA1 Message Date
David Steele
dafaeb0b82 v0.15: Added archive-get
* Added archive-get functionality to aid in restores.

* Added option to force a checkpoint when starting the backup (start_fast=y).
2014-03-29 18:16:08 -04:00
David Steele
ba2590dc8e archive-get function is working.
Added option to force checkpoint at backup start.
2014-03-28 15:20:36 -04:00
David Steele
8ae6c2719d Better code in file_list_get(). 2014-03-28 11:59:44 -04:00
David Steele
23df43e781 file_list_get now works locally and remotely
Conflicts:
	README.md
	pg_backrest.pl
	pg_backrest_db.pm
	pg_backrest_file.pm
2014-03-28 11:40:37 -04:00
David Steele
9d949b7ad6 v0.11: Minor fixes
Tweaking a few settings after running backups for about a month.

* Removed master_stderr_discard option on database SSH connections.  There have been occasional lockups and they could be related issues originally seen in the file code.

* Changed lock file conflicts on backup and expire commands to ERROR.  They were set to DEBUG due to a copy-and-paste from the archive locks.
2014-03-26 10:53:49 -04:00
David Steele
126b39855d Working on archive-get 2014-03-25 18:56:05 -04:00
David Steele
e68f836b2e Documented enhancements planned for v0.15 2014-03-08 09:12:53 -05:00
David Steele
bbb3b48cf8 Added comments. 2014-03-05 20:06:13 -05:00
David Steele
587bd1f8d9 v0.10: Backup and archiving are functional
This version has been put into production at Resonate, so it does work, but there are a number of major caveats.

* No restore functionality, but the backup directories are consistent Postgres data directories.  You'll need to either uncompress the files or turn off compression in the backup.  Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.

* Archiving is single-threaded.  This has not posed an issue on our multi-terabyte databases with heavy write volume.  Recommend a large WAL volume or to use the async option with a large volume nearby.

* Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% threadsafe so it will very occasionally lock up on a thread.  There is an overall process timeout that resolves this issue by killing the process.  Yes, very ugly.

* Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes.  Checksums from previous backups are correctly recorded and a full backup will reset everything.

* The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well.  Would definitely like to save these as human-readable text.

* Absolutely no documentation (outside the code).  Well, excepting these release notes.

* Lots of other little things and not so little things.  Much refactoring to follow.
2014-03-05 19:53:13 -05:00
David Steele
1617afb704 Initial commit. 2014-03-05 19:51:03 -05:00
David Steele
1a4f2527e3 Fixed version number. 2014-03-05 19:49:05 -05:00
David Steele
ae30ec98e9 First release merge from master. 2014-03-05 19:47:05 -05:00
David Steele
81616e19bd v0.01: Backup and archiving are functional
This version has been put into production at Resonate, so it does work, but there are a number of major caveats.

* No restore functionality, but the backup directories are consistent Postgres data directories.  You'll need to either uncompress the files or turn off compression in the backup.  Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.

* Archiving is single-threaded.  This has not posed an issue on our multi-terabyte databases with heavy write volume.  Recommend a large WAL volume or to use the async option with a large volume nearby.

* Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% threadsafe so it will very occasionally lock up on a thread.  There is an overall process timeout that resolves this issue by killing the process.  Yes, very ugly.

* Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes.  Checksums from previous backups are correctly recorded and a full backup will reset everything.

* The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well.  Would definitely like to save these as human-readable text.

* Absolutely no documentation (outside the code).  Well, excepting these release notes.

* Lots of other little things and not so little things.  Much refactoring to follow.
2014-03-05 19:45:01 -05:00
David Steele
951419178b Initial commit. 2014-03-05 19:43:23 -05:00
David Steele
2b55b8ca5e Merge branch 'master' of github.com:dwsteele/pg_backrest into dev 2014-03-05 19:10:23 -05:00
David Steele
1eb8e1be2d v0.01: Backup and archiving are functional
This version has been put into production at Resonate, so it does work, but there are a number of major caveats.

* No restore functionality, but the backup directories are consistent Postgres data directories.  You'll need to either uncompress the files or turn off compression in the backup.  Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.

* Archiving is single-threaded.  This has not posed an issue on our multi-terabyte databases with heavy write volume.  Recommend a large WAL volume or to use the async option with a large volume nearby.

* Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% threadsafe so it will very occasionally lock up on a thread.  There is an overall process timeout that resolves this issue by killing the process.  Yes, very ugly.

* Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes.  Checksums from previous backups are correctly recorded and a full backup will reset everything.

* The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well.  Would definitely like to save these as human-readable text.

* Absolutely no documentation (outside the code).  Well, excepting these release notes.

* Lots of other little things and not so little things.  Much refactoring to follow.
2014-03-05 18:56:15 -05:00
David Steele
80d5332fa0 Squashed commit of the following:
commit f0d82151b9
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Mar 3 16:38:01 2014 -0500

    Changed thread timeout from WARN to ERROR

commit 571d449717
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 27 19:15:00 2014 -0500

    Redirect find error output to /dev/null.  Sometimes files are removed from the db while find is running.  We only want to error if the find process errors.

commit 84c4cec257
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 27 13:00:41 2014 -0500

    Put redirects in common places where errors are be output but we don't want to see them.

commit ae782d0884
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 25 07:30:45 2014 -0500

    Made archiving single-threaded so it will work

commit a84478045e
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 24 18:04:42 2014 -0500

    No attempt to kill threads - just abort

commit da41eb3883
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 24 17:59:42 2014 -0500

    Added more warnings

commit 21b15b7cf6
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 24 09:33:47 2014 -0500

    Added thread timeout.

commit 5de8f275f0
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 22 11:14:01 2014 -0500

    Removed stderr redirects from the master ssh connection

commit 0e4f0f1308
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 22 10:54:02 2014 -0500

    Fixed issue with archive path expiration

commit e8c7d7aca6
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 22 09:05:58 2014 -0500

    When archive_retention_type is full, use the oldest backup if there are fewer than required

commit 9270a67892
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 21 16:09:34 2014 -0500

    Removed some more stderr redirects

commit 6d975d2830
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 21 08:49:27 2014 -0500

    No longer discarding stderr

commit 0387a8ee09
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 21 07:34:17 2014 -0500

    replaced process id with thread id.  Added use thread to all modules.

commit ac3ce81621
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 21 06:42:42 2014 -0500

    Add process id to logging, don't generate checksum for 0 length files

commit 3d8e933343
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 19 16:06:21 2014 -0500

    More threads

commit 857e177429
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 19 15:13:07 2014 -0500

    Using separate master queues

commit 800cd89831
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 19 13:37:18 2014 -0500

    Thread files are no longer in an array

commit 81433c0fdd
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 19 13:03:52 2014 -0500

    Now using one control master for threading.

commit b4ac9552d2
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 18 19:29:57 2014 -0500

    Skipped files removed from manifest, checksums stored

commit 27b820cfb4
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 18 15:42:51 2014 -0500

    Backup handles files that were removed by the db

commit 187fdaf49b
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 16 19:01:06 2014 -0500

    Better compression choices.

commit 5c695a6522
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 16 18:20:55 2014 -0500

    Improved archive logging and put in max limit

commit 13086e9626
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 16 17:02:25 2014 -0500

    Renamed processes.

commit eb25dbdb9d
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 16 16:58:25 2014 -0500

    Added stanza to command string

commit b45d9c946e
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 16 16:51:05 2014 -0500

    Improved command string

commit 6d7f89fc98
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 16 01:31:24 2014 -0500

    change process string

commit 577c5af33f
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 16 01:24:54 2014 -0500

    testing

commit b514058cee
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 15 16:46:13 2014 -0500

    Changed to perl storable - at least it works

commit 392da91620
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 15 15:28:22 2014 -0500

    Remove old local archive dirs.

commit 0bc40d206c
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 15 14:33:31 2014 -0500

    Some fixes for config

commit 0979841f1a
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 15 14:18:15 2014 -0500

    Async compress, thread kill improvements

commit bd00fb7f0c
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 15 10:45:01 2014 -0500

    Still working on thread kill

commit 3b2dd68aac
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 15 10:31:57 2014 -0500

    More robust join code.

commit 27bf9b6572
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 15 10:09:47 2014 -0500

    First SSH session is shared between objects and thread 0

commit c7aebf05c4
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 14 21:39:18 2014 -0500

    Handle term signals gracefully.

commit d100294894
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 14 19:56:28 2014 -0500

    Ability to resume failed backups, better locking

commit 308652cc65
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 14 09:05:14 2014 -0500

    Remove a lot of path creates - percent logging.

commit e14d52c42a
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 14 08:23:25 2014 -0500

    Improved backup threading

commit e93a444a41
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 23:37:52 2014 -0500

    Changes to manifest.

commit 5c393a0df9
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 21:44:19 2014 -0500

    Increased time to wait for log files

commit dccdec2ffa
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 21:41:20 2014 -0500

    trying to make labels stick

commit ae3dcc57f0
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 21:31:57 2014 -0500

    Thread errors now exit

commit 6d9e7db1c6
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 20:54:41 2014 -0500

    Increased thread total.

commit 405d36497d
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 20:33:05 2014 -0500

    Less logging

commit a78d677b1e
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 20:29:42 2014 -0500

    Wait for archive files.

commit d4ba078183
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 18:14:15 2014 -0500

    Only save manifest once

commit 1bec36b65b
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 18:07:58 2014 -0500

    Better thread logging.

commit 2bafa5cf26
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 17:52:38 2014 -0500

    Logging tweak

commit 8453681ccd
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 17:50:14 2014 -0500

    Logging tweaks

commit 4349f0caff
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 17:47:53 2014 -0500

    Fixes to log level and more logging

commit 7d55ce865b
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 17:11:53 2014 -0500

    Work on log levels

commit 133e27d813
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 15:26:07 2014 -0500

    Archive tmp file fix.

commit 19f9a5193f
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 14:54:28 2014 -0500

    Minor config changes.

commit 13a1f0cf24
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 13:54:43 2014 -0500

    Lots and lots of logging improvements

commit 14b13d4840
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 12:02:45 2014 -0500

    Async archive log backup working.

commit f0df759bb9
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 13 00:21:25 2014 -0500

    Small debug fix

commit 7c4d463b1d
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 23:49:40 2014 -0500

    More balanced backup.

commit b770c8ebec
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 22:33:09 2014 -0500

    More work on archive-pull

commit 8e60a3111f
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 20:28:27 2014 -0500

    Lots of archive-pull plumbing

commit 16df430b73
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 17:34:34 2014 -0500

    New constants and basic fork.

commit 564bd65b3e
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 16:01:48 2014 -0500

    Basic plumbing for archive-pull function

commit 5181282b74
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 08:04:42 2014 -0500

    Command line parameter checks

commit 729b07fed5
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 07:45:29 2014 -0500

    Hardlink now a conf option.

commit 219120ba81
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 12 07:39:42 2014 -0500

    Archiving and flag changes.

commit b9a2e7093e
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 11 21:17:30 2014 -0500

    Some more file debug info

commit 1a9f73b781
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 11 17:07:49 2014 -0500

    Added timestamp to log

commit 49dbc897a6
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 11 16:57:51 2014 -0500

    Small file allocation improvements.

commit 9a1de327a1
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 11 16:37:20 2014 -0500

    Turned ssh compression on

commit 71812fca99
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 11 16:21:33 2014 -0500

    Thread total is defined in the conf file.

commit 171ca709d6
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 11 16:08:55 2014 -0500

    Better file ordering for threads.

commit c58ccb588e
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 11 15:31:16 2014 -0500

    Parallel backup now working

commit a55b37efc3
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Feb 7 19:34:14 2014 -0500

    Working on parallel backup

commit 820791cf92
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 6 17:31:33 2014 -0500

    FF log is skipped for pre-9.3 databases.

commit 60b2be58ac
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 6 16:37:37 2014 -0500

    Finished db object abstraction

commit f20f3b3ee6
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 6 15:54:22 2014 -0500

    Started db object abstraction.

commit 9d08f2a644
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Feb 6 12:49:54 2014 -0500

    More robust config.  Retention is read from config.

commit a2b0d7a674
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 22:26:10 2014 -0500

    Backup expiration working again.  Other changes.

commit 27986e0c10
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 21:39:08 2014 -0500

    File is now an object.

commit 3e7faf0e5f
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 16:53:25 2014 -0500

    Some path fixes.

commit 7bee0f3141
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 15:56:05 2014 -0500

    Moved backup location.

commit a15fcb8c23
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 13:10:36 2014 -0500

    Moved tmp path.

commit 7ffade2453
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 11:35:09 2014 -0500

    Fixed file mod time and permissions

commit c86a61ca3e
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 10:40:49 2014 -0500

    Relative links working for tablespaces.

commit 12a960f91a
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Feb 5 08:21:27 2014 -0500

    File copy supports decompression

commit a810499958
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Feb 4 18:48:39 2014 -0500

    Remote backup fully working.

commit e38c482a26
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 3 20:48:02 2014 -0500

    Cleanup

commit 946b8261cf
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 3 20:29:03 2014 -0500

    Moved archive_push function

commit d5832a70c0
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 3 20:23:04 2014 -0500

    Moved last backup function.

commit aef6730267
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 3 19:57:21 2014 -0500

    Moved backup functions.

commit 316a5b5598
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 3 19:03:17 2014 -0500

    Working on remote

commit fa668d5007
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Feb 3 14:50:23 2014 -0500

    Move file init code to file module

commit aef4adf1f3
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 2 19:03:05 2014 -0500

    Splitting main perl file

commit 4c2e4f7e49
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 2 12:45:59 2014 -0500

    Lots more work on remote functionality

commit bf8ac7abe6
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 2 11:12:55 2014 -0500

    manifest_get() now works remotely

commit 4556de4722
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Feb 2 10:40:05 2014 -0500

    Source and destination can now be remote.

commit de972ef5d3
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 1 12:57:27 2014 -0500

    More work on file_copy

commit 52ba8801ef
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Feb 1 11:04:33 2014 -0500

    Working on remote file copy

commit 2ad7a3d815
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 31 00:04:08 2014 -0500

    Formatting.

commit c5de8fa6e6
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 30 23:58:26 2014 -0500

    Created variable for compression extension

commit 151ff8ba60
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 30 23:24:34 2014 -0500

    Removed unused strPath param from path_get()

commit 52004d3c84
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 30 23:16:19 2014 -0500

    Comments

commit 447d83fcdc
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 30 23:09:34 2014 -0500

    Removing more copies.

commit f86618377c
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 30 11:35:34 2014 -0500

    Lots of backup refactoring.

commit e13a706c0a
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Jan 29 15:38:57 2014 -0500

    More moving.  Links now also get references.

commit 54c73b3813
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Jan 29 15:03:57 2014 -0500

    Moving stuff around

commit c26b47d4c7
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Jan 29 12:06:30 2014 -0500

    Abstracting file functions.

commit 47c3c82540
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Jan 29 06:40:15 2014 -0500

    Fixed checksum function.

commit 12f6d889de
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Jan 28 17:16:07 2014 -0500

    More modular file copy.

commit 6625ea1405
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 23 19:39:51 2014 -0500

    Coded paths needed for archive logging

commit cdd1ef7306
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Jan 22 21:40:56 2014 -0500

    Added general path function

commit 161de23c3b
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Jan 22 19:03:20 2014 -0500

    Starting new structure

commit 89b96954f6
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Jan 21 22:02:14 2014 -0500

    Allow any archive file start starts with 24 hex characters.

commit 7e73cf22cf
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Jan 21 21:59:54 2014 -0500

    Changed where .backup files were copied

commit 62ae46e213
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Jan 21 21:12:24 2014 -0500

    Archive expiration working.

commit a83e94ba27
Author: David Steele <github@thelabyrinth.net>
Date:   Tue Jan 21 20:38:58 2014 -0500

    Fixed error in the backup regexp

commit faaaa5690f
Author: David Steele <github@thelabyrinth.net>
Date:   Mon Jan 20 20:09:50 2014 -0500

    Working on archive retention.

commit df7ada85c2
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Jan 18 10:14:35 2014 -0500

    Working on expiration

commit 1e0f2485db
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Jan 18 00:03:32 2014 -0500

    Working on expiration

commit 3719d54cc9
Author: David Steele <github@thelabyrinth.net>
Date:   Wed Jan 15 20:41:29 2014 -0500

    Some notes. removed dead .backup move

commit a7de734105
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Jan 12 20:12:49 2014 -0500

    Some cleanup - fixed tablespace path regression.

commit 05dcbc2246
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Jan 12 15:28:19 2014 -0500

    Working on making backups consistent

commit c1e5904880
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Jan 12 01:46:27 2014 -0500

    Fixed xlog copy - working on correctness

commit 3ed83e78f2
Author: David Steele <github@thelabyrinth.net>
Date:   Sat Jan 11 02:16:14 2014 -0500

    Working on archive log copy

commit 387439ac8e
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 10 16:01:25 2014 -0500

    Working on processing archive logs

commit e23820098d
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 10 14:02:56 2014 -0500

    Notices are not emitted on start and stop backup

commit 8b73a26b84
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 10 00:49:19 2014 -0500

    Some notes for things to do.

commit 49c1fcd3eb
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 10 00:43:26 2014 -0500

    Backup file is being copied to pg_xlog

commit 8e966c50dd
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 10 00:22:51 2014 -0500

    Removed commented code

commit b5f965942b
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 10 00:21:46 2014 -0500

    Added optional hard-linking.

commit cceac444ad
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 21:55:09 2014 -0500

    Added comment

commit c47ec3d260
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 21:40:33 2014 -0500

    All backup references are now stored.

commit ed5f8226a4
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 20:12:46 2014 -0500

    References are from oldest backup

commit bb8fb5b54f
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 20:00:34 2014 -0500

    Sparse backup dirs implemented

commit 854972dde6
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 18:23:50 2014 -0500

    Basic deltas work.

commit c89984cd59
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 18:02:42 2014 -0500

    Revert 2914a6d..e31b21f

    This rolls back to commit 2914a6dcc6.

commit e31b21f95a
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 18:01:34 2014 -0500

    Revert 52d8697..e0be213

    This rolls back to commit 52d8697118.

commit e0be213206
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 18:01:12 2014 -0500

    Revert 2914a6d..9633e17

    This rolls back to commit 2914a6dcc6.

commit 9633e17d04
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 17:58:40 2014 -0500

    Revert "Removed tmp file."

    This reverts commit 52d8697118.

commit 2914a6dcc6
Author: David Steele <github@thelabyrinth.net>
Date:   Thu Jan 9 17:58:01 2014 -0500

    Working on delta

commit 52d8697118
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 3 21:24:13 2014 -0500

    Removed tmp file.

commit ba5075a3e0
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 3 21:19:41 2014 -0500

    New implementation for archive logging

commit 6379b723de
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 3 20:10:53 2014 -0500

    Set the backup label

commit 78457a2f42
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 3 19:59:47 2014 -0500

    Simplified incremental naming

commit ef846ac8db
Author: David Steele <github@thelabyrinth.net>
Date:   Fri Jan 3 19:28:49 2014 -0500

    Working on backup rotation.

commit e6e995ec19
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 18 22:08:49 2013 -0500

    Fixed typo in config.

commit ba72eda700
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 18 21:30:51 2013 -0500

    Commented out dir mod time variable.

commit 9907c8b7b7
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Tue Dec 17 07:47:24 2013 -0500

    Set file mtime.

commit b1ffa94190
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 16 21:59:22 2013 -0500

    New storage structure, saved archive start/stop location.

commit 6d992855d2
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sun Dec 15 20:56:42 2013 -0500

    Cleaning up start stop backup code.

commit 0d3fa39117
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sun Dec 15 20:04:07 2013 -0500

    Added start and stop backup.

commit 1d21f4f9fa
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sun Dec 15 18:18:54 2013 -0500

    Added diff to commands.

commit 31fca50eae
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sun Dec 15 17:18:50 2013 -0500

    Backup more or less works - no start/stop backup or archiving.

commit 21b7a3f27a
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sat Dec 14 15:02:47 2013 -0500

    Added manifest load and save.

commit 8b40ea7d59
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sat Dec 14 14:03:08 2013 -0500

    Tablespace names are now recorded

commit 2ba26ffc26
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sat Dec 14 13:14:59 2013 -0500

    More refactoring.

commit d12773ca85
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 11 20:39:07 2013 -0500

    Added generic config command

commit 4aa480b858
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 11 20:13:39 2013 -0500

    Cleaned up some var names.

commit 299d402209
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 11 19:57:54 2013 -0500

    More cleanup

commit aa6e72bb9c
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 11 19:22:41 2013 -0500

    Refactoring

commit 0dba2ab8c7
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Tue Dec 10 20:16:43 2013 -0500

    Added backup command

commit 0bdccd2982
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Tue Dec 10 19:18:41 2013 -0500

    Reversed order of tests.

commit f09505f60e
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Tue Dec 10 19:11:54 2013 -0500

    Working on unit tests for archiving.

commit 39989dd01c
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 9 17:26:47 2013 -0500

    Working on manifest.

commit 02a9a693b8
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Fri Dec 6 20:24:14 2013 -0500

    Working on backup.

commit 4d42d448e9
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Thu Dec 5 23:29:05 2013 -0500

    Working on backup.

commit 12518e7296
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Thu Dec 5 08:59:39 2013 -0500

    Working on backup.

commit ae6886c8e0
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 4 22:30:26 2013 -0500

    Started backup command.

commit c9c7e340a7
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Wed Dec 4 21:37:45 2013 -0500

    Added config file.

commit 53b2aae106
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 2 21:37:01 2013 -0500

    Added note.

commit 4a2b009c2e
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 2 21:34:25 2013 -0500

    Removed unused include.

commit 1e114cdc9d
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 2 21:26:32 2013 -0500

    Fix comments.

commit 9553baf540
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 2 21:23:10 2013 -0500

    Basic error checking and options for archive-local command

commit c7c049902f
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 2 16:21:40 2013 -0500

    Added warnings, comments, removed dead code.

commit 5761832df5
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 2 15:10:18 2013 -0500

    Implemented strict.

commit a2c98391df
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Mon Dec 2 14:34:37 2013 -0500

    Starting on configuration.

commit 2be8c5d9e7
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sat Nov 23 20:05:04 2013 -0500

    Working on unit tests.

commit 5d0251be2f
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Sat Nov 23 19:16:09 2013 -0500

    Working on archive-local and beefing up unit tests.

commit 8b37786e5b
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Fri Nov 22 23:24:37 2013 -0500

    Now connecting to db.

commit 8db52c115a
Author: David Steele <gitlab@thelabyrinth.net>
Date:   Fri Nov 22 17:29:01 2013 -0500

    Fixed whitespace.

commit d86e7eb627
Author: David Steele <dsteele@Davids-MacBook-Pro.local>
Date:   Wed Nov 20 22:30:26 2013 -0500

    Added checksums.

commit bc46aefe61
Author: David Steele <dsteele@Davids-MacBook-Pro.local>
Date:   Wed Nov 20 22:24:30 2013 -0500

    Fixed for OSX.  Do not every use TextEditor on code!

commit e67821a230
Author: David Steele <dsteele@Davids-MacBook-Pro.local>
Date:   Wed Nov 20 21:49:07 2013 -0500

    Working on OSX port.

commit 9ce37308b5
Author: David Steele <github@thelabyrinth.net>
Date:   Sun Nov 17 21:48:53 2013 -0500

    Working on archive-local command.

commit 548578c8c9
Author: David Steele <dsteele@laptop-dev.(none)>
Date:   Sun Nov 17 13:58:21 2013 -0500

    Working on unit test.
2014-03-04 08:00:51 -05:00
David Steele
f0d82151b9 Changed thread timeout from WARN to ERROR 2014-03-03 16:38:01 -05:00
David Steele
571d449717 Redirect find error output to /dev/null. Sometimes files are removed from the db while find is running. We only want to error if the find process errors. 2014-02-27 19:15:00 -05:00
David Steele
84c4cec257 Put redirects in common places where errors are be output but we don't want to see them. 2014-02-27 13:00:41 -05:00
David Steele
ae782d0884 Made archiving single-threaded so it will work 2014-02-25 07:30:45 -05:00
David Steele
a84478045e No attempt to kill threads - just abort 2014-02-24 18:04:42 -05:00
David Steele
da41eb3883 Added more warnings 2014-02-24 17:59:42 -05:00
David Steele
21b15b7cf6 Added thread timeout. 2014-02-24 09:33:47 -05:00
David Steele
5de8f275f0 Removed stderr redirects from the master ssh connection 2014-02-22 11:14:01 -05:00
David Steele
0e4f0f1308 Fixed issue with archive path expiration 2014-02-22 10:54:02 -05:00
David Steele
e8c7d7aca6 When archive_retention_type is full, use the oldest backup if there are fewer than required 2014-02-22 09:05:58 -05:00
David Steele
9270a67892 Removed some more stderr redirects 2014-02-21 16:09:34 -05:00
David Steele
6d975d2830 No longer discarding stderr 2014-02-21 08:49:27 -05:00
David Steele
0387a8ee09 replaced process id with thread id. Added use thread to all modules. 2014-02-21 07:34:17 -05:00
David Steele
ac3ce81621 Add process id to logging, don't generate checksum for 0 length files 2014-02-21 06:42:42 -05:00
David Steele
3d8e933343 More threads 2014-02-19 16:06:21 -05:00
David Steele
857e177429 Using separate master queues 2014-02-19 15:13:07 -05:00
David Steele
800cd89831 Thread files are no longer in an array 2014-02-19 13:37:18 -05:00
David Steele
81433c0fdd Now using one control master for threading. 2014-02-19 13:03:52 -05:00
David Steele
b4ac9552d2 Skipped files removed from manifest, checksums stored 2014-02-18 19:29:57 -05:00
David Steele
27b820cfb4 Backup handles files that were removed by the db 2014-02-18 15:42:51 -05:00
David Steele
187fdaf49b Better compression choices. 2014-02-16 19:01:06 -05:00
David Steele
5c695a6522 Improved archive logging and put in max limit 2014-02-16 18:20:55 -05:00
David Steele
13086e9626 Renamed processes. 2014-02-16 17:02:25 -05:00
David Steele
eb25dbdb9d Added stanza to command string 2014-02-16 16:58:25 -05:00
David Steele
b45d9c946e Improved command string 2014-02-16 16:51:05 -05:00
David Steele
6d7f89fc98 change process string 2014-02-16 01:31:24 -05:00
David Steele
577c5af33f testing 2014-02-16 01:24:54 -05:00
David Steele
b514058cee Changed to perl storable - at least it works 2014-02-15 16:46:13 -05:00
David Steele
392da91620 Remove old local archive dirs. 2014-02-15 15:28:22 -05:00
David Steele
0bc40d206c Some fixes for config 2014-02-15 14:33:31 -05:00
David Steele
0979841f1a Async compress, thread kill improvements 2014-02-15 14:18:15 -05:00
David Steele
bd00fb7f0c Still working on thread kill 2014-02-15 10:45:01 -05:00
David Steele
3b2dd68aac More robust join code. 2014-02-15 10:31:57 -05:00