David Steele
43d86e64a4
First pass at tests comparing rsync to backrest. Decent results, but room for improvement.
...
All tests local over SSH with rsync default compression, 4 threads and default compression on backrest. Backrest default is gzip = 6, assuming rsync is the same.
On a 1GB DB:
rsync time = 32.82
backrest time = 19.48
backrest is 171% faster.
On a 5GB DB:
rsync time = 171.16
backrest time = 86.97
backrest is 196% faster.
2015-04-07 18:36:59 -04:00
David Steele
7081c8b867
New model where threads are created early and destroyed late.
...
Backups now work like restores in terms of how jobs are queued.
Split out BackupFile and RestoreFile for easier multi-threading/processing.
2015-04-07 07:34:37 -04:00
David Steele
3f651a8ce8
Unit tests will now work across all installed versions of Postgres.
...
Created a function to list all supported versions. Now used for all version checking.
2015-04-02 22:07:23 -04:00
David Steele
984fceea67
Moved archive functions from pg_backrest.pl and Backup.pl to Archive.pm.
...
Moved Remote code from pg_backrest.pl to Config.pm.
Added version specific code to regression tests and Db.pm.
archive-push checks for duplicate WAL in the archive.
archive-push reads the db sys id to match up WAL to the correct archive.
2015-04-01 15:58:33 -04:00
David Steele
a998119cba
Various option and unit test fixes.
2015-03-24 18:50:14 -04:00
David Steele
a3604bc512
Changed retention section to expire.
2015-03-24 11:43:37 -04:00
David Steele
58d6c37f4a
Changed permission to mode in manifest.
2015-03-23 17:17:43 -04:00
David Steele
8a3c6388b2
Removed WAL from some option names - still waffling on this naming thing.
2015-03-23 16:01:15 -04:00
David Steele
1729fa1c51
Added unit tests for expiration.
2015-03-17 18:31:05 -04:00
David Steele
f4f31302f0
Broke archive-required into two options: backup-wal-check and backup-wal-store.
2015-03-16 14:45:53 -04:00
David Steele
258fb9c6e2
More work on automated docs. Merging this to go back to some feature/bug work for a while.
2015-03-16 14:01:01 -04:00
David Steele
882f068254
All config options can now be provided on the command-line.
...
Lots of name changes, but more thought needed there.
Still needs cleanup.
2015-03-12 12:15:19 -04:00
David Steele
ae6bdecfaf
Split command-line parameter processing out into a separate file. This is in preparation allowing all parameters to be specified/overridden on the command line, with pg_backrest.conf being option.
2015-03-08 13:26:09 -04:00
David Steele
d19baefdb9
Removed db-timestamp-start and stop from manifest. Better to get these values from
2015-03-03 22:33:41 -05:00
David Steele
7509b01e22
Added format to manifest to identify a manifest/dir structure version.
2015-03-03 21:21:07 -05:00
David Steele
f5761a41c1
Modules cleanup.
2015-03-03 10:03:55 -05:00
David Steele
3d0d308a98
Made all warnings FATAL.
...
Fixed one bug that was exposed.
2015-03-03 00:57:20 -05:00
David Steele
c0ec62b131
Fixed bug in File->hash() and removed improper last statements from do loops.
2015-03-02 23:42:55 -05:00
David Steele
7dbb11e8b8
Added buffer-size param and added config validation improvements.
2015-03-02 22:58:32 -05:00
David Steele
8e8f2f3c77
New config options compress-level and compress-level-network to control compression levels.
2015-03-02 20:36:12 -05:00
David Steele
49112eb6d3
Better compression window settings.
2015-03-02 09:46:11 -05:00
David Steele
d874e122aa
Optimized compression to disable headers/crc when the source and destination are both uncompressed.
...
Set different compression levels based on usage.
2015-03-01 23:41:01 -05:00
David Steele
0df1b7c473
Changed file sizes are now detected and stored in the manifest.
...
Remove thread file minimums - they are unrealistic for a real db and hinder unit tests.
2015-03-01 22:43:32 -05:00
David Steele
23102f19e5
All copy() cases generate checksums during copy.
...
Unit tests for bAppendChecksum option for copy().
2015-03-01 22:23:52 -05:00
David Steele
76391dde90
Archive checksums are not calculated in stream.
2015-03-01 16:42:27 -05:00
David Steele
393525d280
Full refactor of binary_xfer.
...
Compression now uses Compress::Raw::Zlib.
2015-03-01 15:17:57 -05:00
David Steele
77bc4238dc
ZLib stuff starting to look good. All references removed from File and using binary_xfer for all de/compression.
2015-03-01 13:41:35 -05:00
David Steele
7ede058b45
Backup checksums are no longer optional.
2015-02-28 20:31:56 -05:00
David Steele
28326d6b4c
File->copy now returns hash and size in all cases, though the local copies are not optimal. They just call hash_size().
2015-02-28 19:07:29 -05:00
David Steele
260a6cb8f1
Working on calculating hash and file size in binary_xfer().
2015-02-28 15:54:49 -05:00
David Steele
0f94476a1f
Changed buffer sizes.
2015-02-28 12:10:50 -05:00
David Steele
f6b9ab7326
Minor param fix.
2015-02-28 10:28:25 -05:00
David Steele
f93c6caec2
Backup/restore copy will be run in the main process when thread-max=1. I've resisted this change because it adds complexity, but I have to accept that threads are not stable on all platforms. Or maybe any platform.
2015-02-28 10:23:33 -05:00
David Steele
d6205d9501
Looks like all unit tests pass - now for a long test run to see if that is really true. And to see if the old lockup is gone.
2015-02-27 23:31:39 -05:00
David Steele
25442655c8
Hash of compressed file is working.
...
Something still broken in binary_xfer because some 0 length archive files are showing up. Investigating.
2015-02-27 18:42:28 -05:00
David Steele
53f783d3fe
binary_xfer compress/decompression working without threads. All unit tests passing. Hooray.
2015-02-27 16:36:40 -05:00
David Steele
c18c629878
Compress looks good, back to decompress.
2015-02-27 14:11:59 -05:00
David Steele
bd44b2c471
Making binary xfer more modular.
2015-02-27 13:48:29 -05:00
David Steele
2506bbb21a
Working on decompression.
2015-02-26 19:12:22 -05:00
David Steele
64e1b1cea7
Simple compression case is working.
...
Still need to implement decompress, handle flags, and do checksum decompression.
2015-02-26 14:16:12 -05:00
David Steele
b9378d94a2
Working on removing threads from instream compress/decompression.
2015-02-26 09:22:05 -05:00
David Steele
d2602a5c07
Tracking down a lockup in the restore threads. It doesn't happen in backup - they are the same except that restore uses the ThreadGroup object. I'm beginning to think that threads and objects don't play together very nicely. Objects in threads seems OK, but threads in objects, not so much.
2015-02-03 20:33:33 -05:00
David Steele
d4aae88bec
Try non-blocking dequeue in backup.
2015-02-02 20:11:32 -05:00
David Steele
d93b246a69
Empty restore queues in a non-blocking way.
2015-02-02 19:34:34 -05:00
David Steele
7f38461c68
Remove ThreadQueue->end(). Not supported on all platforms.
2015-02-02 18:48:33 -05:00
David Steele
bde8943517
Fixes to restore compare unit tests.
2015-02-02 18:33:11 -05:00
David Steele
a6d3b7e1a9
Working on checking restores against the manifest. Current issue is that the manifest does not always record the final size of the file - it may change while the file is being copied. This is fine in principal but makes testing a pain.
2015-01-31 23:04:24 -05:00
David Steele
018a2afaca
Moved bNoStartStop tablespace mapping to Manifest object.
2015-01-31 18:23:37 -05:00
David Steele
d7b669d560
Moved manifest build to manifest object.
2015-01-31 18:15:08 -05:00
David Steele
267efbd71d
Fixed restore::thread-max
2015-01-31 15:37:59 -05:00
