1
0
mirror of https://github.com/pgbackrest/pgbackrest.git synced 2024-12-14 10:13:05 +02:00
Commit Graph

224 Commits

Author SHA1 Message Date
David Steele
79b7552940 Rename archive-queue-max option to archive-push-queue-max.
This avoids confusion with the archive-get-queue-max option. The old option name will continue to be accepted.
2018-04-18 19:37:35 -04:00
David Steele
0381945caa Show command parameters as well as command options in initial info message. 2018-04-17 18:47:14 -04:00
David Steele
f0250dab4b Move async forking and more error handling to C.
The Perl process was exiting directly when called but that interfered with proper locking for the forked async process. Now Perl returns results to the C process which handles all errors, including signals.
2018-04-12 20:42:26 -04:00
David Steele
6fd0c3dcaa Improved lock implementation written in C.
Now only two types of locks can be taken: archive and backup. Most commands use one or the other but the stanza-* commands acquire both locks. This provides better protection than the old command-based locking scheme.
2018-04-11 09:36:12 -04:00
David Steele
3bcf637aee Move archive-stop and expire tests to the mock module.
These are mock integration tests so they should be grouped with the other mock integration tests.
2018-04-08 12:57:02 -04:00
David Steele
3b661f4ad5 Move help/version integration tests to mock/all.
Help and version are covered by unit tests, so we really just to need to make sure there is output when called from the command line.
2018-04-08 12:16:32 -04:00
David Steele
348278bb68 Make backup directory sync more efficient.
Scanning the entire backup directory can be very expensive if there are a lot of small tables. The backup manifest contains the backup directory list so use it to perform syncs instead of scanning the backup directory.
2018-04-03 21:30:15 -04:00
David Steele
0dce8ba2fa Make backup.history sync more efficient.
Only the backup.history/[year] directory was being synced, so check if the backup.history is newly created and sync it as well.
2018-04-03 20:54:18 -04:00
David Steele
90f980fe91 Remove documentation describing info --output=json as experimental. 2018-03-30 21:33:30 -04:00
Cynthia Shang
f0a4264f77 Fix manifest test in the check command when tablespaces are present.
Fixed by Cynthia Shang.
Reported by Thomas Flatley.
2018-03-19 19:54:03 -04:00
Cynthia Shang
dd31ae832d Port most of Config::Config::configLoad() from Perl to C.
Contributed by Cynthia Shang.
2018-03-12 11:25:20 -04:00
David Steele
03f1082e86 Improve logging.
Move command begin to C except when it must be called after another command in Perl (e.g. expire after backup). Command begin logs correctly for complex data types like hash and list. Specify which commands will log to file immediately and set the default log level for log messages that are common to all commands. File logging is initiated from C.
2018-03-12 11:03:38 -04:00
David Steele
64a7196cb6 Rename retention-* options to repo-retention-*. 2018-02-19 15:07:24 -05:00
Cynthia Shang
f75ba7db94 Update backup host to repository host in the documentation.
Contributed by Cynthia Shang.
2018-02-19 13:17:58 -05:00
Cynthia Shang
4352407777 Option rename cleanup based on review.
* Remove --reset tags from v1 options.
* Use constants for repo prefix.
* Specify CFGDEF_INDEX_PG in option structure rather than adding in code.
* Fix error message references to "backup host".

Contributed by Cynthia Shang.
2018-02-19 10:12:51 -05:00
David Steele
72cbb9a9d4 Call Perl with built-in interpreter instead of execvp().
Exec'ing Perl worked fine but generated a very large command line in the process list and potentially exposed secrets.
2018-02-14 17:19:54 -05:00
David Steele
7d13087c4b Rename -no to -reset for non-boolean options. 2018-02-06 11:26:06 -05:00
David Steele
d35757c05f Fix missing log-level-stderr option for stanza-* commands. 2018-02-05 15:19:28 -05:00
David Steele
db21b7a360 Cleanup usage of internal options.
Apply internal to options that need to be read to determine locality but should not appear in the help.
2018-02-05 11:50:09 -05:00
David Steele
be90028100 Rename db-* options to pg-* and backup-* options to repo-* to improve consistency.
* repo-* options are now indexed although only one is allowed.
* List deprecated option names in documentation and command-line help.
2018-02-03 18:27:38 -05:00
Cynthia Shang
dc73abdb6e Improve stanza-create command so that it does not error when the stanza already exists.
Contributed by Cynthia Shang.
2018-02-01 14:12:03 -05:00
Cynthia Shang
00f58ec8c0 Fixed inability to restore a single database contained in a tablespace using --db-include.
Fixed by Cynthia Shang.
2018-01-30 16:13:54 -05:00
David Steele
a91a648019 Config parsing no longer implemented in Perl.
Options are passed to Perl as JSON from the C process.
2018-01-28 21:37:09 -05:00
David Steele
b58a45e067 Implement help command in C. 2018-01-23 13:34:24 -05:00
David Steele
404ec30d03 Move Perl help test to real module. 2018-01-23 12:58:35 -05:00
David Steele
1f39a34abc Move lock release later in exitSafe().
This reduce the chance of a new process starting and acquiring a lock before the old process has exited.
2018-01-17 10:31:21 -05:00
David Steele
39cb971afb Add log module. 2018-01-16 13:52:20 -05:00
David Steele
bffc6c49b3 Add perl-bin option to specify the Perl binary location when /usr/bin/env perl won't work. 2018-01-16 13:23:08 -05:00
Cynthia Shang
0e9ba98a50 Add stanza-delete command to cleanup unused stanzas.
Contributed by Cynthia Shang.
2018-01-03 12:23:33 -05:00
David Steele
526acca5bd Config parsing implemented in C. 2017-12-22 23:27:49 -05:00
David Steele
ef3ef85c33 Fixed an issue where WAL was not expired on PostgreSQL 10.
This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X.

Reported by Adam Brusselback.
2017-12-18 18:51:19 -05:00
Cynthia Shang
e5511d0f82 Improve check command to verify that the backup manifest can be built.
Contributed by Cynthia Shang.
2017-12-13 11:16:27 -05:00
David Steele
99a0721631 Allow any non-boolean, non-command-line option to be negated.
This allows specific options in pgbackrest.conf to be ignored (and set to default) which reduces the need to write new configuration files for specific needs.

Note that boolean, non-command-line options are already negatable.
2017-12-05 15:23:32 -05:00
David Steele
c5fc88d340 Fixed an issue that suppressed locality errors for backup and restore.
When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host.

Reported by Lardière Sébastien.
2017-12-05 14:59:09 -05:00
David Steele
c77fc1fa61 Refactor protocol param generation into a new function.
This allows the code to be tested more precisely and doesn't require executing a remote process.
2017-11-21 12:57:00 -05:00
David Steele
062e714307 Disable gzip filter when --compress-level-network=0.
The filter was used with compress level set to 0 which added overhead without any benefit.
2017-11-21 09:31:15 -05:00
David Steele
1da63aff07 Fixed an issue with invalid backup groups being set correctly on restore.
If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
2017-11-20 12:46:35 -05:00
David Steele
8a3425ec04 Automate generation of WAL and pg_control test files.
The existing static files would not work with 32-bit or big-endian systems so create functions to generate these files dynamically rather than creating a bunch of new static files.
2017-11-18 20:02:54 -05:00
David Steele
6ef2167e97 Fixed an issue retrieving WAL for old database versions.
After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed.

Also clean up some issues when the same db version/id appears multiple times in the history.

Fixed by Cynthia Shang.
Reported by Clinton Adams.
2017-11-16 17:18:51 -05:00
David Steele
1cb11a582d Fix flapping protocol timeout test.
It only matters that the correct error code is returned, so disable logging to prevent message ordering from failing the expect test.
2017-11-15 13:17:57 -05:00
David Steele
04e55fe71b Fixed an issue that could cause copying large manifests to fail during restore.
Reported by Craig A. James.
2017-11-14 14:48:44 -05:00
Cynthia Shang
b03c26968a Repository encryption support.
Contributed by Cynthia Shang.
2017-11-06 12:51:12 -05:00
David Steele
ac542788da Convert configuration definitions from auto-generated functions to auto-generated data structures. 2017-11-02 08:14:13 -04:00
David Steele
8674a4f7ae Allow functions with sensitive options to be logged at debug level with redactions.
Previously, functions with sensitive options had to be logged at trace level to avoid exposing them. Trace level logging may still expose secrets so use with caution.
2017-10-24 12:35:36 -04:00
David Steele
d82fe88031 Fix custom settings for compress-level option being ignored.
Reported by Jens Wilke.
2017-10-22 18:05:46 -04:00
David Steele
e1cb9ae86a Improve labeling for errors in helper processes. 2017-10-21 14:40:14 -04:00
David Steele
44c7c0475a Improve protocol error handling.
In particular, "stop" errors are no longer reported as "unexpected".
2017-10-21 14:11:33 -04:00
Cynthia Shang
2d56de00fc Fix instances where database-id was not rendered as an integer in JSON info output.
Fixed by Cynthia Shang.
Reported by Jason O'Donnell.
2017-10-18 08:42:32 -04:00
David Steele
266c9ddcc1 Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. 2017-09-19 10:14:18 -04:00
David Steele
6343fdd584 Additional backup exclusions.
* Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
* Exclude pg_internal.init files from backup since they are rebuilt on startup.
2017-09-04 08:26:57 -04:00
David Steele
fcb7c6fd1d PostgreSQL 10 support. 2017-09-01 12:29:34 -04:00
Cynthia Shang
36e576b483 Up to seven standbys can be configured for backup from standby.`
Contributed by Cynthia Shang.
2017-08-31 19:15:44 -04:00
David Steele
1e0ed07455 Configuration rules are now pulled from the C library when present. 2017-08-25 16:47:47 -04:00
David Steele
cbc026418c v1.21: Improved Info Output and SSH Port Option
Bug Fixes:

* The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
* Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang.)

Features:

* Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang.)
* Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang.)

Refactoring:

* Retry when S3 returns an internal error (500).
* Add bIgnoreMissing parameter to Local->manifest().
2017-08-08 18:09:07 -04:00
Cynthia Shang
b6da509c9f Consolidate stanza-create and stanza-upgrade tests into new stanza test.
Contributed by Cynthia Shang.
2017-08-04 16:10:51 -04:00
Cynthia Shang
ce2b18c5b8 Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports.
Contributed by Cynthia Shang.
2017-08-01 10:34:53 -04:00
David Steele
df7d633683 Improvements to command/command-line help:
* Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
* Fix description of --online based on the command context.
* Remove vestigial repository options from backup command.
2017-07-26 09:33:10 -04:00
David Steele
156fd4d54d Add bIgnoreMissing parameter to Local->manifest(). 2017-07-25 12:44:38 -04:00
Cynthia Shang
cda3b9821b Improved WAL data returned by info command.
Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version.

Contributed by Cynthia Shang.
2017-07-21 17:32:41 -04:00
David Steele
a10a22127d Consolidate archive-get test into new archive test. 2017-06-22 18:51:39 -04:00
David Steele
918c1c6f49 Add s3-repo-ca-path and s3-repo-ca-file options.
The options accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs.

Suggested by Scott Frazer.
2017-06-22 18:22:49 -04:00
David Steele
2348693389 Reduce process redundancy in mock tests. 2017-06-21 18:47:25 -04:00
David Steele
64ec7eb490 Eliminate redundancy in real tests. 2017-06-21 16:07:13 -04:00
David Steele
2337be8636 Consolidate archive-push tests into new archive test. 2017-06-21 08:38:53 -04:00
David Steele
11ee528e0e Rename Archive modules to remove redundancy. 2017-06-21 08:02:21 -04:00
David Steele
692b1e274a Split full module into mock and real to allow better test combinations and save time in CI. 2017-06-19 18:55:00 -04:00
David Steele
f6d4457d58 Full/Synthetic test refactor.
* Combine hardlink and non/compressed in synthetic tests to reduce test time and improve coverage.
* Change log level of hardlink logging to detail.
* Cast size in S3 manifest to integer.
2017-06-15 15:32:10 -04:00
David Steele
051c961151 S3 repository support. 2017-06-12 10:52:32 -04:00
David Steele
de7fc37f88 Storage and IO layer refactor:
Refactor storage layer to allow for new repository filesystems using drivers. (Reviewed by Cynthia Shang.)
Refactor IO layer to allow for new compression formats, checksum types, and other capabilities using filters. (Reviewed by Cynthia Shang.)
2017-06-09 17:51:41 -04:00
Cynthia Shang
7e982f05f5 Fixed config warnings raised from a remote process causing errors in the master process.
Fixed by Cynthia Shang.
2017-05-16 11:28:51 -04:00
David Steele
5296747271 Move backup modules into Backup directory. 2017-05-15 16:01:00 -04:00
David Steele
2a23157bf2 Move modules in Protocol directory in subdirectories. 2017-05-15 11:12:14 -04:00
David Steele
33c87b2a1c Remove flapping protocol timeout test that will be replaced in the upcoming storage patch. 2017-05-15 10:24:46 -04:00
Cynthia Shang
7d0c50fcd6 Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the master) if the standby is not configured and/or reachable.
Fixed by Cynthia Shang.
2017-05-12 16:05:44 -04:00
Cynthia Shang
5bf4f52e3d Fixed the info command so the WAL archive min/max displayed is for the current database version.
Fixed by Cynthia Shang.
2017-05-12 15:49:14 -04:00
David Steele
f207dc7123 Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions.
Suggested by Yogesh Sharma.
2017-04-12 12:41:34 -04:00
David Steele
f5f8d51366 Improved error message when the restore command detects the presence of postmaster.pid.
Suggested by Yogesh Sharma.
2017-04-12 09:53:07 -04:00
David Steele
fe02b40293 Backup and restore commands no longer copy via temp files.
In both cases the files are checksummed on resume so there's no danger of partial copies.
2017-04-12 09:05:57 -04:00
David Steele
3d84f2ce5e Improvements to Ini.pm.
* Refactor Ini.pm to facilitate testing.
* Complete statement/branch coverage for Ini.pm.
* Improved functions used to test/munge manifest and info files.
2017-04-10 13:24:45 -04:00
David Steele
eb4918d018 Fixed locking bug and refactoring.
* Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
* Simplify locking scheme. Now, only the master process will hold write locks (archive-push, backup) and not all the local and remote worker processes as before.
2017-04-07 10:31:49 -04:00
Cynthia Shang
b620e6b282 Improvements and additional tests for stanza-upgrade.
Contributed by Cynthia Shang.
2017-04-06 19:36:37 -04:00
Cynthia Shang
beead043ac Added stanza-upgrade command.
The stanza-upgrade command provides a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL.

Contributed by Cynthia Shang.
2017-04-03 10:42:55 -04:00
David Steele
b0947fdaa8 Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings.
Reported by Stephen Frost.
2017-03-13 15:50:13 -04:00
David Steele
02730526fc Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned.
Reported by blogh.
2017-03-02 13:50:29 -05:00
David Steele
5957f8566f Archive test improvements:
* Fixed flapping archive-stop test with testResult() retries.
* Added final test of archive contents to archive-push test.
2017-03-02 11:47:23 -05:00
David Steele
c2666cc7f3 Fixed an issue where tables over 1GB would report page checksum errors after the first segment.
Reported by Stephen Frost.
2017-03-01 14:15:46 -05:00
David Steele
86fee3da18 Continuous integration using travis-ci. 2017-02-21 08:59:23 -05:00
David Steele
e2ac7e1ea6 Fixed a regression introduced in v1.13 that could cause backups to fail.
This happened if files were removed (e.g. tables dropped) while the manifest was being built.

Reported by Navid Golpayegani.
2017-02-13 19:59:14 -05:00
David Steele
0daeace1cd Fixed alignment issues with multiline logging. 2017-02-05 19:58:52 -05:00
David Steele
cded35791d Added --log-timestamp option.
Allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
2017-02-05 10:59:49 -05:00
David Steele
8b5f37c2a8 Added archive and backup WAL ranges to the info command. 2017-02-05 10:31:29 -05:00
David Steele
7418559a67 Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination. 2017-01-31 10:36:59 -05:00
David Steele
eb1a1e5308 Missed expect logs from commit 455c604. 2017-01-30 16:22:12 -05:00
David Steele
6b99d770b5 Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN. 2017-01-30 13:59:00 -05:00
David Steele
43ada9fb33 Improved, multi-process implementation of asynchronous archiving. 2017-01-27 11:02:27 -05:00
David Steele
2a9ba877e1 Refactor process IO and process master/minion code out from the common protocol code. 2017-01-27 10:06:16 -05:00
David Steele
18db178ac3 Various improvements to the test suite:
* Allow logging to be suppressed via logDisable() and logEnable().
* Added more flexibility in initializing and cleaning up after modules and tests.
* testResult() suppresses logging and reports exceptions.
* testException() allows messages to be matched with regular expressions.
* Refactor name/locations of common modules that setup test environments.
2017-01-27 09:42:30 -05:00
David Steele
7d1347cdec Split the archive-get and archive-push commands into separate modules. 2017-01-10 19:59:32 -05:00
David Steele
b58045f575 Moved the Archive modules to the Archive directory. 2017-01-10 19:51:20 -05:00
David Steele
0b4055e588 Moved File->manifest() into the FileCommon.pm module. 2017-01-10 13:08:03 -05:00